A face expression recognition method with high recognition accuracy

Abstract

The application relates to the technical field of human face expression recognition, and discloses a human face expression recognition method with high recognition accuracy, which comprises the following steps: S1, a model adopts a human face expression recognition public data set RAF-DB and an auxiliary data set RAF-AU, data is extracted according to a 3:1 ratio, and then splicing is carried out to serve as a total data set of each round; S2, according to the fourth stage output of an IR50 model, a feature map of 7x7x512 and a feature output of 7x7x512 of a mobilefacenet are taken as inputs of a double-flow vit model, the output of the IR50 is recorded as x, and the output of the mobilefacenet is recorded as x_landmark. In the current human face expression recognition task, the accuracy based on the RAF-DB data set reaches the highest accuracy, and through improvement of a local relation perception module, an enhanced local relation perception module with better effect is created.

Description

Technical Field

[0001] This invention relates to the field of facial expression recognition technology, and more specifically discloses a facial expression recognition method with high accuracy. Background Technology

[0002] Currently, facial expression recognition is receiving increasing attention from the computer vision research community. Facial expression recognition refers to the use of computers to extract facial expression features from detected faces, enabling computers to understand and process facial expressions according to human thought processes, and to respond to human needs, establishing a friendly and intelligent human-computer interaction environment. Since facial expressions are one of the most important forms of human expression, and one of the most natural, universal, and direct ways for humans to express emotional states and connotations, the ability of computers to understand and recognize human expressions is crucial. Based on this, facial expression recognition has wide applications in human-computer interaction, medical progress monitoring, driver fatigue monitoring, virtual reality, intelligent tutoring systems, and many other fields.

[0003] Current facial expression recognition tasks are based on two main datasets: lab-control and wild. Lab-control facial expression maps are typically generated under specific laboratory conditions. Their advantage is accurate label annotation, but their disadvantage is the limited amount of data. Wild facial expression maps, on the other hand, are collected from the internet by their creators. While this allows for easy access to large amounts of data, it also means that the corresponding labels are manually annotated. Label accuracy is difficult to guarantee, and inconsistencies and errors in labeling due to varying levels of expertise among annotators further hinder the model's learning ability.

[0004] Therefore, the wild dataset makes current facial expression recognition tasks extremely challenging. There are two main reasons for this: intra-class variance and inter-class similarity. In addition, adding auxiliary datasets to aid training can lead to dataset bias, all of which hinder further improvements in model accuracy. In simple terms, intra-class variance and inter-class similarity mean that facial expression images of the same category may be identified as different categories by the model, while facial expression images of different categories may be identified as the same category. This is because, in the same environment, people of different identities, races, skin colors, ages, and other subtle differences can have the same expression, unlike other classification tasks with significantly different categories. Furthermore, the diverse sources of facial expression images in the wild dataset, often occluded or with complex backgrounds, and the inconsistent label levels make it even more difficult for the network to learn discriminative feature maps. Summary of the Invention

[0005] The main technical problem solved by this invention is to provide a facial expression recognition method with high accuracy, which can solve the problems existing in the above background technology.

[0006] To solve the above-mentioned technical problems, according to one aspect of the present invention, more specifically, a facial expression recognition method with high accuracy, comprising the following steps:

[0007] S1. The model uses the public facial expression recognition dataset RAF-DB and the auxiliary dataset RAF-AU, and extracts data in a 3:1 ratio, then concatenates them to form the total dataset for each round.

[0008] S2. Based on the fourth-stage output of the IR50 model, namely the 7×7×512 feature map and the 7×7×512 feature output of MobileFaceNet, use them as the input of the dual-stream VIT model. Denote the output of IR50 as x and the output of MobileFaceNet as x_landmark.

[0009] S3 and x first enter the enhanced local relation perception module. Based on x, two branches are constructed. Each branch reduces the dimension from 512 to 1 through LANet, resulting in two 7×7×1 feature maps, denoted as M1 and M2. Then, M1 and M2 are dropped out and maxed out respectively. Finally, they are multiplied and fused with x to obtain a new 7×7×512 feature map, xout.

[0010] S4 and xout then enter the invariance module, which means passing through two 1×1 convolutions in sequence. The channel dimension of each convolution does not change, and after each convolution, the output is added to the input of the convolution by residual processing. The final output result is still 7×7×512, denoted as x_invariance.

[0011] S5 and x_invariance are then processed by dropout, and then the residuals are added to x. x is then the input to the enhanced local relation awareness module.

[0012] S6. The other input stream, x_landmark, can be processed by dropout.

[0013] S7. At this point, both x and x_landmark are 7×7×512. Reshape them to 49×512 and add the corresponding cls tokens. At the same time, x also needs to be processed by position embedding and dropout.

[0014] S8. Concatenate x and x_landmark in terms of quantity dimension to obtain 100×512, denoted as x_l. First process x_l through the feature pyramid to obtain x_m and x_s, with channel dimensions of 256 and 128 respectively. Input them together into the dual-stream vit model for processing.

[0015] S9. Finally, dropout and LayerNorm processing are added to the model. The clstoken of x (512 channel dimension) is extracted by slicing and then processed by the SE Block attention module mechanism. Finally, an fully connected layer is added.

[0016] Furthermore, in S1, the model output consists of two parts: one part is the output obtained by RAF-DB through the model, which is processed using the cross-entropy loss function; the other part is the output obtained by RAF-AU through the AU Branch of the model, which is processed using the binary cross-entropy loss function. The two types of losses are allocated in a 1:1 ratio to form the total loss.

[0017] Furthermore, in S2, the IR50 model uses pre-trained weights based on the Ms-Celeb-1M dataset.

[0018] Furthermore, in S3, LANet is composed of 1×1 convolution, ReLU, 1×1 convolution, and sigmoid in sequence.

[0019] Furthermore, in S8, the dual-stream vit model consists of 8 layers of hytransformer blocks. When running to the 6th layer, an additional operation is performed, and its output is processed through the AU Branch module. The resulting output contains information on 21 categories of AUs, which are then used to calculate the binary cross-entropy loss with the corresponding AU labels.

[0020] The beneficial effects of the facial expression recognition method with high recognition accuracy of the present invention are as follows: The present invention achieves the highest accuracy in the current facial expression recognition task based on the RAF-DB dataset, and creates an enhanced local relationship perception module with better performance by improving the local relationship perception module. Attached Figure Description

[0021] The present invention will now be described in further detail with reference to the accompanying drawings and specific implementation methods.

[0022] Figure 1 This is a schematic diagram illustrating the principle of the model. Implementation

[0023] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the present application can be combined with each other. Example

[0024] According to one aspect of the invention, such as Figure 1 As shown, a facial expression recognition method with high accuracy is provided, including the following steps:

[0025] The first step involves using the public facial expression recognition dataset RAF-DB and the auxiliary dataset RAF-AU, extracting data in a 3:1 ratio and concatenating them to form the total dataset for each round. The model's output consists of two parts: one part is the output obtained from RAF-DB through the model, processed using the cross-entropy loss function; the other part is the output obtained from RAF-AU through the model's AU branch, processed using the binary cross-entropy loss function. The two types of losses are allocated 1:1 to form the total loss.

[0026] The second step is to use the fourth-stage output of the IR50 model, namely the 7×7×512 feature map, and the 7×7×512 feature output of MobileFaceNet as the input of the dual-stream VIT model. The IR50 model uses pre-trained weights based on the Ms-Celeb-1M dataset. The output of IR50 is denoted as x, and the output of MobileFaceNet is denoted as x_landmark.

[0027] The third step involves x first entering the enhanced local relation perception module, which constructs two branches based on x. Each branch reduces the dimension from 512 to 1 using LANet (LANet consists of 1×1 convolution, ReLU, 1×1 convolution, and sigmoid in sequence) to obtain two 7×7×1 feature maps, denoted as M1 and M2. Then, M1 and M2 are subjected to dropout and max processing respectively, and finally multiplied and fused with x to obtain a new 7×7×512 feature map, xout.

[0028] The fourth step is for xout to enter the invariance module, which means it goes through two 1×1 convolutions in sequence. The channel dimension of each convolution does not change, and after each convolution, the output is added to the input of the convolution by residual processing. The final output result is still 7×7×512, denoted as x_invariance.

[0029] Fifth, x_invariance is then processed by dropout, and then the residual is added to x. x then becomes the input to the enhanced local relation awareness module.

[0030] Step 6: The other input stream, x_landmark, can be processed by dropout.

[0031] Step 7: At this point, both x and x_landmark are 7×7×512. Reshape them to 49×512 and add the corresponding cls tokens. At the same time, x also needs to be processed with position embedding and dropout.

[0032] Step 8: Concatenate x and x_landmark in terms of quantity dimension to obtain 100×512, denoted as x_l. First, process x_l through the feature pyramid to obtain x_m and x_s, with channel dimensions of 256 and 128 respectively. Input them together into the dual-stream VIT model for processing. The dual-stream VIT model consists of 8 layers of hytransformer blocks. When running to the 6th layer, perform an additional operation and take its output to process through the AU Branch module. The output contains information of 21 categories of AUs. Subsequently, perform binary cross-entropy loss calculation with the corresponding AU labels.

[0033] Step 9: Finally, add dropout and LayerNorm processing to the model. Slice out the clstoken of x, which is the 512-channel dimension, and then process it through the SE Block attention module mechanism. Finally, add an fully connected layer. Example

[0034] like Figure 1 As shown, a facial expression recognition method with high accuracy is provided, including the following steps:

[0035] The first step involves constructing a new auxiliary dataset using the publicly available facial expression recognition dataset RAF-DB and OpenFace to predict pseudo-labels for other expression datasets. Data is extracted in a 3:1 ratio and then concatenated to form the total dataset for each round. The model's output consists of two parts: one part is the output obtained from RAF-DB through the model, which is processed using the cross-entropy loss function; the other part is the output obtained from RAF-AU through the model's AU Branch, which is processed using the binary cross-entropy loss function. The two types of losses are allocated 1:1 to form the total loss.

[0036] The second step is to use the fourth-stage output of the IR50 model, namely the 7×7×512 feature map, and the 7×7×512 feature output of MobileFaceNet as the input of the dual-stream VIT model. The IR50 model uses VGGFace2 pre-trained weights as the backbone. The output of IR50 is denoted as x, and the output of MobileFaceNet is denoted as x_landmark.

[0037] The third step involves x first entering the enhanced local relation perception module, which constructs two branches based on x. Each branch reduces the dimension from 512 to 1 using LANet (LANet consists of 1×1 convolution, ReLU, 1×1 convolution, and sigmoid in sequence) to obtain two 7×7×1 feature maps, denoted as M1 and M2. Then, M1 and M2 are subjected to dropout and max processing respectively, and finally multiplied and fused with x to obtain a new 7×7×512 feature map, xout.

[0038] The fourth step is for xout to enter the invariance module, which means it goes through two 1×1 convolutions in sequence. The channel dimension of each convolution does not change, and after each convolution, the output is added to the input of the convolution by residual processing. The final output result is still 7×7×512, denoted as x_invariance.

[0039] Fifth, x_invariance is then processed by dropout, and then the residual is added to x. x then becomes the input to the enhanced local relation awareness module.

[0040] Step 6: The other input stream, x_landmark, can be processed by dropout.

[0041] Step 7: At this point, both x and x_landmark are 7×7×512. Reshape them to 49×512 and add the corresponding cls tokens. At the same time, x also needs to be processed with position embedding and dropout.

[0042] Step 8: Concatenate x and x_landmark in terms of quantity dimension to obtain 100×512, denoted as x_l. First, process x_l through the feature pyramid to obtain x_m and x_s, with channel dimensions of 256 and 128 respectively. Input them together into the dual-stream VIT model for processing. The dual-stream VIT model consists of 8 layers of hytransformer blocks. When running to the 6th layer, perform an additional operation and take its output to process through the AU Branch module. The output contains information of 21 categories of AUs. Subsequently, perform binary cross-entropy loss calculation with the corresponding AU labels.

[0043] Step 9: Finally, add dropout and LayerNorm processing to the model. Slice out the clstoken of x, which is the 512-channel dimension, and then process it through the SE Block attention module mechanism. Finally, add an fully connected layer.

[0044] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention are also within the protection scope of the present invention.

Claims

1. A face expression recognition method with high recognition accuracy, characterized by, Comprise the following steps: S1, the model adopts the public data set RAF-DB and the auxiliary data set RAF-AU of facial expression recognition, and extracts the data in a ratio of 3:1, and then splices as the total data set of each round; S2, according to the output of the fourth stage of the IR50 model, that is, the feature map of 7x7x512 and the feature output 7x7x512 of mobilefacenet as the input of the double-flow vit model, the output of IR50 is recorded as x, and the output of mobilefacenet is recorded as x_landmark; S3, x first enters the enhanced local relationship perception module, constructs two branches according to x, each branch is reduced from 512 dimensions to 1 through LANet, and two 7x7x1 feature maps are obtained, recorded as M1 and M2, then M1 and M2 are respectively subjected to dropout, and then max processing is performed, and finally multiplied with x to obtain a new 7x7x512 feature map, which is xout; S4, xout again enters the invariance module, that is, sequentially through 2 1x1 convolutions, the channel dimension of each convolution does not change, and the output obtained after passing through each convolution is added to the input of the convolution for residual addition processing, and the final output result is still 7x7x512, recorded as x_invariance; S5, x_invariance is subjected to dropout processing, and then added to x for residual addition processing, x is the input of the enhanced local relationship perception module; S6, another input stream x_landmark is subjected to dropout processing; S7, at this time, x and x_landmark are both 7x7x512, which are reshaped into 49x512, and corresponding clstoken is added, and x also needs to add position embedding position coding processing and dropout processing; S8, x and x_landmark are spliced in the quantity dimension to obtain 100x512, recorded as x_l, x_l is first processed by a feature pyramid to obtain x_m and x_s, and the channel dimension sizes are 256 and 128 respectively, which are input into the double-flow vit model for processing; S9, the model is sequentially added with dropout processing and LayerNorm processing, the cls token of x is sliced and taken out, then processed through the SE Block attention module mechanism, and finally added with an fc layer. 2.The facial expression recognition method of claim 1, wherein: In S1, the output of the model is composed of two parts, one part is the output obtained by RAF-DB through the model, which is processed by using a cross-entropy loss function, and the other part is the output obtained by RAF-AU through the AU Branch branch of the model, which is processed by using a binary cross-entropy loss function, and the two types of losses are allocated in a ratio of 1:1 to form the total loss. 3.The facial expression recognition method of claim 1, wherein: In S2, the IR50 model adopts the pre-training weight based on the Ms-Celeb-1M data set.

4. The method of claim 1, wherein the method has high recognition accuracy. In the S3, the LANet is sequentially composed of a 1x1 convolution, a relu, a 1x1 convolution and a sigmoid.

5. The method of claim 1, wherein the method has high recognition accuracy. In the S8, the double-flow vit model is composed of 8 layers of hytransformer blocks, and when running to the 6th layer, an operation is additionally performed, that is, the output is taken and processed through an AU Branch module, and the obtained output contains information of 21 categories of AUs, and subsequent binary cross-entropy loss calculation is performed with the corresponding AU label.

Images