A facial expression recognition method based on inter-class difference reinforcement network

Abstract

The invention discloses a facial expression recognition method based on an inter-class difference reinforcement network, and the steps are as follows: collecting a data set, pre-processing the data set; analyzing the expression similarity relationship; using the similarity information to construct a parallel branch network to extract global features and fine-grained features for distinguishing similar expressions; sending the extracted features to a fully connected layer for dimensionality reduction, and performing feature fusion, and outputting the expression category through a classifier; adding a class balance weighted loss function to expand the class spacing. The method makes full use of the expression similarity information, extracts fine-grained features, and effectively improves the facial expression classification effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of facial expression recognition, and in particular relates to a facial expression recognition method based on a dual-branch attention mechanism. Background Art

[0002] Facial expressions contain rich information about human emotions and are one of the main ways for humans to communicate with each other. Facial expression recognition aims to mine potential emotional features from facial images and classify them. It is a research hotspot in the field of computer vision and has shown broad application prospects in many fields such as autonomous driving, classroom teaching, clinical psychology, and intelligent transportation.

[0003] At present, research on facial expression recognition based on deep learning has made great progress. Researchers have proposed various effective algorithms to improve the network's feature learning ability, including: extracting auxiliary input signals such as facial key point coordinates, assigning different weights to each facial activity unit, extracting facial appearance identity information, designing novel loss functions to optimize network parameters, etc. However, these methods ignore the high similarity between different expression categories. For example, opening the mouth is a common facial feature of fear, happiness, and surprise; frowning is a common facial feature of sadness, disgust, and anger. Therefore, how to learn discriminative fine-grained facial expression features and then distinguish the subtle differences between expressions is crucial to improving the performance of facial expression recognition models. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to address the deficiencies in the prior art, provide a facial expression recognition method based on an inter-class difference enhancement network, design a parallel network structure, enhance the subtle difference features between expression categories, and improve the expression network recognition effect.

[0005] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0006] A method for facial expression recognition based on an inter-class difference enhancement network comprises the following steps:

[0007] Step 1: Obtain public datasets from the Internet and perform image preprocessing and sample enhancement;

[0008] Step 2: Conduct a priori experiments to analyze the similarity relationship of facial expressions and divide the seven basic facial expressions into two strong correlation sets;

[0009] Step 3: Design a parallel network structure. The network consists of a backbone network BBN and a reinforcement branch DRN. The data obtained in step 1 is fed into the parallel network. The backbone network is used to extract global high-level semantic features, and the reinforcement branch is used to extract fine-grained features of similar expressions.

[0010] Step 4: The high-level semantic features and similar expression fine-grained features obtained in step 3 are fused after dimension reduction by the fully connected layer, and the facial expression category is output by the classifier;

[0011] Step 5: Based on the effective sample number theory, design a class-balanced weighted loss function. By assigning different weights to the majority class and the rare class based on Cross-EntropyLoss, we can suppress inter-class competition, alleviate the negative sample supervision signal of the majority class on the rare class, and expand the inter-class distance of expression features;

[0012] Furthermore, the step 1 specifically includes the following steps:

[0013] The facial expression datasets are FER2013 dataset and RAF-DB dataset. This method first preprocesses the datasets. The face area in the image is detected using the MTCNN face detection algorithm, which is cropped from the original image and scaled to a uniform size of 224×224. For single-channel images, the single-channel images are converted into three-channel images by the replication method, and random cropping and random horizontal flipping methods are used for data enhancement.

[0014] Furthermore, the step 2 specifically includes the following steps:

[0015] Based on the preprocessed FER2013 and RAF-DB expression datasets, balanced sets with equal number of samples in each expression category are constructed. The two balanced sets are sent to three classic convolutional neural networks (ResNet18, ResNet34, ResNet50) for training and testing to obtain the corresponding confusion matrices, and the similarity relationship between expressions is analyzed from the confusion matrices. The specific similarity analysis method is:

[0016] Step 2.1: In order to reduce the impact of the long tail problem of the data, the FER2013 dataset and the RAFDB dataset are randomly undersampled to achieve data averaging. Taking the data volume of the shortest class as the benchmark, the same number of pictures are randomly selected from other categories to form a balanced set, which are denoted as: FER2013 Balance ,RAFDB Balance ;

[0017] Step 2.2: Use ResNet18, ResNet34, and ResNet50 as the network model for the prior experiment, and convert the FER2013 obtained in step 2.1 Balance and RAFDB Balance Send them to three networks for training and testing respectively, and get 6 confusion matrices;

[0018] Step 2.3: The elements on the main diagonal of the confusion matrix represent the correct classification results, and the remaining positions are the wrong classification results. For each confusion matrix, arrange the probability of each expression being misclassified from large to small, retain the expression categories corresponding to the first three probability values, and record them as the strongly associated expressions of the true value category. Count the strongly associated expressions in the 6 confusion matrices and draw a similarity relationship matrix. The elements outside the main diagonal in the matrix represent the similarity values ​​of various expressions;

[0019] Step 2.4: Group the two expression categories with similarity values ​​greater than or equal to 4 into a group of strongly associated sets. In this way, the seven types of expressions can be divided into two strongly associated sets, which are recorded as: Group SL1 ={Angry,Fearful,Surprised},Group SL2 ={Happy, Disgusted, Sad, Neutral}.

[0020] Furthermore, the step 3 specifically includes the following steps:

[0021] The network structure is based on the ResNet34 network model, and multiple lightweight attention mechanism modules are embedded in the network to build a backbone network. The backbone network BBN takes the original image as input and learns global high-level semantic features. The reinforcement branch is placed between the second residual block and the third residual block of the backbone network and differentiated into two branches, which further extract fine-grained features of the expression categories contained in different strong association groups. The specific method of constructing a parallel neural network is:

[0022] Step 3.1: The middle part of the ResNet34 network is composed of 4 residual blocks connected in series, and each block contains 3, 4, 6, and 3 residual structures respectively. The backbone network takes the original image of size 224×224×3 as input, embeds a lightweight joint attention module Lightweight Multi Attention Module (LMAM) between each two blocks, and extracts global high-level semantic features;

[0023] Step 3.2: The LMAM described in step 3.1 consists of a Spatial Attention Module (SAM) and a Channel Attention Module (CAM). The feature map F of size c×h×w is input into the LMAM, and the SAM learns the relationship between each spatial position, and then the CAM obtains the weights of different channels to obtain the final feature map;

[0024] SAM first uses AvgPool to compress the input feature map F to obtain a spatial dimension global feature of size 1×h×w, then successively undergoes 5×5 convolution, MaxPool, and 3×3 convolution to reduce the feature size, restores the original size after upsampling, and finally activates it through Sigmoid to obtain a spatial attention map M of size 1×h×w. SA . SA Multiply element-wise with F to obtain the feature map F optimized by the spatial attention map SA It can be expressed as:

[0025] M SA =SAM(F)=δ(Upsample(f conv-pool (AvgPool(x)))) (1)

[0026]

[0027] Where: f conv-pool Represents 5×5 convolution, MaxPool, 3×3 convolution cascade structure. δ represents Sigmoid activation operation;

[0028] Step 3.3: F SA Input into CAM to further obtain the weight information of different channels. First, use AvgPool and MaxPool to calculate F SA After compression, two channel features M with a size of c×1×1 are obtained after being processed by a double-layer fully connected layer. CA_Avg and M CA_Max , M CA_Avg and M CA_Max After element-by-element addition and Sigmoid activation, we get the spatial attention map M CA . CA Multiply element-wise with F to generate a feature map F that is doubly optimized by the spatial attention map and the channel attention map LMAM It can be expressed as:

[0029] M CA =CAM(F)=(δ(f FC1 (AvgPool(F SA )))+δ(f FC2 (MaxPool(F SA )))) (3)

[0030]

[0031] Where: f FC1 and f FC2 represents a double-layer fully connected layer, and δ represents a Sigmoid activation operation;

[0032] Step 3.4: The reinforcement branch network DRN consists of two small branches with the same structure connected in parallel, which respectively strengthen the expression categories in different strongly associated sets. Each branch consists of the third residual block in ResNet34 and a CAM to form a fine-grained feature extraction unit. The extracted fine-grained features of similar expression differences can be expressed as:

[0033]

[0034] Among them: F DRN1 and F DRN2 They represent the fine-grained features of similar expressions extracted by branch 1 and branch 2 respectively.

[0035] Furthermore, the step 4 specifically includes the following steps:

[0036] The high-level semantic features F BBN and similar expression fine-grained features F DRN1 and F DRN2 The fully connected layers are used for dimensionality reduction, and then feature fusion is performed to obtain the final output feature F ALL , the classifier outputs the facial expression category. The specific feature fusion method is:

[0037] Step 4.1: F BBN After being sent to AvgPool and a single fully connected layer for dimensionality reduction, the 7-dimensional output vector can be expressed as:

[0038] V BBN =[c AN ,c DI ,c FE ,c HA ,c SA ,c SU ,c NE ] (6)

[0039] Where: c i ,i∈{AN,DI,FE,HA,SA,SU,NE} represents the output value of each type of expression, and the subscript represents the expression category information corresponding to the value;

[0040] Step 4.2: F DRN1 and F DRN2 After dimensionality reduction by AvgPool and a single-layer fully connected layer, a 7-dimensional output feature vector is obtained. Each element of the feature vector corresponds to the output feature of an expression:

[0041]

[0042] Where: V DRN1 represents the output feature vector of branch 1, V DRN2represents the output feature vector of branch 2;

[0043] a i ,i∈{AN,DI,FE,HA,SA,SU,NE} represents the vector V DRN1 The output value corresponding to each type of expression in the expression, the subscript indicates the expression category corresponding to the value. i ,i∈{AN,DI,FE,HA,SA,SU,NE} represents the vector V DRN2 The output value corresponding to each type of expression in , the subscript indicates the expression category information corresponding to the value. In order to achieve the enhancement effect, V DRN1 Keep only Group SL1 The output values ​​corresponding to the three expression categories contained in V, and the elements in other positions are set to zero. DRN2 Only keep strong groups SL2 The output values ​​corresponding to the four expression categories contained in , and the elements in the remaining positions are set to zero. According to the strong correlation set divided in step 2, the corrected output vector can be expressed as:

[0044]

[0045] The final output vector ECV of the enhanced branch is obtained by adding the two corrected eigenvectors at the corresponding positions, which can be expressed as:

[0046] V DRN =[a AN ,b DI ,a FE ,b HA ,b SA ,a SU ,b NE ] (9)

[0047] Among them, a i ,i∈{AN,FE,SU} is a vector V DRN1 The output values ​​corresponding to the various expressions in the expression, the subscripts represent the corresponding expression categories b j ,j∈{DI,HA,SA,NE} is a vector V DRN2 The output values ​​corresponding to the various expressions in , the subscripts represent the corresponding expression categories;

[0048] Step 4.3: V BBN and V DRN The elements are added in a certain ratio. During the training process, the fusion ratio is continuously adjusted by controlling the dynamic balance coefficient α, so that the network's attention is transferred between the backbone network and the reinforcement branch. During the test, α is set to a constant value of 0.9. The functional relationship between the balance parameter α of feature fusion and the number of iterations can be expressed as:

[0049]

[0050] Among them: Epoch Current and Epoch Max Respectively represent the current number of iterations and the maximum number of iterations, Epoch Current ≥1. The final feature obtained after fusion can be expressed as:

[0051] V ALL =αV BBN +(1-α)V DRN (11)

[0052] Furthermore, the step 5 specifically includes the following steps:

[0053] From the perspective of class-balanced combination weighting, rare classes are protected on the basis of the cross entropy loss function, inter-class differences are strengthened, and inter-class distances are increased. The specific class-balanced combination weighting method is:

[0054] Step 5.1: First, give a heuristic definition of the effective sample number F(n):

[0055]

[0056] Where: k is a coefficient, n represents the actual number of samples, and F(n) represents the effective number of samples. This heuristic definition can better describe the marginal diminishing effect. When n is large enough, further increasing n cannot meet the effective sample capacity.

[0057]

[0058] Based on this definition, the weights of each expression category are redefined:

[0059]

[0060] Where: w c , c∈[1,7] represents the weight of each expression category, c represents the cth category, n c Represents the actual number of samples in the cth category.

[0061] Step 5.2: Based on the definition of weighted weights for each expression category in step 5.1, construct Weighted Cross-Entropy Loss:

[0062]

[0063] Where: c represents the true value label of the expression category, represents the predicted output probability value of the expression, w c represents the weight of the c-th category expression defined in step 5.1.

[0064] Beneficial effects: The present invention provides a facial expression recognition method based on an inter-class difference enhancement network, which analyzes the similarity of expressions, uses parallel branches embedded in a lightweight attention mechanism module to simultaneously extract global high-level semantic features and fine-grained features for distinguishing similar expressions, and uses a class-balanced weighted loss function to optimize network parameters. This method has the following advantages.

[0065] 1) Based on the similarity relationship of expressions, a parallel branch network architecture is designed, which takes into account the extraction of global semantic features and fine-grained features of similar expressions, thereby improving the classification effect of similar expression categories;

[0066] 2) The lightweight attention module extracts features from the input feature map in spatial and channel dimensions, allowing the neural network to pay more attention to facial areas that are highly correlated with expression, effectively reducing the impact of redundant facial information on facial expression recognition;

[0067] 3) Dynamic feature fusion method, which continuously adjusts the network's feature learning attention to generate more robust overall features;

[0068] 4) Combining the principle of effective sample number, a class-balanced weighted loss function is designed, which to a certain extent suppresses the negative sample supervision signal of the category with more samples acting on the category with fewer samples, increases the class distance of expression features, and achieves better classification effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is an overall flow chart of the facial expression recognition method based on the inter-class difference enhancement network provided by the present invention;

[0070] Figure 2 It is a network structure diagram of the facial expression recognition method based on the inter-class difference enhancement network provided by the present invention;

[0071] Figure 3 This is the network structure diagram of the attention module Lightweight Multi Attention Module (LMAM);

[0072] Figure 4 This is the network structure diagram of the Channel Attention Module (CAM);

[0073] Figure 5 This is the network structure diagram of the Spatial Attention Module (SAM). DETAILED DESCRIPTION

[0074] The present invention will be further described below in conjunction with the accompanying drawings.

[0075] Implementation Cases

[0076] This implementation case uses Python3.7 and Pytorch deep learning framework as the experimental platform, and uses GeForce RTX 3070 graphics card with 8G video memory as the training tool. For the FER2013 dataset, Training is used as the training set (the number of samples is 28709), PrivateTest is used as the test set (the number of samples is 3589), and PublicTest is used as the validation set (the number of samples is 3589). For the RAF-DB dataset, the initial division of the training set (the number of samples is 12271) and the test set (the number of samples is 3068) using the original data is used as the division basis for this example. This implementation case does not use any dataset to pre-train the model. The training process of the two datasets uses the same hyperparameter settings: the maximum number of training iterations is 150; batch_size is 48; RAdam optimizer is used; plateau_patience is set to 2; the initial learning rate is 0.01; weight_decay is 0.0001. The test indicator is accuracy. The present invention is further described below with respect to the above examples. The process of the present invention includes:

[0077] Step 1: This implementation case uses the FER2013 dataset (FER2013.csv) and the face-aligned images in the RAF-DB dataset (aligned folder), and no longer uses MTCNN for face alignment. Split FER2013.csv into three csv files, named: Train.csv, Test.csv, Val.csv, and feed them into the network during model training, testing, and verification. For the RAF-DB dataset, first put the images into the corresponding expression category folders in the Train folder and the Test folder according to the labels, then convert the training images and test images into csv files, named Train.csv and Test.csv, and feed them into the network during model training and testing. When using Torch.Dataset to read batch data, first convert the single-channel images of the FER2013 dataset into three-channel images by copying, and the RAF-DB dataset retains the original three-channel format. Then, the transforms tool in torchvision is used to perform random rotation of the training data in the angle range of [-30°~30°] and random horizontal flipping with a probability of 50%, and random rotation of the test data in the angle range of [-30°~30°].

[0078] Step 2: Based on the preprocessed FER2013 and RAF-DB expression datasets, construct balanced sets with equal number of samples in each expression category. Send the two balanced sets to three classic convolutional neural networks (ResNet18, ResNet34, ResNet50) for training and testing to obtain the corresponding confusion matrices, and analyze the similarity relationship between expressions from the confusion matrices. The specific similarity analysis method is:

[0079] Step 2.1: In order to reduce the impact of the long tail problem of the data, the FER2013 dataset and the RAF-DB dataset are randomly undersampled to achieve data averaging. Taking the data volume of the shortest class as the benchmark, the same number of pictures are randomly selected from other categories to form a balanced set, which is denoted as: FER2013 Balance ,RAFDB Balance , of which FER2013 Balance The number of samples of each type of expression in the training set, test set, and validation set are 436, 55, and 56 respectively. Balance The number of samples of each type of expression in the training set and test set are 281 and 74 respectively;

[0080] Step 2.2: Use ResNet18, ResNet34, and ResNet50 pre-trained on the ImageNet dataset as the network model for the prior experiment, and convert the FER2013 obtained in step 2.1 into Balance and RAFDB Balance The training was conducted in three networks respectively. The RAdam optimizer was used to optimize the parameters for all training processes, and the same hyperparameter settings were used: the maximum number of iterations was 100; the initial learning rate was 0.001; the weight_decay was set to 0.0001; and the batch_size was 48. After training, the model was called and tested on the corresponding test set to obtain 6 confusion matrices.

[0081] Step 2.3: The vertical axis of the confusion matrix represents the True Label of each expression category, and the horizontal axis represents the Predict Label of each expression category. In each row of the record matrix, except for the expression category corresponding to the vertical axis, the three expression categories with the highest recognition probability are recorded as the strongly associated expressions of the true value category corresponding to the vertical axis. The strongly associated expressions corresponding to each type of expression in the 6 confusion matrices and their frequencies of occurrence are counted, and the similarity relationship matrix is ​​drawn;

[0082] Step 2.4: Divide the two expression categories with similarity values ​​greater than or equal to 4 into a group of strongly associated sets, and divide the 7 types of expressions into two strongly associated sets, which are recorded as: Group SL1 ={Angry,Fearful,Surprised},GroupSL2 ={Happy, Disgusted, Sad, Neutral}.

[0083] Step 3: The network structure is based on the ResNet34 network model. The ResNet34 network consists of a 7×7 convolution layer, 16 residual modules, a single-layer AvgPool and a single-layer fully connected layer. The residual module in the middle of the network is divided into 4 blocks, each of which contains 3, 4, 6, and 3 residual modules respectively. The backbone network retains the 7×7 convolution close to the input layer and the AvgPool close to the output layer and the single-layer fully connected layer of ResNet34, and embeds 3 lightweight attention mechanism modules in the network. The reinforcement branch is placed between the second and third blocks of the backbone network and differentiated into two branches, which further extract fine-grained features of the expression categories contained in different strongly associated groups. The specific method of constructing a parallel neural network is:

[0084] Step 3.1: The backbone network takes the original image of size 224×224×3 as input, embeds a lightweight joint attention module Lightweight Multi Attention Module (LMAM) between every two blocks, and extracts global high-level semantic features;

[0085] Step 3.2: The LMAM described in step 3.1 consists of a Spatial Attention Module (SAM) and a Channel Attention Module (CAM). The feature map F of size c×h×w is input into the LMAM, and the SAM learns the relationship between each spatial position, and then the CAM obtains the weights of different channels to obtain the final feature map;

[0086] SAM first uses the nn.AdaptiveAvgPool3d function in Torch to compress the input feature map F to obtain a spatial dimension global feature of size 1×h×w, then successively undergoes 5×5 convolution, MaxPool, and 3×3 convolution to reduce the feature size, restores the original size after upsampling, and finally activates it with Sigmoid to obtain a spatial attention map M of size 1×h×w. SA . The kernel_size of 5×5 convolution and 3×3 convolution is 1, stride=1, padding=1; the kernel_size of MaxPool operation is 3, stride=2, padding=1; transposed convolution is used to achieve upsampling. SA Multiply element-wise with F to obtain the feature map F optimized by the spatial attention map SA It can be expressed as:

[0087] M SA =SAM(F)=δ(Upsample(f conv-pool (AvgPool(x)))) (1)

[0088]

[0089] Where: f conv-pool It means that the network is processed by 5×5 convolution, MaxPool, and 3×3 convolution in sequence. δ means Sigmoid activation operation;

[0090] Step 3.3: F SA Input into CAM to further obtain the weight information of different channels. First, use AvgPool and MaxPool to calculate F SA After compression, two channel features M with a size of c×1×1 are obtained after being processed by a double-layer fully connected layer. CA_Avg and M CA_Max , M CA_Avg and M CA_Max After element-by-element addition and Sigmoid activation, we get the spatial attention map M CA . CA Multiply element-wise with F to generate a feature map F that is doubly optimized by the spatial attention map and the channel attention map LMAM It can be expressed as:

[0091] M CA =CAM(F)=(δ(f FC1 (AvgPool(F SA )))+δ(f FC2 (MaxPool(F SA )))) (3)

[0092]

[0093] Where: f FC1 and f FC2 represents a double-layer fully connected layer, and δ represents a Sigmoid activation operation;

[0094] Step 3.4: The reinforcement branch network DRN consists of two small branches with the same structure connected in parallel, which respectively strengthen the expression categories in different strongly associated sets. Each branch consists of the third residual block in ResNet34 and a CAM to form a fine-grained feature extraction unit. The specific operation steps and the extracted fine-grained features of similar expression differences can be expressed as:

[0095]

[0096] Among them: F DRN1 and F DRN2 They represent the fine-grained features of similar expressions extracted by branch 1 and branch 2 respectively.

[0097] Step 4: Transform the high-level semantic features F BBN and similar expression fine-grained features F DRN1 and F DRN2 The fully connected layers are used for dimensionality reduction, and then feature fusion is performed to obtain the final output feature F ALL , the SoftMax classifier outputs the facial expression category. The specific feature fusion method is:

[0098] Step 4.1: F BBN After being sent to AvgPool and a single fully connected layer for dimensionality reduction, the 7-dimensional output vector can be expressed as:

[0099] V BBN =[c AN ,c DI ,c FE ,c HA ,c SA ,c SU ,c NE ] (6)

[0100] Where: c i ,i∈{AN,DI,FE,HA,SA,SU,NE} represents the output value of each type of expression, and the subscript represents the expression category information corresponding to the element;

[0101] Step 4.2: F DRN1 and F DRN2 After dimensionality reduction by AvgPool and a single-layer fully connected layer, a 7-dimensional output feature vector is obtained. Each element of the feature vector corresponds to the output feature of an expression:

[0102]

[0103] Where: V DRN1 represents the output feature vector of branch 1, V DRN2 represents the output feature vector of branch 2;

[0104] a i ,i∈{AN,DI,FE,HA,SA,SU,NE} represents the vector V DRN1 Output the output value of each type of expression in the , and the subscript indicates the expression category information corresponding to the element. i ,i∈{AN,DI,FE,HA,SA,SU,NE} represents the vector V DRN2 The subscripts in the elements represent the expression category information corresponding to the element. In order to achieve the enhancement effect, VDRN1 Keep only Group SL1 The element information corresponding to the three expression categories contained in V, and the elements in other positions are set to zero. DRN2 Keep only Group SL2 The element information corresponding to the four expression categories contained in , and the remaining elements are set to zero. According to the strong association set divided in step 2, the corrected output vector can be expressed as:

[0105]

[0106] The final output vector ECV of the enhanced branch is obtained by adding the two corrected eigenvectors at the corresponding positions, which can be expressed as:

[0107] V DRN =[a AN ,b DI ,a FE ,b HA ,b SA ,a SU ,b NE ] (9)

[0108] Among them, a i ,i∈{AN,FE,SU} is a vector V DRN1 The elements in b j ,j∈{DI,HA,SA,NE} is a vector V DRN2 The elements in , the subscripts represent the corresponding expression categories;

[0109] Step 4.3: V BBN and V DRN The elements are added in a certain ratio. During the training process, the fusion ratio is continuously adjusted by controlling the dynamic balance coefficient α, so that the network's attention is transferred between the backbone network and the reinforcement branch. During the test, α is set to a constant value of 0.9. The functional relationship between the balance parameter α of feature fusion and the number of iterations can be expressed as:

[0110]

[0111] Among them: Epoch Current and Epoch Max Respectively represent the current number of iterations and the maximum number of iterations, Epoch Current ≥1. The final feature obtained after fusion can be expressed as:

[0112] V ALL =αV BBN +(1-α)V DRN (11)

[0113] Step 5: From the perspective of class-balanced combination weighting, rare classes are protected based on the cross entropy loss function, inter-class differences are strengthened, and inter-class distances are increased. The specific class-balanced combination weighting method is:

[0114] Step 5.1: First, give a heuristic definition of the effective sample number F(n):

[0115]

[0116] Where: k is a coefficient, n represents the actual number of samples, and F(n) represents the effective number of samples. This heuristic definition can better describe the marginal diminishing effect. When n is large enough, further increasing n cannot meet the effective sample capacity.

[0117]

[0118] Based on this definition, the weights of each expression category are redefined:

[0119]

[0120] Where: w c , c∈[1,7] represents the weight of each expression category, c represents the cth category, n c Represents the actual number of samples in the cth category.

[0121] In the implementation case, the FER2013 dataset and the RAF-DB dataset are based on w c The calculated weighted weights of various expression categories are: FER2013 =[0.0158,0.0478,0.0156,0.0118,0.0144,0.0178,0.0142], w RAF-DB =[0.0376,0.0373,0.0595,0.0145,0.0225,0.0278,0.0199]. The expression categories corresponding to each value in the list are {Angry, Disgusted, Fearful, Happy, Sad, Surprised, Neutral}.

[0122] Step 5.2: Based on the definition of weighted weights for each expression category in step 5.1, construct Weighted Cross-Entropy Loss:

[0123]

[0124] Where: c represents the true value label of the expression category, represents the predicted output probability value of the expression, w crepresents the weight of the c-th category expression defined in step 5.1.

[0125] It should be pointed out that the above-described implementation cases are only preferred implementation modes of the present invention, rather than limitations on the implementation modes. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A facial expression recognition method based on inter-class difference enhancement network, characterized in that: The following steps are involved: Step 1: Obtain public datasets from the Internet and perform image preprocessing and sample enhancement; Step 2: Conduct a priori experiments to analyze the similarity relationship of facial expressions and divide the seven basic facial expressions into two strong correlation sets; Step 3: Design a parallel network structure. The network consists of a backbone network BBN and a reinforcement branch DRN. The data obtained in step 1 is fed into the parallel network. The backbone network is used to extract global high-level semantic features, and the reinforcement branch is used to extract fine-grained features of similar expressions. Step 4: The high-level semantic features and similar expression fine-grained features obtained in step 3 are fused after dimension reduction by the fully connected layer, and the facial expression category is output by the classifier; Step 5: Based on the effective sample number theory, a class-balanced weighted loss function is designed. By assigning different weights to the majority class and the rare class on the basis of Cross-Entropy Loss, inter-class competition is suppressed, the negative sample supervision signal of the majority class acting on the rare class is alleviated, and the inter-class distance of the expression feature is expanded; The specific method of step 1 is: The public datasets are FER2013 dataset and RAF-DB dataset. First, the datasets are preprocessed. The face area in the image is detected using the MTCNN face detection algorithm, which is cropped from the original image and scaled to a uniform size of 224×224. For single-channel images, the single-channel images are converted into three-channel images by the replication method, and data enhancement is performed by random cropping and random horizontal flipping. The specific method of step 3 is: The network structure is based on the ResNet34 network model, and multiple lightweight attention mechanism modules are embedded in the network to construct a backbone network. The backbone network BBN takes the original image as input to learn global high-level semantic features. The strengthening branch is placed between the second residual block and the third residual block of the backbone network and differentiated into two branches, which further extract fine-grained features of the expression categories contained in different strong association groups. The specific method of constructing a parallel neural network is as follows: Step 3.1: The middle part of the ResNet34 network is composed of 4 residual blocks connected in series, each of which contains 3, 4, 6, and 3 residual structures respectively. The backbone network takes the original image of size 224×224×3 as input, embeds a lightweight joint attention module LMAM between each two blocks, and extracts global high-level semantic features; Step 3.2: The LMAM described in step 3.1 is composed of SAM and CAM. The feature map F with a size of c×h×w is input into LMAM. SAM learns the relationship between various spatial positions, and then CAM obtains the weights of different channels to obtain the final feature map. SAM first uses AvgPool to compress the input feature map F to obtain a spatial dimension global feature of size 1×h×w, then successively undergoes 5×5 convolution, MaxPool, and 3×3 convolution to reduce the feature size, restores the original size after upsampling, and finally activates it through Sigmoid to obtain a spatial attention map M of size 1×h×w. SA , M SA Multiply element-wise with F to obtain the feature map F optimized by the spatial attention map SA It is expressed as: M SA =SAM(F)=δ(Upsample(f conv-pool (AvgPool(x)))) (1) Where: f conv-pool represents 5×5 convolution, MaxPool, 3×3 convolution cascade structure, δ represents Sigmoid activation operation; Step 3.3: F SA Input into CAM to further obtain the weight information of different channels. First, use AvgPool and MaxPool to calculate F SA After compression, two channel features M with a size of c×1×1 are obtained after being processed by a double-layer fully connected layer. CA_Avg and M CA_Max , M CA_Avg and M CA_Max After element-by-element addition and Sigmoid activation, we get the spatial attention map M CA , M CA Multiply element-wise with F to generate a feature map F that is doubly optimized by the spatial attention map and the channel attention map LMAM It is expressed as: M CA =CAM(F)=(δ(f FC1 (AvgPool(F SA )))+δ(f FC2 (MaxPool(F SA )))) (3) Where: f FC1 and f FC2 represents a double-layer fully connected layer, and δ represents a Sigmoid activation operation; Step 3.4: The reinforcement branch network DRN consists of two small branches with the same structure in parallel, which respectively strengthen the expression categories in different strongly associated sets. Each branch is composed of the third residual block in ResNet34 and a CAM to form a fine-grained feature extraction unit. The extracted fine-grained features of similar expression differences are expressed as: Among them: F DRN1 and F DRN2 They represent the fine-grained features of similar expressions extracted by branch 1 and branch 2 respectively.

2. The method for facial expression recognition based on inter-class difference enhancement network according to claim 1, characterized in that: The specific method of step 2 is: Based on the preprocessed FER2013 and RAF-DB expression datasets, balanced sets with equal number of samples in each expression category were constructed respectively. The two balanced sets were sent to three classic convolutional neural networks for training and testing to obtain the corresponding confusion matrices, and the expression similarity relationship was analyzed from the confusion matrices. Among them, the three classic convolutional neural networks include ResNet18, ResNet34, and ResNet50. The specific similarity analysis method is as follows: Step 2.1: In order to reduce the impact of the long tail problem of the data, the FER2013 dataset and the RAF-DB dataset are randomly undersampled to achieve data averaging. Taking the data volume of the shortest class as the benchmark, the same number of pictures are randomly selected from other categories to form a balanced set, which is denoted as: FER2013 Balance ,RAFDB Balance ; Step 2.2: Use ResNet18, ResNet34, and ResNet50 as the network model for the prior experiment, and convert the FER2013 obtained in step 2.1 Balance and RAFDB Balance Send them to three networks for training and testing respectively, and get 6 confusion matrices; Step 2.3: The elements on the main diagonal of the confusion matrix represent the correct classification results, and the remaining positions are the wrong classification results. For each confusion matrix, the probability of each expression being misclassified is arranged from large to small, and the expression categories corresponding to the first three probability values ​​are retained and recorded as the strongly associated expressions of the category. The strongly associated expressions in the 6 confusion matrices are counted and the similarity relationship matrix is ​​drawn. The elements outside the main diagonal of the matrix represent the similarity values ​​between various expressions; Step 2.4: Group the two expression categories with similarity values ​​greater than or equal to 4 into a group of strongly associated sets. In this way, the seven types of expressions can be divided into two strongly associated sets, which are recorded as: Group SL1 ={Angry,Fearful,Surprised},Group SL2 ={Happy, Disgusted, Sad, Neutral}.

3. The facial expression recognition method based on the inter-class difference enhancement network according to claim 1 is characterized in that: The specific method of step 4 is: The high-level semantic features F BBN and similar expression fine-grained features F DRN1 and F DRN2 The fully connected layers are used for dimensionality reduction, and then feature fusion is performed to obtain the final output feature F ALL , the classifier outputs the facial expression category, and the specific feature fusion method is: Step 4.1: F BBN After being sent to AvgPool and a single fully connected layer for dimensionality reduction, the 7-dimensional output vector is expressed as: V BBN =[c AN ,c DI ,c FE ,c HA ,c SA ,c SU ,c NE ] (6) Where: c i ,i∈{AN,DI,FE,HA,SA,SU,NE} represents the output value of each type of expression, and the subscript represents the expression category information corresponding to the value; Step 4.2: F DRN1 and F DRN2 After dimensionality reduction by AvgPool and a single-layer fully connected layer, a 7-dimensional output feature vector is obtained. Each element of the feature vector corresponds to the output value of an expression: Where: V DRN1 represents the output feature vector of branch 1, V DRN2 represents the output feature vector of branch 2; a i ,i∈{AN,DI,FE,HA,SA,SU,NE} represents the vector V DRN1 The output value of each type of expression in the expression, the subscript indicates the expression category information corresponding to the value, b i ,i∈{AN,DI,FE,HA,SA,SU,NE} represents the vector V DRN2 The output value of each type of expression in the subscript indicates the expression category information corresponding to the value; in order to achieve the strengthening effect, V DRN1 Keep only Group SL1 The numerical information corresponding to the three expression categories contained in V DRN2 Keep only Group SL2 The numerical information corresponding to the four expression categories contained in , and the elements in the other positions are set to zero. According to the strong association set divided in step 2, the corrected output vector is expressed as: The final output vector ECV of the enhanced branch is obtained by adding the two corrected eigenvectors at the corresponding positions, which is expressed as: V DRN =[a AN ,b DI ,a FE ,b HA ,b SA ,a SU ,b NE ] (9) Among them, a i ,i∈{AN,FE,SU} is a vector V DRN1 The output values ​​corresponding to the various expressions in the expression, the subscript indicates the expression category corresponding to the value, b j ,j∈{DI,HA,SA,NE} is a vector V DRN2 The output values ​​corresponding to the various expressions in , the subscript indicates the expression category corresponding to the value; Step 4.3: V BBN and V DRN The elements are added element by element in a certain ratio. During the training process, the fusion ratio is continuously adjusted by controlling the dynamic balance coefficient α, so that the network's attention is transferred between the backbone network and the reinforcement branch. During the test, α is set to a constant value of 0.

9. The functional relationship between the balance parameter α of feature fusion and the number of iterations is expressed as: Among them: Epoch Current and Epoch Max Respectively represent the current number of iterations and the maximum number of iterations, Epoch Current ≥1, the final feature obtained after fusion is expressed as: V ALL =αV BBN +(1-α)V DRN (11).

4. The method for facial expression recognition based on inter-class difference enhancement network according to claim 1, characterized in that: The specific method of step 5 is: From the perspective of class-balanced combination weighting, rare classes are protected on the basis of the cross entropy loss function, inter-class differences are strengthened, and inter-class distances are increased. The specific class-balanced combination weighting method is as follows: Step 5.1: First, give a heuristic definition of the effective sample number F(n): Where: k is a coefficient, n represents the actual number of samples, and F(n) represents the effective number of samples. This heuristic definition better describes the marginal diminishing effect. When n is large enough, further increasing n cannot meet the effective sample capacity. Based on this definition, the weights of each expression category are redefined: Where: w c , c∈[1,7] represents the weight of each expression category, c represents the cth category, n c represents the actual number of samples in category c; Step 5.2: Based on the definition of weighted weights for each expression category in step 5.1, construct Weighted Cross-Entropy Loss: Where: c represents the true value label of the expression category, represents the predicted output probability value of the expression, w c represents the weight of the c-th category expression defined in step 5.1.

Images