A facial expression recognition method and system

Abstract

The invention discloses a facial expression recognition method and system, which belongs to the field of expression recognition technology, including: processing video data to obtain a face image frame data set; inputting each face image frame into a residual network to extract facial expression texture features and local fineness features; inputting facial expression texture features into a feature pyramid module to obtain multi-scale three-dimensional space features, superimposing local fineness features on them, and obtaining multi-scale and spatial context features; inputting multi-scale and spatial context features into a dual cascade module to obtain global key spatial domain context features; inputting global key spatial domain context features into a recurrent neural network to extract multiple context features including time features; inputting multiple context features into a fully connected layer to classify and recognize facial expressions. The invention can effectively improve the accuracy of facial expression recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of expression recognition, and more specifically, relates to a facial expression recognition method and system. Background Art

[0002] Facial expression is a form of non-verbal communication. It is the main means of expressing social information between humans. It is a physiological and psychological reaction and is usually used to convey emotions. The main application areas of facial expression recognition technology include intelligent control, security, medical treatment, communication, education and other fields.

[0003] At present, there are two main branches of facial expression recognition technology, namely static facial expression frame recognition and dynamic video recognition containing facial expressions. Most of the existing facial expression recognition uses static facial expression frame recognition, that is, through video frame extraction, facial expression frames are collected and facial expression related features (such as: the movement of the eyes and eyebrows in the eye area, the movement of the mouth area or the small movement features of the cheeks on both sides) are extracted to classify the facial expressions. Static facial expression recognition only considers the current image features, but facial expression is a dynamic process. Static image recognition ignores the changes in expression in the time dimension and space dimension, and has limitations.

[0004] At present, the mainstream method for facial recognition based on static images is the convolutional neural network (CNN network). The CNN network uses convolution operations to extract features from facial images. Through the convolution operation of the CNN network, low-level features such as color and edges can be learned at the shallow level. The features learned in the middle layer of the network will become slightly more complex, and some texture features (such as grid texture features) will be learned. In the high layer of the network, key features with distinction will be learned. However, when learning the CNN network, only the local features of the face image can be learned, and the overall spatial domain features of the image cannot be learned, and the context information of the image is lacking. If the number of layers of the CNN network is too high, it will lead to problems such as gradient disappearance or gradient explosion, and the CNN network cannot effectively suppress the factors that affect facial expression recognition.

[0005] The residual network is a type of convolutional neural network, which is easy to optimize and can improve accuracy by increasing the depth. The residual block inside it uses skip connections, which alleviates the gradient vanishing and gradient exploding problems caused by increasing the depth in the deep neural network, and solves the problem of network degradation to a certain extent. The fully connected layers (FC) play the role of "classifier" in the entire convolutional neural network, and the fully connected layers play the role of mapping the learned "feature representation" to the sample label space.

[0006] Feature pyramid is a basic component in the recognition system for detecting objects of different scales. Recognizing objects at multiple scales is a challenge in computer vision. By extracting multi-scale feature information and fusing them, the accuracy of the model can be improved. Feature pyramid has been widely used in target detection, instance segmentation, posture recognition, face recognition and other fields.

[0007] In summary, although the existing technology can achieve good recognition performance, it still has shortcomings. Many methods use experimentally induced expression data to verify the effect of the method, but it is extremely difficult to improve the accuracy of facial expression recognition. In this context, overcoming the influence of the above factors and increasing the robustness of the method is a major issue in facial expression recognition under experimental conditions. Summary of the invention

[0008] In view of the defects of the prior art, the purpose of the present invention is to provide a facial expression recognition method and system, aiming to solve the problem of the accuracy of existing facial expression recognition.

[0009] To achieve the above object, the present invention provides a facial expression recognition method, comprising the following steps:

[0010] (1) Video data processing step: performing frame extraction operation on the video data to obtain a set of single image frames, extracting key feature points from each image frame, correcting and aligning each image frame based on the position coordinates of the key feature points, and generating a face image frame data set. The following steps are all performed on each face image frame in the face image frame data set;

[0011] (2) Facial expression feature extraction step: input the face image frame into the residual network to extract facial expression texture features and local fineness features;

[0012] (3) Multi-scale and spatial context feature extraction step: Perform feature pyramid extraction on facial expression texture features to obtain multi-scale three-dimensional spatial features; superimpose and fuse them with local fineness features to obtain multi-scale and spatial context features;

[0013] (4) Global key spatial domain context feature extraction step: The multi-scale and spatial context features are directly or compressed into the self-attention and spatial attention for processing in turn to obtain the global key spatial domain context features;

[0014] (5) Temporal feature extraction step: inputting the global key spatial domain context features into the recurrent neural network to extract multiple context features of facial expressions including temporal context;

[0015] (6) Facial expression recognition step: The extracted multiple context features of facial expressions are input into the fully connected layer, and the facial expressions are classified and recognized using the probability distribution calculation method in the fully connected layer.

[0016] In an optional embodiment, the video data processing step includes the following sub-steps:

[0017] (1.1) Image frame extraction sub-step: separating the video data into a group of single image frames, extracting them at intervals and saving them as key image frames;

[0018] (1.2) Key feature point extraction sub-step: Detect the extracted key image frame, obtain the position coordinates of the four endpoints of the face detection rectangular frame and the key feature points, and use the face detection rectangular frame to crop the key image frame to obtain a face image frame; the key feature points are respectively located in the eye, nose, and mouth corner areas;

[0019] (1.3) Face alignment sub-step: Correct the face based on the key feature points so that the line connecting the coordinates of the two eyes is parallel to the horizontal coordinate axis. Using the coordinates of the key feature points of the first face image frame as a reference, align the key feature points of subsequent face image frames on the same horizontal line.

[0020] (1.4) Generate a face image frame dataset sub-step: adjust each face image frame to a uniform size and perform normalization processing to obtain a face image frame dataset.

[0021] In an optional embodiment, the facial expression feature extraction step is specifically:

[0022] The face image frame is input into the residual network to obtain a set of facial expression features from the primary level to the high level, including the color and edge features of the facial expression at the primary level, the texture features of the facial expression at the middle level, and the high level local fineness features.

[0023] In an optional embodiment, the multi-scale and spatial context feature extraction step includes the following sub-steps:

[0024] (3.1) Obtaining feature maps of different scales: scaling the facial expression texture features into multiple feature maps of different scales;

[0025] (3.2) Obtaining multi-scale stereoscopic spatial features sub-step: Inputting feature maps of different scales into scale-agnostic attention for processing, obtaining spatial context features of feature maps of different scales, wherein the scale-agnostic attention uses feature similarities at different scales to weight spatial context features of different scales; the spatial context features of each large-scale feature map are also input into the scale-agnostic attention connected to the adjacent small-scale feature maps for processing, thereby obtaining multi-scale stereoscopic spatial features;

[0026] (3.3) Obtaining multi-scale and spatial context features sub-step: superimpose and fuse the multi-scale stereoscopic spatial features and the local fineness features to obtain multi-scale and spatial context features.

[0027] In an optional embodiment, the step of extracting global key spatial domain context features includes the following sub-steps:

[0028] (4.1) Self-attention processing sub-step. Input the multi-scale and spatial context features into the self-attention and perform linear layer operations to obtain self-attention weights, assign the weights to the multi-scale and spatial context features, and obtain new multi-scale and spatial context features;

[0029] (4.2) Spatial attention processing sub-step. The new multi-scale and spatial context features are input into the spatial attention for convolution and average pooling operations, the spatial attention weights in the frame dimension are extracted, and the features of multiple image frames are fused. The expression changes between adjacent frames are calculated to obtain the global key spatial domain context features.

[0030] In an optional embodiment, in step (3), the multi-scale and spatial context features are compressed and input into the self-attention and spatial attention for processing in sequence, and the compression is specifically as follows:

[0031] The multi-scale and spatial context features outputted in step (3) are inputted into the global average pooling layer for feature compression to obtain compressed multi-scale and spatial context features.

[0032] In a second aspect, the present invention provides a facial expression recognition system, comprising the following units:

[0033] Video data processing unit: performs frame extraction operation on video data to obtain a set of single image frames, extracts key feature points from each image frame, corrects and aligns each image frame based on the position coordinates of the key feature points, and generates a face image frame data set. The following units all process each face image frame in the face image frame data set;

[0034] Facial expression feature extraction unit: input the face image frame into the residual network to extract facial expression texture features and local fineness features;

[0035] Multi-scale and spatial context feature extraction unit: Perform feature pyramid extraction on facial expression texture features to obtain multi-scale three-dimensional spatial features; superimpose and fuse them with local fineness features to obtain multi-scale and spatial context features;

[0036] Global key spatial domain context feature extraction unit: directly or compressedly input multi-scale and spatial context features into self-attention and spatial attention for processing in sequence to obtain global key spatial domain context features;

[0037] Temporal feature extraction unit: inputs global key spatial domain context features into the recurrent neural network to extract multiple context features of facial expressions including temporal context;

[0038] Facial expression recognition unit: The extracted multiple context features of facial expressions are input into the fully connected layer, and the facial expressions are classified and recognized using the probability distribution calculation method in the fully connected layer.

[0039] In an optional embodiment, the video data processing unit includes the following subunits:

[0040] Image frame extraction subunit: separates the video data into a group of single image frames, extracts them at intervals and saves them as key image frames;

[0041] Key feature point extraction subunit: detect the extracted key image frame, obtain the position coordinates of the four endpoints of the face detection rectangular frame and the key feature points, and use the face detection rectangular frame to crop the key image frame to obtain a face image frame; the key feature points are respectively located in the eye, nose, and mouth corner areas;

[0042] Face alignment subunit: Correct the face based on key feature points so that the line connecting the two eye coordinates is parallel to the horizontal coordinate axis. Based on the key feature point coordinates of the first face image frame, align the key feature points of subsequent face image frames on the same horizontal line.

[0043] Generate a face image frame data set subunit: adjust each face image frame to a uniform size and perform normalization processing to obtain a face image frame data set.

[0044] In an optional embodiment, the global key spatial domain context feature extraction unit includes the following subunits:

[0045] Self-attention processing subunit: input the multi-scale and spatial context features into the self-attention subunit, use the linear layer operation to obtain the self-attention weight, assign the weight to the multi-scale and spatial context features, and obtain new multi-scale and spatial context features;

[0046] Spatial attention processing subunit: The new multi-scale and spatial context features are input into the spatial attention subunit for convolution and average pooling operations, the spatial attention weights in the frame dimension are extracted, and the features of multiple image frames are fused, and the expression changes between adjacent frames are calculated to obtain the global key spatial domain context features.

[0047] In an optional embodiment, in the global key spatial domain context feature extraction unit, the multi-scale and spatial context features are compressed and input into the self-attention and spatial attention for processing in sequence, and the compression is specifically:

[0048] The multi-scale and spatial context features output by the multi-scale and spatial context feature extraction unit are input into the global average pooling layer for feature compression to obtain compressed multi-scale and spatial context features.

[0049] Compared with the prior art, the above technical solution conceived by the present invention has the following beneficial effects:

[0050] The present invention provides a facial expression recognition method and system, which extracts facial expression texture features and local fineness features through a residual network, inputs the facial texture features into a feature pyramid extraction module to obtain multi-scale three-dimensional spatial features, fuses them with local fineness features, highlights key areas, eliminates irrelevant interference features, and obtains richer facial expression features. The present invention adds self-attention and spatial attention to a convolutional neural network model, and fuses the multi-scale three-dimensional spatial features extracted by the feature pyramid module, effectively improving the accuracy of facial expression recognition under the influence of multiple factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the facial expression recognition method of the present invention;

[0052] Figure 2 is a flow chart of an embodiment of the present invention;

[0053] Figure 3 A structural diagram of a feature pyramid extraction module provided by an embodiment of the present invention;

[0054] Figure 4 A diagram of the self-attention and spatial attention structures provided by an embodiment of the present invention;

[0055] Figure 5 The confusion matrix result diagram of expression classification on the CK+ dataset by the method provided in the embodiment of the present invention;

[0056] Figure 6 A confusion matrix result diagram of expression classification on the JAFFE dataset for the method provided in an embodiment of the present invention;

[0057] Figure 7 The confusion matrix result diagram of expression classification on the Oulu-CASIA dataset by the method provided in the embodiment of the present invention;

[0058] Figure 8 Flow chart of the facial expression recognition system of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0060] Figure 1 It is a flow chart of the facial expression recognition method of the present invention, as shown in Figure 1 As shown, the following steps are included:

[0061] (1) Video data processing step: performing frame extraction operation on the video data to obtain a set of single image frames, extracting key feature points from each image frame, correcting and aligning each image frame based on the position coordinates of the key feature points, and generating a face image frame data set. The following steps are all performed on each face image frame in the face image frame data set;

[0062] (2) Facial expression feature extraction step: input the face image frame into the residual network to extract facial expression texture features and local fineness features;

[0063] (3) Multi-scale and spatial context feature extraction step: Perform feature pyramid extraction on facial expression texture features to obtain multi-scale three-dimensional spatial features; superimpose and fuse them with local fineness features to obtain multi-scale and spatial context features;

[0064] (4) Global key spatial domain context feature extraction step: The multi-scale and spatial context features are directly or compressed into the self-attention and spatial attention for processing in turn to obtain the global key spatial domain context features;

[0065] (5) Temporal feature extraction step: inputting the global key spatial domain context features into the recurrent neural network to extract multiple context features of facial expressions including temporal context;

[0066] (6) Facial expression recognition step: The extracted multiple context features of facial expressions are input into the fully connected layer, and the facial expressions are classified and recognized using the probability distribution calculation method in the fully connected layer.

[0067] Specifically, the facial expression recognition method provided by the present invention is as follows: Figure 2 is a flow chart of an embodiment of the present invention; Figure 2As shown, the following steps are included:

[0068] (1) Video data processing steps: Perform frame extraction operations on the video data to obtain a set of single image frames, extract key feature points from each image frame, correct and align each image frame based on the position coordinates of the key feature points, and generate a face image frame dataset.

[0069] (1.1) Image frame extraction sub-step: Use the open source multimedia tool library FFmpeg to separate the video data into a group of single image frames, extract every 5 frames and save them as key image frames.

[0070] (1.2) Key feature point extraction sub-step: Use the MTCNN method to continuously resize the images in the key image frame to obtain images of different sizes. Stack these images to form an image pyramid and input them into the PNet network to obtain face candidates. Then, screen the candidates using the classification score and NMS method to obtain candidates greater than the set threshold. Input the candidates obtained from PNet into RNet for fine-tuning. Use the classification score and NMS method and a higher threshold to screen the candidates. RNet avoids deformation of the face image and retains more details. Input the candidates screened by RNet into ONet. After the classification score and NMS method, the coordinate sequence of the four endpoints of the face detection rectangular box, bounding_box, is output. t And the coordinate sequence p of the five key feature points (corresponding to the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth) t :

[0071]

[0072] Among them, t is the key image frame number, bounding box t [k] is the coordinates of the endpoints of the face rectangle in the tth frame, 0≤k≤3; is the position coordinate of the key feature point in the tth frame, 1≤i≤5.

[0073] (1.3) Face alignment sub-step: crop the key image frame based on the coordinates of the four endpoints of the face detection rectangle to obtain the face image frame; correct the face based on the key feature points so that the line connecting the coordinates of the two eyes is parallel to the horizontal coordinate axis; use the coordinates of the key feature points in the first face image frame as a reference to align the key feature points in subsequent face image frames on the same horizontal line.

[0074] (1.4) Generate face image frame dataset sub-step: Use the transforms method in the Pytorch framework to resize the aligned face image frame to 256×256, then use the center cropping method to resize the face image frame to 224×224, normalize the RGB channels of the face image frame to the range of [-1, 1], and obtain the normalized face image frame x t :

[0075]

[0076] mean(x)=[0.485, 0.456, 0.406], std(x)=[0.229, 0.224, 0.225]

[0077] Among them, mean(x) is the mean of the three color channels, and std(x) is the standard deviation of each channel.

[0078] Use the Dataloader method in the Pytorch framework to encapsulate the normalized face image frame into a data iterator to obtain the face image frame dataset dataloader:

[0079] dataloader=Dataloader(dataset(x t ), batchsize, transforms)

[0080] Among them, Dataloader is set to batchsize, data processing and normalization operations, dataset(x t ) is the aligned face image frame, batchsize is the amount of data for one training round, and transforms is the method of cropping and normalizing the face image frame.

[0081] (2) Facial expression feature extraction step: Input the face image frame dataset dataloader into the residual network, extract the facial expression features in the face image frame, and obtain the extracted feature data F(x t ):

[0082] F(x t )=ResNet(dataloader)

[0083] Among them, ResNet represents a 50-layer residual network. Ordinary residual networks are troubled by the gradient vanishing and gradient exploding problems caused by too many layers, but the 50-layer residual network uses shortcut-connected residual blocks to cleverly solve the problems of gradient vanishing and gradient exploding. Batch Normalization layers and L2 regularization are also used in the network to prevent network overfitting and other problems.

[0084] (3) Multi-scale and spatial context feature extraction step: Perform feature pyramid extraction operation on facial expression texture features to obtain multi-scale three-dimensional spatial features; superimpose and fuse them with local fineness features to obtain multi-scale and spatial context features.

[0085] (3.1) Sub-steps for obtaining feature maps of different scales: Figure 3 is a structural diagram of a feature pyramid extraction module provided by an embodiment of the present invention; Figure 3 As shown in the figure, the local texture feature map output by the residual network is scaled into five feature maps of different scales {S 1 ;...;S 5}.

[0086] (3.2) Obtaining multi-scale stereoscopic spatial features: Input feature maps of different scales into the scale-agnostic attention respectively to obtain spatial context features of feature maps of different scales. At the same time, the spatial context features of large-scale feature maps are also input into the scale-agnostic attention connected to the adjacent small-scale to obtain multi-scale stereoscopic spatial features F. pa (F C3-layer (x t )):

[0087]

[0088] Among them, σ represents the sigmoid function, S i represents the scaled i-th feature map, F C3-layer (x t ) represents the facial expression texture feature, m represents the number of scales, φ represents the spatial context feature function of feature maps of different scales, z j represents the regional features of feature maps of different scales, and θ represents the correlation weights of context features of different regions in feature maps of different scales.

[0089] (3.3) Obtaining multi-scale and spatial context features sub-step: The output multi-scale stereoscopic spatial features and local fineness features F C4-layer (x t ) to perform superposition and fusion operations to obtain multi-scale and spatial context features F pc (x t ):

[0090]

[0091] in, Represents the superposition operation of features.

[0092] (4) Feature compression step: The multi-scale and spatial context features are input into the global average pooling layer for feature compression to obtain compressed multi-scale and spatial context features.

[0093] (5) Global key spatial domain context feature extraction step: The compressed multi-scale and spatial context features are input into the eye attention and spatial attention for processing in sequence to obtain the global key spatial domain context features.

[0094] (5.1) Self-attention processing sub-step: Figure 4 is a structural diagram of self-attention and spatial attention provided by an embodiment of the present invention. Figure 4 As shown, the multi-scale and spatial context features are input into the self-attention and the self-attention weight AC is obtained by linear layer operation. t (F pc (x t )):

[0095] AC t (F pc (x t ))=σ(Linear(F pc (x t )))

[0096] Among them, Linear represents the linear layer operation.

[0097] The obtained self-attention weights are used to weight the input features to obtain new multi-scale and spatial context features FC t (F pc (x t )):

[0098]

[0099] Wherein, t represents the serial number of the face image frame, where the serial number of the face image frame is consistent with the serial number of the key image frame in the key feature point extraction sub-step, and n represents the total number of frames of the face image frame dataset.

[0100] (5.2) Spatial attention processing sub-step: input it into the spatial attention for convolution and average pooling operations to obtain the spatial attention weight AS t (F pc (x t )),

[0101] ASt (F pc (x t ))=σ(f 1 (AvgPool(FC t (F pc (x t )))))

[0102] Among them, f 1 represents the convolution operation, and Avgpool represents the average pooling operation.

[0103] The spatial attention weights are weighted to obtain the global key spatial domain context feature FS t (F pc (x t )):

[0104]

[0105] (6) Temporal feature extraction step: The present invention uses a gated recurrent unit (GRU) as a recurrent neural network to extract temporal features. The gated recurrent unit is simpler than other recurrent neural network structure models, especially in models with deeper networks. The GRU can simultaneously perform forgetting and selective memory through a single gate, which significantly reduces the number of parameters and is more efficient. The global key spatial domain context feature FS t (F pc (x t )) Input to GRU to obtain the facial expression multi-context feature Y including the temporal context t :

[0106] Y t =GRU(FS t (F pc (x t )))

[0107] (7) Facial expression recognition step: Feature Y obtained by GRU unit t The data is input into the fully connected layer, and the final facial expression classification result is obtained through the calculation of the Sigmoid function of the fully connected layer.

[0108] After the above steps, facial expression recognition of face image frames is realized. Cross entropy loss is used as the loss function in the network training process, and the loss function value is optimized by stochastic gradient descent optimizer (SGD). Sigmoid is used as the activation function, the weight decay hyperparameter is set to 1e-4, and the initial value of the learning rate is 0.001. The learning rate is dynamically adjusted during the network training process, and finally the optimal result is achieved.

[0109] The method of the present invention uses accuracy, confusion matrix, and receiver operating characteristic curve (ROC) area as specific evaluation indicators for facial expression recognition. The larger the accuracy value and the receiver operating characteristic curve ROC area, the better the recognition performance; the confusion matrix is ​​an indicator used for accuracy evaluation, which is used to compare the prediction results and the true classification value to obtain the prediction accuracy of each expression.

[0110] Figure 5 This is the confusion matrix result diagram of expression classification using the method of the present invention on the CK+ dataset. Figure 6 This is the confusion matrix result diagram of expression classification based on the JAFFE dataset by the method of the present invention. Figure 7 It is the result diagram of the confusion matrix of expression classification of the method of the present invention on the Oulu-CASIA data set. Among them, the CK+ data set and the JAFFE data set both include seven facial expressions, and the Oulu-CASIA data set includes six facial expressions. The average result is obtained by five-fold cross validation as the accuracy of the method of the present invention on each data set. The five-fold cross validation is to divide the data set into five equal parts, take one part for testing each time, and use the rest for training, and the average value is calculated by five experiments.

[0111] Specifically, the average results of the method of the present invention obtained by five-fold cross validation on the CK+ dataset and the accuracy comparison of other methods are shown in Table 1:

[0112] Table 1: Comparison of the accuracy of various methods on the CK+ dataset

[0113]

[0114] Specifically, the average results of the method of the present invention obtained through a five-fold cross validation experiment on the JAFFE dataset and the accuracy comparison of other methods are shown in Table 2:

[0115] Table 2: Comparison of the accuracy of various methods on the JAFFE dataset

[0116]

[0117] Specifically, the average results of the method of the present invention obtained by a five-fold cross validation experiment on the Oulu-CASIA dataset and the accuracy comparison of other methods are shown in Table 3:

[0118] Table 3: Comparison of the accuracy of various methods on the Oulu-CASIA dataset

[0119]

[0120] It can be seen from Tables 1, 2 and 3 that the performance of the facial expression recognition method constructed by the present invention on the three data sets can reach a relatively good level.

[0121] Table 4 shows the ROC area of ​​the present invention on three data sets. The ROC curve is one of the performance indicators for measuring the quality of deep learning methods. The ROC area is between 0.5 and 1. The larger the area, the better the method effect. It can be seen from Table 4 that the ROC area of ​​the present invention on the three data sets is much larger than 0.5, indicating that the classification effect of the present method on facial expression recognition has reached a relatively good level.

[0122] Table 4: ROC area on different data sets

[0123]

[0124] Figure 8 FIG. 1 is a flow chart of a facial expression recognition system provided by an embodiment of the present invention. Figure 8 As shown, it includes the following units:

[0125] Video data processing unit: performs frame extraction operation on video data to obtain a set of single image frames, extracts key feature points from each image frame, corrects and aligns each image frame based on the position coordinates of the key feature points, and generates a face image frame data set. The following units all process each face image frame in the face image frame data set;

[0126] Facial expression feature extraction unit: input the face image frame into the residual network to extract facial expression texture features and local fineness features;

[0127] Multi-scale and spatial context feature extraction unit: Perform feature pyramid extraction on facial expression texture features to obtain multi-scale three-dimensional spatial features; superimpose and fuse them with local fineness features to obtain multi-scale and spatial context features;

[0128] Global key spatial domain context feature extraction unit: directly or compressedly input multi-scale and spatial context features into self-attention and spatial attention for processing in sequence to obtain global key spatial domain context features;

[0129] Temporal feature extraction unit: inputs global key spatial domain context features into the recurrent neural network to extract multiple context features of facial expressions including temporal context;

[0130] Facial expression recognition unit: The extracted multiple context features of facial expressions are input into the fully connected layer, and the facial expressions are classified and recognized using the probability distribution calculation method in the fully connected layer.

[0131] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A facial expression recognition method, characterized in that: The steps are as follows: (1) Video data processing step: Perform frame extraction operation on the video data to obtain a set of single image frames, extract key feature points from each image frame, correct and align each image frame based on the position coordinates of the key feature points, and generate a face image frame dataset; (2) Facial expression feature extraction step: input each face image frame in the face image frame dataset into the residual network to extract facial expression texture features and local fineness features; (3) Multi-scale and spatial context feature extraction step: Perform feature pyramid extraction on facial expression texture features to obtain multi-scale three-dimensional spatial features, which are then superimposed and fused with local fineness features to obtain multi-scale and spatial context features; (4) Global key spatial domain context feature extraction step: The multi-scale and spatial context features are directly or compressed into the self-attention and spatial attention for processing in sequence to obtain the global key spatial domain context features; (5) Temporal feature extraction step: inputting the global key spatial domain context features into the recurrent neural network to extract multiple context features of facial expressions including temporal context; (6) Facial expression recognition step: The extracted multiple context features of facial expressions are input into the fully connected layer, and the facial expressions are classified and recognized using the probability distribution calculation method in the fully connected layer; The facial expression feature extraction step is specifically as follows: inputting the face image frame into the residual network to obtain a set of facial expression features, including primary-level facial expression color and edge features, mid-level facial expression texture features, and high-level local fineness features; The multi-scale and spatial context feature extraction step includes the following sub-steps: (3.1) Obtaining feature maps of different scales: scaling the facial expression texture features into multiple feature maps of different scales; (3.2) Obtaining multi-scale stereoscopic spatial features sub-step: Inputting feature maps of different scales into scale-agnostic attention for processing, obtaining spatial context features of feature maps of different scales, wherein the scale-agnostic attention uses feature similarities at different scales to weight spatial context features of different scales; the spatial context features of each large-scale feature map are also input into the scale-agnostic attention connected to the adjacent small-scale feature maps for processing, thereby obtaining multi-scale stereoscopic spatial features; (3.3) Obtaining multi-scale and spatial context features sub-step: superimposing and fusing the multi-scale stereoscopic spatial features and the local fineness features to obtain the multi-scale and spatial context features; The global key spatial domain context feature extraction step includes the following sub-steps: (4.1) Self-attention processing sub-step: input the multi-scale and spatial context features into the self-attention and perform linear layer operations to obtain self-attention weights, assign the weights to the multi-scale and spatial context features, and obtain new multi-scale and spatial context features; (4.2) Spatial attention processing sub-step: Input the new multi-scale and spatial context features into the spatial attention for convolution and average pooling operations, extract the spatial attention weights in the frame dimension, fuse the features of multiple image frames, calculate the expression changes between adjacent frames, and obtain the global key spatial domain context features; In the step (3), the multi-scale and spatial context features are compressed and input into the self-attention and spatial attention for processing in sequence. The compression is specifically as follows: The multi-scale and spatial context features outputted in step (3) are inputted into the global average pooling layer for feature compression to obtain compressed multi-scale and spatial context features.

2. A facial expression recognition method according to claim 1, characterized in that: The video data processing step includes the following sub-steps: (1.1) Image frame extraction sub-step: separating the video data into a group of single image frames, extracting them at intervals and saving them as key image frames; (1.2) Key feature point extraction sub-step: extracting the position coordinates of the four endpoints of the face detection rectangular frame and the key feature points from the key image frame, and using the face detection rectangular frame to crop the key image frame to obtain a face image frame; the key feature points are respectively located in the eye, nose, and mouth corner areas; (1.3) Face alignment sub-step: Correct the face based on the key feature points so that the line connecting the two eye coordinates is parallel to the horizontal coordinate axis, and align the key feature points of subsequent face image frames on the same horizontal line based on the key feature point coordinates of the first face image frame; (1.4) Generate a face image frame dataset sub-step: adjust each face image frame to a uniform size and perform normalization processing to obtain a face image frame dataset.

3. A facial expression recognition system, characterized in that: The following units are included: Video data processing unit: performs frame extraction operation on video data to obtain a set of single image frames, extracts key feature points from each image frame, corrects and aligns each image frame based on the position coordinates of the key feature points, and generates a face image frame data set. The following units all process each face image frame in the face image frame data set; Facial expression feature extraction unit: input the face image frame into the residual network to extract facial expression texture features and local fineness features; Multi-scale and spatial context feature extraction unit: performs feature pyramid extraction operation on facial expression texture features to obtain multi-scale three-dimensional spatial features; It is superimposed and fused with the local fineness feature to obtain multi-scale and spatial context features; Global key spatial domain context feature extraction unit: directly or compressedly input multi-scale and spatial context features into self-attention and spatial attention for processing in sequence to obtain global key spatial domain context features; Temporal feature extraction unit: inputs global key spatial domain context features into the recurrent neural network to extract multiple context features of facial expressions including temporal context; Facial expression recognition unit: The extracted multiple context features of facial expressions are input into the fully connected layer, and the facial expressions are classified and recognized using the probability distribution calculation method in the fully connected layer; The multi-scale and spatial context feature extraction unit comprises: Feature map sub-units of different scales: scaling facial expression texture features into multiple feature maps of different scales; Multi-scale stereoscopic spatial feature subunit: input feature maps of different scales into scale-agnostic attention for processing, and obtain spatial context features of feature maps of different scales. The scale-agnostic attention uses feature similarities at different scales to weight spatial context features of different scales; the spatial context features of each large-scale feature map are also input into the scale-agnostic attention connected to the adjacent small-scale feature maps for processing, and obtain multi-scale stereoscopic spatial features; and Multi-scale and spatial context feature subunit: multi-scale stereoscopic features and local fineness features are superimposed and fused to obtain multi-scale and spatial context features; The global key spatial domain context feature extraction unit comprises: Self-attention processing subunit: inputs the multi-scale and spatial context features into the self-attention for linear layer operation, obtains the self-attention weights, assigns the weights to the multi-scale and spatial context features, and obtains new multi-scale and spatial context features; and Spatial attention processing subunit: The new multi-scale and spatial context features are input into the spatial attention for convolution and average pooling operations, the spatial attention weights in the frame dimension are extracted, and the features of multiple image frames are fused, and the expression changes between adjacent frames are calculated to obtain the global key spatial domain context features.

4. A facial expression recognition system according to claim 3, characterized in that: After the video data processing unit, the following sub-units are also included: Image frame extraction subunit: separates the video data into a group of single image frames, extracts them at intervals and saves them as key image frames; Key feature point extraction subunit: detect the extracted key image frame, obtain the position coordinates of the four endpoints of the face detection rectangular frame and the key feature points, and use the face detection rectangular frame to crop the key image frame to obtain a face image frame; the key feature points are respectively located in the eye, nose, and mouth corner areas; Face alignment subunit: Correct the face based on key feature points so that the line connecting the two eye coordinates is parallel to the horizontal coordinate axis. Based on the key feature point coordinates of the first face image frame, align the key feature points of subsequent face image frames on the same horizontal line. Generate a face image frame data set subunit: adjust each face image frame to a uniform size and perform normalization processing to obtain a face image frame data set.

5. A facial expression recognition system according to claim 3, characterized in that: In the global key spatial domain context feature extraction unit, the multi-scale and spatial context features are compressed and input into the self-attention and spatial attention for processing in sequence. The compression is specifically as follows: The multi-scale and spatial context features output by the multi-scale and spatial context feature extraction unit are input into the global average pooling layer for feature compression to obtain compressed multi-scale and spatial context features.

Images