Mixed emotion recognition method based on facial expressions and body postures

Abstract

The invention relates to a multi-modal emotion recognition method based on facial expressions and body postures. The method specifically comprises the following steps: firstly, extracting human body key point coordinates of a video frame, and respectively taking change values of facial key point coordinates relative to a natural state and body posture key point coordinates as original information features of facial expressions and body postures so as to reduce data dimensionality and calculation cost; secondly, using fast Fourier transform to capture periodic change of input data, converting one-dimensional data into two-dimensional data, and then using a two-dimensional convolution kernel to perform spatial-temporal feature coding and extraction on two groups of features to enhance the characterization capability of the data; and finally, dynamically distributing each modal weight by using a fusion algorithm to obtain an optimal fusion effect. By enhancing the extraction of the spatio-temporal features, the multi-modal features can be effectively fused, and the accuracy and stability of emotion recognition can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence emotion recognition, and in particular to a multimodal emotion recognition method based on facial expressions and body postures. Background Art

[0002] Multimodal data describes objects from different perspectives and provides more information than unimodal data. Facial expressions and body movements are the most natural, intuitive and commonly used ways for humans to express their emotions, and therefore are two important modalities for studying user emotions. In addition, emotion recognition through facial expressions and body gestures. Vision-based emotion recognition methods have played an increasingly important role in psychopathology, public safety, human-computer interaction and other fields due to their advantages such as accessibility, intuitiveness, non-contact, and accuracy. Thanks to the rapid development of deep learning, the field of emotion recognition has made significant progress in recent years. However, the field of emotion recognition based on facial expressions and body gestures still has the following problems:

[0003] (1) Methods that directly analyze the RGB pixel features of image sequences. However, due to factors such as occlusion, angle deviation, and image blur, which lead to the loss of pixel information, many methods are often limited to a single dataset and have weak generalization capabilities.

[0004] (2) Deep learning models require a lot of computing resources and time to process raw video pixels, and it is difficult to effectively highlight the role of key parts of the face and body in emotion recognition.

[0005] (3) Some people use the CNN+RNN architecture to model the spatial and temporal relationships of features respectively. However, independent learning of spatial and temporal features may cause partial information loss, and the resulting cascaded spatiotemporal features are often not the best features for emotion recognition.

[0006] (4) Deep learning algorithms such as recursive neural networks and convolutional neural networks have gradually shown their advantages in the field of body posture emotion recognition. Although this method performs well in distinguishing high positive emotions from high negative emotions, there is still room for improvement in the accuracy of fine-grained emotion recognition. Summary of the invention

[0007] The purpose of the present invention is to provide a multimodal emotion recognition algorithm based on facial expressions and body postures with enhanced spatiotemporal feature extraction, so as to solve the above-mentioned problems such as poor generalization, high computational cost, and low attention to temporal features, and to more effectively improve the accuracy and stability of emotion recognition.

[0008] The present invention can be implemented by the following technical scheme: a multimodal emotion recognition method based on facial expression and body posture, comprising the following steps:

[0009] S1. Use the open source OpenPose model to obtain the 2D coordinates of the facial and skeleton key points of each frame as the original feature information, and perform preprocessing:

[0010] The coordinate origin is unified from the upper left corner of the screen to the tip of the nose, so that the representation of the eigenvalues ​​is more uniform, the position difference is eliminated, the eigenvalues ​​are more consistent, and the geometric relationship between each feature point is better reflected.

[0011] The facial and posture features are rotated, and the rotation angle is calculated based on the position of the eyes and chest to achieve horizontal alignment of feature points and eliminate angle deviation during shooting.

[0012] Based on the distance between the chin and forehead key points and the ratio of the pre-set parameters, the face size is scaled proportionally to unify the effect of inconsistent distance from the lens.

[0013] S2. The change values ​​of facial key point coordinates relative to the natural state and the coordinates of body posture key points are used as the original information features of facial expression and body posture, and input into the defined FB-TimesNet model for training:

[0014] For facial feature points, the distance between the feature points in each frame and the coordinates of the feature points in the normal state is calculated as the facial information features of the model input.

[0015] For posture features, since some frames do not contain all key points, the missing values ​​are set to 0, and the body posture features used as model input are obtained.

[0016] S3. Construct the feature extraction layer, feature fusion layer and classification layer of the multimodal emotion recognition deep learning model to capture rich spatiotemporal information and facilitate the subsequent classification of emotion categories. Use the multimodal fusion module to adjust the weights of the two modalities to improve the accuracy of emotion recognition after feature fusion:

[0017] The input data is discrete Fourier transformed, and the periods of all frequencies are sorted in descending order according to the amplitude. After removing meaningless high-frequency periods, the first k frequencies are extracted as the periodic features of the time series data.

[0018] The data sequence is segmented according to the cycle length, and the cutting results under the same cycle length are spliced ​​to obtain two-dimensional data. Different cycle lengths correspond to different two-dimensional results.

[0019] The feature extraction layer processes k frequency features using three two-dimensional convolutional modules with the same input and output sizes. Different convolutional kernels are used to extract features, and the results of the three convolutional modules are combined to extract temporal features. Then, two two-dimensional convolutional blocks are used to extract spatial features using a similar method.

[0020] The feature fusion layer adaptively fuses the output results of the feature extraction layer, calculates the relative importance of the selected frequencies and periods, and uses the aggregated tensor after residual connection as input again to obtain deeper information through multi-layer extraction.

[0021] The classification layer is processed by the fully connected layer. GeLU() is the activation function, which facilitates the convergence of the gradient descent optimization algorithm. The Dropout() method is used to reduce the excessive dependence of the neural network on specific neurons, thereby improving the generalization ability of the model.

[0022] The multimodal fusion module uses decision-level fusion to process multimodal features and dynamically changes the weights according to the importance of each modality, thereby flexibly adjusting the contribution between modalities and improving the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0024] Figure 2 Schematic diagram of the multimodal model process constructed in the embodiment. DETAILED DESCRIPTION

[0025] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Example

[0027] like Figure 1 As shown, an emotion recognition method based on facial expression and body posture includes the following steps:

[0028] S1. Use the open source OpenPose model recognition to obtain the 2D coordinates of the facial and skeleton key points of each frame as the original feature information, and perform preprocessing.

[0029] S2. The change values ​​of facial key point coordinates relative to the natural state and the coordinates of body posture key points are used as the original information features of facial expressions and body postures, and input into the defined FB-TimesNet model for awakening training.

[0030] S3. Construct a multimodal emotion recognition deep learning model FB-TimesNet, which includes feature extraction layer, feature fusion layer and classification layer, to capture rich spatiotemporal information and facilitate the subsequent classification of emotion categories. Use a multimodal fusion module to adjust the weights of the two modalities and improve the accuracy of emotion recognition after feature fusion.

[0031] This embodiment applies the above technical solution and selects the common BRED emotion dataset containing facial and body posture features for testing. BRED is a multimodal dataset that can be obtained for free. The dataset records the six basic emotions expressed by children after interacting with two robots in a laboratory environment: surprise, sadness, happiness, fear, disgust and anger. The BRED dataset contains a total of 215 valid sequences with an average of 72 frames. Each set of data contains facial feature vectors extracted by Resnet50 and key point coordinates captured by OpenPose from video frames.

[0032] For facial features, the key point coordinates are first normalized and preprocessed. The coordinates of the nose tip key point are subtracted from the coordinates of each video frame, so that the coordinate origin is uniformly set to the nose tip. The rotation angle is calculated based on the coordinates of the three key points in the middle of the left and right eyes and the nose tip to correct the face direction. Finally, the distance between the chin and forehead key points is proportionally scaled according to the ratio of the pre-set parameters to unify the face size. After normalization, the coordinate points BaseFrame in the natural state and the representation F describing the facial features of the video are obtained. f :

[0033] BaseFrame={(x 1 ,y 1 ),(x 2 ,y 2 ),(x 3 ,y 3 ),...,(x n ,y n )} (1)

[0034]

[0035] For posture feature processing, since some frames do not contain all key points, the missing values ​​are set to 0. Similar to facial expressions, posture features are normalized in the same way to obtain the representation F describing the body posture features. b

[0036]

[0037] Then calculate the distance between the feature points of each frame and the coordinates of the feature points in the normal state, that is, F f -BaseFrame, to get the change value F of the two-dimensional coordinate f ', taking it as the input model

[0038]

[0039] F b and F f'As facial expression features and body posture features input into the FB-TimesNet model for training.

[0040] Then build the FB-TimesNet model as shown in the attached Figure 2

[0041] The data conversion layer processes the processed one-dimensional input data X 1D Perform discrete Fourier transform, sort the periods of all frequencies in descending order according to the amplitude, extract the first K frequencies as the periodic features of the time series data, segment the data sequence according to the period length, and splice the cutting results under the same period length to obtain two-dimensional data

[0042] The feature extraction layer processes k frequency features as shown in formula (5). 1 It includes three two-dimensional convolution modules, whose input and output sizes are the same, namely the last dimension of X and 1024, and the step size is 1. The sizes of the convolution kernels are (1, 9), (1, 7) and (1, 5), respectively. The results extracted by the three convolution modules are fused to extract the temporal features. Then, the Conv2d is processed by formula (6). 2 It contains two 2D convolution blocks, the output is the last dimension of x, the stride is 1, and the sizes of the convolution kernels are (5,1) and (3,1) respectively.

[0043] Finally, the original shape is restored using equation (7) to extract the spatial features.

[0044]

[0045] The feature fusion layer adaptively fuses the output results of the feature extraction layer and calculates the relative importance of the selected frequency and period, as shown in formula (8). Among them, the amplitude A is used as the weight coefficient after the Softmax (·) aggregation operation to obtain the extracted features, and the aggregated tensor after the residual connection is used as input again to obtain deeper information through multi-layer extraction.

[0046] In the classification layer, as shown in formula (9), firstly, X is processed by the fully connected layer 1D GeLU() is an activation function. Since this activation function is smooth within the nonlinear range, it helps the gradient descent optimization algorithm converge. The Dropout() method randomly sets the output of some neurons to 0 during the training process, thereby reducing the excessive dependence of the neural network on specific neurons and improving the generalization ability of the model.

[0047]

[0048] Output = Linear(Dropout(GeLU(X 1D ))) (9)

[0049] The multimodal fusion layer uses decision-level fusion to process multimodal features, as shown in formula (10). It gives different weights to different modalities, where Output f Represents the output of the facial expression module, Output b represents the output of the body posture module, λ 1 and λ 2 is the weight of the two modules. By changing the weight, the contribution degree between the modes can be flexibly adjusted.

[0050] Output=λ 1 Output f +λ 2 Output b (10)

[0051] To evaluate the proposed multimodal emotion recognition method, the following comparative experiments were conducted. The experiments were completed on the Windows 10 operating system, using Python 3.8.12, the deep learning framework Torch 1.10.1+cu113, and the hardware used CPU i9-10900k@3.70GHz and NVIDIA GeForce RTX 3090. The model uses two Timesblock modules, stacked in the structure of a residual network, and uses the Adam optimizer for parameter optimization. The learning rate is set to 0.0001, the number of samples for each training is 64, and the L2 regularization term and Dropout mechanism are used to prevent overfitting. LeakyReLU and GeLU are used as activation functions, and cross entropy is used as loss function.

[0052] The experimental results are shown in Table 1 below. When the weights of facial recognition results and body posture recognition results are 0.75 and 0.25 respectively, the model performs best. Obviously, compared with SOTA (GB-FB), the recognition accuracy of FB-TimesNet on face, body posture and multimodal fusion has increased by 14.02%, 13.04% and 4.89% respectively. This shows that FB-TimesNet has more excellent recognition performance. In addition, compared with single modality, the effect after multimodal fusion is significantly improved.

[0053] Table 1.

[0054] Comparison results of FB-TimesNet after fusing face and posture

Claims

1. An emotion recognition method based on facial expression and body posture, characterized in that , including the following steps: S1, using the open source OpenPose model recognition to obtain the 2D coordinates of the facial and skeleton key points of each frame of the fragment as the original feature information, and preprocessing; S2, using the change value of the facial key point coordinates relative to the natural state, and the body posture key point coordinates as the original information features of facial expressions and body postures, respectively, and input them into the defined FB-TimesNet model for awakening training; S3, constructing a multimodal emotion recognition deep learning model FB-TimesNet, which includes a feature extraction layer, a feature fusion layer, and a classification layer to capture rich spatiotemporal information and facilitate the subsequent classification of emotion categories; using a multimodal fusion module to adjust the weights of the two modalities to improve the accuracy of emotion recognition after feature fusion.

2. The method for emotion recognition based on facial expression and body posture according to claim 1, characterized in that , the pre-processing steps in steps S1 and S2 include, Facial feature preprocessing module and posture feature preprocessing module.

3. The method for emotion recognition based on facial expression and body posture according to claim 2 is characterized in that ,The working process of the feature preprocessing module is as follows: for facial expressions, noise is removed and standardization is performed, and the ,change of the key point coordinates relative to the natural state is used as the original ,feature information of the face; For body posture, the same noise removal and standardization process is performed, and the coordinates of posture key points are used as the original feature information of the body posture module. This greatly reduces the calculation and time costs and improves the efficiency of emotion recognition.

4. The emotion recognition method based on facial expression and body posture according to claim 1 is characterized in that The preprocessing step in step S3 includes a data conversion layer, a feature extraction layer, a feature fusion layer, a classification layer and a multimodal fusion module. The data conversion layer is used to convert one-dimensional data into two-dimensional data according to periodic features; the feature extraction layer is used to extract spatiotemporal features in two-dimensional data; the feature fusion layer is used to fuse spatiotemporal features extracted from different two-dimensional data; the classification layer is used to classify according to the fused spatiotemporal features; the multimodal fusion module is used to fuse the output features of facial expression modality and body posture modality.

5. The method for emotion recognition based on facial expression and body posture according to claim 4, characterized in that The working process of the data conversion layer is to perform fast Fourier transform on the one-dimensional data, sort the periods of all frequencies in descending order according to the amplitude, extract the first k frequencies as the periodic features of the time series data, segment the data sequence according to the period length, and splice the cutting results under the same period length to obtain two-dimensional data.

6. The method for emotion recognition based on facial expression and body posture according to claim 4, characterized in that The working process of the feature extraction layer is to use a universal convolution block of dimensional vision to extract different information in the two-dimensional data through parallel calculation of convolution kernels of different sizes, and finally stack the results in the depth dimension to obtain output, thereby improving the feature learning ability.

7. The method for emotion recognition based on facial expression and body posture according to claim 4, characterized in that The working process of the feature fusion layer is to calculate the relative importance of the selected frequency and period according to the amplitude. The amplitude is aggregated and used as the weight coefficient to obtain the extracted features. The aggregated tensor after residual connection is used as input again to obtain deeper information through multi-layer extraction.

8. The method for emotion recognition based on facial expression and body posture according to claim 4, characterized in that ,The working process of the classification layer is as follows, after the fully connected layer processing, a suitable activation function is selected to help the gradient descent optimization algorithm to converge, and some neurons’ outputs are randomly set to 0 during the training process, thereby reducing the excessive dependence of the neural network on specific neurons, thereby improving the generalization ability of the model.

9. The method for emotion recognition based on facial expression and body posture according to claim 4, characterized in that ,The working process of the multimodal fusion module is to use decision-level fusion to process multimodal features, which gives different weights to different modalities, and flexibly adjusts the contribution degree between modalities by changing the weights.

Images