An emotion recognition method and system based on face data and electrocardio data

Abstract

The application discloses an emotion recognition method and system based on face data and electrocardio data. Compared with the prior art, the application combines the convenience of face information acquisition and the objectivity and easy detection of electrocardio signals, so that accurate emotion recognition can be carried out even in the case of insufficient light or poor network quality. Secondly, the problem of low prediction credibility and poor robustness of an emotion recognition system using only a single signal as input in extreme cases is overcome, so that the accuracy of emotion recognition in normal and extreme cases is ensured in a balanced manner. Thirdly, the improved SC-CBAM network model reduces the number of parameters and the amount of calculation in the model, so that the complexity of the model can be further reduced, and the inference efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition, specifically to an emotion recognition method and system based on facial data and electrocardiogram data. Background Technology

[0002] Emotions are ubiquitous in people's daily lives. They can reflect a person's psychological state to a certain extent and have a significant impact on people's cognition and behavior. In personalized recommendation systems, emotions often reflect a user's satisfaction with the recommendation results. Therefore, identifying a user's emotions through their external and internal manifestations is of great significance.

[0003] Recognizing human emotions is a popular research area. Current mainstream emotion recognition technologies include image-based emotion recognition and physiological information-based emotion recognition. Image-based emotion recognition systems typically use video streams from a camera as input, extract faces from the video stream, perform grayscale conversion, and then call a model for emotion recognition. However, in low-light conditions or with poor network quality, face-based emotion recognition systems cannot guarantee the reliability of emotion recognition; therefore, judging a person's emotion solely based on facial expressions may produce incorrect results. Physiological information-based emotion recognition currently relies on collecting physiological signals such as electroencephalogram (EEG), eye movements, electromyography (EMG), electroskin conductance (ESF), electrocardiogram (ECG), and respiration. However, this physiological information is often susceptible to interference from the external environment and noise during collection, resulting in lower reliability and less accurate emotion recognition results. Summary of the Invention

[0004] This application provides an emotion recognition method and system based on facial data and electrocardiogram data to solve the problems of poor reliability and inaccurate emotion recognition results in existing single-signal-based emotion recognition technologies.

[0005] To achieve the above objectives, this application adopts the following technical solution.

[0006] On the one hand, an emotion recognition method based on facial data and electrocardiogram data is provided, including the following steps:

[0007] S1. Collect the training dataset D = {(x1,y1),(x2,y2),…,(x...} i ,y i ),…,(x N ,y N )}, where x i Let y be the i-th bimodal feature vector sample, which consists of face data and electrocardiogram data. i For the labeled sample x ithe emotion category, Y is the emotion category set, y i ∈Y, N is the number of samples in the training dataset;

[0008] S2. Initialize the weight of the i-th bimodal feature vector sample in the training dataset: W₁(i)=1 / (N·|y i |·|Y-y i |); where |y i | has a value of 1;

[0009] S3. Calculate the weight α of the t-th generation sub-classifier t :

[0010]

[0011]

[0012] where p t (x i ,k0) is the probability that the t-th generation sub-classifier predicts the sample x i as category k0, k0≠y i , that is, inconsistent with the labeled emotion category y i , p t (x i ,k1) is the probability that the t-th generation sub-classifier predicts the sample x i as category k1, k1=y i ; that is, consistent with the labeled emotion category y i ;

[0013] Each generation of sub-classifiers includes a first sub-classifier for processing face data and a second sub-classifier for processing electrocardiogram data; W t (i) is the weight of the i-th bimodal feature vector sample in the t-th iteration process;

[0014] S4. Update the sample weight W t+1 (i):

[0015]

[0016] where Z t is a normalization parameter,

[0017] S5. If t<T, increment the value of t by 1, and return to S3; if t=T, take the T-th generation sub-classifier as the emotion recognition classifier; T is the preset total number of iterations.

[0018] The emotion recognition classifier constructed using the above method combines the convenience of facial information acquisition with the objectivity and ease of detection of electrocardiogram (ECG) signals, enabling accurate emotion recognition even in low-light or poor network conditions. It overcomes the problems of low prediction reliability and poor robustness in extreme situations faced by emotion recognition systems that use only a single signal as input, ensuring accuracy in both normal and extreme conditions in a balanced way.

[0019] In some embodiments, the following steps are included after S5:

[0020] The facial data is input into the first sub-classifier to obtain the first classification result, and the electrocardiogram data is input into the second sub-classifier to obtain the second classification result.

[0021] The first and second classification results are input into the emotion recognition classifier to obtain the emotion recognition classification result;

[0022] The expression for the emotion recognition classifier is as follows: x represents a bimodal feature vector sample composed of facial data and electrocardiogram data, k represents the category of the emotion recognition classification result, and p t (x,k) represents the probability that the t-th generation sub-classifier will predict sample x as class k.

[0023] In some embodiments, the step of inputting facial data into a first sub-classifier to obtain a first classification result includes:

[0024] After processing the collected face data using the SC-CBAM module, further convolution processing, global pooling processing, and activation processing are performed to obtain the first classification result;

[0025] The SC-CBAM module includes multiple cascaded hierarchical structures and SC-CBAM sub-modules with the same convolution kernel size. The SC-CBAM sub-modules include SC modules and CBAM modules.

[0026] The SC module includes a first SC module and a second SC module. The face data after convolution and normalization are used as the inputs of the first SC module and the second SC module, respectively. The first SC module is used to perform one convolution process, and the second SC module is used to perform two convolution processes and one max pooling process in sequence.

[0027] The CBAM module has two components, which are used to process the outputs of the first SC module and the second SC module, respectively; the CBAM module includes a channel attention module and a spatial attention module;

[0028] The channel attention module includes a first channel attention module, a second channel attention module, and a first operation module. The first channel attention module is used to perform global max pooling and two convolution processes in sequence. The second channel attention module is used to perform global average pooling and two convolution processes in sequence. The first operation module is used to add the outputs of the first channel attention module and the second channel attention module, and multiply the result of the addition with the output of the SC module as the output of the channel attention module.

[0029] The spatial attention module uses the torch.mean and torch.max functions to process the output of the channel attention module, obtaining the mean feature map and the maximum feature map respectively. The mean feature map and the maximum feature map are concatenated and then subjected to convolution and normalization. The result is multiplied with the output of the channel attention module and used as the output of the CBAM module.

[0030] The outputs of the two CBAM modules are added together and then convolved again. The result is used as the output of the SC-CBAM submodule.

[0031] The improved SC-CBAM network model introduces a deep convolutional separable module based on the existing CBAM attention mechanism. Its key feature is a reduction in the number of parameters and computational cost, further reducing model complexity and improving inference efficiency. While CBAM itself already demonstrates good performance in attention mechanisms, further reducing model complexity is crucial for deployment in resource-constrained scenarios. CBAM using a deep convolutional separable module provides a more lightweight model, making it suitable for resource-constrained environments such as mobile devices and embedded systems.

[0032] In some embodiments, the following steps are included before processing the acquired face data using the SC-CBAM module:

[0033] Image pyramid technology is used to resize the acquired facial data, generating images of different scales.

[0034] A multi-task convolutional neural network is used to process images of different scales to obtain RGB images of the face region;

[0035] Convert the RGB image of the face region to a grayscale image.

[0036] In some embodiments, before processing the acquired face data using the SC-CBAM module, the following steps are further included: performing convolution processing and normalization processing on the grayscale image.

[0037] In some embodiments, the step of inputting electrocardiogram data into a second sub-classifier to obtain a second classification result includes:

[0038] A portion of the electrocardiogram (ECG) data samples were selected from the ECG database to obtain the first ECG data sample and the second ECG data sample.

[0039] Gaussian random noise was added to the first ECG data sample, and the second ECG data sample was scaled.

[0040] After the processed first and second ECG data samples are inserted into the ECG data for random sorting, they are input into the continuous ECG recognition convolutional neural network model for emotion recognition, and the output is the second classification result.

[0041] In some embodiments, Gaussian random noise is added to the first ECG data sample, and a preprocessing step is included before scaling the second ECG data sample. The preprocessing includes downsampling and bandpass filtering.

[0042] In some embodiments, the training process of the continuous ECG recognition convolutional neural network model includes the following steps:

[0043] Divide all datasets into M parts, and take one part as the test set each time without repetition, and use the other M-1 parts as the training set to train the model;

[0044] Calculate the mean squared error (MES) of the model on the test set after the j-th training iteration. j The mean squared error obtained after M training cycles is averaged to obtain the MSE.

[0045] Repeat the above steps multiple times to obtain multiple MSE values, and take the model corresponding to the lowest MSE value as the trained continuous ECG recognition convolutional neural network model.

[0046] in, y j The output is the predicted value, where y is the true label. j Both y and y are of the form Where V represents the level of emotional pleasure and A represents the level of emotional arousal. The range of V and A after one-hot encoding is 1-5, and after normalization, both range from 0 to 1; y i -y is defined as: ‖y i -y‖2 is defined as: (V i -V) 2 +(A i -A) 2 .

[0047] In another aspect, a system is provided that includes a computer device, the computer device including a memory, a processor, and program instructions stored in the memory that are executable by the processor, wherein the processor executes the program instructions to implement the steps of the above-described emotion recognition method based on facial data and electrocardiogram data.

[0048] This application has at least the following technical effects or advantages:

[0049] 1. It combines the convenience of obtaining facial information with the objectivity and ease of detection of electrocardiogram signals, enabling accurate emotion recognition even in low light or poor network conditions.

[0050] 2. It overcomes the problems of low prediction reliability and poor robustness of emotion recognition systems that use only a single signal as input in extreme situations, and ensures the accuracy of emotion recognition in both normal and extreme situations in a balanced way.

[0051] 3. The improved SC-CBAM network model reduces the number of parameters and computational load in the model, which can further reduce the complexity of the model and improve inference efficiency. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating an emotion recognition method based on facial data and electrocardiogram data in one embodiment of this application.

[0053] Figure 2 This is a schematic diagram of the process of performing emotion recognition on facial data in one embodiment of this application;

[0054] Figure 3 This is a schematic diagram of the SC-CBAM module in one embodiment of this application;

[0055] Figure 4 This is a schematic diagram of the SC module in one embodiment of this application;

[0056] Figure 5 This is a schematic diagram of the CBAM module in one embodiment of this application;

[0057] Figure 6 This is a schematic diagram of the SC-CBAM module in one embodiment of this application;

[0058] Figure 7 This is a schematic diagram of the structure of the convolution / global pooling / activation module in one embodiment of this application;

[0059] Figure 8 This is a schematic diagram of the training process of a continuous electrocardiogram recognition convolutional neural network model in one embodiment of this application;

[0060] Figure 9This is a schematic diagram of the structure of a convolutional neural network model for continuous electrocardiogram recognition in one embodiment of this application. Detailed Implementation

[0061] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0062] Example 1

[0063] See Figures 1-9 An emotion recognition method based on facial data and electrocardiogram data includes the following steps:

[0064] S1. Collect the training dataset D = {(x1,y1),(x2,y2),…,(x...} i ,y i ),…,(x N ,y N )}, where x i Let y be the i-th bimodal feature vector sample, which consists of face data and electrocardiogram data. i For the labeled sample x i Regarding the emotional categories, this application selected eight emotional categories: pleasure, sadness, fright, fear, anger, disdain, disgust, and calmness. i ∈Y={0,1,2,3,4,5,6,7}. N is the number of samples in the training dataset;

[0065] S2. Initialize the weights of the i-th bimodal feature vector sample in the training dataset: W1(i) = 1 / (N·|y i |·|Yy i |); where |y i The value of | is 1;

[0066] S3. Calculate the weights of the t-th generation sub-classifier. t :

[0067]

[0068]

[0069] Where, p t (x i (k0) represents the sub-classifier of generation t that classifies sample x. i The probability of predicting class k0, where k0 ≠ y i That is, the emotion category y that is labeled i Inconsistent, p t (x i k1) represents the sub-classifier of generation t that classifies sample x. ithe probability of being predicted as category k1, k1=y i ; that is, it is consistent with the labeled emotion category y i ;

[0070] Each generation of sub-classifiers includes a first sub-classifier for processing face data and a second sub-classifier for processing electrocardiogram data; W t (i) is the weight of the i-th bimodal feature vector sample in the t-th iteration process;

[0071] S4. Update the weight W of the sample t+1 (i):

[0072]

[0073] Wherein, Z t is a normalization parameter,

[0074] S5. If t<T, add 1 to the value of t, and return to S3; if t=T, take the T-th generation sub-classifier as the emotion recognition classifier; T is the preset total number of iterations;

[0075] S6. Input the face data into the first sub-classifier to obtain a first classification result, and input the electrocardiogram data into the second sub-classifier to obtain a second classification result;

[0076] Input the first classification result and the second classification result into the emotion recognition classifier to obtain an emotion recognition classification result;

[0077] Wherein, the expression of the emotion recognition classifier is: x is a bimodal feature vector sample composed of face data and electrocardiogram data, k is the category of the emotion recognition classification result, p t (x,k) represents the probability that the t-th generation sub-classifier predicts the sample x as category k, where k∈Y, and its predicted category is h t (x i )=argmax k {p t (x i ,k)}. For each sample (x in the training set i ,y i ), 2 sets of probabilities p can be obtained t (x i ,k0) and p t (x i ,k1), where k0≠y i , k1=y i .

[0078] It can be seen from the sample weight update formula that for samples with wrong prediction categories, that is, p t(x i ,k0)>p t (x i The weights of samples with values ​​greater than k1 will increase, and they will receive more attention in the next round of training. This will affect the training results of the new weak classifier. Because the proportion of these misclassified samples in the resampled training set increases, the new weak classifier is more likely to focus on these difficult-to-classify samples in order to better correct the errors.

[0079] By continuously iterating through training and updating sample weights, the entire ensemble model can gradually improve its ability to identify misclassified samples after multiple rounds of training, thereby reducing the overall error rate and forming a more powerful ensemble classifier. The final output is a strong classifier.

[0080]

[0081] For both ECG and facial recognition inputs, the strong classifier outputs the accuracy of emotion prediction. Therefore, we only need to select the result with the higher accuracy. Thus, the final predicted emotion category is:

[0082] The steps for inputting facial data into the first sub-classifier to obtain the first classification result specifically include:

[0083] S61.1 The image pyramid technique is used to resize the collected face data and generate images of different scales.

[0084] When performing face detection on the original image set, the images are first resized using image pyramid techniques to generate image pyramids of different scales. These images of different scales are then fed into two cascaded networks for training to achieve multi-scale face detection.

[0085] S61.2. A multi-task convolutional neural network is used to process images of different scales to obtain RGB images of the face region.

[0086] After completing the image pyramid, the image enters the first P-Net layer of MTCNN (Multi-task convolutional neural network). This layer initially extracts image features and provides preliminary bounding boxes. These bounding boxes are then preliminarily screened, discarding those that do not meet the criteria. Face regions within the bounding boxes output by P-Net are then further processed.

[0087] Next is the second R-Net network layer of MTCNN. R-Net performs more refined selection of the input values ​​and discards most of the erroneous bounding boxes. In addition, R-Net uses a facial landmark locator to locate facial landmarks and performs bounding box regression to further optimize the results. Finally, R-Net outputs a high-confidence RGB image of the face region with a size of 48x48 pixels.

[0088] S61.3 Convert the RGB image of the face region to a grayscale image.

[0089] In step S61.2, we obtain an RGB image output M with a size of 48x48 pixels. To convert it to a grayscale image, we can use the psychological grayscale formula: Gray = 0.299R + 0.587G + 0.114B. By applying the above formula to each pixel in M, we can obtain a grayscale image where each pixel has only one grayscale value, representing the brightness information of the original color image.

[0090] S61.4 Perform convolution and normalization processing on the grayscale image.

[0091] We need to perform convolution / normalization processing on the 48x48 face image data output from step S61.3, and use the processed output as the input for the next stage. Specifically, we use two Conv2D layers to construct the convolution / normalization module, with each Conv2D layer including one convolution operation and one normalization operation.

[0092] In the convolution operation, we use 3x3 filters to perform convolution operations on the input data to extract image feature information. Each filter performs a convolution operation on the input data once, obtaining an output value. The output feature information can be controlled by adjusting the filter weights as needed. By filtering the entire input data, we can obtain an output feature image of the same size as the input.

[0093] In the normalization operation, we normalize the output obtained from the convolution. Normalization is mainly used to control the amplitude range of the output to avoid problems such as gradient explosion or vanishing gradients. Common normalization methods include Local Response Normalization (LRN) and Batch Normalization (BN). In this step, we use BN for normalization.

[0094] S61.5. The SC-CBAM module is used to further process the face data.

[0095] The SC-CBAM module consists of four cascaded SC-CBAM sub-modules. The overall architecture of step S3 is described in detail below. Figure 2 .

[0096] The SC-CBAM submodule consists of an SC (Seperable Convolution) module based on a deep separable neural network and a CBAM (Convolution Block Attention Module) module. See the architecture diagram of the SC-CBAM submodule for reference. Figure 3 .

[0097] The SC module includes a first SC module and a second SC module. The face data after convolution and normalization are used as the inputs of the first SC module and the second SC module, respectively. The first SC module is used to perform one convolution process, and the second SC module is used to perform two convolution processes and one max pooling process in sequence.

[0098] The SC module uses the X-ception network architecture with depthwise separable convolutions as its foundation and incorporates the idea of ​​depthwise separability. Unlike the original X-ception network, the SC module first performs a 1x1 convolution on the input, then a 3x3 convolution on each channel, and finally performs a concatenation operation. Furthermore, the SC module introduces a CBAM module that integrates channel attention and spatial attention mechanisms to further optimize the results. See the architecture diagram of the SC module for reference. Figure 4 .

[0099] In the SC module, the input 44*44 data undergoes convolution and normalization, followed by two separate processing steps. The first step uses a 1*1 convolution kernel to generate a 22*22 output data X. The second step uses two 3*3 convolution kernels and one MaxPool2D operation to similarly generate a 22*22 output data Y. These two output data sets are used as input to the CBAM module.

[0100] The CBAM module has two parts, used to process the output X of the first SC module and the output Y of the second SC module, respectively. The CBAM module includes a channel attention module and a spatial attention module. The CBAM module utilizes an attention mechanism, mimicking the tendency of the human visual system to focus attention on the more important parts of the image and ignore irrelevant information. (See CBAM module structure diagram for reference.) Figure 5 .

[0101] The channel attention module includes a first channel attention module, a second channel attention module, and a first operation module. The first channel attention module is used to perform global max pooling and two convolution processes in sequence. The second channel attention module is used to perform global average pooling and two convolution processes in sequence. The first operation module is used to add the outputs of the first channel attention module and the second channel attention module, and multiply the result by the output of the SC module as the output of the channel attention module.

[0102] The input to the channel attention module is the 22x22 output of the SC module, denoted as X and Y. First, global average pooling and global max pooling are performed on X and Y to obtain 1x1 outputs. These two outputs are then processed by Conv2D with a 1x1 kernel and ReLU activation function, and by Conv2D with a 1x1 kernel and Sigmoid activation function, respectively.

[0103] The spatial attention module processes the output of the channel attention module using the `torch.mean` and `torch.max` functions to obtain the mean feature map and the maximum feature map, respectively. The mean and maximum feature maps are concatenated, then convolved and normalized. The result is multiplied by the output of the channel attention module, becoming the output of the CBAM module. The outputs of the two CBAM modules are then summed and convolved again; the final result is the output of the SC-CBAM submodule.

[0104] The input to the spatial attention module is the 22x22 output data of the channel attention module. Two feature maps are obtained using the `torch.mean` and `torch.max` functions, concatenated using `torch.cat`, processed by a 1x1 convolution kernel and normalized, then multiplied with the original output of the spatial attention module to obtain 22x22 data X and Y. X and Y are then added together and processed again by a 1x1 convolution kernel to obtain the 22x22 output data of the first SC-CBAM submodule. This data is used as the input to the next SC-CBAM module, and the above steps are repeated until four SC-CBAM submodules with the same hierarchical structure and convolution kernel size have been completed, ultimately yielding 3x3 output data. For a detailed diagram of the SC-CBAM submodule hierarchy, please refer to [link to diagram]. Figure 6 .

[0105] S61.6 Further convolution processing, global pooling processing, and activation processing are performed to obtain the first classification result.

[0106] The analysis results processed by the SC-CBAM module undergo final pooling / activation operations to obtain the final recognition result. This module consists of three parts: convolution, global pooling, and activation. For the specific structure, please refer to [reference needed]. Figure 7 .

[0107] For the 3x3 data output by the SC-CBAM neural network model, a 3x3 convolution kernel is first applied to process the data. Then, pooling / activation operations are performed. Pooling can be global pooling to reduce the data size to 1x1; activation operations can use appropriate activation functions such as ReLU to add non-linearity. Finally, the final recognition result is obtained from the pooling / activation results.

[0108] II. The steps for inputting ECG data into the second sub-classifier to obtain the second classification result include:

[0109] S62.1 Select a portion of ECG data samples from the ECG database to obtain the first ECG data sample and the second ECG data sample;

[0110] This embodiment uses the publicly available dataset Dreamer, with an original sample size of 27,588, from which samples were randomly selected. That is, 4,598 electrocardiogram data samples.

[0111] S62.2 Add Gaussian random noise to the first ECG data sample and perform scaling transformation on the second ECG data sample;

[0112] The specific process of adding Gaussian random noise to the first ECG data sample is as follows: Let the original ECG sample be e(t), the mean be 0, and the standard deviation be E. αvg The Gaussian distribution of is p(t), where Where mean is a function from the NumPy library, e(t) 2 The dataset is a new dataset formed by squaring each sample data in the original ECG sample e(t). Then, after adding noise, e′(t) = e(t) + p(t).

[0113] The specific process of scaling the second ECG data sample is as follows: Let the original ECG signal be e(t), then after scaling, e′(t) = αe(t), where α is a parameter set by the user.

[0114] S62.3 After randomly sorting the processed first and second ECG data samples into the ECG data, the data is input into the continuous ECG recognition convolutional neural network model for emotion recognition, and the second classification result is output.

[0115] Preferably, before scaling the second ECG data sample, Gaussian random noise is added to the first ECG data sample, and a preprocessing step is performed on both the first and second ECG data samples. This preprocessing includes downsampling and bandpass filtering. The downsampling factor is denoted as P, which is generally an integer or fraction greater than 1, representing a multiple of the sampling period or a factor of the sampling frequency. When P is an integer, the downsampling process involves: removing high-frequency components from the signal using a digital low-pass filter; downsampling the filtered high-frequency signal by a factor of P, in other words, retaining sampling points with an interval of P in the original signal. When P is a fraction, it is written as... The downsampling process is as follows: upsampling at L times the frequency; Frequency downsampling. The bandpass filtering process involves using the `butter` and `filtfilt` functions from the `signal` class in the scipy library to filter out data below 0.05Hz and above 100Hz.

[0116] For the specific network structure of the Continuous ECG Recognition Convolutional Neural Network Model (CER-CNN), please refer to [link / reference needed]. Figure 9 The input to CER-CNN is (x, 2560, 2), where x is the number of samples, 2560 is the window size, and 2 is the number of ECG signal acquisition devices. This application uses two ECG signal acquisition devices, connected to the left chest-right arm and the right chest-left arm respectively. There is some overlap between the windows; in this embodiment, the overlap size is 1280. The sampling time for each window is 5 seconds (related to the sampling frequency).

[0117] The ECG data first passes through (5*2*32) and (5*32*32) convolutional layers with ReLU activation, followed by average pooling and batch normalization layers. Then, the 1D vector is processed using Dropout. The processed result is then passed through (5*32*64) and (5*64*64) convolutional layers, followed by average pooling, normalization, and Dropout layers. The final result is then passed through (3*64*128) and (3*128*128) convolutional layers, followed by average pooling, normalization, and Dropout layers. Finally, the result is input into a (3*128*512) convolutional layer, followed by average pooling, normalization, and Dropout layers. The result is input into a fully connected layer (512*64) with the full output layer and the activation function TanH. The result is then input into a fully connected layer (64*8) with the activation function TanH. The result is then dropped out. Finally, the result is input into two fully connected layers (8*6) to obtain the VA value.

[0118] The applicant's research revealed significant individual variations in electrocardiogram (ECG) signals, thus placing high demands on the model's generalization ability. Based on this, this application employs k-fold cross-validation to optimize model performance. A larger k value results in smaller model bias, but simultaneously decreases generalization ability. Based on experience, this application selects k = 10. The training process of the continuous ECG recognition convolutional neural network model includes the following steps:

[0119] Divide all datasets into M parts, and take one part as the test set each time without repetition, and use the other M-1 parts as the training set to train the model;

[0120] Calculate the mean squared error (MES) of the model on the test set after the j-th training iteration. j The mean squared error obtained after M training cycles is averaged to obtain the MSE.

[0121] The above steps are called one 10-fold cross-validation. We adjust the hyperparameters such as the convolution kernel and repeat the above 10-fold cross-validation 5 times. We record the MSE of each time and select the model corresponding to the smallest MSE as the optimal model, that is, the trained continuous ECG recognition convolutional neural network model.

[0122] in, y j The output is the predicted value, where y is the true label. j Both y and y are of the form Where V represents the level of emotional pleasure and A represents the level of emotional arousal. The range of V and A after one-hot encoding is 1-5, and after normalization, both range from 0 to 1; y i -y is defined as: ‖y i -y‖2 is defined as: (V i -V) 2 +(A i -A) 2 Its mathematical meaning is the second normal distance between the trained result and the true result.

[0123] III. Effect Verification.

[0124] To verify the superiority of our algorithm in situations where facial information is scarce, such as insufficient light or network congestion, we propose a method for emotion recognition using electrocardiogram (ECG) data. Consider a scenario where emotion classification based on facial recognition suffers from frame drops in insufficient light or poor network quality, resulting in continuous loss of facial data collected by the camera. In the most extreme case, the accuracy of emotion recognition based on facial information can drop to zero. To address this situation, we propose the following method.

[0125] 1. Twenty volunteers were selected and each watched one of ten videos. After watching the videos, each volunteer's facial video and electrocardiogram (ECG) data were recorded. Volunteers were also asked to fill in the values ​​of Arousal and Valence on a form after watching each video, with the evaluation index ranging from 1 to 5. Then, the accuracy of emotion recognition with and without ECG data was compared under different frame sampling rates.

[0126] 2. Construct an emotion recognition subclassifier based on face data, denoted as A.

[0127] 3. Construct an emotion recognition sub-classifier based on electrocardiogram data, denoted as B.

[0128] 4. Perform random frame dropping on the original video data.

[0129] 5. Mark the extracted frames, find the face data corresponding to the timestamp at the extracted frame, denoted as inA, and find the ECG data within 5 seconds before and after the timestamp (related to the sampling frequency; the sampling frequency in this application is 512Hz), denoted as inB. Use inA and inB as inputs to the strong classifier obtained in step S8 to obtain two weights, representing the prediction accuracy based on these two signals, i.e., weights. The recognition results of sub-classifiers A and B are weighted and averaged with their respective weights to obtain the final recognition accuracy. This step is to use ECG signals to compensate for the loss of face data. The more information lost in the face signal in each frame, the lower its output in the strong classifier will be, meaning it contributes less to the final recognition accuracy. When the face information is complete, its output in the strong classifier is higher, playing a major role in the final accurate recognition. Considering that face signals are not completely lost in real life, this application uses a weighted average method of face signals and ECG signals to balance the weights between them.

[0130] Experimental results (all data are percentages):

[0131]

[0132] Table 1. Recognition accuracy without ECG data assistance under different frame drop rates.

[0133]

[0134]

[0135] Table 2. Recognition accuracy with ECG data assistance under different frame drop rates.

[0136] The data above shows that, without the aid of ECG data, the overall accuracy of emotion recognition decreases as the frame drop rate increases, which is in line with our expectations. However, some videos, such as video 2, exhibited a situation where the recognition accuracy increased locally as the frame drop rate decreased. This is because we used a random frame drop method, and the dropped frames may have been noise in the video, causing a local increase in emotion recognition accuracy. However, overall, the accuracy of emotion recognition still decreases as the frame drop rate increases.

[0137] With the assistance of ECG data, the accuracy of emotion recognition decreased as the frame drop rate increased. However, compared to Table 1, the rate of decline slowed significantly, eventually remaining around 68%, an average increase of 6-8 percentage points compared to the method without ECG data assistance, which meets our expectations. This sufficiently demonstrates the effectiveness of our method. It should be noted that the random frame drop method has a high degree of randomness. If the dropped video frame is a key frame, it may cause a sharp drop in recognition accuracy; conversely, if the dropped video frame is a noisy frame, it may lead to a local increase in recognition accuracy.

[0138] In addition, in the event of frame loss, the ECG signal and facial signal are each passed through a strong classifier, receiving corresponding weights. These weights are then averaged with the outputs of sub-classifiers A and B to obtain the final recognition accuracy. The more facial information is lost, the greater the impact of the ECG-based sub-classifier B on the result. Therefore, in extreme cases, the accuracy of emotion recognition depends to some extent on the accuracy of sub-classifier B itself. The recognition accuracy of sub-classifier B is around 60%-80%, which meets our needs.

[0139] Experiments have shown that even when insufficient light sources or network congestion lead to a lack of facial information, the method of this invention can still predict a person's emotions within a reliable range.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An emotion recognition method based on facial data and electrocardiogram data, comprising the following steps: S1. Collect the training dataset ,in, For the first A bimodal feature vector sample, wherein the bimodal feature vector sample consists of facial data and electrocardiogram data. For the labeled samples Emotional categories, For a set of emotion categories, The number of samples in the training dataset; S2. Initialize the weights of the i-th bimodal feature vector sample in the training dataset: ;in, The value is 1; S3, Calculate the first Weights of the subclassifier : ； ； in, For the first Subclassifier will classify samples Predicted as category The probability, That is, the labeled emotion category Inconsistent For the first Subclassifier will classify samples Predicted as category The probability, That is, the emotions categorized as labeled. Consistent; Each generation of subclassifiers includes a first subclassifier for processing face data and a second subclassifier for processing electrocardiogram data; Let be the weight of the i-th bimodal feature vector sample during the t-th iteration; S4. Update the weights of the samples. : ； in, For normalization parameters, ; S5, if ,but Increment the value by 1 and return S3; if , will the Subclassifiers are used as emotion recognition classifiers; This is the preset total number of iterations; S6. Input the face data into the first sub-classifier to obtain the first classification result, and input the ECG data into the second sub-classifier to obtain the second classification result; The first and second classification results are input into the emotion recognition classifier to obtain the emotion recognition classification result; The expression for the emotion recognition classifier is as follows: ; This is a dual-modal feature vector sample composed of facial data and electrocardiogram data. The categories for emotion recognition classification results. Indicates the first Subclassifier will classify samples Predicted as category The probability of.

2. The emotion recognition method based on facial data and electrocardiogram data according to claim 1, characterized in that, The steps to input facial data into the first sub-classifier and obtain the first classification result include: After processing the collected face data using the SC-CBAM module, further convolution processing, global pooling processing, and activation processing are performed to obtain the first classification result; The SC-CBAM module includes multiple cascaded hierarchical structures and SC-CBAM sub-modules with the same convolution kernel size. The SC-CBAM sub-modules include SC modules and CBAM modules. The SC module includes a first SC module and a second SC module. The face data after convolution and normalization are used as the inputs of the first SC module and the second SC module, respectively. The first SC module is used to perform one convolution process, and the second SC module is used to perform two convolution processes and one max pooling process in sequence. The CBAM module has two components, which are used to process the outputs of the first SC module and the second SC module, respectively; the CBAM module includes a channel attention module and a spatial attention module; The channel attention module includes a first channel attention module, a second channel attention module, and a first operation module. The first channel attention module is used to perform global max pooling and two convolution processes in sequence. The second channel attention module is used to perform global average pooling and two convolution processes in sequence. The first operation module is used to add the outputs of the first channel attention module and the second channel attention module, and multiply the result of the addition with the output of the SC module as the output of the channel attention module. The spatial attention module uses the torch.mean and torch.max functions to process the output of the channel attention module, obtaining the mean feature map and the maximum feature map respectively. The mean feature map and the maximum feature map are concatenated and then subjected to convolution and normalization. The result is multiplied with the output of the channel attention module and used as the output of the CBAM module. The outputs of the two CBAM modules are added together and then convolved again. The result is used as the output of the SC-CBAM submodule.

3. The emotion recognition method based on facial data and electrocardiogram data according to claim 2, characterized in that, Before processing the acquired face data using the SC-CBAM module, the following steps are also included: Image pyramid technology is used to resize the acquired facial data, generating images of different scales. A multi-task convolutional neural network is used to process images of different scales to obtain RGB images of the face region; Convert the RGB image of the face region to a grayscale image.

4. The emotion recognition method based on facial data and electrocardiogram data according to claim 3, characterized in that: Before processing the acquired face data using the SC-CBAM module, the following steps are also included: performing convolution processing and normalization processing on the grayscale image.

5. The emotion recognition method based on facial data and electrocardiogram data according to claim 1, characterized in that, The steps for inputting ECG data into the second sub-classifier to obtain the second classification result include: A portion of the electrocardiogram (ECG) data samples were selected from the ECG database to obtain the first ECG data sample and the second ECG data sample. Gaussian random noise was added to the first ECG data sample, and the second ECG data sample was scaled. After the processed first and second ECG data samples are inserted into the ECG data for random sorting, they are input into the continuous ECG recognition convolutional neural network model for emotion recognition, and the output is the second classification result.

6. The emotion recognition method based on facial data and electrocardiogram data according to claim 5, characterized in that, Before scaling the second ECG data sample, Gaussian random noise is added to the first ECG data sample. The process also includes a preprocessing step for the first and second ECG data samples, which includes downsampling and bandpass filtering.

7. An emotion recognition system based on facial data and electrocardiogram data, comprising a computer device, the computer device including a memory, a processor, and program instructions stored in the memory that are executable by the processor, wherein the processor executes the program instructions to implement the steps of any one of claims 1-6.

Images