Emotion recognition system based on heart activity physiological signals

Abstract

The invention provides an emotion recognition system based on heart activity physiological signals, and belongs to the technical field of emotion recognition, and the emotion recognition system comprises an all-polymer multimode polymer optical fiber sensing unit based on a speckle principle, and the all-polymer multimode polymer optical fiber sensing unit can detect heart physiological activities with high sensitivity and generate heart vibration signals (SCG signals). Through a signal extraction algorithm robust to motion artifacts, a method of combining frequency domain amplitude selection and sample entropy selection is utilized, and heart physiological signals are accurately extracted from signals influenced by the motion artifacts. Further, an emotion recognition machine learning algorithm based on complex network feature engineering is adopted, an Euclidean distance matrix is processed through threshold binarization to generate an SCG binary network, network topology features are extracted, and time-domain heart rate variability features and nonlinear heart rate variability features are combined and input into a support vector machine model for training and emotion classification. The system can overcome the difference of tested individuals, realizes cross-tested emotion measurement, and has good robustness and accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and particularly relates to an emotion recognition system based on physiological signals of cardiac activity. Background Art

[0002] An accurate heart rate measurement method is the basis of an emotion measurement system based on physiological signals of cardiac activity. Among various heart rate measurement methods, fiber optic sensors have many advantages: high sensitivity, flexibility, low response time, and good biocompatibility. In addition, since the sensing part of the fiber optic sensor is completely designed based on optical fibers, it can be realized without electrical components and without any metals, so it has good resistance to electromagnetic interference and electrochemical corrosion, and can even be used in extreme electromagnetic interference environments such as nuclear magnetic resonance. However, since most fiber optic-based sensing systems use the body vibration caused by cardiovascular activity as the signal source, they are easily affected by motion artifacts caused by the body and muscles of the subject, which affects the accurate measurement of cardiac activity. Currently, in the field of cardiac activity monitoring using fiber optic sensors, there is no generally recognized effective method for removing motion artifacts. In the photoplethysmography (PPG) technology that also detects cardiovascular activity by light, the commonly used motion artifact removal techniques usually require the assistance of a reference signal, which is lacking in convenience.

[0003] Therefore, it is of great significance to develop an algorithm that does not rely on a reference signal and can extract accurate physiological signals of cardiac activity from signals affected by motion artifacts end-to-end.

[0004] The connection between cardiac activity and emotion provides a theoretical basis for emotion measurement based on cardiac physiological signals. With the development of wearable heart rate monitoring devices in recent years, the convenience of heart rate monitoring has been greatly improved. A system that can conveniently collect human cardiac physiological signals in daily life and then achieve accurate emotion measurement has a wide range of application prospects.

[0005] However, there is a common challenge in the research field of emotion measurement based on physiological signals: the decrease in cross-subject classification accuracy caused by subject differences. Since there are significant differences in physiological conditions among different subjects, the accuracy of cross-subject emotion measurement is often lower than that of non-cross-subject emotion measurement. Cross-subject emotion measurement means that the data faced by the model in the training stage and the testing stage comes from different subjects. It provides convenience for developing a large-scale emotion measurement system in engineering, and scientifically means that the features used by the model for emotion measurement have a similar distribution among different subjects.

[0006] Therefore, it is of great significance to develop an emotion measurement system that can have a similar accuracy in cross / non-cross-subject emotion measurement. Summary of the Invention

[0007] To solve the above technical problems, the present invention proposes an emotion recognition system based on physiological signals of cardiac activity to solve the problems existing in the above prior art.

[0008] To achieve the above object, the present invention provides an emotion recognition system based on physiological signals of cardiac activity, including:

[0009] A sensing module, configured to collect chest wall vibration signals caused by cardiac beats;

[0010] A signal processing module, connected to the sensing unit, and configured to extract clean SCG signals from the chest wall vibration signals through frequency domain amplitude selection and sample entropy screening;

[0011] An emotion recognition module, connected to the signal processing unit, and configured to extract SCG signal features from the SCG signals and perform emotion classification according to the SCG signal features.

[0012] Preferably, the sensing unit includes:

[0013] A laser light source, configured to provide coherent light;

[0014] A multimode polymer optical fiber speckle sensing unit, obtained by winding a multimode polymer optical fiber into a ring and encapsulating it with polydimethylsiloxane;

[0015] A camera, configured to record the speckle image at the distal end of the optical fiber;

[0016] A computer, configured to calculate the speckle displacement according to the speckle image and obtain the chest wall vibration signal according to the speckle displacement.

[0017] Preferably, the preparation steps of the multimode polymer optical fiber speckle sensing unit include: mixing a dimethylsiloxane base material and a curing agent, pouring the mixture into a mold, standing for curing and then demolding to obtain a multimode polymer optical fiber fixed as three rings.

[0018] Preferably, the signal processing module includes:

[0019] An original signal extraction unit, configured to convert the speckle image into a grayscale image and obtain the original SCG signal through mask processing and frame-by-frame difference;

[0020] A signal preprocessing unit, configured to extract clean SCG signals from the original SCG signal through windowing, filtering, fast Fourier transform and sample entropy screening.

[0021] Preferably, the steps of sample entropy screening include:

[0022] Performing amplitude screening on the frequency domain signal to retain candidate frequency components;

[0023] Perform inverse Fourier transform on the candidate frequency components to obtain the time-domain signal;

[0024] Calculate the sample entropy of each time-domain signal, and select the signal with the minimum sample entropy as the clean SCG signal.

[0025] Preferably, the emotion classification module includes:

[0026] A feature extraction unit for constructing an SCG binary network and extracting network topology features;

[0027] A machine learning unit for emotion classification based on the extracted network topology features.

[0028] Preferably, the method for constructing the SCG binary network is:

[0029] Segment the SCG signal into RR intervals according to the R peaks;

[0030] Calculate the Euclidean distance matrix between the RR intervals;

[0031] Generate an SCG binary network by threshold binary processing of the Euclidean distance matrix.

[0032] Preferably, the network topology features include assortativity coefficient, average shortest path, average betweenness centrality, and clustering coefficient.

[0033] Compared with the prior art, the present invention has the following advantages and technical effects:

[0034] The present invention discloses an emotion recognition system based on cardiac activity physiological signals, including a fully polymer multimode polymer optical fiber sensing unit based on the speckle principle, which can detect cardiac physiological activities with high sensitivity and generate a seismocardiogram signal (SCG signal). Through a signal extraction algorithm that is robust to motion artifacts, a method combining frequency-domain amplitude selection and sample entropy selection is used to accurately extract cardiac physiological signals from signals affected by motion artifacts. Further, an emotion recognition machine learning algorithm based on complex network feature engineering is adopted. An SCG binary network is generated by threshold binary processing of the Euclidean distance matrix, and network topology features are extracted. Combining time-domain heart rate variability features and non-linear heart rate variability features, the input support vector machine model is trained and emotion classified. This system can overcome individual differences among subjects, achieve cross-subject emotion measurement, and has good robustness and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings forming a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:

[0036] Figure 1Schematic diagram of the system structure according to an embodiment of the present invention;

[0037] Figure 2 Structure and physical diagram of the all-polymer multimode polymer optical fiber sensing module according to an embodiment of the present invention;

[0038] Figure 3 Flow chart for preparing the all-polymer multimode polymer optical fiber sensing module according to an embodiment of the present invention;

[0039] Figure 4 Main flow chart for constructing the SCG binary network based on the SCG signal according to an embodiment of the present invention;

[0040] Figure 5 Experimental result graph of the response time and recovery time of the sensing module according to an embodiment of the present invention;

[0041] Figure 6 Accuracy result graph of the durability experiment of the sensing module according to an embodiment of the present invention;

[0042] Figure 7 Signal-to-noise ratio result graph of the durability experiment of the sensing module according to an embodiment of the present invention;

[0043] Figure 8 Accuracy result graph of the experiment on the robustness of the heart rate monitoring function of the system according to an embodiment of the present invention to ambient light;

[0044] Figure 9 Signal-to-noise ratio result graph of the experiment on the robustness of the SCG signal extraction function of the system according to an embodiment of the present invention to ambient light;

[0045] Figure 10 Experimental result graph of the robustness to motion artifacts according to an embodiment of the present invention;

[0046] Figure 11 Schematic diagram of the performance of the system according to an embodiment of the present invention in an emotion recognition task. Detailed implementation manners

[0047] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0048] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0049] Embodiment 1

[0050] As Figure 1As shown in the figure, in this embodiment, an emotion recognition system based on physiological signals of cardiac activity is provided, including:

[0051] A sensing module, configured to collect chest wall vibration signals caused by heartbeats;

[0052] A signal processing module, connected to the sensing unit, configured to extract clean heart vibration signals from the chest wall vibration signals through frequency domain amplitude selection and sample entropy screening;

[0053] An emotion recognition module, connected to the signal processing unit, configured to extract heart vibration signal features from the heart vibration signals and perform emotion classification according to the heart vibration signal features.

[0054] The specific structure is as follows:

[0055] As Figure 2 shown, the sensing module is configured with a multi-mode polymer optical fiber into a three-ring structure and encapsulated with PDMS to enhance sensitivity and robustness. Among them, the outer diameter of the multi-mode polymer optical fiber is 1 mm, and the core diameter is 0.5 mm. The diameter of the sensor is 35 mm, and the thickness is 4 mm.

[0056] As Figure 3 shown, the preparation steps of the multi-mode polymer optical fiber speckle sensing module are as follows:

[0057] Weigh 4.0 grams of dimethyl silicone base material and 0.4 grams of curing agent with an analytical balance. After fully mixing with a stirring rod, let the base material curing agent mixture stand to eliminate air bubbles. Fix the multi-mode polymer optical fiber into three rings in a 3D printed mold. Pour the degassed base material curing agent mixture into the mold, let it stand and cure, and then demold to obtain the multi-mode polymer optical fiber speckle sensing module.

[0058] The relevant hardware required for the multi-mode polymer sensing unit also includes: a laser light source, a camera, and a computer. Among them, the laser light source provides coherent light input to the sensor, exciting multiple modes of the multi-mode polymer optical fiber. The camera is arranged at the far end of the sensor to record the speckles output by the sensor, and then transmits the speckle video to the computer. The sum of the pixel displacements of the speckles is calculated through the following formula to detect the chest wall vibration caused by heartbeats, and realize the detection of cardiac physiological activities:

[0059] ;

[0060] The signal processing module is mainly divided into two parts: The first part is the original signal extraction stage, which converts the collected remote speckle images into grayscale images. After Gaussian filtering the first frame image, the ROI region is selected by the OTSU method to generate a binary mask, and the binary mask is multiplied with all the images pixel by pixel to obtain an image with the original grayscale values inside the ROI and all zeros outside the ROI. After taking the element-by-element difference of these processed images frame by frame and then squaring each element, the original cardiac activity signal is obtained.

[0061] The second stage is the signal preprocessing stage. The main objective in this stage is to obtain a clean SCG signal, which also means that the noise caused by motion artifacts needs to be overcome in this stage to obtain a clean SCG signal. First, considering the non-uniform signal frequencies, windowing is performed on the signal. After that, the noise existing outside the heart rate frequency range is filtered out by a Butterworth filter with a passband of 0.6 Hz - 3.0 Hz. Then, the time-domain signal is converted into a frequency-domain signal using the fast Fourier transform. Accurately determining the frequency components of the SCG signal means making the correct selection among the signal peaks in the frequency-domain signal. Generally, the selection basis is the frequency-domain amplitude, and it is considered that the frequency corresponding to the largest frequency-domain amplitude is the heart rate signal. However, when there is noise caused by motion artifacts with frequencies in the passband, relying solely on the frequency-domain amplitude for screening may wrongly select the frequency corresponding to the noise, causing errors in heart rate measurement. Therefore, in this invention, sample entropy screening is introduced. In the field of physiological signal monitoring, sample entropy is often used to detect the complexity of physiological signals, and its calculation formula can be expressed as:

[0062] ;

[0063] where ϕ is the mean value of the proportion of signal segments of length m with pairwise Chebyshev distances less than the threshold r over the entire signal. In this invention , .

[0064] Since sample entropy can reflect the complexity of time-series signals, and a correct heart rate signal should belong to a kind of pseudo-random time-series signal with a lower complexity compared to noise, its time-domain signal sample entropy should be lower. Based on this principle, after screening on the frequency-domain signal based on the frequency-domain amplitude, the selected frequency components and the background noise are retained. The inverse fast Fourier transform (IFFT) is performed on the selected frequency components respectively to obtain the corresponding time-domain signals, and then the sample entropy is used to select the correct and clean SCG signal.

[0065] The emotion classification module includes the following parts: feature extraction, model training, and model testing.

[0066] Among them, in the feature extraction part, in order to achieve cross-subject emotion recognition, this invention innovatively introduces the method of the SCG binary network, such asFigure 4 First, the SCG signal is segmented into individual RR intervals based on the R peak, and one RR interval represents a complete cardiac cycle. Then, the Euclidean distance between pairwise RR intervals is calculated to obtain the distance matrix between the RR intervals of the signal. To better statistically analyze the topological features of the network and better relativize the network, the present invention performs a binarization operation on the SCG binary network. Specifically, considering a binarization threshold, when the distance between two RR intervals is greater than this threshold, the edge weight is set to 0, that is, the edge is removed; conversely, the edge weight is set to 1 and the edge is retained.

[0067] After constructing the SCG binary network, this embodiment will extract the topological features of the network and measure the specific properties demonstrated at the macroscopic level of the network by the microscopic similarity between pairwise heartbeats in the heart rate signal according to the properties of the network. Specifically, the present invention extracts the following four features of the network:

[0068] 1. Assortativity coefficient: , which reflects whether the nodes in the SCG network are connected to nodes with degrees close to their own. Among them, is the probability that the degrees of the two endpoints are j and k respectively, represents the probability that the neighbor node of a random node is k, is The variance r > 0 represents assortative mixing of the network, and r < 0 represents disassortative mixing of the network.

[0069] 2. Average shortest path: , which reflects the small-world property of the SCG network. The calculation method is the average of the shortest paths between different heartbeat nodes in the network.

[0070] 3. Average betweenness centrality: , which reflects the importance of the heartbeat nodes in the SCG network. The betweenness centrality of the network is the average of the betweenness centralities of the nodes. The average betweenness centrality of the network nodes can be used as a distinguishing index for the smoothness of heartbeats.

[0071] 4. Clustering coefficient: , which is used to measure the degree of clustering of each heartbeat node in the SCG signal network. Qualitatively speaking, when there are more similar heartbeat times in the SCG signal, the nodes in the SCG network are more inclined to cluster.

[0072] In addition to the features of the network, the present invention also introduces time-domain heart rate variability features and traditional non-linear heart rate variability features based on Poincaré scatter plots. After completing the feature extraction, the present invention inputs the features into a support vector machine model and trains the model with the existing data in the training set.

[0073] The effect verification is as follows:

[0074] Verification of Sensor Response Time and Recovery Time. The design and materials used in the sensor facilitate rapid response and recovery times. The experiment evaluated the performance of the sensor by measuring the signal response and recovery time when a table tennis ball was dropped from a height of 5 cm and hit the sensor. The results of this experiment (see Figure 5 ) showed that the average response time for the three experiments was 15.09 milliseconds, and the average recovery time was 11.32 milliseconds.

[0075] In the durability evaluation phase, the experiment evaluated the performance of the sensor immediately after manufacturing and after 3 months of use. The results of the accuracy of heart rate monitoring and the signal-to-noise ratio (SNR) of the SCG signal are shown respectively in Figure 6 and Figure 7 . It was observed that both the accuracy and the signal-to-noise ratio remained stable. The average accuracy of heart rate monitoring was 99.64% at the time of manufacturing completion and 99.66% after 3 months of use, while the average signal-to-noise ratio values were 14.92 dB and 14.75 dB respectively. These results demonstrate the high durability of the sensor in obtaining clean and accurate SCG signals.

[0076] In the evaluation of the sensor's robustness to ambient light, this embodiment evaluated the performance of the sensor in bright and dark environments. The results of these experiments are shown in Figure 8 and Figure 9 . Obviously, regardless of the presence of ambient light, the accuracy of heart rate monitoring and the signal-to-noise ratio (SNR) of the SCG signal remained stable. The average accuracy of heart rate monitoring was 99.64% under bright conditions and 99.50% under dark conditions, while the average signal-to-noise ratio values were 14.92 dB and 14.15 dB respectively. It is worth noting that under bright conditions with ambient light, both the accuracy and the signal-to-noise ratio were higher than those under dark conditions. It can be concluded that the POF sensor is basically not affected by ambient light.

[0077] Finally, the experiment evaluated the robustness of the signal processing module to motion artifacts. In this embodiment, the subjects were required to perform three types of actions: moving forward and backward, moving left and right, and rotating the upper body. To ensure that the noise generated by motion artifacts fell within the bandwidth of the band-pass filter and to test the effectiveness of the sample entropy method, the subjects were required to perform these actions at a frequency greater than 1 Hz and less than 1.5 Hz, which corresponded to the range of resting heart rate. In addition, to simulate motion artifacts in the real world, the order and number of these actions were not fixed in the trial. Similar to other experiments, this study involved five subjects, and each subject participated in five trials.

[0078] The results of this embodiment are shown in Figure 10Among them, the control group for heart rate monitoring without motion artifacts showed an average accuracy of 99.64%. The average accuracy of heart rate monitoring in the experimental group involving motion artifacts was 98.32%. In contrast, in the presence of motion artifacts, the traditional method without applying sample entropy only achieved an average accuracy of 91.55%, thus verifying the effectiveness of introducing sample entropy in the method of this embodiment.

[0079] In this embodiment, the performance of the present invention in emotion recognition was evaluated through experiments. At this stage, the subjects were required to watch video clips designed to evoke four different emotions: fear, happiness, relaxation, and sadness. Each video was divided into two-minute segments. The first minute was designed to evoke the emotional response of the subjects, and the second minute recorded the SCG signals. Each emotion corresponded to six video clips. The experiment involved eight subjects, and finally 192 SCG signal records were obtained. After the above feature extraction process, a support vector machine model with an RBF kernel and hyperparameters of , was used for emotion recognition. To verify the cross-subject ability of the present invention, four verification methods were adopted. Among them, non-cross-subject cross-validation: 4-fold cross-validation, 8-fold cross-validation; cross-subject cross-validation: leave-two-out cross-validation, leave-one-out cross-validation. Specifically, 4-fold cross-validation means retaining 25% of the data as the test set for cross-validation, and leave-two-out cross-validation means retaining two subjects as the test set for cross-validation. Since this embodiment will be carried out on 8 subjects, these two cross-validation methods have test sets of the same size; while 8-fold cross-validation means retaining 12.5% of the data as the test set for cross-validation, and leave-one-out cross-validation means retaining one subject as the test set for cross-validation. These two test set division methods also have test sets of equal size. Therefore, the comparison of cross-subject / non-cross-subject emotion classification accuracies will be carried out on these two combinations with the same test set size.

[0080] The experimental results are as Figure 11 , and the average accuracies of leave-one-out cross-validation, 8-fold cross-validation, leave-two-out cross-validation, and 4-fold cross-validation are 80.21%, 82.81%, 78.50%, and 80.73% respectively. Among them, the leave-one-out cross-validation of cross-subject and the 8-fold cross-validation of non-cross-subject have the same test set size, and the leave-two-out cross-validation of cross-subject and the 4-fold cross-validation of non-cross-subject have the same test set size. The differences in accuracies between the cross / non-cross-subject cross-validation methods with the same test set size in the two groups are 2.60% and 2.23% respectively. The close accuracies prove the ability of the present invention to overcome subject differences.

[0081] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An emotion recognition system based on physiological signals of cardiac activity, characterized in that, Including: A sensing module, the sensing module includes a multimode polymer optical fiber speckle sensing unit obtained by winding a multimode polymer optical fiber into a ring and encapsulating it with polydimethylsiloxane, for collecting chest wall vibration signals caused by heartbeats through speckle images; A signal processing module, connected to the sensing unit, for extracting clean SCG signals from the chest wall vibration signals through frequency-domain amplitude selection and sample entropy screening; An emotion recognition module, connected to the signal processing unit, for extracting SCG signal features from the SCG signals and classifying emotions according to the SCG signal features; The emotion recognition module includes: A feature extraction unit, for constructing an SCG binary network and extracting network topology features; A machine learning unit, for classifying emotions based on the extracted network topology features.

2. The system according to claim 1, characterized in that, The sensing unit includes: A laser light source, for providing coherent light; A multimode polymer optical fiber speckle sensing unit, obtained by winding a multimode polymer optical fiber into a ring and encapsulating it with polydimethylsiloxane; A camera, for recording the speckle image at the distal end of the optical fiber; A computer, for calculating the speckle displacement according to the speckle image and obtaining the chest wall vibration signal according to the speckle displacement.

3. The system according to claim 2, characterized in that, The preparation steps of the multimode polymer optical fiber speckle sensing unit include: mixing dimethylsiloxane base material and curing agent, pouring them into a mold, standing for curing and then demolding to obtain a multimode polymer optical fiber fixed as three rings.

4. The system according to claim 1, characterized in that, The signal processing module includes: An original signal extraction unit, for converting the speckle image into a grayscale image and obtaining the original SCG signal through mask processing and frame-by-frame difference; A signal preprocessing unit, for extracting clean SCG signals from the original SCG signals through windowing, filtering, fast Fourier transform and sample entropy screening.

5. The system according to claim 4, characterized in that, The steps of the sample entropy screening include: Performing amplitude screening on the frequency-domain signal to retain candidate frequency components; Performing inverse Fourier transform on the candidate frequency components to obtain the time-domain signal; Calculating the sample entropy of each time-domain signal and selecting the signal with the minimum sample entropy as the clean SCG signal.

6. The system according to claim 1, characterized in that, The construction method of the SCG binary network is: Segmenting the SCG signal into RR intervals according to the R peak; Calculating the Euclidean distance matrix between RR intervals; Generating an SCG binary network by threshold binary processing of the Euclidean distance matrix.

7. The system according to claim 1, characterized in that, The network topology features include assortativity coefficient, average shortest path, average betweenness centrality and clustering coefficient.

Images