An emotion recognition system based on cardiac activity physiological signals

Abstract

The present invention provides an emotion recognition system based on cardiac activity physiological signals, belonging to the field of emotion recognition technology. The system includes an all-polymer multimode polymer optical fiber sensing unit based on the speckle principle, which can detect cardiac physiological activity with high sensitivity and generate a cardiac tremor signal (SCG signal). A signal extraction algorithm robust to motion artifacts is used, combining frequency domain amplitude selection with sample entropy selection to accurately extract cardiac physiological signals from signals affected by motion artifacts. Furthermore, an emotion recognition machine learning algorithm based on complex network feature engineering is employed. The Euclidean distance matrix is ​​processed by threshold binarization to generate an SCG binary network. The network topology features are extracted, combined with time-domain heart rate variability features and nonlinear heart rate variability features, and then input into a support vector machine model for training and emotion classification. This system can overcome individual differences among subjects and achieve cross-subject emotion measurement, demonstrating excellent robustness and accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and in particular relates to an emotion recognition system based on cardiac activity physiological signals. Background Art

[0002] Accurate heart rate measurement is fundamental to emotion measurement systems based on physiological signals from cardiac activity. Among various heart rate measurement methods, fiber-optic sensors offer numerous advantages: high sensitivity, flexibility, short response times, and good biocompatibility. Furthermore, because the sensing portion of fiber-optic sensors is entirely fiber-optic, they contain no electrical components and no metal, offering excellent resistance to electromagnetic interference and electrochemical corrosion, and can even be used in extreme electromagnetic interference environments such as MRI. However, because fiber-optic sensing systems typically use body vibrations caused by cardiovascular activity as their signal source, they are susceptible to motion artifacts caused by the subject's body and muscles, which can affect accurate cardiac activity measurements. Currently, there is no recognized and effective method for motion artifact reduction in fiber-optic sensor-based cardiac activity monitoring. In photoplethysmography (PPG), which also detects cardiovascular activity through light, commonly used motion artifact reduction techniques typically require the assistance of a reference signal, making them less convenient.

[0003] Therefore, it is of great significance to develop an algorithm that does not rely on reference signals and can extract accurate cardiac activity physiological signals from signals affected by motion artifacts in an end-to-end manner.

[0004] The connection between heart activity and emotions provides a theoretical basis for measuring emotions based on cardiac physiological signals. The recent development of wearable heart rate monitoring devices has greatly improved the convenience of heart rate monitoring. Systems that can conveniently collect human cardiac physiological signals in daily life and accurately measure emotions have broad application prospects.

[0005] However, the field of emotion measurement based on physiological signals faces a common challenge: decreased cross-subject classification accuracy due to subject variability. Due to significant physiological variations between subjects, cross-subject emotion measurement accuracy is often lower than non-cross-subject emotion measurement. Cross-subject emotion measurement means that the model is trained and tested on data from different subjects. This facilitates the development of large-scale emotion measurement systems in engineering terms and scientifically means that the features used by the model for emotion measurement have a similar distribution across subjects.

[0006] Therefore, it is of great significance to develop an emotion measurement system that can measure emotions with similar accuracy across / non-cross subjects. Summary of the Invention

[0007] In order to solve the above technical problems, the present invention proposes an emotion recognition system based on cardiac activity physiological signals to solve the problems existing in the above-mentioned prior art.

[0008] To achieve the above objectives, the present invention provides an emotion recognition system based on cardiac activity physiological signals, comprising:

[0009] A sensor module is used to collect chest wall vibration signals caused by heartbeat;

[0010] a signal processing module connected to the sensing unit, configured to extract a clean cardiotremor signal from the chest wall vibration signal through frequency domain amplitude selection and sample entropy screening;

[0011] The emotion recognition module is connected to the signal processing unit and is used to extract the heart shock signal features from the heart shock signal and perform emotion classification according to the heart shock signal features.

[0012] Preferably, the sensing unit includes:

[0013] A laser light source for providing coherent light;

[0014] A multimode polymer optical fiber speckle sensing unit is obtained by winding a multimode polymer optical fiber into a ring and encapsulating it with polydimethylsiloxane;

[0015] a camera for recording the speckle image at the far end of the optical fiber;

[0016] The computer is used for calculating the speckle displacement according to the speckle image and obtaining the chest wall vibration signal according to the speckle displacement.

[0017] Preferably, the preparation steps of the multimode polymer optical fiber speckle sensing unit include: mixing a dimethylsiloxane base material and a curing agent, pouring the mixture into a mold, standing and curing, and then demoulding to obtain a multimode polymer optical fiber fixed into three rings.

[0018] Preferably, the signal processing module includes:

[0019] The original signal extraction unit is used to convert the speckle image into a grayscale image and obtain the original SCG signal through mask processing and frame-by-frame difference;

[0020] The signal preprocessing unit is used to extract a clean SCG signal from the original SCG signal through windowing, filtering, fast Fourier transform and sample entropy screening.

[0021] Preferably, the step of sample entropy screening includes:

[0022] Perform amplitude screening on the frequency domain signal and retain the candidate frequency components;

[0023] Perform inverse Fourier transform on the candidate frequency components to obtain the time domain signal;

[0024] Calculate the sample entropy of each time domain signal and select the signal with the smallest sample entropy as the clean SCG signal.

[0025] Preferably, the emotion classification module includes:

[0026] Feature extraction unit, used to construct the SCG binary network and extract network topology features;

[0027] A machine learning unit for sentiment classification based on the extracted network topology features.

[0028] Preferably, the method for constructing the SCG binary network is:

[0029] The SCG signal was divided into RR intervals based on the R peak;

[0030] Calculate the Euclidean distance matrix between RR intervals;

[0031] The Euclidean distance matrix is ​​processed by threshold binarization to generate the SCG binary network.

[0032] Preferably, the network topology characteristics include assortativity coefficient, average shortest path, average betweenness centrality and clustering coefficient.

[0033] Compared with the prior art, the present invention has the following advantages and technical effects:

[0034] The present invention discloses an emotion recognition system based on cardiac activity physiological signals, including an all-polymer multimode polymer optical fiber sensing unit based on the speckle principle, which can detect cardiac physiological activity with high sensitivity and generate a cardiac tremor signal (SCG signal). Through a signal extraction algorithm that is robust to motion artifacts, a method combining frequency domain amplitude selection with sample entropy selection is used to accurately extract cardiac physiological signals from signals affected by motion artifacts. Furthermore, an emotion recognition machine learning algorithm based on complex network feature engineering is adopted. The Euclidean distance matrix is ​​processed by threshold binarization to generate an SCG binary network, and the network topology features are extracted. Combined with the time domain heart rate variability features and nonlinear heart rate variability features, the network is input into a support vector machine model for training and emotion classification. The system can overcome individual differences among subjects and realize cross-subject emotion measurement with good robustness and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0036] Figure 1Schematic diagram of the system structure of an embodiment of the present invention;

[0037] Figure 2 The structure and physical diagram of the all-polymer multimode polymer optical fiber sensing module according to an embodiment of the present invention;

[0038] Figure 3 This is a flow chart for preparing an all-polymer multimode polymer optical fiber sensing module according to an embodiment of the present invention;

[0039] Figure 4 This is a main flow chart of constructing an SCG binary network based on SCG signals according to an embodiment of the present invention;

[0040] Figure 5 This is a diagram showing the experimental results of the response time and recovery time of the sensor module according to an embodiment of the present invention;

[0041] Figure 6 This is a graph showing the accuracy results of a durability experiment of a sensor module according to an embodiment of the present invention;

[0042] Figure 7 This is a graph showing the signal-to-noise ratio results of a durability test of a sensor module according to an embodiment of the present invention;

[0043] Figure 8 This is a graph showing the accuracy results of an experiment on the robustness of the heart rate monitoring function of the system according to an embodiment of the present invention to ambient light;

[0044] Figure 9 Graph showing the signal-to-noise ratio results of an experiment on the robustness of the SCG signal extraction function of the system according to an embodiment of the present invention to ambient light;

[0045] Figure 10 FIG is an experimental result diagram of the robustness to motion artifacts of an embodiment of the present invention;

[0046] Figure 11 Schematic diagram of the performance of the system according to an embodiment of the present invention on the emotion recognition task. DETAILED DESCRIPTION

[0047] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0048] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0049] Example 1

[0050] like Figure 1As shown, this embodiment provides an emotion recognition system based on cardiac activity physiological signals, including:

[0051] A sensor module is used to collect chest wall vibration signals caused by heartbeat;

[0052] A signal processing module is connected to the sensing unit and is used to extract a clean cardiotremor signal from the chest wall vibration signal through frequency domain amplitude selection and sample entropy screening;

[0053] The emotion recognition module is connected to the signal processing unit and is used to extract the heart shock signal features from the heart shock signal and perform emotion classification based on the heart shock signal features.

[0054] The specific structure is as follows:

[0055] like Figure 2 As shown, the sensing module consists of a multimode polymer optical fiber configured in a three-ring structure and encapsulated with PDMS to enhance sensitivity and robustness. The multimode polymer optical fiber has an outer diameter of 1 mm and a core diameter of 0.5 mm. The sensor has a diameter of 35 mm and a thickness of 4 mm.

[0056] like Figure 3 As shown in FIG, the preparation steps of the multimode polymer optical fiber speckle sensing module are as follows:

[0057] An analytical balance was used to weigh 4.0 g of dimethylsiloxane base and 0.4 g of curing agent. The mixture was thoroughly mixed with a stirring rod and allowed to stand to eliminate bubbles. The multimode polymer optical fiber was fixed into three rings in a 3D-printed mold. The defoamed base-curing agent mixture was poured into the mold, allowed to stand and cure, and then demolded to obtain a multimode polymer optical fiber speckle sensing module.

[0058] The multimode polymer sensor unit also requires hardware including a laser light source, a camera, and a computer. The laser light source provides coherent light for the sensor, stimulating the various modes of the multimode polymer optical fiber. The camera, located at the far end of the sensor, records the sensor's speckle pattern and transmits the speckle pattern video to the computer. The computer calculates the sum of the pixel-by-pixel displacements of the speckle pattern using the following formula to detect chest wall vibration caused by the heartbeat, enabling the detection of cardiac physiological activity:

[0059] ;

[0060] The signal processing module consists of two main parts. The first part is the raw signal extraction stage, which converts the acquired distal speckle image into a grayscale image. After Gaussian filtering the first frame, the ROI region is selected using the OTSU method to generate a binary mask. This mask is then multiplied by the entire image pixel-wise, resulting in an image with the original grayscale values ​​within the ROI and zero values ​​outside the ROI. These processed images are then subtracted frame-by-frame and element-by-element squared to obtain the original cardiac activity signal.

[0061] The second stage is the signal preprocessing stage. The main goal in this stage is to obtain a clean SCG signal, which also means that in this stage it is necessary to overcome the noise caused by motion artifacts and obtain a clean SCG signal. First, in order to consider the non-uniform frequency of the signal, the signal is windowed. After that, the noise outside the heart rate frequency range is filtered out by a Butterworth filter with a passband of 0.6Hz-3.0Hz. The time domain signal is then converted into a frequency domain signal using fast Fourier transform. Accurately determining the frequency component of the SCG signal means that the correct selection needs to be made in the signal peak of the frequency domain signal. Generally speaking, the basis for selection is the frequency domain amplitude, and the frequency domain amplitude with the largest value is considered to be the frequency corresponding to the heart rate signal. However, when there is noise caused by motion artifacts with a frequency in the passband, relying solely on the frequency domain amplitude for screening may incorrectly select the frequency corresponding to the noise, causing errors in heart rate measurement. Therefore, the present invention introduces sample entropy screening. In the field of physiological signal monitoring, sample entropy is often used to detect the complexity of physiological signals, and its calculation formula can be expressed as:

[0062] ;

[0063] Where ϕ is the average value of the ratio of signal segments with length m whose pairwise Chebyshev distance is less than the threshold r on the entire signal. , .

[0064] Since sample entropy reflects the complexity of a time-series signal, and a correct heart rate signal is a pseudo-random time-series signal with less complexity than noise, its time-domain sample entropy should be lower. Based on this principle, after filtering the frequency-domain signal based on its amplitude, the filtered frequency components and noise floor are retained. These filtered frequency components are then subjected to an inverse fast Fourier transform (IFFT) to obtain the corresponding time-domain signal. Sample entropy is then used to select the correct and clean SCG signal.

[0065] The emotion classification module includes the following parts: feature extraction, model training, and model testing.

[0066] Among them, in the feature extraction part, in order to realize emotion recognition across subjects, the present invention innovatively introduces the SCG binary network method, such as Figure 4 . First, the SCG signal is divided into RR intervals according to the R peak, and one RR interval represents a complete heartbeat cycle. Then the Euclidean distance between each RR interval is calculated to obtain the distance matrix between the signal RR intervals. In order to better count the topological characteristics of the network and better realize the relativization of the network, the present invention performs a binarization operation on the SCG binary network. Specifically, considering a binarization threshold, when the distance between two RR intervals is greater than this threshold, the edge weight is reset to 0, that is, the edge is removed. Otherwise, the edge weight is reset to 1 and the edge is retained.

[0067] After constructing the SCG binary network, this embodiment extracts the network's topological features and measures the specific properties of the network's macroscopic representation of the microscopic similarities between heartbeats in the heart rate signal based on the network's properties. Specifically, the present invention extracts the following four network features:

[0068] 1. Assortative coefficient: , reflects whether the node in the SCG network is connected to the node with a degree close to itself. is the probability that the degrees of the two endpoints are j and k respectively, represents the probability that the neighbor node of a random node is k, for A variance r > 0 indicates homogamy, and r < 0 indicates heterogamy.

[0069] 2. Average shortest path: , which reflects the small-world nature of the SCG network and is calculated as the average of the shortest paths between different heartbeat nodes in the network.

[0070] 3. Average betweenness centrality: , which reflects the importance of the heartbeat nodes in the SCG network. The betweenness centrality of the network is the average value of the betweenness centrality of the nodes. The average betweenness centrality of the network nodes can be used as a distinguishing indicator of the stability of heartbeat.

[0071] 4. Clustering coefficient: , which is used to measure the degree of clustering of each heartbeat node in the SCG signal network. Qualitatively speaking, the more similar heartbeats there are in the SCG signal, the more nodes in the SCG network tend to cluster.

[0072] In addition to network features, the present invention also incorporates time-domain heart rate variability features and traditional nonlinear heart rate variability features based on Poincare scatter plots. After feature extraction, the present invention inputs the features into a support vector machine model and trains the model using existing data from the training set.

[0073] The effect verification is as follows:

[0074] Sensor response time and recovery time verification. The design and materials used in the sensor promote fast response and recovery times. The sensor performance was evaluated by measuring the signal response and recovery time when a ping-pong ball dropped from a height of 5 cm hit the sensor. The results of this experiment (see Figure 5 ) shows that the average response time of the three experiments is 15.09 milliseconds and the average recovery time is 11.32 milliseconds.

[0075] In the durability evaluation phase, the sensor performance was evaluated immediately after manufacturing and after 3 months of use. The accuracy of heart rate monitoring and the signal-to-noise ratio (SNR) of the SCG signal are shown in Figure 6 and Figure 7 It was observed that both accuracy and signal-to-noise ratio remained stable. The average accuracy of heart rate monitoring was 99.64% upon manufacture and 99.66% after three months of use, while the average signal-to-noise ratio values ​​were 14.92 dB and 14.75 dB, respectively. These results demonstrate the sensor's high durability in acquiring clean and accurate SCG signals.

[0076] In the evaluation of the sensor's robustness to ambient light, this embodiment evaluates the sensor's performance in bright and dark environments. These experimental results are presented in Figure 8 and Figure 9 Clearly, heart rate monitoring accuracy and the SCG signal's signal-to-noise ratio (SNR) remain stable regardless of ambient light. The average accuracy of heart rate monitoring is 99.64% in bright conditions and 99.50% in dark conditions, while the average SNR values ​​are 14.92 dB and 14.15 dB, respectively. Notably, both accuracy and SNR are higher in bright conditions with ambient light than in dark conditions. This suggests that POF sensors are largely unaffected by ambient light.

[0077] Finally, the experiment evaluated the robustness of the signal processing module to motion artifacts. In this example, the subjects were asked to perform three types of movements: forward and backward movement, left and right movement, and upper body rotation. To ensure that the noise generated by the motion artifacts fell within the bandwidth of the bandpass filter and to test the effectiveness of the sample entropy method, the subjects were asked to perform these movements at a frequency greater than 1 Hz and less than 1.5 Hz, which corresponds to the range of resting heart rate. In addition, to simulate motion artifacts in the real world, the order and number of these movements in the experiment were not fixed. Similar to other experiments, this study involved five subjects, and each subject participated in five trials.

[0078] The results of this example are shown in Figure 10The control group, which performed heart rate monitoring without motion artifacts, showed an average accuracy of 99.64%. The experimental group, which included motion artifacts, had an average accuracy of 98.32%. In contrast, the traditional method without sample entropy achieved an average accuracy of only 91.55% in the presence of motion artifacts, thus validating the effectiveness of introducing sample entropy in the method of this embodiment.

[0079] This embodiment experimentally evaluates the performance of the present invention in emotion recognition. In this stage, subjects were asked to watch video clips designed to induce four different emotions: fear, happiness, relaxation, and sadness. Each video was divided into two-minute segments. The first minute was designed to induce the subject's emotional response, and the second minute was used to record SCG signals. Each emotion corresponded to six video clips. The experiment involved eight subjects in total, and ultimately 192 SCG signal records were obtained. After the above feature extraction process, the hyperparameters were used. , At the same time, a support vector machine model with an RBF kernel is used for emotion recognition. In order to verify the cross-subject ability of the present invention, four verification methods were adopted. Among them, non-cross-subject cross-validation: 4-fold cross-validation, 8-fold cross-validation; cross-subject cross-validation: leave-two-out cross-validation, leave-one-out cross-validation. Specifically, 4-fold cross-validation refers to retaining 25% of the data as the test set for cross-validation, and leave-two-out cross-validation refers to retaining two subjects as the test set for cross-validation. Since this embodiment will be carried out on 8 subjects, these two cross-validation methods have test sets of the same size; and 8-fold cross-validation refers to retaining 12.5% ​​of the data as the test set for cross-validation, and leave-one-out cross-validation refers to retaining one subject as the test set for cross-validation. These two test set division methods also have test sets of equal size. Therefore, the comparison of the accuracy of cross-subject / non-cross-subject emotion classification will be carried out on a combination of the same size of these two test sets.

[0080] The experimental results are as follows Figure 11 , leave-one-out cross-validation, 8-fold cross-validation, leave-two-out cross-validation, and the average accuracy of 4-fold cross-validation were 80.21%, 82.81%, 78.50%, and 80.73%, respectively. The cross-subject leave-one-out cross-validation had the same test set size as the non-cross-subject 8-fold cross-validation, while the cross-subject leave-two-out cross-validation had the same test set size as the non-cross-subject 4-fold cross-validation. The accuracy differences between the cross-subject / non-cross-subject cross-validation methods of the same test set size were 2.60% and 2.23%, respectively. The close accuracy rates demonstrate the ability of the present invention to overcome subject differences.

[0081] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An emotion recognition system based on cardiac activity physiological signals, characterized in that: include: A sensing module, comprising a multimode polymer optical fiber speckle sensing unit obtained by winding a multimode polymer optical fiber into a ring structure and encapsulating it with polydimethylsiloxane, for collecting chest wall vibration signals caused by heartbeats through speckle images; The sensing module further includes: A laser light source for providing coherent light; a camera for recording a speckle image at the far end of a multimode polymer optical fiber; a computer, used for calculating speckle displacement according to the speckle image, and obtaining a chest wall vibration signal according to the speckle displacement; a signal processing module connected to the sensing module, configured to extract a clean cardiotremor signal from the chest wall vibration signal through frequency domain amplitude selection and sample entropy screening; The signal processing module includes: The original signal extraction unit is used to convert the speckle image into a grayscale image and obtain the original cardiac shock signal through mask processing and frame-by-frame difference; A signal preprocessing unit is used to extract a clean cardiac seismic signal from the original cardiac seismic signal through windowing, filtering, fast Fourier transform and sample entropy screening; Extracting a clean cardiac seismic signal from the original cardiac seismic signal comprises: Perform amplitude screening on the frequency domain signal and retain the candidate frequency components; Perform inverse Fourier transform on the candidate frequency components to obtain the time domain signal; Calculate the sample entropy of each time domain signal and select the signal with the smallest sample entropy as the clean cardiac shock signal; an emotion recognition module, connected to the signal processing module, configured to extract cardiac tremor signal features from the clean cardiac tremor signal and perform emotion classification based on the cardiac tremor signal features; The emotion recognition module includes: A feature extraction unit is used to construct a heart seismic binary network and extract network topology features, wherein the network topology features include assortativity coefficient, average shortest path, average betweenness centrality and clustering coefficient; The method for constructing the heart shock binary network is as follows: The cardiac signal is divided into RR intervals according to the R peak; Calculate the Euclidean distance matrix between RR intervals; The Euclidean distance matrix is ​​processed by threshold binarization to generate a heart seismic binary network; The machine learning unit is used to classify emotions based on the extracted network topology features, thereby achieving emotion recognition across subjects.

Images