Multi-modal emotion analysis method and system

Abstract

The invention discloses a multi-modal emotion analysis method and system, and the method comprises the steps: carrying out the heart rate variability analysis of a collected vital sign signal, so as to extract a time domain feature, a frequency domain feature and a nonlinear feature corresponding to a heart rate variability signal obtained after the analysis, fusing the time domain feature, the frequency domain feature and the nonlinear feature to generate a heart rate variability feature corresponding to the vital sign signal; calculating a power spectrum and a power spectrum entropy corresponding to the collected electroencephalogram signals, and extracting electroencephalogram characteristics corresponding to the electroencephalogram signals according to the power spectrum and the power spectrum entropy; extracting skin electric characteristics corresponding to the collected skin electric signals; respectively inputting the heart rate variability feature, the electroencephalogram feature and the galvanic skin feature into a pre-trained emotion recognition model, and outputting a plurality of emotion analysis results; and acquiring an emotion regulation scheme matched with each emotion analysis result to output a corresponding emotion regulation strategy, thereby improving the accuracy of the emotion analysis and regulation strategy.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal sentiment analysis method and system. Background Art

[0002] With the fast-paced development of society, people's life pressure is also increasing, resulting in frequent mood swings and physical and mental exhaustion. Emotional problems not only affect personal mental health, but may also have a negative impact on work and social life. Therefore, real-time monitoring of individual emotions and timely physical and mental adjustment has become an urgent need.

[0003] To address this problem, the existing technology uses wearable devices to obtain user heart rate data or brain wave data, and based on a pre-established recognition model, identifies different types of negative emotions based on the above data, and then can promptly prompt the user when it is determined that the user has negative emotions.

[0004] However, the above recognition method needs to recognize different types and levels of negative emotions, such as anger, very angry, extremely angry, depression, very depressed and extremely depressed, etc., so the training process of the recognition model is complicated and the recognition accuracy is not high; if the recognition model can only recognize one or several types of negative emotions, for example, it can only recognize anger or depression, etc., it is impossible to effectively and accurately regulate the user's emotions according to the degree of the user's negative emotions. Summary of the invention

[0005] In order to solve the above technical problems, the present invention discloses a multimodal emotion analysis method and system, which are used to improve the accuracy of emotion analysis and provide accurate regulation strategies.

[0006] In order to achieve the above object, the present invention discloses a multimodal sentiment analysis method, comprising:

[0007] Acquire vital sign signals, EEG signals and skin electrical signals collected within a predetermined time period;

[0008] Performing heart rate variability analysis on the vital sign signal to extract time domain features, frequency domain features and nonlinear features corresponding to the heart rate variability signal obtained after the analysis, and fusing the time domain features, frequency domain features and nonlinear features to generate heart rate variability features corresponding to the vital sign signal;

[0009] Calculating a power spectrum and a power spectrum entropy corresponding to the EEG signal, so as to extract EEG features corresponding to the EEG signal according to the power spectrum and the power spectrum entropy;

[0010] Analyzing the skin electrical signal to extract skin electrical features corresponding to the skin electrical signal;

[0011] Inputting the heart rate variability feature, the electroencephalogram feature and the electrodermal feature into a pre-trained emotion recognition model respectively, and outputting a plurality of emotion analysis results through the emotion recognition model;

[0012] Acquire an emotion regulation scheme that matches each of the emotion analysis results, and output a corresponding emotion regulation strategy according to the emotion analysis results and the emotion regulation scheme.

[0013] The present invention discloses a multimodal emotion analysis method, which first obtains multimodal physiological signals collected within a predetermined time period, wherein the multimodal physiological signals include vital sign signals, EEG signals and skin electrical signals, so as to comprehensively and accurately feedback the emotional state of the individual according to the multimodal physiological signals, thereby improving the accuracy of emotion recognition. After obtaining the multimodal physiological signals, different features are extracted from the vital sign signals, EEG signals and skin electrical signals respectively; wherein, the vital sign signals are subjected to heart rate variability analysis, and the time domain features, frequency domain features and nonlinear features corresponding to the heart rate variability signals are extracted, and then the heart rate variability features corresponding to the vital sign signals are comprehensively and accurately reflected in combination with the time domain features, frequency domain features and nonlinear features, thereby improving the accuracy of feature extraction.

[0014] While extracting the heart rate variability feature, the power spectrum and power spectrum entropy of the EEG signal are calculated and the skin electrical feature of the skin electrical signal is extracted, so that the different emotional states of the individual can be identified according to the different features corresponding to the different signals, thereby improving the comprehensiveness of the emotion analysis results finally obtained. In the present invention, the emotion analysis results corresponding to the above different features are respectively identified using a pre-trained emotion recognition model, and the emotion regulation scheme corresponding to each of the emotion analysis results is obtained to generate a corresponding emotion regulation strategy to ensure the regulation accuracy of the strategy.

[0015] As a preferred example, the time domain features, frequency domain features and nonlinear features corresponding to the heart rate variability signal obtained after the extraction and analysis include:

[0016] Extracting a plurality of signals located in the RR interval sequence from the heart rate variability signal to form a heart rate variability time series segment;

[0017] Calculating the mean of the heart rate variability time series segments to calculate the overall standard deviation of the heart rate variability time series segments according to the mean;

[0018] Calculating the root mean square of the difference between adjacent RR intervals and the standard deviation between all adjacent RR intervals in the heart rate variability time series segment;

[0019] Obtaining the number of sampling points corresponding to the difference between adjacent RR intervals in the heart rate variability time series segment being greater than a preset time threshold, and calculating the ratio of the number to the total number of RR intervals in the heart rate variability time series segment;

[0020] The overall standard deviation, the root mean square of the adjacent RR interval differences, the standard deviation of the total adjacent RR interval differences and the ratio are integrated to obtain the time domain features corresponding to the heart rate variability signal.

[0021] The present invention calculates the time domain features by intercepting the heart rate variability signal of the RR interval sequence, avoiding the interference of useless data and improving the efficiency of emotion recognition. Furthermore, the average level of the RR interval sequence data is reflected by calculating the mean value, the overall change of the heart rate variability is reflected by calculating the overall standard deviation, the fast-changing components in the heart rate variability are reflected by calculating the root mean square of the difference between adjacent RR intervals, and the sudden change of the RR interval is reflected by calculating the standard deviation of the difference between all adjacent RR intervals and the ratio, thereby improving the comprehensiveness and accuracy of the time series features.

[0022] As a preferred example, the time domain features, frequency domain features and nonlinear features corresponding to the heart rate variability signal obtained after the extraction and analysis include:

[0023] Inputting the heart rate variability time series segment into a preset autoregressive model, and obtaining the power spectrum rate corresponding to the heart rate variability time series segment through the autoregressive model and a preset Z-change transfer function;

[0024] Calculating the parameter value of the autoregressive model by Burg recursion method, so as to calculate the power spectrum density corresponding to the heart rate variability signal according to the parameter value and the power spectrum rate;

[0025] The frequency domain features corresponding to the heart rate variability signal are calculated according to the power spectrum density; wherein the frequency domain features include very low frequency power, low frequency power, high frequency power, total power and low to high frequency ratio.

[0026] The present invention adopts an autoregressive model to calculate the power spectrum and power spectrum density corresponding to the heart rate variability signal, which can not only process short data, but also overcome the shortcomings of spectrum aliasing and low resolution, and improve the accuracy of the frequency domain features.

[0027] As a preferred example, the time domain features, frequency domain features and nonlinear features corresponding to the heart rate variability signal obtained after the extraction and analysis include:

[0028] Performing Poincare scatter plot analysis on the heart rate variability time series segment to obtain a Poincare scatter plot corresponding to the heart rate variability time series segment;

[0029] The nonlinear parameters corresponding to the heart rate variability time series segment are calculated according to the Poincare scatter plot; wherein the nonlinear parameters include a first scale change value and a second scale change value corresponding to the Poincare scatter plot.

[0030] Based on the limitations of the linear analysis method, the present invention performs nonlinear analysis on the heart rate variability time series fragments, so as to combine the results of the nonlinear analysis and the linear analysis to comprehensively and accurately reflect the heart rate variability characteristics corresponding to the physiological characteristic signal, and then identify the emotional state through precise characteristics, thereby improving the accuracy of emotion recognition.

[0031] As a preferred example, the training process of the emotion recognition model includes:

[0032] An initial emotion recognition model for emotion classification is constructed through support vector machine (SVM);

[0033] Collect EEG signals, skin electrical signals and vital sign signals for training, and extract features in turn to obtain corresponding EEG features, skin electrical features and heart rate variability features to construct a feature training set;

[0034] Input different features corresponding to different emotional states into the initial emotion recognition model, continuously update the penalty parameters and kernel function parameters of the initial emotion recognition model through the PSO optimization algorithm, and calculate the loss value between the predicted value and the true value of the model after the parameter update;

[0035] When the loss value is the smallest, the emotion recognition model is obtained according to the current model parameters and in combination with the emotion recognition model.

[0036] The present invention utilizes the PSO optimization algorithm and the SVM algorithm to construct the emotion recognition model, thereby improving the accuracy of the emotion recognition model and further improving the accuracy of the emotion analysis result output by the model.

[0037] As a preferred example, the heart rate variability feature, the electroencephalogram feature and the electrodermal feature are respectively input into a pre-trained emotion recognition model, and a plurality of emotion analysis results are outputted by the emotion recognition model, including:

[0038] Inputting the heart rate variability feature into the emotion recognition model, and calculating the emotion index through the emotion recognition model; wherein the emotion index includes an emotion stress index and a fatigue index;

[0039] Inputting the EEG feature into the emotion recognition model, and obtaining a brain state index through the emotion recognition model; wherein the brain state index includes an attention index, a relaxation index, and a brain fatigue index;

[0040] Inputting the electrodermal characteristics into the emotion recognition model, and obtaining a psychological state index through the emotion recognition model; wherein the psychological state index includes a psychological stress index and a tension degree;

[0041] The emotion index, brain state index and psychological state index are compared with preset thresholds respectively to determine the emotion analysis result of the user according to the comparison result.

[0042] The present invention utilizes the different characteristic parameters to respectively identify different emotional states, accurately and comprehensively reflect the emotional state of an individual, and simultaneously sets different thresholds to monitor the emotional state, determine whether the emotional state needs to be adjusted, and thus construct an accurate adjustment strategy.

[0043] As a preferred example, the obtaining of an emotion regulation scheme matching each of the emotion analysis results to output a corresponding emotion regulation strategy according to the emotion analysis results and the emotion regulation scheme includes:

[0044] When the emotional stress index is less than a preset emotional stress index threshold, selecting a first brainwave music that matches the emotional stress index from a preset brainwave music library, and generating a first emotional regulation scheme that matches the emotional stress index;

[0045] When the attention index is less than a preset attention index threshold, a second brainwave music matching the attention index is selected from a preset brainwave music library to generate a second emotion regulation scheme matching the attention index.

[0046] The present invention sets different emotion regulation schemes based on different emotion recognition results, so that the emotion regulation strategy can comprehensively and accurately regulate the emotional state.

[0047] On the other hand, the present invention discloses a multimodal emotion analysis system, including a multimodal signal acquisition component, a signal monitoring module, an emotion recognition module and a feedback regulation module;

[0048] The multimodal signal acquisition component is connected to the signal monitoring module signal, and is used to obtain vital sign signals, EEG signals and skin electrical signals collected within a predetermined time period, and send the EEG signals, skin electrical signals and vital sign signals to the signal monitoring module, so that the signal monitoring module performs the following steps:

[0049] Performing heart rate variability analysis on the vital sign signal to extract time domain features, frequency domain features and nonlinear features corresponding to the heart rate variability signal obtained after the analysis, and fusing the time domain features, frequency domain features and nonlinear features to generate heart rate variability features corresponding to the vital sign signal;

[0050] Calculating a power spectrum and a power spectrum entropy corresponding to the EEG signal, so as to extract EEG features corresponding to the EEG signal according to the power spectrum and the power spectrum entropy;

[0051] Analyzing the skin electrical signal to extract skin electrical features corresponding to the skin electrical signal;

[0052] The signal monitoring module is connected to the emotion recognition module by signal, and is used to send the heart rate variability feature, the EEG feature and the skin electrical feature to the emotion recognition module, so that the emotion recognition module inputs the heart rate variability feature, the EEG feature and the skin electrical feature into the pre-trained emotion recognition model respectively, and outputs a plurality of emotion analysis results through the emotion recognition model;

[0053] The emotion recognition module is signal-connected to the feedback regulation module, and is used to send the emotion analysis results to the feedback regulation module, so that the feedback regulation module obtains an emotion regulation scheme matching each of the emotion analysis results, and outputs a corresponding emotion regulation strategy according to the emotion analysis results and the emotion regulation scheme.

[0054] The present invention discloses a multimodal emotion analysis system, which first acquires multimodal physiological signals collected within a predetermined time period, wherein the multimodal physiological signals include vital sign signals, EEG signals and skin electrical signals, so as to comprehensively and accurately feedback the emotional state of an individual based on the multimodal physiological signals, thereby improving the accuracy of emotion recognition. After acquiring the multimodal physiological signals, different features are extracted from the vital sign signals, EEG signals and skin electrical signals, respectively; wherein, the vital sign signals are subjected to heart rate variability analysis, and the time domain features, frequency domain features and nonlinear features corresponding to the heart rate variability signals are extracted, and then the heart rate variability features corresponding to the vital sign signals are comprehensively and accurately reflected in combination with the time domain features, frequency domain features and nonlinear features, thereby improving the accuracy of feature extraction.

[0055] While extracting the heart rate variability feature, the power spectrum and power spectrum entropy of the EEG signal are calculated and the skin electrical feature of the skin electrical signal is extracted, so that the different emotional states of the individual can be identified according to the different features corresponding to the different signals, thereby improving the comprehensiveness of the emotion analysis results finally obtained. In the present invention, the emotion analysis results corresponding to the above different features are respectively identified using a pre-trained emotion recognition model, and the emotion regulation scheme corresponding to each of the emotion analysis results is obtained to generate a corresponding emotion regulation strategy to ensure the regulation accuracy of the strategy.

[0056] As a preferred example, the multimodal signal acquisition component includes a blood oxygen pulse sensor, a forehead temperature sensor, an electroencephalogram sensor, and a skin electrical sensor;

[0057] The blood oxygen pulse sensor is connected to the signal monitoring module by signal, and is used to collect and send the user's blood oxygen signal, pulse signal, heart rate signal and breathing signal to the signal monitoring module within a predetermined time period;

[0058] The forehead temperature sensor is connected to the signal monitoring module signal, and is used to collect and send the user's forehead temperature signal to the signal monitoring module within a predetermined time period; wherein the vital sign signal includes the blood oxygen signal, pulse signal, heart rate signal, breathing signal and forehead temperature signal;

[0059] The EEG sensor is signal-connected to the signal monitoring module, and is used to collect and send the user's EEG signal to the signal monitoring module within a predetermined time period;

[0060] The skin electrical sensor is signal-connected to the signal monitoring module, and is used to collect and send the user's skin electrical signal to the signal monitoring module within a predetermined time period.

[0061] The present invention sets different types of sensors for collecting physiological data of different modalities, so as to utilize the diversity of the data to comprehensively and accurately extract feature data reflecting the emotional state, and then utilize the high-precision and highly comprehensive feature data to perform emotion recognition, thereby improving the accuracy of emotion recognition.

[0062] As a preferred example, the signal monitoring module further includes a time domain analysis unit;

[0063] The time domain analysis unit is used to intercept several signals located in the RR interval sequence from the heart rate variability signal to form a heart rate variability time series segment; calculate the mean of the heart rate variability time series segment to calculate the overall standard deviation of the heart rate variability time series segment based on the mean; calculate the root mean square of the difference between adjacent RR intervals in the heart rate variability time series segment and the standard deviation between all adjacent RR intervals; obtain the number of sampling points corresponding to the difference between adjacent RR intervals in the heart rate variability time series segment that is greater than a preset time threshold, and calculate the ratio of the number to the total number of RR intervals in the heart rate variability time series segment; integrate the overall standard deviation, the root mean square of the difference between adjacent RR intervals, the standard deviation of the difference between the total adjacent RR intervals and the ratio to obtain the time domain features corresponding to the heart rate variability signal.

[0064] The present invention calculates the time domain features by intercepting the heart rate variability signal of the RR interval sequence, avoiding the interference of useless data and improving the efficiency of emotion recognition. Furthermore, the average level of the RR interval sequence data is reflected by calculating the mean value, the overall change of the heart rate variability is reflected by calculating the overall standard deviation, the fast-changing components in the heart rate variability are reflected by calculating the root mean square of the difference between adjacent RR intervals, and the sudden change of the RR interval is reflected by calculating the standard deviation of the difference between all adjacent RR intervals and the ratio, thereby improving the comprehensiveness and accuracy of the time series features.

[0065] As a preferred example, the signal monitoring module further includes a frequency domain analysis unit;

[0066] The frequency domain analysis unit is used to input the heart rate variability time series segment into a preset autoregressive model, and obtain the power spectrum rate corresponding to the heart rate variability time series segment through the autoregressive model and a preset Z-change transfer function;

[0067] Calculating the parameter value of the autoregressive model by Burg recursion method, so as to calculate the power spectrum density corresponding to the heart rate variability signal according to the parameter value and the power spectrum rate;

[0068] The frequency domain features corresponding to the heart rate variability signal are calculated according to the power spectrum density; wherein the frequency domain features include very low frequency power, low frequency power, high frequency power, total power and low to high frequency ratio.

[0069] The present invention adopts an autoregressive model to calculate the power spectrum and power spectrum density corresponding to the heart rate variability signal, which can not only process short data, but also overcome the shortcomings of spectrum aliasing and low resolution, and improve the accuracy of the frequency domain features.

[0070] As a preferred example, the signal monitoring module further includes a nonlinear analysis unit;

[0071] The nonlinear unit is used to perform Poincare scatter plot analysis on the heart rate variability time series segment to obtain the Poincare scatter plot corresponding to the heart rate variability time series segment;

[0072] The nonlinear parameters corresponding to the heart rate variability time series segment are calculated according to the Poincare scatter plot; wherein the nonlinear parameters include a first scale change value and a second scale change value corresponding to the Poincare scatter plot.

[0073] Based on the limitations of the linear analysis method, the present invention performs nonlinear analysis on the heart rate variability time series fragments, so as to combine the results of the nonlinear analysis and the linear analysis to comprehensively and accurately reflect the heart rate variability characteristics corresponding to the physiological characteristic signal, and then identify the emotional state through precise characteristics, thereby improving the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 : A flowchart of a multimodal sentiment analysis method disclosed in an embodiment of the present invention;

[0075] Figure 2 : is a structural diagram of a multimodal emotion analysis system disclosed in an embodiment of the present invention;

[0076] Figure 3 : A flowchart of a multimodal sentiment analysis method disclosed in another embodiment of the present invention;

[0077] Figure 4 : is a structural schematic diagram of a multi-modal data acquisition headband disclosed in yet another embodiment of the present invention;

[0078] Figure 5 : A schematic diagram of the relationship between a characteristic parameter and a pressure index disclosed in another embodiment of the present invention;

[0079] Figure 6 : A schematic diagram of a trend of SDNN and pressure index disclosed in another embodiment of the present invention;

[0080] Figure 7 : A schematic diagram of the trend of HF and pressure index disclosed in another embodiment of the present invention;

[0081] Figure 8 : A schematic diagram of a model training method disclosed in another embodiment of the present invention. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] Embodiment 1

[0084] This embodiment discloses a multimodal sentiment analysis method. For details, please refer to Figure 1 , mainly including steps 101 to 106, the steps mainly include:

[0085] Step 101: Obtain vital sign signals, EEG signals and skin electrical signals collected within a predetermined time period.

[0086] Step 102: Perform heart rate variability analysis on the vital sign signal to extract time domain features, frequency domain features and nonlinear features corresponding to the heart rate variability signal obtained after analysis, and fuse the time domain features, frequency domain features and nonlinear features to generate heart rate variability features corresponding to the vital sign signal.

[0087] In this embodiment, the step mainly includes: intercepting a number of signals located in the RR interval sequence from the heart rate variability signal to form a heart rate variability time series segment;

[0088] Calculate the mean of the heart rate variability time series segment to calculate the overall standard deviation of the heart rate variability time series segment based on the mean; calculate the root mean square of the difference between adjacent RR intervals in the heart rate variability time series segment and the standard deviation between all adjacent RR intervals; obtain the number of sampling points corresponding to the difference between adjacent RR intervals in the heart rate variability time series segment that is greater than a preset time threshold, and calculate the ratio of the number to the total number of RR intervals in the heart rate variability time series segment; integrate the overall standard deviation, the root mean square of the difference between adjacent RR intervals, the standard deviation of the difference between the total adjacent RR intervals and the ratio to obtain the time domain features corresponding to the heart rate variability signal.

[0089] At the same time, the heart rate variability time series segment is input into a preset autoregressive model, and the power spectrum rate corresponding to the heart rate variability time series segment is obtained through the autoregressive model and a preset Z-change transfer function;

[0090] The parameter values ​​of the autoregressive model are calculated by the Burg recursion method to calculate the power spectrum density corresponding to the heart rate variability signal according to the parameter values ​​and the power spectrum rate; the frequency domain characteristics corresponding to the heart rate variability signal are calculated according to the power spectrum rate density; wherein the frequency domain characteristics include extremely low frequency power, low frequency power, high frequency power, total power and low to high frequency ratio.

[0091] Furthermore, a Poincare scatter plot analysis is performed on the heart rate variability time series segment to obtain a Poincare scatter plot corresponding to the heart rate variability time series segment; and nonlinear parameters corresponding to the heart rate variability time series segment are calculated based on the Poincare scatter plot; wherein the nonlinear parameters include a first scale change value and a second scale change value corresponding to the Poincare scatter plot.

[0092] In this embodiment, this step calculates the time domain features by intercepting the heart rate variability signal of the RR interval sequence, avoiding the interference of useless data and improving the efficiency of emotion recognition. Further, the mean value is calculated to reflect the average level of the RR interval sequence data, the overall standard deviation is calculated to reflect the overall change of heart rate variability, the root mean square of the difference between adjacent RR intervals is calculated to reflect the fast-changing components in heart rate variability, and the standard deviation of the difference between all adjacent RR intervals and the ratio are calculated to reflect the sudden change of the RR interval, thereby improving the comprehensiveness and accuracy of the time series features.

[0093] At the same time, an autoregressive model is used to calculate the power spectrum and power spectrum density corresponding to the heart rate variability signal, which can not only process short data, but also overcome the shortcomings of spectrum aliasing and low resolution, and improve the accuracy of the frequency domain features.

[0094] Furthermore, based on the limitations of the linear analysis method, a nonlinear analysis is performed on the heart rate variability time series segment, so as to combine the results of the nonlinear analysis and the linear analysis to comprehensively and accurately reflect the heart rate variability characteristics corresponding to the physiological characteristic signal, and then identify the emotional state through precise characteristics, thereby improving the accuracy of emotion recognition.

[0095] Step 103: Calculate the power spectrum and power spectrum entropy corresponding to the EEG signal, so as to extract the EEG features corresponding to the EEG signal according to the power spectrum and the power spectrum entropy.

[0096] Step 104: Analyze the skin electrical signal and extract skin electrical features corresponding to the skin electrical signal.

[0097] Step 105: input the heart rate variability feature, the electroencephalogram feature and the electrodermal feature into a pre-trained emotion recognition model respectively, and output a plurality of emotion analysis results through the emotion recognition model.

[0098] In this embodiment, before executing this step, the emotion recognition model is first trained, and the training process of the emotion recognition model includes:

[0099] An initial emotion recognition model for emotion classification is constructed by support vector machine (SVM); EEG signals, skin electrodermal signals and vital sign signals for training are collected, and feature extraction is performed in sequence to obtain corresponding EEG features, skin electrodermal features and heart rate variability features to construct a feature training set; different features corresponding to different emotional states are input into the initial emotion recognition model, and the penalty parameters and kernel function parameters of the initial emotion recognition model are continuously updated by the PSO optimization algorithm, and the loss value between the predicted value and the true value of the model after the parameters are updated is calculated; when the loss value is the smallest, the emotion recognition model is obtained according to the current model parameters in combination with the emotion recognition model.

[0100] Furthermore, the heart rate variability characteristics are input into the emotion recognition model, and the emotion index is calculated through the emotion recognition model; wherein the emotion index includes an emotional stress index, a fatigue index and an anxiety index; the EEG characteristics are input into the emotion recognition model, and the brain state index is obtained through the emotion recognition model; wherein the brain state index includes an attention index, a relaxation index and a brain fatigue index; the electrodermal characteristics are input into the emotion recognition model, and the psychological state index is obtained through the emotion recognition model; wherein the psychological state index includes a psychological stress index and a degree of tension; the emotion index, brain state index and psychological state index are compared with preset thresholds respectively, so as to determine the user's emotion analysis result according to the comparison result.

[0101] In this embodiment, in this step, the PSO optimization algorithm and the SVM algorithm are used to construct the emotion recognition model, thereby improving the accuracy of the emotion recognition model, and then improving the accuracy of the emotion analysis results output by the model. Furthermore, the different characteristic parameters are used to respectively identify different emotional states, accurately and comprehensively reflect the emotional state of the individual, and at the same time, different thresholds are set to monitor the emotional state to determine whether the emotional state needs to be adjusted, so as to construct an accurate adjustment strategy.

[0102] Step 106: Acquire an emotion regulation scheme that matches each of the emotion analysis results, so as to output a corresponding emotion regulation strategy according to the emotion analysis results and the emotion regulation scheme.

[0103] In this embodiment, the step is: when the emotional stress index is less than a preset emotional stress index threshold, selecting a first brain wave music that matches the emotional stress index from a preset brain wave music library, and generating a first emotion regulation plan that matches the emotional stress index; when the attention index is less than a preset attention index threshold, selecting a second brain wave music that matches the attention index from a preset brain wave music library, and generating a second emotion regulation plan that matches the attention index.

[0104] In this embodiment, this step sets different emotion regulation schemes based on different emotion recognition results, so that the emotion regulation strategy can comprehensively and accurately regulate the emotional state.

[0105] On the other hand, this embodiment also discloses a multimodal emotion analysis system. For the specific structure of the analysis system, please refer to Figure 2 , including a multimodal signal acquisition component 201, a signal monitoring module 202, an emotion recognition module 203 and a feedback adjustment module 204.

[0106] The multimodal signal acquisition component 201 is connected to the signal monitoring module 202 for acquiring vital sign signals, EEG signals and skin electrical signals acquired within a predetermined time period, and sending the EEG signals, skin electrical signals and vital sign signals to the signal monitoring module 202, so that the signal monitoring module 202 performs the following steps:

[0107] Performing heart rate variability analysis on the vital sign signal to extract time domain features, frequency domain features and nonlinear features corresponding to the heart rate variability signal obtained after the analysis, and fusing the time domain features, frequency domain features and nonlinear features to generate heart rate variability features corresponding to the vital sign signal.

[0108] The power spectrum and the power spectrum entropy corresponding to the EEG signal are calculated to extract the EEG features corresponding to the EEG signal according to the power spectrum and the power spectrum entropy.

[0109] The skin electrical signal is analyzed to extract skin electrical features corresponding to the skin electrical signal.

[0110] The signal monitoring module 202 is signal-connected to the emotion recognition module 203, and is used to send the heart rate variability characteristics, the EEG characteristics and the electrodermal characteristics to the emotion recognition module 203, so that the emotion recognition module 203 inputs the heart rate variability characteristics, the EEG characteristics and the electrodermal characteristics into the pre-trained emotion recognition model respectively, and outputs a number of emotion analysis results through the emotion recognition model.

[0111] The emotion recognition module 203 is signal-connected to the feedback regulation module 204, and is used to send the emotion analysis results to the feedback regulation module 204, so that the feedback regulation module 204 obtains an emotion regulation scheme matching each of the emotion analysis results, and outputs a corresponding emotion regulation strategy according to the emotion analysis results and the emotion regulation scheme.

[0112] In this embodiment, the multimodal signal acquisition component 201 includes a blood oxygen pulse sensor, a forehead temperature sensor, an electroencephalogram sensor, and a skin electrical sensor.

[0113] The blood oxygen pulse sensor is signal-connected to the signal monitoring module, and is used to collect and send the user's blood oxygen signal, pulse signal, heart rate signal and breathing signal to the signal monitoring module within a predetermined time period.

[0114] The forehead temperature sensor is signal-connected to the signal monitoring module, and is used to collect and send the user's forehead temperature signal to the signal monitoring module within a predetermined time period; wherein the vital sign signal includes the blood oxygen signal, pulse signal, heart rate signal, breathing signal and forehead temperature signal.

[0115] The EEG sensor is signal-connected to the signal monitoring module and is used to collect and send the user's EEG signal to the signal monitoring module within a predetermined time period.

[0116] The skin electrical sensor is signal-connected to the signal monitoring module, and is used to collect and send the user's skin electrical signal to the signal monitoring module within a predetermined time period.

[0117] In this embodiment, the signal monitoring module 202 further includes a time domain analysis unit.

[0118] The time domain analysis unit is used to intercept several signals located in the RR interval sequence from the heart rate variability signal to form a heart rate variability time series segment; calculate the mean of the heart rate variability time series segment to calculate the overall standard deviation of the heart rate variability time series segment based on the mean; calculate the root mean square of the difference between adjacent RR intervals in the heart rate variability time series segment and the standard deviation between all adjacent RR intervals; obtain the number of sampling points corresponding to the difference between adjacent RR intervals in the heart rate variability time series segment that is greater than a preset time threshold, and calculate the ratio of the number to the total number of RR intervals in the heart rate variability time series segment; integrate the overall standard deviation, the root mean square of the difference between adjacent RR intervals, the standard deviation of the difference between the total adjacent RR intervals and the ratio to obtain the time domain features corresponding to the heart rate variability signal.

[0119] In this embodiment, the signal monitoring module 202 includes a frequency domain analysis unit.

[0120] The frequency domain analysis unit is used to input the heart rate variability time series segment into a preset autoregressive model, and obtain the power spectrum rate corresponding to the heart rate variability time series segment through the autoregressive model and a preset Z-change transfer function.

[0121] The parameter values ​​of the autoregressive model are calculated by Burg recursion method, so as to calculate the power spectrum density corresponding to the heart rate variability signal according to the parameter values ​​and the power spectrum rate.

[0122] The frequency domain features corresponding to the heart rate variability signal are calculated according to the power spectrum density; wherein the frequency domain features include very low frequency power, low frequency power, high frequency power, total power and low to high frequency ratio.

[0123] In this embodiment, the signal monitoring module 203 includes a nonlinear analysis unit.

[0124] The nonlinear unit is used to perform Poincare scatter plot analysis on the heart rate variability time series segment to obtain a Poincare scatter plot corresponding to the heart rate variability time series segment.

[0125] The nonlinear parameters corresponding to the heart rate variability time series segment are calculated according to the Poincare scatter plot; wherein the nonlinear parameters include a first scale change value and a second scale change value corresponding to the Poincare scatter plot.

[0126] Embodiment 2

[0127] Specifically, this embodiment further discloses a multimodal sentiment analysis method. Specifically, the implementation process of the analysis method is shown in FIG. Figure 3 , mainly including steps 301 to 304, the steps are mainly:

[0128] Step 301: Acquire vital sign signals, EEG signals and galvanic skin signals within a preset time period according to a preset multimodal physiological signal acquisition headband.

[0129] Specifically, in the steps, the vital sign signals include physiological signals such as heart rate, skin electricity, blood oxygen, respiration, and forehead temperature.

[0130] Optionally, a multi-modal physiological signal acquisition headband is provided, wherein the structure of the acquisition headband is shown in Figure 4 , including 401 blood oxygen pulse sensor, which collects blood oxygen and pulse at the center of the forehead; 402 forehead temperature sensor, which is used to collect forehead temperature at the center of the forehead; 403 EEG sensor, which is used to collect EEG signals of left forehead FP1 and right forehead FP2; 404 skin electricity sensor, which is used to collect skin electricity signals at both ends of the fingers; 405 ear electrode.

[0131] Furthermore, the blood oxygen pulse sensor 401 adopts a reflective pulse blood oxygen sensor, and the light emission wavelength is 660nm±3nm (red light) and 940nm±10nm (infrared light).

[0132] The 402 forehead temperature sensor adopts a non-contact infrared temperature sensor with a measurement temperature range of 0-100°C and an accuracy of 0.1. The thermopile infrared temperature sensor is mainly used for non-contact temperature measurement. The basic working principle is to convert the thermal radiation of the target object into a voltage output.

[0133] The 403 EEG sensor adopts the KS1092 dual-channel EEG chip, a dual-channel high-precision, low-power analog front-end chip used for measuring EEG and other weak electrical signals.

[0134] The 404 skin electrical sensor adopts a medical silicone finger sleeve with a stainless steel sheet embedded inside.

[0135] The 405 ear electrode is made of soft silicone and has a silver chloride solid powder sintered electrode embedded inside.

[0136] In this embodiment, if Figure 4 The circuit of the modal physiological signal acquisition headband shown uses a flexible FPC, wherein the part in contact with the human epidermis uses soft medical silicone.

[0137] In the specific emotion analysis process, open the software on the tablet or computer to establish a communication connection with the headband, and turn on the Bluetooth connection device to establish a signal connection with the headband. After the device is successfully connected to the headband, first detect the quality of the EEG signal collected by the headband. If the signal quality is good, start collecting multimodal physiological signals for a period of time (set to 3 minutes), and perform HRV, EEG and GSR data analysis at the same time. The data analysis includes dynamic real-time display of EEG waveforms, breathing waveforms, HRV curves, blood oxygen, heart rate, body temperature and other physiological signals.

[0138] Step 302: Perform HRV time domain analysis, HRV frequency domain analysis and HRV nonlinear analysis on the vital sign signal to extract the HRV characteristic parameters corresponding to the vital sign signal, and simultaneously calculate the power spectrum and power spectrum entropy corresponding to the EEG signal to extract the EEG features corresponding to the EEG signal and the electrical skin features corresponding to the electrical skin signal based on the power spectrum and the power spectrum entropy.

[0139] Specifically, a heart rate variability analysis is performed on the vital sign signal to obtain a heart rate variability signal, and the signal located in the RR interval sequence in the heart rate variability signal is extracted as RR interval sequence data, so as to calculate the mean of the RR interval sequence data, the overall standard deviation, the root mean square of the difference between adjacent RR intervals, the standard deviation between all adjacent RR intervals, and the ratio of the number of adjacent RR intervals with a difference greater than a preset time threshold to the total number of RR intervals as HRV time domain feature parameters.

[0140] Optionally, statistical analysis is used to extract HRV time domain feature values, and the heart rate variability is analyzed by calculating the statistical indicators of the RR interval (calculated by the interval between the peak R and R points in the ECG signal wave group). The mean (MEAN) reflects the average level of the RR interval, and the calculation formula is:

[0141]

[0142] Wherein, N is the number of RR intervals within the HRV detection time; is the average value of N RR intervals; RR i is the i-th RR interval.

[0143] The overall standard deviation (SDNN) reflects the overall change of HRV and is calculated as follows:

[0144]

[0145] The root mean square difference between adjacent RR intervals (RMSSD) reflects the fast-changing components in HRV and is calculated as follows:

[0146]

[0147] where RR i+1 is the i+1th RR interval.

[0148] The standard deviation of the differences between all adjacent RR intervals (SDSD) is calculated as:

[0149]

[0150] PNN50 represents the ratio of the number of adjacent RR intervals whose difference is greater than the preset time interval (set to 50ms) to the total number of RR intervals, reflecting the sudden change of the RR interval. The calculation formula is:

[0151] PNN50 = NN50 ÷ TotalNN × 100%

[0152] The NN50 represents the number of adjacent RR intervals whose difference is greater than 50 ms; and the TotalNN represents the total number of RR intervals.

[0153] Specifically, the heart rate variability time series segment is input into a preset autoregressive model, and the power spectrum rate corresponding to the heart rate variability time series segment is obtained through the autoregressive model and a preset Z-change transfer function; the parameter value of the autoregressive model is calculated by the Burg recursion method, so as to calculate the power spectrum density corresponding to the heart rate variability signal according to the parameter value and the power spectrum rate; the frequency domain characteristics corresponding to the heart rate variability signal are calculated according to the power spectrum rate density; wherein the frequency domain characteristics include extremely low frequency power, low frequency power, high frequency power, total power and low to high frequency ratio.

[0154] Optionally, the frequency domain analysis method is used to analyze HRV, which mainly analyzes the energy distribution of the signal at different frequencies. Frequency domain analysis is related to time domain analysis and reflects certain laws hidden in HRV. Extracting HRV features through frequency domain analysis can reflect the vagus nerve and sympathetic nerve activities of the heart, among which the low frequency (LF) and high frequency (HF) of HRV and their ratio (LF / HF) can indicate the balance of the vagus nerve and the sympathetic nerve.

[0155] When analyzing HRV from the frequency domain, the power spectrum of HRV must first be estimated, then the power spectrum in different frequency bands must be integrated, and finally the HRV eigenvalues ​​must be extracted. In this embodiment, the autoregressive model method is used for frequency domain analysis.

[0156] Furthermore, the AR model method is used for frequency domain analysis, and the calculation formula of the AR model is as follows:

[0157]

[0158] The transfer function is generated by Z transform, and the transfer function calculation formula is as follows:

[0159]

[0160] In the formula, x(n) is a random signal, x(nk) is its own past values, u(n) is white noise, H(z) is the system function, and p is the system order.

[0161] Furthermore, the Burg recursion method is used to calculate the parameter values ​​of the AR model to obtain the HRV power spectrum density PSD, so as to extract the features of HRV. The power spectrum density PSD calculation formula is as follows:

[0162]

[0163] x n is the nth sample of the signal, N is the length of the signal, W is the frequency, j is the imaginary unit, E is the expected value, e = 2.7

[0164] Among them, the quantitative indicators of the power spectrum density curve include VLF (very low frequency power), LF (low frequency power) (0.04-0.15Hz), HF high frequency power (0.15-0.4Hz), TP (total power), LF / HF (low-high frequency ratio). When LF / HF is between 0.6 and 1.5, it indicates that the sympathetic nerve and the vagus nerve are in a balanced state; when it is less than 0.6, it indicates that the vagus nerve is dominant; and when it is greater than 1.5, the sympathetic nerve is dominant.

[0165] Further, when the quantitative index is calculated as the HRV frequency domain characteristic parameter, the calculation formula of the VLF is:

[0166] The calculation formula of LF is:

[0167] The calculation formula of HF is:

[0168] The calculation formula of TP is:

[0169] Specifically, a Poincare scatter plot analysis is performed on the heart rate variability time series segment to obtain a Poincare scatter plot corresponding to the heart rate variability time series segment; and nonlinear parameters corresponding to the heart rate variability time series segment are calculated based on the Poincare scatter plot; wherein the nonlinear parameters include a first scale change value and a second scale change value corresponding to the Poincare scatter plot.

[0170] Optionally, a Poincare scatter plot analysis method is used to perform HRV nonlinear analysis, wherein the Poincare scatter plot is a point distribution diagram, and the distribution diagram marks the positions of points in a rectangular coordinate system. Furthermore, in this embodiment, when using the Poincare scatter plot analysis, the positions of all adjacent RR intervals are marked. The marked distribution diagram can intuitively reflect the overall characteristics of HRV, and can also reflect the heart rate during instantaneous changes, thereby revealing the nonlinear law of HRV.

[0171] Furthermore, the Poincare scatter plot method is used to extract nonlinear features of HRV. Using the scatter plot method, two main parameters SD1 and SD2 can be obtained. SD1 reflects the changes in the low-frequency and very low-frequency components of HRV, which is related to the changes in the sympathetic nerves: SD2 is the rapid change of HRV, which is related to the activity of the vagus nerve. The calculation formulas of SDI and SD2 are as follows:

[0172]

[0173] Wherein, SD1 and SD2 represent the first scale change value and the second scale change value corresponding to the Poincare scatter plot.

[0174] Furthermore, after extracting the HRV characteristic parameters corresponding to the vital sign signal, in a certain implementation of this embodiment, the state of the brain can be reflected based on the specific frequency band in the electroencephalogram (EEG) signal, such as theta wave (Theta): at 4-7 Hz, usually associated with deep relaxation and sleep; alpha wave (Alpha): at 8-13 Hz, usually associated with a quiet, relaxed alert state; beta wave (Beta): at 13-30 Hz, usually associated with active thinking, concentration or anxiety; gamma wave (Gamma): at 30-100 Hz, usually associated with higher-level brain functions such as information processing, learning and memory formation.

[0175] The artifact-removed EEG signal is sampled through the EEG acquisition module, and the sampling rate is set to 100 Hz. Then, a non-overlapping window is added to the 2-channel EEG signal X={X1,X2} every 10 seconds. The processed EEG data is differentiated and Fourier transformed, and the EEG offline data is subjected to power frequency denoising and electrooculogram removal.

[0176] Furthermore, the EMD algorithm performs a series of stabilization processes on the EEG signal, that is, the original signal is screened to decompose the fluctuations or trends of single components of different scales, and a set of data is obtained with linear and steady-state characteristics. The steps of the EMD algorithm are as follows:

[0177] Step 1. Calculate all extreme points of the induced EEG signal, including the maximum and minimum values, through the time scale formula x(t)=0 at the zero point position;

[0178] Step 2. Use the interpolation function to fit, and the lower envelope e(t) of the time series is formed by the local minimum points, and the upper envelope of the time series is formed by the local maximum points.

[0179] Step 3. Remember the mean of the two envelopes, denoted as m1(t), as shown in the formula:

[0180]

[0181] Step 4. Extract the new component and remove the mean m1(t) of the envelope from the original time series to obtain a new signal component. Such as the formula:

[0182]

[0183] Step 5. Check whether the extracted new component meets the two requirements of the modal eigenfunction. If so, it is the first-order intrinsic mode function of the original signal. If not, repeat Step 1 to Step 4 until the conditions are met to obtain the first-order intrinsic mode function of the time series.

[0184] Step 6. Now assume h1 1 (t) meets the requirements after multiple cycles and is recorded as the first-order eigenmode function c1(t), as shown in the formula:

[0185] c1(t)=imf1(t)=h1 k (t)

[0186] Where K is the number of cycles;

[0187] Step 7. Calculate the residual function r1(t) after removing the first-order intrinsic mode function c1(t), as follows:

[0188] r1(t)=x(t)-c1(t)

[0189] Step 8. Take the residual function r1(t) as input and repeat the above steps until the intrinsic mode functions c2(t) of all orders are obtained. n (t), the remaining residual component r n (t) satisfies monotonic change and is less than a given threshold, at which point the EMD decomposition process ends.

[0190] Finally, the multimodal x(t) is decomposed into the empirical mode as follows:

[0191]

[0192] Furthermore, the hybrid feature extraction module first performs empirical mode decomposition on the EEG signal after preprocessing to make it adaptively decomposed, and reconstructs the decomposed modal eigenfunctions in sequence to form a new time series. The new EEG time series is subjected to linear and nonlinear feature extraction. The linear features include fluctuation index and energy, and the nonlinear features are fused by approximate entropy, sample entropy and fuzzy entropy. Finally, all the features are taken as the total feature vector of the induced EEG signal time series. The hybrid feature vector describes the information of the EEG signal more comprehensively, which is convenient for better classification. The specific steps of hybrid feature extraction are as follows:

[0193] Step 1. Preprocess the EEG signal time series to remove noise and improve the signal-to-noise ratio:

[0194] Step 2. Select EEG signal sample X, the number of samples is m, the sample length is N, x i One of the samples is represented as

[0195] x i =[x i (1),x i (2),...,x i (n)],1≤i≤m;

[0196] Step 3. Perform EMD decomposition on the EEG signal time series X. Each sample is decomposed into a series of intrinsic mode function sequences IMF. Take the first IMF of each sample for sequence reconstruction to obtain a new time series Y. The number of samples of Y is m.

[0197] Step 4. For the above reconstructed time series, calculate the linear characteristic fluctuation index F1 and energy index Energy of the EEG signal, denoted as FI(Y)=[FI1,FI2,FI m ],Energy(Y)=[Energy1,...Energy m ];

[0198] Step 5. Calculate the nonlinear feature approximate entropy, sample entropy and fuzzy entropy respectively.

[0199] FastAPEn(Y)=[FastAPEn1,...,FastAPEn m ],

[0200] sampEn(Y)=[sampEn1,...,sampEn m ],

[0201] FuzzyEn(Y)=[FuzzyEn1,...,FuzzyEnm ]

[0202] Where m = 2, r = 0.2, N is the length of the sample data;

[0203] Step 6. Fusion of linear features and nonlinear features Features(Y) = [FI(Y), Energy(Y), FastAPEn(Y), sampEn(Y), FuzzEn(Y)];

[0204] Features(Y) in the formula is the feature of sample Y, that is, the mixed linear and nonlinear features of the EEG signal.

[0205] In one implementation of this embodiment, the skin electrical signal with artifacts removed is sampled by the skin electrical acquisition module, the sampling rate is set to 100 Hz, and then a non-overlapping window is added to the 1-channel skin electrical signal every 10 seconds, and the processed skin electrical data is differentiated and Fourier transformed, and the skin electrical features are extracted by differential entropy. The following is the calculation formula of differential entropy.

[0206]

[0207] Where K is the number of samples of skin electrical signal, X i is the sample value of the skin electrical signal at the i-th time point, X I+1 is the sample value of the skin electrical signal at the i+1th time point.

[0208] Step 303: input the heart rate variability feature, the electroencephalogram feature and the electrodermal feature into a pre-trained emotion recognition model respectively, and output a plurality of emotion analysis results through the emotion recognition model.

[0209] Specifically, in this embodiment, the heart rate variability characteristics are input into the emotion recognition model, and the emotion index is calculated through the emotion recognition model; wherein the emotion index includes an emotional stress index, a fatigue index and an anxiety index; the EEG characteristics are input into the emotion recognition model, and the brain state index is obtained through the emotion recognition model; wherein the brain state index includes an attention index, a relaxation index and brain fatigue; the electrodermal characteristics are input into the emotion recognition model, and the psychological state index is obtained through the emotion recognition model; wherein the psychological state index includes a psychological stress index and a degree of tension; the emotion index, brain state index and psychological state index are compared with preset thresholds respectively to determine the user's emotion analysis results based on the comparison results.

[0210] Optionally, the mean of the heart rate can reflect the speed of a human heartbeat: Therefore, the extracted HRV characteristic parameters are closely related to different states of human emotions. Figure 5is the statistical result of characteristic data under different pressure conditions. Figure 5 As shown in the figure, by comparing the HRV feature values ​​extracted under the two stress states, it can be seen that the RR mean value under the low-pressure state is 0.8286 seconds, and the RR mean value under the high-pressure state is 0.9612 seconds. The RR mean value under high pressure is 0.1326 seconds higher than that under low pressure. It can be seen that the RR mean value will gradually decrease with the increase in the degree of emotion, indicating that cardiovascular activity and corresponding metabolic activities are enhanced under high-pressure conditions: comparing the heart rate mean value under the two stress states, that is, HR mean, the heart rate beats 86.2425 times per minute under high-pressure conditions and 69.7633 times per minute under low-pressure conditions, indicating that the fluctuation of heart rate under high-pressure conditions is greater than that under low-pressure conditions. Correspondingly, the PNN50 under the two states is 0.2563 under low pressure and 0.3462 under high pressure. Since PNN50 is related to the vagus nerve of the human body and can reflect the heartbeat, the numerical comparison of PNN50 under high and low pressure states shows that the stability of the heartbeat under low pressure is better than that under high pressure, that is, the heartbeat is faster under high pressure. Further, by comparing the LF value and LF / HF value under high pressure and low pressure, it can be seen that the LF value and LF / HF value under high pressure are smaller than those under low pressure, indicating that when the pressure level increases, the energy of the low-frequency component in HRV is reduced. Although the calculated characteristic values ​​can be used to compare the differences between different pressure states, in order to more accurately illustrate this difference, the emotional state is judged by the significance of the HRV characteristic parameters based on the numerical comparison.

[0211] Further, by Figure 5 It can be seen that SDNN shows obvious differences under the two pressure levels. The SDNN under high pressure is significantly lower than that under low pressure. As the pressure increases, HF shows an upward trend, and the high pressure state is higher than the low pressure state. Figure 6 and Figure 7 As the pressure level increases, the two characteristic parameters of SDNN and HF change significantly, which means that the SDNN and HF characteristic values ​​can effectively detect pressure and can be better applied to pressure recognition algorithms.

[0212] Specifically, in this embodiment, the PSO optimization algorithm and the SVM algorithm are combined to perform pressure identification. When the SVM is used for classification, in order to improve the classification accuracy, the SVM parameters are optimized by the PSO optimization algorithm to obtain the optimal eigenvalue, and then the optimal eigenvalue is sent to the classifier for training and prediction. In the process of model optimization, the penalty parameter c and the kernel function parameter g of the SVM are optimized. The process of optimizing the SVM parameters based on PSO is as follows: Figure 8 shown.

[0213] The HRV feature parameters extracted by time domain, frequency domain and nonlinear methods, EEG features and skin electricity (GSR) feature groups are used as data samples. There are 120 groups of training samples and test sample data, including 50 groups of high-voltage sample data and 50 groups of low-voltage sample data. 40 groups of data are extracted from the low-voltage samples as training data and 10 groups of data are used as test data: 40 groups of data are extracted from the high-voltage samples as training data and 10 groups of data are used as test data to test the classification ability. The pressure recognition algorithm process of PS0 algorithm optimizing SVM is as follows Figure 8 shown.

[0214] The detailed steps of pressure recognition based on PSO optimized SVM are as follows:

[0215] Step 1: Convert the pressure data into an N×M feature matrix, where N is the number of pressure data and M is the feature dimension, where N is 40 and M is 10:

[0216] Step 2: Divide the data into 40 sets of training data and 10 sets of test data sets:

[0217] Step 3: Normalize the training and test data sets:

[0218] Step 4: Initialize the .PS0 parameters and the selected SVM model:

[0219] Step 5: Combine 50 sets of low-pressure data and 50 sets of high-pressure data in the training set to train the SVM model, and combine the remaining 10 sets of low-pressure data and 10 sets of high-pressure data for data testing of the model:

[0220] Step 6: Determine whether each data set is used for verification testing. If it has been verified, go to Step 7, otherwise go to Step 5:

[0221] Stp7: Obtain the classification accuracy of the validation data set through the above steps, and use the maximum classification accuracy as the fitness value of PSO:

[0222] Step 8: Determine whether the fitness value meets the end condition. If so, calculate the best accuracy and the optimized parameter value. Otherwise, jump to Step 9:

[0223] Step 9: Update the fitness value of PS0, update the speed and position of the particle, and then execute Step 5 to let PSO continue to optimize and repeat the cycle until the conditions of Step 8 are met:

[0224] Step 10: Use the obtained optimal parameters c, g and training set data to train the final classifier model:

[0225] Step 11: Use the final SVM classifier model to recognize emotions and calculate the recognition accuracy.

[0226] Model training and verification: Use labeled emotion datasets to train the model, and verify and optimize the model to ensure the accuracy and stability of emotion recognition.

[0227] Furthermore, the average recognition rate of emotions by the PS0-SVM algorithm tested 20 times during the model training process was 90.17%, and the recognition rate of each test was above 85%.

[0228] Furthermore, by combining the heart rate variability features, EEG features and skin electrical features, the multimodal features are synthesized according to the DS evidence theory synthesis algorithm to obtain a more accurate emotion fusion feature vector. The DS evidence theory synthesis algorithm is:

[0229] 1) Define the set, using the traditional recognition framework: R = {A1, A2, ..., An}, Z = 2R is the power set of R, for any subset A of Z, let its corresponding number m, m(A)

[0230] For any subset A belonging to , let it correspond to number m, m(A)∈[0,1], and m satisfies:

[0231]

[0232] Then m is called the Basic Probability Assignment function (BPA) defined on 2R, and m(A) is called the basic credibility number of A, which reflects the degree of support for the occurrence of proposition A. The proposition with m(A)>0 is defined as the focal element or focal element of evidence.

[0233] Compute the mean of the evidence:

[0234]

[0235] Then, calculate the distance of each evidence to the average evidence:

[0236]

[0237] Finally, calculate the credibility of each piece of evidence:

[0238]

[0239] Among them, C(m i) as evidence i The weights satisfy ∑ i＝1,2,...,n c(m i )=1Here, the support degree of other evidence for this evidence is called the weight of the evidence. If the support degree of the evidence is higher, the corresponding weight is higher, and the influence on the combination result is greater; conversely, the corresponding weight is lower, and the influence on the combination result is small.

[0240] M n =c1m1+c2m2+…+c n m n

[0241] Construct a confidence matrix M (n × m) dimensional matrix based on basic probability distribution:

[0242]

[0243] in:

[0244] m i1 +…+m im =1(i=1,2,…n)

[0245] Transpose a row i in the matrix M and then multiply it with another row j in the matrix, that is:

[0246]

[0247] Expand to get a new m×m dimensional matrix:

[0248]

[0249] From the above formula, we can see that according to the numerator of the DS combination formula, the elements of the main diagonal are the product of the basic probability distribution of sensor i and sensor j for the target, and the sum of all other elements in the matrix constitutes the uncertainty factor of the evidence, that is:

[0250]

[0251] Let Zj = ajj, then we can get the fusion index, namely:

[0252]

[0253] According to the DS evidence theory synthesis algorithm rules, the three features are synthesized to calculate the multimodal fusion emotional stress index.

[0254] Furthermore, the multimodal fusion features are classified by Support Vector Machine (SVM) to obtain the dynamic emotion category of the target subject.

[0255] Specifically, in this embodiment, the SVM model preprocesses the online multimodal data to obtain online multimodal information, and performs fusion classification processing on the online multimodal information to obtain an online emotion classification result of the online multimodal data.

[0256] Specifically, Support Vector Machine (SVM) is a classification method suitable for small samples and high-dimensional features, and has good generalization ability. Its basic principle is to transform the input space into a high-dimensional space through nonlinear transformation, and then find the optimal linear distribution in this new space. Let the sample set be x i ,i=1,…,l,y i are the corresponding category labels, (x1,y1),…,(x l ,y l )∈R D ×{-1,1}. SVM is an optimization problem that solves the following quadratic equation

[0257]

[0258] subject to you i (w·x i +b)≥1-ξ i ,ξ i >0(i=1,…,l)

[0259] where C is the regularization parameter, ξ i is the slack variable, w is the weight vector, and b is the bias term.

[0260] The discriminant function f(x) of SVM is as follows

[0261]

[0262] Where N s is the number of support vectors, α i >0 is the Lagrange multiplier, G(x,x i ) is the kernel function, and b is the domain value of the classification.

[0263] Furthermore, the SVM model needs to convert the multimodal fusion emotional stress index into a feature vector and standardize the data. The standardization process uses the Z-score standardization method, and its formula is:

[0264]

[0265] Among them, χ original is the original eigenvalue, χ std is the standardized eigenvalue, μ is the mean of the feature, and σ is the standard deviation of the feature.

[0266] Furthermore, the normalized feature vector is input into the trained SVM model for emotion prediction. The prediction process of the SVM model is expressed as: * =Z·β; where γ* is the predicted emotional state; Z is the input feature matrix, and β is the coefficient vector of the SVM model.

[0267] Specifically, the emotional state is interpreted based on the prediction results of the SVM model. If the prediction result is a continuous value, it is used to represent the intensity of the emotion; if it is a discrete value, it is used to represent the emotion category.

[0268] The present invention fuses multimodal features according to the DS evidence theory synthesis algorithm through the collection of heart rate variability features, EEG features and skin electrical features to obtain a more accurate fusion feature emotional stress index. Then it is input into the SVM model to obtain the corresponding recognition result, thereby improving the accuracy and real-time performance of emotion recognition and making up for the shortcomings of traditional emotion recognition methods.

[0269] Step 304: Obtain brainwave music that matches each of the emotion analysis results, so as to output a corresponding emotion regulation strategy according to the emotion analysis result and the brainwave music.

[0270] Specifically, in this embodiment, when the emotional stress index is less than the emotional stress index threshold, a first brain wave music matching the emotional stress index is selected from a preset brain wave music library, and a first emotion regulation scheme matching the emotional stress index is generated; when the attention index is less than the preset attention index threshold, a second brain wave music matching the attention index is selected from a preset brain wave music library, and a second emotion regulation scheme matching the attention index is generated.

[0271] Optionally, a normal range of concentration, an emotional range, a fatigue index range, etc. are set to determine the index range of the user's current emotional state, so as to judge whether the user needs to adjust his emotions according to the range. Preferably, the normal range of concentration is set to [30, 100], and the larger the value, the more focused the attention and the stronger the concentration. If the average concentration is lower than 30, it means that the user is in a state of inattention and anxiety, then it is determined that the user needs to formulate an emotion regulation strategy.

[0272] Furthermore, the user can choose a customized adjustment scheme, or follow a matching adjustment scheme, and the user's emotional state is fed back in real time in the form of a curve while adjusting. Then the music feedback emotion adjustment monitors the intervention effect in real time, adjusts and optimizes the body and mind adjustment method, and ensures the best intervention effect. Specifically, the adjustment scheme is as follows:

[0273] If the concentration index is not within the preset range, Alpha brainwave music and Beta brainwave music will be used;

[0274] If the brain fatigue index is not within the preset range, white noise, Alpha wave brainwave music, Theta wave brainwave music, etc. will be used; if fatigue is present, Theta wave brainwave music training will be used; to relieve psychological stress, Alpha wave brainwave music and Theta wave brainwave music training will be used; to relieve anxiety, Alpha wave brainwave music and Theta wave brainwave music training will be used; to regulate depression, Theta wave brainwave music training will be used.

[0275] In this embodiment, the above adjustment scheme can be used to improve focus, improve sleep, relieve fatigue, relieve psychological pressure, relieve anxiety, and treat depression. Furthermore, when adjusting emotions, the EEG rhythm is extracted from the collected EEG data in real time, and the corresponding brainwave music is selected for training according to the EEG rhythm scheme set by the system. While training, users can adjust their body and mind according to the feedback map.

[0276] The present embodiment discloses an emotion regulation method, which obtains multimodal physiological signals collected within a predetermined time period, wherein the multimodal physiological signals include vital sign signals, EEG signals and skin electrical signals, so as to comprehensively and accurately feedback the emotional state of the individual based on the multimodal physiological signals, thereby improving the accuracy of emotion recognition. After obtaining the multimodal physiological signals, different features are extracted from the vital sign signals, EEG signals and skin electrical signals respectively; wherein, the vital sign signals are subjected to heart rate variability analysis, and the time domain features, frequency domain features and nonlinear features corresponding to the heart rate variability signals are extracted, and then the heart rate variability features corresponding to the vital sign signals are comprehensively and accurately reflected in combination with the time domain features, frequency domain features and nonlinear features, thereby improving the accuracy of feature extraction.

[0277] While extracting the heart rate variability feature, the power spectrum and power spectrum entropy of the EEG signal are calculated and the skin electrical feature of the skin electrical signal is extracted, so that the different emotional states of the individual can be identified according to the different features corresponding to the different signals, thereby improving the comprehensiveness of the emotion analysis results finally obtained. In the present invention, the emotion analysis results corresponding to the above different features are respectively identified using a pre-trained emotion recognition model, and the emotion regulation scheme corresponding to each of the emotion analysis results is obtained to generate a corresponding emotion regulation strategy to ensure the regulation accuracy of the strategy.

[0278] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal sentiment analysis method, characterized in that: include: Acquire vital sign signals, EEG signals and skin electrical signals collected within a predetermined time period; Performing heart rate variability analysis on the vital sign signal to extract time domain features, frequency domain features and nonlinear features corresponding to the heart rate variability signal obtained after the analysis, and fusing the time domain features, frequency domain features and nonlinear features to generate heart rate variability features corresponding to the vital sign signal; Calculating a power spectrum and a power spectrum entropy corresponding to the EEG signal, so as to extract EEG features corresponding to the EEG signal according to the power spectrum and the power spectrum entropy; Analyzing the skin electrical signal to extract skin electrical features corresponding to the skin electrical signal; Inputting the heart rate variability feature, the electroencephalogram feature and the electrodermal feature into a pre-trained emotion recognition model respectively, and outputting a plurality of emotion analysis results through the emotion recognition model; Acquire an emotion regulation scheme that matches each of the emotion analysis results, and output a corresponding emotion regulation strategy according to the emotion analysis results and the emotion regulation scheme.

2. A multimodal sentiment analysis method according to claim 1, characterized in that: The time domain features, frequency domain features and nonlinear features corresponding to the heart rate variability signal obtained after the extraction and analysis include: Extracting a plurality of signals located in the RR interval sequence from the heart rate variability signal to form a heart rate variability time series segment; Calculating the mean of the heart rate variability time series segments to calculate the overall standard deviation of the heart rate variability time series segments according to the mean; Calculating the root mean square of the difference between adjacent RR intervals and the standard deviation between all adjacent RR intervals in the heart rate variability time series segment; Obtaining the number of sampling points corresponding to the difference between adjacent RR intervals in the heart rate variability time series segment being greater than a preset time threshold, and calculating the ratio of the number to the total number of RR intervals in the heart rate variability time series segment; The overall standard deviation, the root mean square of the adjacent RR interval differences, the standard deviation of the total adjacent RR interval differences and the ratio are integrated to obtain the time domain features corresponding to the heart rate variability signal.

3. A multimodal sentiment analysis method according to claim 2, characterized in that: The time domain features, frequency domain features and nonlinear features corresponding to the heart rate variability signal obtained after the extraction and analysis include: Inputting the heart rate variability time series segment into a preset autoregressive model, and obtaining the power spectrum rate corresponding to the heart rate variability time series segment through the autoregressive model and a preset Z-change transfer function; Calculating the parameter value of the autoregressive model by Burg recursion method, so as to calculate the power spectrum density corresponding to the heart rate variability signal according to the parameter value and the power spectrum rate; The frequency domain features corresponding to the heart rate variability signal are calculated according to the power spectrum density; wherein the frequency domain features include very low frequency power, low frequency power, high frequency power, total power and low to high frequency ratio.

4. A multimodal sentiment analysis method according to claim 2, characterized in that: The time domain features, frequency domain features and nonlinear features corresponding to the heart rate variability signal obtained after the extraction and analysis include: Performing Poincare scatter plot analysis on the heart rate variability time series segment to obtain a Poincare scatter plot corresponding to the heart rate variability time series segment; The nonlinear parameters corresponding to the heart rate variability time series segment are calculated according to the Poincare scatter plot; wherein the nonlinear parameters include a first scale change value and a second scale change value corresponding to the Poincare scatter plot.

5. A multimodal sentiment analysis method according to claim 1, characterized in that: The training process of the emotion recognition model includes: An initial emotion recognition model for emotion classification is constructed through support vector machine (SVM); Collect EEG signals, skin electrical signals and vital sign signals for training, and extract features in turn to obtain corresponding EEG features, skin electrical features and heart rate variability features to construct a feature training set; Input different features corresponding to different emotional states into the initial emotion recognition model, continuously update the penalty parameters and kernel function parameters of the initial emotion recognition model through the PSO optimization algorithm, and calculate the loss value between the predicted value and the true value of the model after the parameter update; When the loss value is the smallest, the emotion recognition model is obtained according to the current model parameters and in combination with the emotion recognition model.

6. A multimodal sentiment analysis method according to claim 1, characterized in that: The heart rate variability feature, the electroencephalogram feature and the electrodermal feature are respectively input into a pre-trained emotion recognition model, and a plurality of emotion analysis results are outputted through the emotion recognition model, including: Inputting the heart rate variability feature into the emotion recognition model, and calculating the emotion index through the emotion recognition model; wherein the emotion index includes an emotion stress index and a fatigue index; Inputting the EEG feature into the emotion recognition model, and obtaining a brain state index through the emotion recognition model; wherein the brain state index includes an attention index, a relaxation index, and a brain fatigue index; Inputting the electrodermal characteristics into the emotion recognition model, and obtaining a psychological state index through the emotion recognition model; wherein the psychological state index includes a psychological stress index and a tension degree; The emotion index, brain state index and psychological state index are compared with preset thresholds respectively to determine the emotion analysis result of the user according to the comparison result.

7. A multimodal sentiment analysis method according to claim 6, characterized in that: The obtaining of an emotion regulation scheme matching each of the emotion analysis results to output a corresponding emotion regulation strategy according to the emotion analysis results and the emotion regulation scheme includes: When the emotional stress index is less than a preset emotional stress index threshold, selecting a first brainwave music that matches the emotional stress index from a preset brainwave music library, and generating a first emotional regulation scheme that matches the emotional stress index; When the attention index is less than a preset attention index threshold, a second brainwave music matching the attention index is selected from a preset brainwave music library to generate a second emotion regulation scheme matching the attention index.

8. A multimodal sentiment analysis system, characterized in that: It includes a multimodal signal acquisition component, a signal monitoring module, an emotion recognition module and a feedback regulation module; The multimodal signal acquisition component is connected to the signal monitoring module signal, and is used to obtain vital sign signals, EEG signals and skin electrical signals collected within a predetermined time period, and send the EEG signals, skin electrical signals and vital sign signals to the signal monitoring module, so that the signal monitoring module performs the following steps: Performing heart rate variability analysis on the vital sign signal to extract time domain features, frequency domain features and nonlinear features corresponding to the heart rate variability signal obtained after the analysis, and fusing the time domain features, frequency domain features and nonlinear features to generate heart rate variability features corresponding to the vital sign signal; Calculating a power spectrum and a power spectrum entropy corresponding to the EEG signal, so as to extract EEG features corresponding to the EEG signal according to the power spectrum and the power spectrum entropy; Analyzing the skin electrical signal to extract skin electrical features corresponding to the skin electrical signal; The signal monitoring module is connected to the emotion recognition module by signal, and is used to send the heart rate variability feature, the EEG feature and the skin electrical feature to the emotion recognition module, so that the emotion recognition module inputs the heart rate variability feature, the EEG feature and the skin electrical feature into the pre-trained emotion recognition model respectively, and outputs a plurality of emotion analysis results through the emotion recognition model; The emotion recognition module is signal-connected to the feedback regulation module, and is used to send the emotion analysis results to the feedback regulation module, so that the feedback regulation module obtains an emotion regulation scheme matching each of the emotion analysis results, and outputs a corresponding emotion regulation strategy according to the emotion analysis results and the emotion regulation scheme.

9. A multimodal emotion analysis system according to claim 8, characterized in that: The multimodal signal acquisition component includes a blood oxygen pulse sensor, a forehead temperature sensor, an electroencephalogram sensor, and a skin electrical sensor; The blood oxygen pulse sensor is connected to the signal monitoring module by signal, and is used to collect and send the user's blood oxygen signal, pulse signal, heart rate signal and breathing signal to the signal monitoring module within a predetermined time period; The forehead temperature sensor is connected to the signal monitoring module signal, and is used to collect and send the user's forehead temperature signal to the signal monitoring module within a predetermined time period; wherein the vital sign signal includes the blood oxygen signal, pulse signal, heart rate signal, breathing signal and forehead temperature signal; The EEG sensor is signal-connected to the signal monitoring module, and is used to collect and send the user's EEG signal to the signal monitoring module within a predetermined time period; The skin electrical sensor is signal-connected to the signal monitoring module, and is used to collect and send the user's skin electrical signal to the signal monitoring module within a predetermined time period.

10. A multimodal sentiment analysis system according to claim 8, characterized in that: The signal monitoring module includes a time domain analysis unit; The time domain analysis unit is used to intercept several signals located in the RR interval sequence from the heart rate variability signal to form a heart rate variability time series segment; calculate the mean of the heart rate variability time series segment to calculate the overall standard deviation of the heart rate variability time series segment based on the mean; calculate the root mean square of the difference between adjacent RR intervals in the heart rate variability time series segment and the standard deviation between all adjacent RR intervals; obtain the number of sampling points corresponding to the difference between adjacent RR intervals in the heart rate variability time series segment that is greater than a preset time threshold, and calculate the ratio of the number to the total number of RR intervals in the heart rate variability time series segment; integrate the overall standard deviation, the root mean square of the difference between adjacent RR intervals, the standard deviation of the difference between the total adjacent RR intervals and the ratio to obtain the time domain features corresponding to the heart rate variability signal.

11. A multimodal sentiment analysis system according to claim 10, characterized in that: The signal monitoring module includes a frequency domain analysis unit; The frequency domain analysis unit is used to input the heart rate variability time series segment into a preset autoregressive model, and obtain the power spectrum rate corresponding to the heart rate variability time series segment through the autoregressive model and a preset Z-change transfer function; Calculating the parameter value of the autoregressive model by Burg recursion method, so as to calculate the power spectrum density corresponding to the heart rate variability signal according to the parameter value and the power spectrum rate; The frequency domain features corresponding to the heart rate variability signal are calculated according to the power spectrum density; wherein the frequency domain features include very low frequency power, low frequency power, high frequency power, total power and low to high frequency ratio.

12. A multimodal sentiment analysis system according to claim 10, characterized in that: The signal monitoring module includes a nonlinear analysis unit; The nonlinear unit is used to perform Poincare scatter plot analysis on the heart rate variability time series segment to obtain the Poincare scatter plot corresponding to the heart rate variability time series segment; The nonlinear parameters corresponding to the heart rate variability time series segment are calculated according to the Poincare scatter plot; wherein the nonlinear parameters include a first scale change value and a second scale change value corresponding to the Poincare scatter plot.

Images