A Multimodal Sentiment Analysis Method and System

Abstract

This invention discloses a multimodal emotion analysis method and system, comprising: performing heart rate variability analysis on collected vital sign signals to extract time-domain features, frequency-domain features, and nonlinear features corresponding to the analyzed heart rate variability signals, and fusing the time-domain features, frequency-domain features, and nonlinear features to generate heart rate variability features corresponding to the vital sign signals; calculating the power spectrum and power spectrum entropy corresponding to the collected electroencephalogram (EEG) signals to extract EEG features corresponding to the EEG signals based on the power spectrum and power spectrum entropy; extracting skin conductance features corresponding to the collected electroskin conductance (ESC) signals; inputting the heart rate variability features, the EEG features, and the ESC features into a pre-trained emotion recognition model, and outputting several emotion analysis results; obtaining an emotion regulation scheme matching each emotion analysis result to output the corresponding emotion regulation strategy, thereby improving the accuracy of emotion analysis and regulation strategies.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a multimodal sentiment analysis method and system. Background Technology

[0002] With the fast pace of modern life, people are experiencing increasing stress, leading to frequent mood swings and mental exhaustion. Emotional problems not only affect individual mental health but can also negatively impact work and social life. Therefore, real-time monitoring of individual emotions and timely mental and physical well-being regulation have become an urgent need.

[0003] To address this issue, existing technologies use wearable devices to acquire users' heart rate data or brainwave data. Based on a pre-established recognition model, different types of negative emotions can be identified from the data, and users can be promptly alerted when negative emotions are detected.

[0004] However, the above-mentioned identification methods need to identify different types and levels of negative emotions, such as anger, very angry, extremely angry, depressed, very depressed, and extremely depressed. Therefore, the training process of the identification model is complicated and the identification accuracy is not high. If the identification model can only identify one or a few types of negative emotions, such as only being able to identify anger or depression, it cannot effectively and accurately regulate the user's emotions according to the degree of the user's negative emotions. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention discloses a multimodal emotion analysis method and system to improve the accuracy of emotion analysis and provide precise regulation strategies.

[0006] To achieve the above objectives, this invention discloses a multimodal emotion analysis method, comprising:

[0007] Acquire vital signs, electroencephalogram (EEG) signals, and electrical skin signals collected within a predetermined time period;

[0008] Heart rate variability analysis is performed on the vital signs signal to extract the time-domain features, frequency-domain features, and nonlinear features corresponding to the heart rate variability signal obtained after analysis, and the time-domain features, frequency-domain features, and nonlinear features are fused to generate the heart rate variability features corresponding to the vital signs signal.

[0009] Calculate the power spectrum and power spectrum entropy corresponding to the EEG signal, and extract the EEG features corresponding to the EEG signal based on the power spectrum and power spectrum entropy;

[0010] The skin electrical signal is analyzed to extract the corresponding skin electrical characteristics;

[0011] The heart rate variability features, electroencephalogram (EEG) features, and electrodermal conductance features are respectively input into a pre-trained emotion recognition model, and the emotion recognition model outputs several emotion analysis results.

[0012] Obtain an emotion regulation scheme that matches each of the emotion analysis results, and output a corresponding emotion regulation strategy based on the emotion analysis results and the emotion regulation scheme.

[0013] This invention discloses a multimodal emotion analysis method. First, it acquires multimodal physiological signals collected within a predetermined time period. These signals include vital signs, electroencephalogram (EEG), and electrodermal (EDT) signals. The method aims to comprehensively and accurately reflect an individual's emotional state based on these signals, thereby improving the accuracy of emotion recognition. After acquiring the multimodal physiological signals, different features are extracted from the vital signs, EEG, and EDT signals. Specifically, heart rate variability analysis is performed on the vital signs, extracting time-domain, frequency-domain, and nonlinear features corresponding to the heart rate variability signals. These time-domain, frequency-domain, and nonlinear features are then combined to comprehensively and accurately reflect the heart rate variability characteristics corresponding to the vital signs, further improving the accuracy of feature extraction.

[0014] While extracting the heart rate variability features, the power spectrum and power spectrum entropy of the electroencephalogram (EEG) signals are calculated, and the skin conductance features of the electrodermal (ED) signals are extracted. This allows for the identification of different emotional states of individuals based on the different features corresponding to different signals, improving the comprehensiveness of the final emotion analysis results. In this invention, a pre-trained emotion recognition model is used to identify the emotion analysis results corresponding to the different features, and an emotion regulation scheme corresponding to each emotion analysis result is obtained to generate a corresponding emotion regulation strategy, ensuring the regulation accuracy of the strategy.

[0015] As a preferred example, the time-domain features, frequency-domain features, and nonlinear features corresponding to the heart rate variability signal obtained after extraction and analysis include:

[0016] Several signals located in the RR interval sequence are extracted from the heart rate variability signal to form a heart rate variability time series segment;

[0017] Calculate the mean of the heart rate variability time series segment to calculate the population standard deviation of the heart rate variability time series segment based on the mean;

[0018] Calculate the root mean square of the difference between adjacent RR intervals in the heart rate variability time series segment and the standard deviation between all adjacent RR intervals;

[0019] The number of sampling points in the heart rate variability time series segment where the difference between adjacent RR intervals is greater than a preset time threshold is obtained, and the ratio of the number to the total number of RR intervals in the heart rate variability time series segment is calculated.

[0020] The overall standard deviation, the root mean square of the difference between adjacent RR intervals, the standard deviation between all adjacent RR intervals, and the ratio are used as the time-domain features corresponding to the heart rate variability signal.

[0021] This invention calculates temporal features by extracting heart rate variability signals from the RR interval sequence, avoiding interference from useless data and improving the efficiency of emotion recognition. Furthermore, by calculating the mean to reflect the average level of the RR interval sequence data, by calculating the overall standard deviation to reflect the overall change in heart rate variability, by calculating the root mean square of the difference between adjacent RR intervals to reflect the fast-changing component of heart rate variability, and by calculating the standard deviation and ratio of the difference between all adjacent RR intervals to reflect sudden changes in RR intervals, the comprehensiveness and accuracy of the temporal features are improved.

[0022] As a preferred example, the time-domain features, frequency-domain features, and nonlinear features corresponding to the heart rate variability signal obtained after extraction and analysis include:

[0023] The heart rate variability time series segment is input into a preset autoregressive model, and the power spectrum corresponding to the heart rate variability time series segment is obtained through the autoregressive model and the preset Z-transformation transfer function.

[0024] The parameters of the autoregressive model are calculated using the Burg recursion method, and the power spectral density corresponding to the heart rate variability signal is calculated based on the parameters and the power spectrum.

[0025] The frequency domain characteristics corresponding to the heart rate variability signal are calculated based on the power spectral density; wherein, the frequency domain characteristics include very low frequency power, low frequency power, high frequency power, total power, and low-to-high frequency ratio.

[0026] This invention uses an autoregressive model to calculate the power spectrum and power spectral density corresponding to the heart rate variability signal. It can not only process short data, but also overcome the shortcomings of spectrum aliasing and low resolution, thereby improving the accuracy of the frequency domain features.

[0027] As a preferred example, the time-domain features, frequency-domain features, and nonlinear features corresponding to the heart rate variability signal obtained after extraction and analysis include:

[0028] Poincaré scatter plot analysis was performed on the heart rate variability time series segment to obtain the Poincaré scatter plot corresponding to the heart rate variability time series segment;

[0029] The nonlinear parameters corresponding to the heart rate variability time series segment are calculated based on the Poincaré scatter plot; wherein the nonlinear parameters include the first scale change value and the second scale change value corresponding to the Poincaré scatter plot.

[0030] Based on the limitations of linear analysis methods, this invention performs nonlinear analysis on the heart rate variability time series segments. By combining the results of the nonlinear analysis and the linear analysis, the heart rate variability characteristics corresponding to the physiological feature signals are comprehensively and accurately reflected. In turn, the precise features are used to identify emotional states, thereby improving the accuracy of emotion recognition.

[0031] As a preferred example, the training process of the emotion recognition model includes:

[0032] An initial emotion recognition model for emotion classification is constructed using a support vector machine (SVM).

[0033] Electroencephalogram (EEG), electrodermal (ED) signals, and vital signs signals were collected for training and feature extraction was performed sequentially to obtain corresponding EEG features, EED features, and heart rate variability features, in order to construct a feature training set.

[0034] Different features corresponding to different emotional states are input into the initial emotion recognition model. The penalty parameters and kernel function parameters of the initial emotion recognition model are continuously updated through the PSO optimization algorithm, and the loss value between the model's predicted value and the true value after updating the parameters is calculated.

[0035] When the loss value is minimized, the emotion recognition model is obtained based on the current model parameters and the emotion recognition model.

[0036] This invention utilizes the PSO optimization algorithm and the SVM algorithm to construct the emotion recognition model, thereby improving the accuracy of the emotion recognition model and the accuracy of the emotion analysis results output by the model.

[0037] As a preferred example, the heart rate variability features, electroencephalogram (EEG) features, and electrodermal conductance features are respectively input into a pre-trained emotion recognition model, and the emotion recognition model outputs several emotion analysis results, including:

[0038] The heart rate variability features are input into the emotion recognition model, and an emotion index is calculated through the emotion recognition model; wherein, the emotion index includes an emotional stress index and a fatigue index;

[0039] The EEG features are input into the emotion recognition model, and the brain state index is obtained through the emotion recognition model; wherein, the brain state index includes attention index, relaxation index, and brain fatigue level;

[0040] The skin conductance feature is input into the emotion recognition model, and a psychological state index is obtained through the emotion recognition model; wherein, the psychological state index includes a psychological stress index and a level of tension;

[0041] The emotion index, brain state index, and psychological state index are compared with preset thresholds to determine the user's emotion analysis result based on the comparison results.

[0042] This invention utilizes different feature parameters to identify different emotional states, accurately and comprehensively reflecting an individual's emotional state. At the same time, it sets different thresholds to monitor the emotional state and determine whether the emotional state needs to be regulated, so as to construct a precise regulation strategy.

[0043] As a preferred example, the step of obtaining an emotion regulation scheme matching each of the emotion analysis results, and outputting a corresponding emotion regulation strategy based on the emotion analysis results and the emotion regulation scheme, includes:

[0044] When the emotional stress index is less than the preset emotional stress index threshold, a first brainwave music matching the emotional stress index is selected from the preset brainwave music library, and a first emotion regulation scheme matching the emotional stress index is generated.

[0045] When the attention index is less than a preset attention index threshold, a second brainwave music matching the attention index is selected from a preset brainwave music library, and a second emotion regulation scheme matching the attention index is generated.

[0046] This invention sets different emotion regulation schemes based on different emotion recognition results, so that the emotion regulation strategy can comprehensively and accurately regulate the emotional state.

[0047] On the other hand, the present invention discloses a multimodal emotion analysis system, including a multimodal signal acquisition component, a signal monitoring module, an emotion recognition module, and a feedback adjustment module;

[0048] The multimodal signal acquisition component is signal-connected to the signal monitoring module and is used to acquire vital sign signals, electroencephalogram (EEG) signals, and electrodermal signal (EDS) signals collected within a predetermined time period, and to send the EEG signals, EDS signals, and vital sign signals to the signal monitoring module, so that the signal monitoring module performs the following steps:

[0049] Heart rate variability analysis is performed on the vital signs signal to extract the time-domain features, frequency-domain features, and nonlinear features corresponding to the heart rate variability signal obtained after analysis, and the time-domain features, frequency-domain features, and nonlinear features are fused to generate the heart rate variability features corresponding to the vital signs signal.

[0050] Calculate the power spectrum and power spectrum entropy corresponding to the EEG signal, and extract the EEG features corresponding to the EEG signal based on the power spectrum and power spectrum entropy;

[0051] The skin electrical signal is analyzed to extract the corresponding skin electrical characteristics;

[0052] The signal monitoring module is signal-connected to the emotion recognition module and is used to send the heart rate variability feature, the electroencephalogram (EEG) feature, and the electrodermal conductance feature to the emotion recognition module, so that the emotion recognition module inputs the heart rate variability feature, the EEG feature, and the electrodermal conductance feature into a pre-trained emotion recognition model, and outputs several emotion analysis results through the emotion recognition model;

[0053] The emotion recognition module is signal-connected to the feedback adjustment module and is used to send the emotion analysis results to the feedback adjustment module so that the feedback adjustment module can obtain an emotion adjustment scheme matching each emotion analysis result and output a corresponding emotion adjustment strategy based on the emotion analysis results and the emotion adjustment scheme.

[0054] This invention discloses a multimodal emotion analysis system. First, it acquires multimodal physiological signals collected within a predetermined time period. These signals include vital signs, electroencephalogram (EEG), and electrodermal (EDS) signals. The system comprehensively and accurately reflects an individual's emotional state based on these signals, improving the accuracy of emotion recognition. After acquiring the multimodal physiological signals, different features are extracted from the vital signs, EEG, and EDS signals. Specifically, heart rate variability analysis is performed on the vital signs, extracting time-domain, frequency-domain, and nonlinear features corresponding to the heart rate variability signals. These time-domain, frequency-domain, and nonlinear features are then combined to comprehensively and accurately reflect the heart rate variability characteristics corresponding to the vital signs, further improving the accuracy of feature extraction.

[0055] While extracting the heart rate variability features, the power spectrum and power spectrum entropy of the electroencephalogram (EEG) signals are calculated, and the skin conductance features of the electrodermal (ED) signals are extracted. This allows for the identification of different emotional states of individuals based on the different features corresponding to different signals, improving the comprehensiveness of the final emotion analysis results. In this invention, a pre-trained emotion recognition model is used to identify the emotion analysis results corresponding to the different features, and an emotion regulation scheme corresponding to each emotion analysis result is obtained to generate a corresponding emotion regulation strategy, ensuring the regulation accuracy of the strategy.

[0056] As a preferred example, the multimodal signal acquisition component includes a blood oxygen pulse sensor, a forehead temperature sensor, an electroencephalogram (EEG) sensor, and a skin conductance sensor.

[0057] The blood oxygen pulse sensor is connected to the signal monitoring module and is used to collect and send the user's blood oxygen signal, pulse signal, heart rate signal and respiratory signal to the signal monitoring module within a predetermined time period.

[0058] The forehead temperature sensor is connected to the signal monitoring module and is used to collect and send the user's forehead temperature signal to the signal monitoring module within a predetermined time period; wherein, the vital signs signal includes the blood oxygen signal, pulse signal, heart rate signal, respiratory signal and forehead temperature signal;

[0059] The EEG sensor is connected to the signal monitoring module and is used to collect and send the user's EEG signals to the signal monitoring module within a predetermined time period.

[0060] The skin conductance sensor is connected to the signal monitoring module and is used to collect and send the user's skin conductance signal to the signal monitoring module within a predetermined time period.

[0061] This invention sets up different types of sensors to collect physiological data of different modalities, so as to comprehensively and accurately extract feature data reflecting emotional state by utilizing the diversity of the data, and then use the high-precision and comprehensive feature data to perform emotion recognition, thereby improving the accuracy of emotion recognition.

[0062] As a preferred example, the signal monitoring module further includes a time-domain analysis unit;

[0063] The time-domain analysis unit is used to extract several signals located in the RR interval sequence from the heart rate variability signal to form a heart rate variability time series segment; calculate the mean of the heart rate variability time series segment, and calculate the overall standard deviation of the heart rate variability time series segment based on the mean; calculate the root mean square of the difference between adjacent RR intervals in the heart rate variability time series segment and the standard deviation between all adjacent RR intervals; obtain the number of sampling points in the heart rate variability time series segment whose difference between adjacent RR intervals is greater than a preset time threshold, and calculate the ratio of the number to the total number of RR intervals in the heart rate variability time series segment; and use the overall standard deviation, the root mean square of the difference between adjacent RR intervals, the standard deviation between all adjacent RR intervals, and the ratio as the time-domain features corresponding to the heart rate variability signal.

[0064] This invention calculates temporal features by extracting heart rate variability signals from the RR interval sequence, avoiding interference from useless data and improving the efficiency of emotion recognition. Furthermore, by calculating the mean to reflect the average level of the RR interval sequence data, by calculating the overall standard deviation to reflect the overall change in heart rate variability, by calculating the root mean square of the difference between adjacent RR intervals to reflect the fast-changing component of heart rate variability, and by calculating the standard deviation and ratio of the difference between all adjacent RR intervals to reflect sudden changes in RR intervals, the comprehensiveness and accuracy of the temporal features are improved.

[0065] As a preferred example, the signal monitoring module further includes a frequency domain analysis unit;

[0066] The frequency domain analysis unit is used to input the heart rate variability time series segment into a preset autoregressive model, and obtain the power spectrum corresponding to the heart rate variability time series segment through the autoregressive model and the preset Z-transformation transfer function.

[0067] The parameters of the autoregressive model are calculated using the Burg recursion method, and the power spectral density corresponding to the heart rate variability signal is calculated based on the parameters and the power spectrum.

[0068] The frequency domain characteristics corresponding to the heart rate variability signal are calculated based on the power spectral density; wherein, the frequency domain characteristics include very low frequency power, low frequency power, high frequency power, total power, and low-to-high frequency ratio.

[0069] This invention uses an autoregressive model to calculate the power spectrum and power spectral density corresponding to the heart rate variability signal. It can not only process short data, but also overcome the shortcomings of spectrum aliasing and low resolution, thereby improving the accuracy of the frequency domain features.

[0070] As a preferred example, the signal monitoring module further includes a nonlinear analysis unit;

[0071] The nonlinear unit is used to perform Poincaré scatter plot analysis on the heart rate variability time series segment to obtain the Poincaré scatter plot corresponding to the heart rate variability time series segment;

[0072] The nonlinear parameters corresponding to the heart rate variability time series segment are calculated based on the Poincaré scatter plot; wherein the nonlinear parameters include the first scale change value and the second scale change value corresponding to the Poincaré scatter plot.

[0073] Based on the limitations of linear analysis methods, this invention performs nonlinear analysis on the heart rate variability time series segments. By combining the results of the nonlinear analysis and the linear analysis, the heart rate variability characteristics corresponding to the physiological feature signals are comprehensively and accurately reflected. In turn, the precise features are used to identify emotional states, thereby improving the accuracy of emotion recognition. Attached Figure Description

[0074] Figure 1 This is a flowchart illustrating a multimodal emotion analysis method disclosed in an embodiment of the present invention.

[0075] Figure 2 This is a schematic diagram of the structure of a multimodal emotion analysis system disclosed in an embodiment of the present invention;

[0076] Figure 3 This is a flowchart illustrating a multimodal emotion analysis method disclosed in another embodiment of the present invention.

[0077] Figure 4 This is a schematic diagram of the structure of a multimodal acquisition headband disclosed in another embodiment of the present invention;

[0078] Figure 5 This is a schematic diagram illustrating the relationship between a characteristic parameter and a pressure index, as disclosed in another embodiment of the present invention.

[0079] Figure 6 This is a schematic diagram illustrating the trend of SDNN and stress index disclosed in another embodiment of the present invention;

[0080] Figure 7 This is a schematic diagram illustrating the trend of HF and pressure index according to another embodiment of the present invention;

[0081] Figure 8 This is a schematic diagram of a model training method disclosed in another embodiment of the present invention. Detailed Implementation

[0082] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0083] Example 1

[0084] This embodiment discloses a multimodal sentiment analysis method. Specifically, the implementation process of the analysis method is described in [reference needed]. Figure 1 It mainly includes steps 101 to 106, and the steps are mainly as follows:

[0085] Step 101: Acquire vital signs, electroencephalogram (EEG) signals, and electrical skin signals collected within a predetermined time period.

[0086] Step 102: Perform heart rate variability analysis on the vital signs signal to extract the time-domain features, frequency-domain features, and nonlinear features corresponding to the heart rate variability signal obtained after analysis, and fuse the time-domain features, frequency-domain features, and nonlinear features to generate the heart rate variability features corresponding to the vital signs signal.

[0087] In this embodiment, the main step is to extract several signals located in the RR interval sequence from the heart rate variability signal to form a heart rate variability time series segment;

[0088] The mean of the heart rate variability time series segment is calculated, and the overall standard deviation of the heart rate variability time series segment is calculated based on the mean. The root mean square of the difference between adjacent RR intervals in the heart rate variability time series segment and the standard deviation between all adjacent RR intervals are calculated. The number of sampling points in the heart rate variability time series segment whose difference between adjacent RR intervals is greater than a preset time threshold is obtained, and the ratio of the number to the total number of RR intervals in the heart rate variability time series segment is calculated. The overall standard deviation, the root mean square of the difference between adjacent RR intervals, the standard deviation between all adjacent RR intervals, and the ratio are used as the time-domain features corresponding to the heart rate variability signal.

[0089] Simultaneously, the heart rate variability time series segment is input into a preset autoregressive model, and the power spectrum corresponding to the heart rate variability time series segment is obtained through the autoregressive model and the preset Z-transformation transfer function.

[0090] The parameters of the autoregressive model are calculated using the Burg recursion method, and the power spectral density corresponding to the heart rate variability signal is calculated based on the parameter values ​​and the power spectrum. The frequency domain features corresponding to the heart rate variability signal are calculated based on the power spectral density. The frequency domain features include very low frequency power, low frequency power, high frequency power, total power, and low-to-high frequency ratio.

[0091] Further, a Poincaré scatter plot analysis is performed on the heart rate variability time series segment to obtain the Poincaré scatter plot corresponding to the heart rate variability time series segment; the nonlinear parameters corresponding to the heart rate variability time series segment are calculated based on the Poincaré scatter plot; wherein, the nonlinear parameters include the first scale change value and the second scale change value corresponding to the Poincaré scatter plot.

[0092] In this embodiment, this step calculates time-domain features by extracting the heart rate variability signal from the RR interval sequence, avoiding interference from useless data and improving the efficiency of emotion recognition. Furthermore, by calculating the mean to reflect the average level of the RR interval sequence data, by calculating the overall standard deviation to reflect the overall change in heart rate variability, by calculating the root mean square of the difference between adjacent RR intervals to reflect the fast-changing component in heart rate variability, and by calculating the standard deviation and ratio of the difference between all adjacent RR intervals to reflect sudden changes in RR intervals, the comprehensiveness and accuracy of the time-series features are improved.

[0093] Simultaneously, an autoregressive model is used to calculate the power spectrum and power spectral density corresponding to the heart rate variability signal. This not only enables the processing of short data but also overcomes the shortcomings of spectral aliasing and low resolution, thereby improving the accuracy of the frequency domain features.

[0094] Furthermore, based on the limitations of linear analysis methods, nonlinear analysis is performed on the heart rate variability time series segments. The results of the nonlinear analysis and the linear analysis are combined to comprehensively and accurately reflect the heart rate variability characteristics corresponding to the physiological characteristic signals. In turn, the precise features are used to identify emotional states, thereby improving the accuracy of emotion recognition.

[0095] Step 103: Calculate the power spectrum and power spectrum entropy corresponding to the EEG signal, so as to extract the EEG features corresponding to the EEG signal based on the power spectrum and the power spectrum entropy.

[0096] Step 104: Analyze the skin electrical signal and extract the skin electrical characteristics corresponding to the skin electrical signal.

[0097] Step 105: Input the heart rate variability features, the electroencephalogram (EEG) features, and the electrodermal conductance features into the pre-trained emotion recognition model, and output several emotion analysis results through the emotion recognition model.

[0098] In this embodiment, before performing this step, the emotion recognition model is first trained. The training process of the emotion recognition model includes:

[0099] An initial emotion recognition model for emotion classification is constructed using Support Vector Machines (SVM). Electroencephalogram (EEG), electrodermal (EDS), and vital sign (VRS) signals are collected for training, and features are extracted sequentially to obtain corresponding EEG features, EDS features, and heart rate variability features, thus constructing a feature training set. Different features corresponding to different emotional states are input into the initial emotion recognition model. The penalty parameters and kernel function parameters of the initial emotion recognition model are continuously updated using the PSO optimization algorithm, and the loss value between the model's predicted value and the true value after parameter updates is calculated. When the loss value is minimized, the emotion recognition model is obtained based on the current model parameters and the established emotion recognition model.

[0100] Further, the heart rate variability features are input into the emotion recognition model, and an emotion index is calculated through the emotion recognition model; wherein, the emotion index includes an emotional stress index, a fatigue index, and an anxiety index; the electroencephalogram (EEG) features are input into the emotion recognition model, and a brain state index is obtained through the emotion recognition model; wherein, the brain state index includes an attention index, a relaxation index, and a brain fatigue level; the electrodermal conductance (EDC) features are input into the emotion recognition model, and a psychological state index is obtained through the emotion recognition model; wherein, the psychological state index includes a psychological stress index and a tension level; the emotion index, the brain state index, and the psychological state index are compared with preset thresholds respectively, and the user's emotion analysis result is determined based on the comparison results.

[0101] In this embodiment, the PSO optimization algorithm and SVM algorithm are used to construct the emotion recognition model in this step, thereby improving the accuracy of the emotion recognition model and the accuracy of the emotion analysis results output by the model. Furthermore, different feature parameters are used to identify different emotional states, accurately and comprehensively reflecting the individual's emotional state. At the same time, different thresholds are set to monitor the emotional state and determine whether the emotional state needs to be adjusted, so as to construct a precise adjustment strategy.

[0102] Step 106: Obtain an emotion regulation scheme that matches each of the emotion analysis results, so as to output the corresponding emotion regulation strategy based on the emotion analysis results and the emotion regulation scheme.

[0103] In this embodiment, the step is as follows: when the emotional stress index is less than a preset emotional stress index threshold, a first brainwave music matching the emotional stress index is selected from a preset brainwave music library, and a first emotion regulation scheme matching the emotional stress index is generated; when the attention index is less than a preset attention index threshold, a second brainwave music matching the attention index is selected from a preset brainwave music library, and a second emotion regulation scheme matching the attention index is generated.

[0104] In this embodiment, this step sets different emotion regulation schemes based on different emotion recognition results, so that the emotion regulation strategy can comprehensively and accurately regulate the emotional state.

[0105] On the other hand, this embodiment also discloses a multimodal emotion analysis system. For the specific structural composition of the analysis system, please refer to... Figure 2 It includes a multimodal signal acquisition component 201, a signal monitoring module 202, an emotion recognition module 203, and a feedback adjustment module 204.

[0106] The multimodal signal acquisition component 201 is signal-connected to the signal monitoring module 202, and is used to acquire vital sign signals, electroencephalogram (EEG) signals, and electrodermal signal (EDS) signals collected within a predetermined time period, and send the EEG signals, EDS signals, and vital sign signals to the signal monitoring module 202, so that the signal monitoring module 202 performs the following steps:

[0107] Heart rate variability analysis is performed on the vital signs signal to extract the time-domain features, frequency-domain features, and nonlinear features corresponding to the heart rate variability signal obtained after analysis, and the time-domain features, frequency-domain features, and nonlinear features are fused to generate the heart rate variability features corresponding to the vital signs signal.

[0108] Calculate the power spectrum and power spectral entropy corresponding to the EEG signal, and extract the EEG features corresponding to the EEG signal based on the power spectrum and power spectral entropy.

[0109] The skin electrical signal is analyzed to extract the corresponding skin electrical characteristics.

[0110] The signal monitoring module 202 is connected to the emotion recognition module 203 and is used to send the heart rate variability feature, the electroencephalogram (EEG) feature and the electrodermal conductance feature to the emotion recognition module 203, so that the emotion recognition module 203 inputs the heart rate variability feature, the EEG feature and the electrodermal conductance feature into a pre-trained emotion recognition model, and outputs several emotion analysis results through the emotion recognition model.

[0111] The emotion recognition module 203 is signal-connected to the feedback adjustment module 204 and is used to send the emotion analysis results to the feedback adjustment module 204 so that the feedback adjustment module 204 can obtain an emotion adjustment scheme matching each emotion analysis result and output a corresponding emotion adjustment strategy based on the emotion analysis results and the emotion adjustment scheme.

[0112] In this embodiment, the multimodal signal acquisition component 201 includes a blood oxygen pulse sensor, a forehead temperature sensor, an electroencephalogram (EEG) sensor, and a skin conductance sensor.

[0113] The blood oxygen pulse sensor is connected to the signal monitoring module and is used to collect and send the user's blood oxygen signal, pulse signal, heart rate signal and respiratory signal to the signal monitoring module within a predetermined time period.

[0114] The forehead temperature sensor is connected to the signal monitoring module and is used to collect and send the user's forehead temperature signal to the signal monitoring module within a predetermined time period; wherein, the vital signs signal includes the blood oxygen signal, pulse signal, heart rate signal, respiration signal and forehead temperature signal.

[0115] The EEG sensor is connected to the signal monitoring module and is used to collect and send the user's EEG signals to the signal monitoring module within a predetermined time period.

[0116] The skin conductance sensor is connected to the signal monitoring module and is used to collect and send the user's skin conductance signal to the signal monitoring module within a predetermined time period.

[0117] In this embodiment, the signal monitoring module 202 further includes a time-domain analysis unit.

[0118] The time-domain analysis unit is used to extract several signals located in the RR interval sequence from the heart rate variability signal to form a heart rate variability time series segment; calculate the mean of the heart rate variability time series segment, and calculate the overall standard deviation of the heart rate variability time series segment based on the mean; calculate the root mean square of the difference between adjacent RR intervals in the heart rate variability time series segment and the standard deviation between all adjacent RR intervals; obtain the number of sampling points in the heart rate variability time series segment whose difference between adjacent RR intervals is greater than a preset time threshold, and calculate the ratio of the number to the total number of RR intervals in the heart rate variability time series segment; and use the overall standard deviation, the root mean square of the difference between adjacent RR intervals, the standard deviation between all adjacent RR intervals, and the ratio as the time-domain features corresponding to the heart rate variability signal.

[0119] In this embodiment, the signal monitoring module 202 includes a frequency domain analysis unit.

[0120] The frequency domain analysis unit is used to input the heart rate variability time series segment into a preset autoregressive model, and obtain the power spectrum corresponding to the heart rate variability time series segment through the autoregressive model and the preset Z-transformation transfer function.

[0121] The parameters of the autoregressive model are calculated using the Burg recursion method, and the power spectral density corresponding to the heart rate variability signal is calculated based on the parameters and the power spectrum.

[0122] The frequency domain characteristics corresponding to the heart rate variability signal are calculated based on the power spectral density; wherein, the frequency domain characteristics include very low frequency power, low frequency power, high frequency power, total power, and low-to-high frequency ratio.

[0123] In this embodiment, the signal monitoring module 203 includes a nonlinear analysis unit.

[0124] The nonlinear unit is used to perform Poincaré scatter plot analysis on the heart rate variability time series segment to obtain the Poincaré scatter plot corresponding to the heart rate variability time series segment.

[0125] The nonlinear parameters corresponding to the heart rate variability time series segment are calculated based on the Poincaré scatter plot; wherein the nonlinear parameters include the first scale change value and the second scale change value corresponding to the Poincaré scatter plot.

[0126] Example 2

[0127] Specifically, this embodiment discloses a multimodal sentiment analysis method. For details on the implementation process of this analysis method, please refer to [link / reference needed]. Figure 3 It mainly includes steps 301 to 304, and the steps are mainly as follows:

[0128] Step 301: Acquire vital signs, electroencephalogram (EEG) signals, and electrodermal signals within a preset time period using a preset multimodal physiological signal acquisition headband.

[0129] Specifically, in the steps described, the vital signs signals include physiological signals such as heart rate, skin conductance, blood oxygenation, respiration, and forehead temperature.

[0130] Optionally, a multimodal physiological signal acquisition headband is provided, wherein the structural composition of the acquisition headband is described in reference to... Figure 4 It includes a 401 blood oxygen pulse sensor to collect blood oxygen and pulse at the center of the forehead; a 402 forehead temperature sensor to collect forehead temperature at the center of the forehead; a 403 electroencephalogram (EEG) sensor to collect EEG signals from the left frontal FP1 and right frontal FP2; a 404 electrical skin conductance (ESC) sensor to collect ESC signals from the ends of the fingers; and a 405 ear electrode.

[0131] Furthermore, the 401 pulse oximeter is a reflective pulse oximeter with an emission wavelength of 660nm±3nm (red light) and 940nm±10nm (infrared light).

[0132] The 402 forehead temperature sensor is a non-contact infrared temperature sensor with a temperature measurement range of 0-100℃ and an accuracy of 0.1. The thermopile infrared temperature sensor is mainly used for non-contact temperature measurement. Its basic working principle is to convert the thermal radiation of the target object into a voltage output.

[0133] The 403 EEG sensor uses the KS1092 dual-channel EEG chip, a dual-channel high-precision, low-power analog front-end chip for measuring EEG and other weak electrical signals.

[0134] The 404 skin conductance sensor uses a medical silicone finger sleeve with a stainless steel sheet embedded inside.

[0135] The 405 ear electrode is made of soft silicone, with a silver chloride solid powder sintered electrode embedded inside.

[0136] In this embodiment, as Figure 4 The circuit of the modal physiological signal acquisition headband shown is made of flexible FPC, and the part that comes into contact with the human skin is made of soft medical silicone.

[0137] During the specific emotion analysis process, the software for establishing a communication connection with the headband is opened on a tablet or computer, and a Bluetooth connection is established between the device and the headband. After the device successfully connects to the headband, the quality of the EEG signals collected by the headband is first checked. If the signal quality is good, multimodal physiological signals are collected for a period of time (set to 3 minutes), and HRV, EEG, and GSR data analysis are performed simultaneously. The data analysis includes dynamic real-time display of physiological signals such as EEG waveforms, respiratory waveforms, HRV curves, blood oxygenation, heart rate, and body temperature.

[0138] Step 302: Perform HRV time-domain analysis, HRV frequency-domain analysis, and HRV nonlinear analysis on the vital signs signal to extract the HRV feature parameters corresponding to the vital signs signal. At the same time, calculate the power spectrum and power spectrum entropy corresponding to the EEG signal to extract the EEG features corresponding to the EEG signal and the skin conductance features corresponding to the skin conductance signal based on the power spectrum and the power spectrum entropy.

[0139] Specifically, heart rate variability analysis is performed on the vital signs signals to obtain heart rate variability signals, and signals located in the RR interval sequence from the heart rate variability signals are extracted as RR interval sequence data. The mean, overall standard deviation, root mean square of the difference between adjacent RR intervals, standard deviation between all adjacent RR intervals, and the ratio of the number of adjacent RR intervals with a difference greater than a preset time threshold to the total number of RR intervals are used as HRV time-domain feature parameters.

[0140] Optionally, statistical analysis can be used to extract HRV time-domain feature values, and heart rate variability can be analyzed by calculating statistical indices of the RR interval sequence (calculated by the interval between the peak R and R point in the ECG signal waveform). The mean (MEAN) reflects the average level of the RR interval, and its calculation formula is:

[0141]

[0142] Where N is the number of RR intervals within the HRV detection time; The average of N RR intervals; Let be the i-th RR interval.

[0143] The population standard deviation (SDNN) reflects the overall variation in HRV, and is calculated using the following formula:

[0144]

[0145] The root mean square difference (RMSSD) between adjacent RR intervals reflects the fast-changing component of HRV, and is calculated using the following formula:

[0146]

[0147] in This is the (i+1)th RR interval.

[0148] The standard deviation (SDSD) of the difference between all adjacent RR intervals is calculated using the following formula:

[0149]

[0150] PNN50 represents the ratio of the number of adjacent RR intervals with a difference greater than a preset time interval (set to 50ms) to the total number of RR intervals, reflecting sudden changes in the RR interval. The calculation formula is as follows:

[0151]

[0152] Wherein, NN50 represents the number of adjacent RR intervals with a difference greater than 50ms; TotalNN represents the total number of RR intervals.

[0153] Specifically, the heart rate variability time series segment is input into a preset autoregressive model, and the power spectrum corresponding to the heart rate variability time series segment is obtained through the autoregressive model and a preset Z-transformation transfer function; the parameter values ​​of the autoregressive model are calculated using the Burg recursion method, and the power spectral density corresponding to the heart rate variability signal is calculated based on the parameter values ​​and the power spectrum; the frequency domain characteristics corresponding to the heart rate variability signal are calculated based on the power spectral density; wherein, the frequency domain characteristics include very low frequency power, low frequency power, high frequency power, total power, and low-to-high frequency ratio.

[0154] Optionally, frequency domain analysis can be used to analyze HRV, primarily by analyzing the energy distribution of the signal at different frequencies. Frequency domain analysis is related to time domain analysis and also reflects certain patterns inherent in HRV. By extracting HRV features through frequency domain analysis, the activity of the vagus and sympathetic nerves in the heart can be reflected. Specifically, the low frequency (LF) and high frequency (HF) of HRV, as well as their ratio (LF / HF), can represent the balance between the vagus and sympathetic nerves.

[0155] When analyzing HRV from a frequency domain perspective, the first step is to estimate the power spectrum of the HRV, then integrate the power spectrum over different frequency bands, and finally extract the HRV eigenvalues. This embodiment employs an autoregressive model for frequency domain analysis.

[0156] Furthermore, frequency domain analysis is performed using the AR model method, and the calculation formula for the AR model is as follows:

[0157]

[0158] The transfer function is generated through the Z-transform, and the formula for calculating the transfer function is as follows:

[0159]

[0160] In the formula, x(n) is a random signal, x(n−k) is a number of past values ​​of itself, u(n) is white noise, H(z) is the system function, and p is the system order.

[0161] Furthermore, the Burg recursive method is used to calculate the parameter values ​​of the AR model, obtaining the HRV power spectral density (PSD), thereby performing feature extraction on the HRV. The formula for calculating the power spectral density (PSD) is as follows:

[0162]

[0163] x n Let N be the nth sample of the signal, N be the length of the signal, W be the frequency, j be the imaginary unit, E be the expected value, and e = 2.7.

[0164] The quantitative indicators of the power spectral density curve include VLF (Very Low Frequency Power), LF (Low Frequency Power) (0.04-0.15Hz), HF (High Frequency Power) (0.15-0.4Hz), TP (Total Power), and LF / HF (Low-High Frequency Ratio). When LF / HF is between 0.6 and 1.5, it indicates that the sympathetic and vagus nerves are in a balanced state. When it is less than 0.6, it indicates that the vagus nerve is dominant. When it is greater than 1.5, it indicates that the sympathetic nerve is dominant.

[0165] Furthermore, when calculating the quantization index as a frequency domain characteristic parameter of HRV, the calculation formula for VLF is as follows: ;

[0166] The formula for calculating LF is as follows: ;

[0167] The formula for calculating HF is: ;

[0168] The formula for calculating TP is: ;

[0169] Specifically, a Poincaré scatter plot analysis is performed on the heart rate variability time series segment to obtain the Poincaré scatter plot corresponding to the heart rate variability time series segment; the nonlinear parameters corresponding to the heart rate variability time series segment are calculated based on the Poincaré scatter plot; wherein, the nonlinear parameters include the first scale change value and the second scale change value corresponding to the Poincaré scatter plot.

[0170] Optionally, the Poincaré scatter plot analysis method is used for HRV nonlinear analysis. The Poincaré scatter plot is a point distribution plot in which the positions of points are marked in a rectangular coordinate system. In this embodiment, when using the Poincaré scatter plot analysis, the positions of all adjacent RR intervals are marked. The marked distribution plot can intuitively reflect the overall characteristics of HRV and reflect the heart rate during instantaneous changes, thereby revealing the nonlinear law of HRV.

[0171] Furthermore, the Poincaré scatter plot method was used to extract nonlinear features of HRV. The scatter plot analysis yielded two main parameters, SD1 and SD2. SD1 reflects the changes in low-frequency and very low-frequency components of HRV and is related to sympathetic nerve activity; SD2 represents rapid changes in HRV and is related to vagal nerve activity. The formulas for calculating SDI and SD2 are as follows:

[0172]

[0173]

[0174] Wherein, SD1 and SD2 represent the first scale change value and the second scale change value corresponding to the Poincaré scatter plot.

[0175] Furthermore, after extracting the HRV characteristic parameters corresponding to the vital signs signals, in one embodiment of this example, the brain state can be reflected based on specific frequency bands in the electroencephalogram (EEG) signals, such as theta waves (4-7 Hz), which are usually associated with deep relaxation and sleep; alpha waves (8-13 Hz), which are usually associated with a quiet and relaxed state of alertness; beta waves (13-30 Hz), which are usually associated with active thinking, focus, or anxiety; and gamma waves (30-100 Hz), which are usually associated with higher brain functions such as information processing, learning, and memory formation.

[0176] The EEG signals after artifact removal were sampled using an EEG acquisition module at a sampling rate of 100 Hz. Then, two-channel EEG signals X={X} were collected every 10 seconds. 1 , X2} is added with a non-overlapping window, and differential operations and Fourier transforms are performed on the processed EEG data to perform power frequency denoising and ocular denoising on the offline EEG data.

[0177] Furthermore, the EMD algorithm performs a series of stabilization processes on the EEG signal, namely, filtering the original signal to decompose the fluctuations or trends of single components at different scales, resulting in a set of data with linear and steady-state characteristics. The steps of the EMD algorithm are as follows:

[0178] Step 1. Using the time scale formula for the zero point position. All extreme points of the induced EEG signal, including maxima and minima, were calculated;

[0179] Step 2. Fit the data using an interpolation function, forming the lower envelope of the time series based on local minima. The upper envelope of a time series is formed by local maxima.

[0180] Step 3. Remember the mean of the two envelopes, denoted as . For example, in the formula:

[0181]

[0182] Step 4. Extract new components and remove the mean of the envelope from the original time series. Thus, a new signal component is obtained. For example, in the formula:

[0183]

[0184] Step 5. Check whether the extracted new component meets the two requirements of the modal eigenfunction. If it does, it is the first-order eigenmode function of the original signal. If it does not, repeat Step 1 to Step 4 until the conditions are met and the first-order eigenmode function of the time series is obtained.

[0185] Step 6. Now assume After multiple iterations, the requirement was met, and this is denoted as the first-order eigenmode function. For example, in the formula:

[0186]

[0187] Where K is the number of iterations;

[0188] Step 7. Calculate the first-order eigenmode functions after removal. The residual function after For example:

[0189]

[0190] Step 8. Transfer the residual function As input, repeat the above steps until the eigenmode functions of all orders are obtained. At this point, the remaining residual component When the condition satisfies monotonic change and is less than a given threshold, the EMD decomposition process ends.

[0191] Finally, multimodal The empirical mode decomposition is as follows:

[0192]

[0193] Furthermore, the hybrid feature extraction module first preprocesses the EEG signal and then performs empirical mode decomposition (EMD) to achieve adaptive decomposition. The decomposed modal eigenfunctions are then reconstructed into a new time series. Linear and nonlinear features are then extracted from this new EEG time series. Linear features include fluctuation index and energy, while nonlinear features are obtained by fusing approximate entropy, sample entropy, and fuzzy entropy. Finally, all extracted features are used as the total feature vector of the induced EEG signal time series. The hybrid feature vector provides a more comprehensive description of the EEG signal information, facilitating better classification. The specific steps of hybrid feature extraction are as follows:

[0194] Step 1. Preprocess the EEG signal time series to remove noise and improve the signal-to-noise ratio:

[0195] Step 2. Select EEG signal samples X, with m samples and N sample lengths. Let one of the samples be represented as

[0196] ;

[0197] Step 3. Perform EMD decomposition on the EEG signal time series X. Each sample is decomposed into a series of intrinsic mode function (IMF) sequences. Take the first IMF of each sample for sequence reconstruction to obtain a new time series Y. The number of samples in Y is m.

[0198] Step 4. For the reconstructed time series above, calculate the linearity fluctuation index F1 and the energy index Energy of the EEG signal, denoted as... ;

[0199] Step 5. Calculate its nonlinear characteristic approximate entropy, sample entropy, and fuzzy entropy respectively;

[0200]

[0201] Where m=2, r=0.2, and N is the length of the sample data;

[0202] Step 6. Integrate linear and nonlinear features ;

[0203] In the formula, Features(Y) represents the features of sample Y, which are the mixed linear and nonlinear features of the EEG signal.

[0204] In one embodiment of this example, the skin conductance signal for artifact removal is sampled using a skin conductance acquisition module at a sampling rate of 100 Hz. Then, a non-overlapping window is applied to each 10-second sample of the single-channel skin conductance signal. Differential operations and Fourier transforms are performed on the processed skin conductance data, and skin conductance features are extracted using differential entropy. The following is the formula for calculating differential entropy.

[0205]

[0206] In the formula, K is the number of samples of the electrodermal signal, and X... i Let X be the sample value of the electrodermal signal at time point i. I+1 This represents the sample value of the electrodermal signal at time point i+1.

[0207] Step 303: Input the heart rate variability features, the electroencephalogram (EEG) features, and the electrodermal conductance features into the pre-trained emotion recognition model, and output several emotion analysis results through the emotion recognition model.

[0208] Specifically, in this embodiment, the heart rate variability features are input into the emotion recognition model, and an emotion index is calculated through the emotion recognition model; wherein, the emotion index includes an emotional stress index, a fatigue index, and an anxiety index; the electroencephalogram (EEG) features are input into the emotion recognition model, and a brain state index is obtained through the emotion recognition model; wherein, the brain state index includes an attention index, a relaxation index, and a brain fatigue level; the electrodermal conductance (EDC) features are input into the emotion recognition model, and a psychological state index is obtained through the emotion recognition model; wherein, the psychological state index includes a psychological stress index and a tension level; the emotion index, the brain state index, and the psychological state index are compared with preset thresholds respectively, so as to determine the user's emotion analysis result based on the comparison results.

[0209] Optionally, the mean heart rate can reflect the speed of a person's heartbeat; therefore, the extracted HRV feature parameters are closely related to different emotional states. Further, refer to... Figure 5 Statistical results of characteristic data under different pressure conditions. For example... Figure 5As shown, comparing the HRV feature values ​​extracted under the two stress states reveals that the mean RR interval under low pressure is 0.8286 seconds, while the mean RR interval under high pressure is 0.9612 seconds. The mean RR interval under high pressure is 0.1326 seconds longer than that under low pressure. This indicates that the mean RR interval gradually decreases with increasing emotional intensity, suggesting that cardiovascular activity and corresponding metabolic activity are enhanced under high pressure. Comparing the mean heart rate (HR mean) under the two stress states, the heart rate is 86.2425 beats per minute under high pressure and 69.7633 beats per minute under low pressure, indicating that the heart rate fluctuates more under high pressure than under low pressure. Correspondingly, comparing the PNN50 values ​​under the two conditions, the value is 0.2563 under low pressure and 0.3462 under high pressure. Since PNN50 is related to the vagus nerve and reflects heart rate, the comparison of PNN50 values ​​under high and low pressure indicates that the stability of the heartbeat is better under low pressure than under high pressure, meaning the heart beats faster under high pressure. Furthermore, comparing the LF and LF / HF values ​​under high and low pressure, it can be seen that the LF and LF / HF values ​​are smaller under high pressure than under low pressure, indicating that as pressure increases, the energy of the low-frequency components in HRV decreases. While calculated characteristic values ​​can compare differences between different pressure states, to more accurately illustrate this difference, we further use the significance of HRV characteristic parameters to determine emotional state in addition to numerical comparisons.

[0210] Furthermore, by Figure 5 It can be seen that SDNN exhibits significant differences under the two pressure levels. SDNN is significantly lower under high pressure than under low pressure, while HF increases with increasing pressure, with higher HF under high pressure than under low pressure. The trends of SDNN and HF with pressure are shown below. Figure 6 and Figure 7 As shown, the two feature parameters, SDNN and HF, changed significantly with increasing pressure, indicating that the SDNN and HF feature values ​​can effectively detect pressure and can be better applied to pressure recognition algorithms.

[0211] Specifically, in this embodiment, stress identification is performed by combining the PSO optimization algorithm and the SVM algorithm. When classifying using SVM, to improve classification accuracy, the SVM parameters are optimized using the PSO optimization algorithm to obtain optimal feature values. These optimal feature values ​​are then fed into the classifier for training and prediction. The optimization process specifically optimizes the SVM's penalty parameter c and kernel function parameter g. The process of optimizing SVM parameters based on PSO is as follows: Figure 8 As shown.

[0212] HRV feature parameters extracted using time-domain, frequency-domain, and nonlinear methods, along with EEG and GSR features, were used as data samples. A total of 120 training and testing samples were provided, including 50 high-pressure samples and 50 low-pressure samples. Forty samples from the low-pressure samples were used for training, and 10 were used for testing; similarly, 40 samples from the high-pressure samples were used for training, and 10 were used for testing to assess classification ability. The PS0 algorithm-optimized SVM stress recognition algorithm flow is as follows: Figure 8 As shown.

[0213] The detailed steps for stress identification based on PSO-optimized SVM are as follows:

[0214] Step 1: Transform the pressure data into an N×M feature matrix, where N is the number of pressure data points and M is the feature dimension, where N is 40 and M is 10.

[0215] Step 2: Divide the data into 40 training datasets and 10 test datasets:

[0216] Step 3: Normalize the training and testing datasets:

[0217] Step 4: Initialize the .PS0 parameters and initialize the selected SVM model:

[0218] Step 5: Combine the 50 sets of low-pressure samples and 50 sets of high-pressure samples in the training set to train the SVM model. Combine the remaining 10 sets of low-pressure samples and 10 sets of high-pressure samples for model testing.

[0219] Step 6: Determine if each dataset is used for validation testing. If it is, proceed to Step 7; otherwise, go to Step 5.

[0220] Stp7: Obtain the classification accuracy of the validation dataset through the above steps, and use the maximum classification accuracy as the fitness value of PSO:

[0221] Step 8: Determine if the fitness value meets the termination condition. If it does, calculate the optimal accuracy and the optimized parameter values; otherwise, proceed to Step 9.

[0222] Step 9: Update the fitness value of PSO, update the particle velocity and position, then execute Step 5 to continue optimizing PSO. Repeat the loop until the condition of Step 8 is met.

[0223] Step 10: Train the final classifier model using the obtained optimal parameters c, g and the training set data:

[0224] Step 11: Use the final SVM classifier model to identify emotions and calculate the recognition accuracy.

[0225] Model training and validation: The model is trained using a labeled emotion dataset, and then validated and optimized to ensure the accuracy and stability of emotion recognition.

[0226] Furthermore, during the training process of the model, the PSO-SVM algorithm, tested 20 times, achieved an average emotion recognition rate of 90.17%, with a recognition rate of over 85% in each test.

[0227] Furthermore, by combining heart rate variability features, electroencephalogram (EEG) features, and skin conductance features, multimodal features are synthesized using the DS evidence theory synthesis algorithm to obtain a more accurate emotion fusion feature vector. The DS evidence theory synthesis algorithm is as follows:

[0228] 1) Define a set, denoted using the traditional identification framework: R = {A1, A2, ..., An}, Z = 2R, where R is the power set. For any subset A of Z to which a set belongs, let its corresponding number be m, m(A).

[0229] For any subset A to which a variable belongs, let its corresponding number m, m(A)∈[0,1], and m satisfies:

[0230] Then m is called the Basic Probability Assignment Function (BPA) defined on 2R, and m(A) is called the basic confidence number of A, reflecting the degree to which proposition A is supported. Propositions with m(A)>0 are defined as focal elements or key elements of evidence.

[0231] Calculate the average value of each piece of evidence:

[0232]

[0233] Then, calculate the distance of each piece of evidence to the average evidence:

[0234]

[0235] Finally, the credibility of each piece of evidence is calculated:

[0236]

[0237] Wherein, C(m) i ) as evidence m i The weights satisfy Here, the degree to which other evidence supports this evidence is called the weight of that evidence. The higher the degree of support of the evidence, the higher the corresponding weight, and the greater the impact on the combined result; conversely, the lower the corresponding weight, the smaller the impact on the combined result.

[0238]

[0239] Construct an M(n×m) dimensional confidence matrix based on the basic probability assignment:

[0240]

[0241] in:

[0242]

[0243] Transpose a row i in matrix M, and then multiply it by another row j of the matrix, i.e.:

[0244]

[0245] Expanding yields a new m×m dimensional matrix:

[0246]

[0247] As can be seen from the formula above, according to the numerator of the DS combination formula, the elements on the main diagonal are the product of the basic probability distributions of sensor i and sensor j with respect to the target, while the sum of all other elements in the matrix constitutes the uncertainty factor of the evidence, i.e.:

[0248]

[0249] Let Zj = ajj, then we can obtain the fusion index, that is:

[0250]

[0251] The multimodal fusion emotional stress index is calculated by synthesizing three features based on the DS evidence theory synthesis algorithm rules.

[0252] Furthermore, the multimodal fusion features are classified using a Support Vector Machine (SVM) to obtain the dynamic emotion category of the target subject.

[0253] Specifically, in this embodiment, the SVM model preprocesses the online multimodal data to obtain online multimodal information, and then performs fusion and classification processing on the online multimodal information to obtain the online sentiment classification result of the online multimodal data.

[0254] Specifically, Support Vector Machine (SVM) is a classification method suitable for small samples and high-dimensional features, and possesses good generalization ability. Its basic principle is to transform the input space into a high-dimensional space through a nonlinear transformation, and then find the optimal linear distribution in this new space. Let the sample set be... , , Assign corresponding category numbers, SVM is an optimization problem for solving the following quadratic equation.

[0255] min

[0256] subject to , (i=1,…, l)

[0257] Where C is the regularization parameter, is the slack variable, w is the weight vector, and b is the bias term.

[0258] SVM discriminant function as follows

[0259]

[0260] in It is the number of support vectors. It is a Lagrange multiplier. is the kernel function, and b is the classification threshold.

[0261] Furthermore, the SVM model needs to transform the multimodal fusion emotional stress index into a feature vector form and then standardize the data. The standardization process uses the Z-score standardization method, with the following formula:

[0262]

[0263] in, These are the original eigenvalues. These are the standardized eigenvalues. It is the mean of the features. It is the standard deviation of the feature.

[0264] Furthermore, the standardized feature vectors are input into a pre-trained SVM model for sentiment prediction. The prediction process of the SVM model is represented as follows: ;in, Z is the predicted emotional state; Z is the input feature matrix; and β is the coefficient vector of the SVM model.

[0265] Specifically, the emotional state is interpreted based on the prediction results of the SVM model. If the prediction result is a continuous value, it is used to represent the intensity of the emotion; if it is a discrete value, it is used to represent the category of the emotion.

[0266] This invention fuses multimodal features—including heart rate variability, electroencephalogram (EEG), and electrodermal conductance (EDC) features—using a DS evidence theory-based synthesis algorithm to obtain a more accurate fused feature, the emotional stress index. This index is then input into an SVM model to obtain the corresponding recognition result, improving the accuracy and real-time performance of emotion recognition and overcoming the shortcomings of traditional emotion recognition methods.

[0267] Step 304: Obtain brainwave music that matches each of the emotion analysis results, and output corresponding emotion regulation strategies based on the emotion analysis results and the brainwave music.

[0268] Specifically, in this embodiment, when the emotional stress index is less than the emotional stress index threshold, a first brainwave music matching the emotional stress index is selected from a preset brainwave music library to generate a first emotion regulation scheme matching the emotional stress index; when the attention index is less than the preset attention index threshold, a second brainwave music matching the attention index is selected from a preset brainwave music library to generate a second emotion regulation scheme matching the attention index.

[0269] Optionally, index ranges such as a normal range for focus, an emotional range, and a fatigue index range can be set to determine the user's current emotional state, and to determine whether the user needs emotional regulation based on these ranges. Preferably, the normal range for focus is set to [30, 100], where a larger value represents greater concentration and stronger focus. If the average focus is below 30, it indicates a state of inattention and anxiety, in which case it is determined that the user needs to develop an emotional regulation strategy.

[0270] Furthermore, users can choose a custom adjustment plan or follow a pre-selected plan. During the adjustment process, the user's emotional state is fed back in real-time via a curve. Music-based emotional regulation is then used to monitor the intervention effect in real time, allowing for adjustments and optimization of the mind-body regulation methods to ensure optimal intervention results. Specifically, the adjustment plan is as follows:

[0271] If the focus index is not within the preset range, then Alpha wave brainwave music or Beta wave brainwave music will be used.

[0272] If the brain fatigue index is not within the preset range, white noise, alpha wave brainwave music, and theta wave brainwave music are used; if fatigue is present, theta wave brainwave music is used for training; for psychological stress relief, alpha wave brainwave music and theta wave brainwave music are used for training; for anxiety relief, alpha wave brainwave music and theta wave brainwave music are used for training; and for depression regulation, theta wave brainwave music is used for training.

[0273] In this embodiment, the above-described adjustment scheme can be used to enhance focus, improve sleep, relieve fatigue, reduce stress, alleviate anxiety, and treat depression. Furthermore, during emotion regulation, brainwave rhythms are extracted in real time from the collected EEG data, and corresponding brainwave music is selected for training according to the system's pre-set brainwave rhythm scheme. During training, users can adjust their mind and body based on feedback graphs.

[0274] This embodiment discloses an emotion regulation method that acquires multimodal physiological signals collected within a predetermined time period. These multimodal physiological signals include vital signs, electroencephalogram (EEG), and electrodermal (EDS) signals. The method aims to comprehensively and accurately reflect an individual's emotional state based on these multimodal physiological signals, thereby improving the accuracy of emotion recognition. After acquiring the multimodal physiological signals, different features are extracted from the vital signs, EEG, and EDS signals. Specifically, heart rate variability analysis is performed on the vital signs, extracting time-domain, frequency-domain, and nonlinear features corresponding to the heart rate variability signals. These time-domain, frequency-domain, and nonlinear features are then combined to comprehensively and accurately reflect the heart rate variability characteristics corresponding to the vital signs, further improving the accuracy of feature extraction.

[0275] While extracting the heart rate variability features, the power spectrum and power spectrum entropy of the electroencephalogram (EEG) signals are calculated, and the skin conductance features of the electrodermal (ED) signals are extracted. This allows for the identification of different emotional states of individuals based on the different features corresponding to different signals, improving the comprehensiveness of the final emotion analysis results. In this invention, a pre-trained emotion recognition model is used to identify the emotion analysis results corresponding to the different features, and an emotion regulation scheme corresponding to each emotion analysis result is obtained to generate a corresponding emotion regulation strategy, ensuring the regulation accuracy of the strategy.

[0276] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A multimodal sentiment analysis method, characterized in that, include: Acquire vital signs, electroencephalogram (EEG) signals, and electrical skin signals collected within a predetermined time period; Heart rate variability analysis is performed on the vital signs signal to extract the time-domain features, frequency-domain features, and nonlinear features corresponding to the heart rate variability signal obtained after analysis, and the time-domain features, frequency-domain features, and nonlinear features are fused to generate the heart rate variability features corresponding to the vital signs signal. The power spectrum and power spectrum entropy corresponding to the EEG signal are calculated to extract the EEG features corresponding to the EEG signal based on the power spectrum and power spectrum entropy. Specifically, the EEG signal is preprocessed and then subjected to empirical mode decomposition (EMD) for adaptive decomposition. The decomposed modal eigenfunctions are then reconstructed into a new EEG time series. Linear and nonlinear features corresponding to the new EEG time series are extracted. The linear and nonlinear features are then fused to obtain the EEG features corresponding to the EEG signal. The linear features are the fluctuation index and energy corresponding to the new EEG time series; the nonlinear features are the approximate entropy, sample entropy, and fuzzy entropy corresponding to the new EEG time series. The skin conductance signal is analyzed to extract the corresponding skin conductance features; wherein, the skin conductance signal is differentiated and Fourier transformed, and the skin conductance features corresponding to the skin conductance signal are extracted through differential entropy. The heart rate variability (HRV) feature, electroencephalogram (EEG) feature, and electrodermal conductance (EDC) feature are respectively input into a pre-trained emotion recognition model, which outputs several emotion analysis results. Specifically, the HRV feature is input into the emotion recognition model to calculate an emotion index; the EEG feature is input into the emotion recognition model to calculate a brain state index; and the EDC feature is input into the emotion recognition model to calculate a psychological state index. The emotion index, brain state index, and psychological state index are each compared with preset thresholds to determine the user's emotion analysis result based on the comparison results. The emotion index includes an emotional stress index and a fatigue index; the brain state index includes an attention index, a relaxation index, and a brain fatigue index; and the psychological state index includes a psychological stress index and a tension level. Obtain an emotion regulation scheme that matches each of the emotion analysis results, and output a corresponding emotion regulation strategy based on the emotion analysis results and the emotion regulation scheme.

2. The multimodal sentiment analysis method according to claim 1, characterized in that, The time-domain features, frequency-domain features, and nonlinear features corresponding to the heart rate variability signal obtained after extraction and analysis include: Several signals located in the RR interval sequence are extracted from the heart rate variability signal to form a heart rate variability time series segment; Calculate the mean of the heart rate variability time series segment to calculate the population standard deviation of the heart rate variability time series segment based on the mean; Calculate the root mean square of the difference between adjacent RR intervals in the heart rate variability time series segment and the standard deviation between all adjacent RR intervals; The number of sampling points in the heart rate variability time series segment where the difference between adjacent RR intervals is greater than a preset time threshold is obtained, and the ratio of the number to the total number of RR intervals in the heart rate variability time series segment is calculated. The overall standard deviation, the root mean square of the difference between adjacent RR intervals, the standard deviation between all adjacent RR intervals, and the ratio are used as the time-domain features corresponding to the heart rate variability signal.

3. The multimodal sentiment analysis method according to claim 2, characterized in that, The time-domain features, frequency-domain features, and nonlinear features corresponding to the heart rate variability signal obtained after extraction and analysis include: The heart rate variability time series segment is input into a preset autoregressive model, and the power spectrum corresponding to the heart rate variability time series segment is obtained through the autoregressive model and the preset Z-transformation transfer function. The parameters of the autoregressive model are calculated using the Burg recursion method, and the power spectral density corresponding to the heart rate variability signal is calculated based on the parameters and the power spectrum. The frequency domain characteristics corresponding to the heart rate variability signal are calculated based on the power spectral density; wherein, the frequency domain characteristics include very low frequency power, low frequency power, high frequency power, total power, and low-to-high frequency ratio.

4. The multimodal sentiment analysis method according to claim 2, characterized in that, The time-domain features, frequency-domain features, and nonlinear features corresponding to the heart rate variability signal obtained after extraction and analysis include: Poincaré scatter plot analysis was performed on the heart rate variability time series segment to obtain the Poincaré scatter plot corresponding to the heart rate variability time series segment; The nonlinear parameters corresponding to the heart rate variability time series segment are calculated based on the Poincaré scatter plot; wherein the nonlinear parameters include the first scale change value and the second scale change value corresponding to the Poincaré scatter plot.

5. The multimodal sentiment analysis method according to claim 1, characterized in that, The training process of the emotion recognition model includes: An initial emotion recognition model for emotion classification is constructed using a support vector machine (SVM). Electroencephalogram (EEG), electrodermal (ED) signals, and vital signs signals were collected for training, and features were extracted sequentially to obtain corresponding EEG features, EED features, and heart rate variability features to construct a feature training set. Different features corresponding to different emotional states are input into the initial emotion recognition model. The penalty parameters and kernel function parameters of the initial emotion recognition model are continuously updated through the PSO optimization algorithm, and the loss value between the model's predicted value and the true value after updating the parameters is calculated. When the loss value is minimized, the emotion recognition model is obtained based on the current model parameters and the emotion recognition model.

6. The multimodal sentiment analysis method according to claim 1, characterized in that, The step of obtaining an emotion regulation scheme matching each emotion analysis result, and outputting a corresponding emotion regulation strategy based on the emotion analysis results and the emotion regulation scheme, includes: When the emotional stress index is less than the preset emotional stress index threshold, a first brainwave music matching the emotional stress index is selected from the preset brainwave music library, and a first emotion regulation scheme matching the emotional stress index is generated. When the attention index is less than a preset attention index threshold, a second brainwave music matching the attention index is selected from a preset brainwave music library, and a second emotion regulation scheme matching the attention index is generated.

7. A multimodal emotion analysis system, characterized in that, It includes a multimodal signal acquisition component, a signal monitoring module, an emotion recognition module, and a feedback adjustment module; The multimodal signal acquisition component is signal-connected to the signal monitoring module and is used to acquire vital sign signals, electroencephalogram (EEG) signals, and electrodermal signal (EDS) signals collected within a predetermined time period, and to send the EEG signals, EDS signals, and vital sign signals to the signal monitoring module, so that the signal monitoring module performs the following steps: Heart rate variability analysis is performed on the vital signs signal to extract the time-domain features, frequency-domain features, and nonlinear features corresponding to the heart rate variability signal obtained after analysis, and the time-domain features, frequency-domain features, and nonlinear features are fused to generate the heart rate variability features corresponding to the vital signs signal. The power spectrum and power spectrum entropy corresponding to the EEG signal are calculated to extract the EEG features corresponding to the EEG signal based on the power spectrum and power spectrum entropy. Specifically, the EEG signal is preprocessed and then subjected to empirical mode decomposition (EMD) for adaptive decomposition. The decomposed modal eigenfunctions are then reconstructed into a new EEG time series. Linear and nonlinear features corresponding to the new EEG time series are extracted. The linear and nonlinear features are then fused to obtain the EEG features corresponding to the EEG signal. The linear features are the fluctuation index and energy corresponding to the new EEG time series; the nonlinear features are the approximate entropy, sample entropy, and fuzzy entropy corresponding to the new EEG time series. The skin conductance signal is analyzed to extract the corresponding skin conductance features; wherein, the skin conductance signal is differentiated and Fourier transformed, and the skin conductance features corresponding to the skin conductance signal are extracted through differential entropy. The signal monitoring module is signal-connected to the emotion recognition module and is used to send the heart rate variability feature, the electroencephalogram (EEG) feature, and the electrodermal conductance (EDC) feature to the emotion recognition module. The emotion recognition module then inputs these features into a pre-trained emotion recognition model and outputs several emotion analysis results. Specifically, the heart rate variability feature is input into the emotion recognition model to calculate an emotion index; the EEG feature is input into the emotion recognition model to calculate a brain state index; and the EDC feature is input into the emotion recognition model to calculate a psychological state index. The emotion index, brain state index, and psychological state index are then compared with preset thresholds to determine the user's emotion analysis result based on the comparison results. The emotion recognition module is signal-connected to the feedback adjustment module and is used to send the emotion analysis results to the feedback adjustment module so that the feedback adjustment module can obtain an emotion adjustment scheme matching each emotion analysis result and output a corresponding emotion adjustment strategy based on the emotion analysis results and the emotion adjustment scheme.

8. A multimodal emotion analysis system according to claim 7, characterized in that, The multimodal signal acquisition component includes a blood oxygen pulse sensor, a forehead temperature sensor, an electroencephalogram (EEG) sensor, and a skin conductance sensor. The blood oxygen pulse sensor is connected to the signal monitoring module and is used to collect and send the user's blood oxygen signal, pulse signal, heart rate signal and respiratory signal to the signal monitoring module within a predetermined time period. The forehead temperature sensor is connected to the signal monitoring module and is used to collect and send the user's forehead temperature signal to the signal monitoring module within a predetermined time period; wherein, the vital signs signal includes the blood oxygen signal, pulse signal, heart rate signal, respiratory signal and forehead temperature signal; The EEG sensor is connected to the signal monitoring module and is used to collect and send the user's EEG signals to the signal monitoring module within a predetermined time period. The skin conductance sensor is connected to the signal monitoring module and is used to collect and send the user's skin conductance signal to the signal monitoring module within a predetermined time period.

9. A multimodal emotion analysis system according to claim 7, characterized in that, The signal monitoring module includes a time-domain analysis unit; The time-domain analysis unit is used to extract several signals located in the RR interval sequence from the heart rate variability signal to form a heart rate variability time series segment; calculate the mean of the heart rate variability time series segment, and calculate the overall standard deviation of the heart rate variability time series segment based on the mean; calculate the root mean square of the difference between adjacent RR intervals in the heart rate variability time series segment and the standard deviation between all adjacent RR intervals; obtain the number of sampling points corresponding to the difference between adjacent RR intervals in the heart rate variability time series segment being greater than a preset time threshold, and calculate the ratio of the number to the total number of RR intervals in the heart rate variability time series segment; The overall standard deviation, the root mean square of the difference between adjacent RR intervals, the standard deviation between all adjacent RR intervals, and the ratio are used as the time-domain features corresponding to the heart rate variability signal.

10. A multimodal emotion analysis system according to claim 9, characterized in that, The signal monitoring module includes a frequency domain analysis unit; The frequency domain analysis unit is used to input the heart rate variability time series segment into a preset autoregressive model, and obtain the power spectrum corresponding to the heart rate variability time series segment through the autoregressive model and the preset Z-transformation transfer function. The parameters of the autoregressive model are calculated using the Burg recursion method, and the power spectral density corresponding to the heart rate variability signal is calculated based on the parameters and the power spectrum. The frequency domain characteristics corresponding to the heart rate variability signal are calculated based on the power spectral density; wherein, the frequency domain characteristics include very low frequency power, low frequency power, high frequency power, total power, and low-to-high frequency ratio.

Images