A method and system for detecting human emotions based on multimodal physiological signals

Abstract

The present invention provides a method and system for detecting human emotions based on multimodal physiological signals, which relates to the field of artificial intelligence. The method includes: using an electrocardiogram sensor and a skin resistance sensor to collect the electrocardiogram signal and the skin resistance signal on the human epidermis. Taking them as inputs, three physiological characteristic indexes, namely SDNN, SCR peak value and SCR frequency, are calculated in real time. And based on the dynamic change law, two indexes for evaluating the sudden change of individual emotions and the corresponding judgment thresholds are comprehensively analyzed. If the real-time emotion representation index is greater than its threshold value, it is considered that the emotion of the person to be detected fluctuates violently in a short time, and the system alarms for it. Otherwise, there is no alarm. The technical problem of difficult to accurately detect the micro-emotion changes hidden by the person to be detected is solved. The technical effect of dynamically perceiving the physiological changes hidden by the person to be detected through will or behavior, and then realizing the accurate detection of their micro-emotion changes is achieved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and particularly to a method and system for detecting human emotions based on multi-modal physiological signals. Background Art

[0002] For some special groups in society, effective emotion detection can be carried out to perceive the emotional state of the person to be detected during the meeting activity, such as the analysis of the state of the person to be detected, psychological diagnosis and treatment, etc. By collecting physiological signals during the process of physiological changes caused by the psychological changes of the person to be detected under the influence of negative events, the monitoring of emotional out-of-control can be effectively carried out through the changes in physiological signals.

[0003] However, in the prior art, when collecting the physiological changes of the person to be detected, it is difficult to dynamically perceive the physiological changes hidden by the person to be detected through will or behavior, resulting in the technical problem that it is difficult to accurately detect the micro-emotional changes hidden by the person to be detected. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for detecting human emotions based on multi-modal physiological signals, which solves the technical problem that when collecting the physiological changes of the person to be detected, it is difficult to dynamically perceive the physiological changes hidden by the person to be detected through will or behavior, resulting in the difficulty in accurately detecting the micro-emotional changes hidden by the person to be detected. It achieves the technical effect that when collecting the physiological changes of the person to be detected, by evaluating the SDNN index and SCR index of the sudden change of individual emotions, as well as the corresponding judgment thresholds, the dynamic perception of the physiological changes hidden by the person to be detected through will or behavior is realized, and further the accurate detection of the micro-emotional changes hidden by the person to be detected is realized.

[0005] In view of the above problems, this application provides a method and system for detecting human emotions based on multi-modal physiological signals.

[0006] In the first aspect of the present application, a method for detecting a person's emotion based on multimodal physiological signals is provided. The method is applied to an emotion detection system, and the system is communicatively connected to an electrocardiogram sensor and a skin resistance sensor. The method includes: using the electrocardiogram sensor to collect electrocardiogram signals from a target user to obtain dynamic electrocardiogram signal data; using the skin resistance sensor to collect skin resistance signals from the target user to obtain dynamic skin resistance signal data; performing preprocessing on the dynamic electrocardiogram signal data and the dynamic skin resistance signal data for a predetermined window length period to respectively obtain an SDNN index and an SCR fusion index; determining whether the SDNN index meets a decreasing trend; if the SDNN index meets the decreasing trend, determining whether the SCR fusion index simultaneously meets an increasing trend; if the SDNN index meets the decreasing trend and the SCR fusion index simultaneously meets the increasing trend, triggering a first alarm instruction to alarm the emotional state of the target user.

[0007] In the second aspect of the present application, a system for detecting a person's emotion based on multimodal physiological signals is provided. The system includes: an electrocardiogram signal acquisition module for using an electrocardiogram sensor to collect electrocardiogram signals from a target user to obtain dynamic electrocardiogram signal data; a skin resistance signal acquisition module for using a skin resistance sensor to collect skin resistance signals from the target user to obtain dynamic skin resistance signal data; a data preprocessing module for performing preprocessing on the dynamic electrocardiogram signal data and the dynamic skin resistance signal data for a predetermined window length period to respectively obtain an SDNN index and an SCR fusion index; an SDNN index determination module for determining whether the SDNN index meets a decreasing trend; an SCR fusion index determination module for determining whether the SCR fusion index simultaneously meets an increasing trend if the SDNN index meets the decreasing trend; a status alarm module for triggering a first alarm instruction to alarm the emotional state of the target user if the SDNN index meets the decreasing trend and the SCR fusion index simultaneously meets the increasing trend.

[0008] One or more technical solutions provided in the present application have at least the following technical effects or advantages:

[0009] A method for detecting human emotions based on multimodal physiological signals provided by this application. By placing an electrocardiogram (ECG) sensor and a galvanic skin response (GSR) sensor on the skin surface of the person to be detected, the electrocardiogram signal and the galvanic skin response signal on the human epidermis are collected. Using the electrocardiogram signal and the galvanic skin response signal as inputs, three physiological characteristic indexes, namely SDNN, SCR peak value, and SCR frequency, are calculated in real time. And based on the dynamic change rules of the three indexes, two indexes that can evaluate the sudden change of individual emotions and the corresponding judgment thresholds are comprehensively analyzed. If the real-time emotion representation index is greater than its threshold value, it is considered that the emotion of the person to be detected has fluctuated violently in a short time, and the system alarms for it, otherwise there is no alarm. When collecting the physiological changes of the person to be detected, by evaluating the SDNN index and SCR index of the sudden change of individual emotions and the corresponding judgment thresholds, the physiological changes hidden by the person to be detected through will or behavior are dynamically perceived, and then the precise detection of the micro-emotion changes hidden by the person to be detected is realized.

[0010] The above description is only an overview of the technical solution of this application. In order to be able to understand the technical means of this application more clearly, it can be implemented according to the content of the description. And in order to make the above and other purposes, features, and advantages of this application more obvious and understandable, the following specific embodiments of this application are specifically given. Brief Description of the Drawings

[0011] Figure 1 It is a schematic flowchart of a method for detecting human emotions based on multimodal physiological signals provided by this application;

[0012] Figure 2 It is a schematic flowchart of preprocessing for a predetermined window length period in a method for detecting human emotions based on multimodal physiological signals provided by this application;

[0013] Figure 3 It is a schematic flowchart of judging whether the SCR fusion index simultaneously meets the increasing trend in a method for detecting human emotions based on multimodal physiological signals provided by this application;

[0014] Figure 4 It is a schematic structural diagram of a system for detecting human emotions based on multimodal physiological signals provided by this application; Detailed Description of the Embodiments

[0015] Due to the technical problem in the prior art that when collecting the physiological changes of the person to be detected, it is difficult to dynamically perceive the physiological changes hidden by the person to be detected through will or behavior, resulting in the difficulty in precisely detecting the micro-emotion changes hidden by the person to be detected.

[0016] In response to the above technical problem, the overall idea of the technical solution provided by this application is as follows:

[0017] This application proposes a method for detecting human emotions based on multimodal physiological signals. Since the person to be detected is vulnerable to negative events, during the family visit activity where emotional out-of-control may occur, an electrocardiogram (ECG) sensor and a galvanic skin response (GSR) sensor are placed on the skin surface of the person to be detected to collect the electrocardiogram signal and the galvanic skin response signal of the human epidermis. Using the electrocardiogram signal and the galvanic skin response signal as inputs, three physiological characteristic indexes, namely SDNN, SCR peak value, and SCR frequency, are calculated in real time. And based on the dynamic change rules of the three indexes, two indexes that can evaluate the sudden change of individual emotions and the corresponding judgment thresholds are comprehensively analyzed. If the real-time emotion representation index is greater than its threshold value, it is considered that the emotion of the person to be detected fluctuates violently in a short time, and the system alarms for it; otherwise, there is no alarm.

[0018] After introducing the basic principle of this application, below, the technical solutions in this application will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments of this application. It should be understood that this application is not limited by the example embodiments described here. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of this application. Additionally, it should be noted that for the sake of description, only the parts related to this application rather than all are shown in the accompanying drawings.

[0019] Embodiment 1

[0020] As Figure 1 shown, this application provides a method for detecting human emotions based on multimodal physiological signals. The method is applied to an emotion detection system, and the system is communicatively connected to an electrocardiogram sensor and a galvanic skin response sensor. The method includes:

[0021] Step S100: Using the electrocardiogram sensor, collect the electrocardiogram signal of the target user to obtain dynamic electrocardiogram signal data;

[0022] Step S200: Using the galvanic skin response sensor, collect the galvanic skin response signal of the target user to obtain dynamic galvanic skin response signal data;

[0023] Specifically, as a special group in society, effective emotion detection of the person to be detected can effectively perceive the emotional state of the person to be detected during the visit activity. By collecting physiological signals during the process of physiological changes caused by the psychological changes of the person to be detected under the influence of negative events, the monitoring of emotional out-of-control can be effectively carried out through the changes of physiological signals.

[0024] However, in the prior art, when collecting the physiological changes of the person to be detected, it is difficult to dynamically perceive the physiological changes hidden by the person to be detected through will or behavior, resulting in the technical problem that it is difficult to accurately detect the micro-emotional changes hidden by the person to be detected.

[0025] To solve the problems existing in the prior art, the present application proposes a method for detecting a person's emotion based on multi-modal physiological signals. Since the person to be detected is easily affected by negative events and may have an emotional outburst during the family visit activity, an electrocardiogram (ECG) sensor and a galvanic skin response (GSR) sensor are placed on the skin surface of the person to be detected to collect the electrocardiogram signal and the galvanic skin response signal of the human epidermis. Taking the electrocardiogram signal and the galvanic skin response signal as inputs, three physiological characteristic indexes, namely SDNN, SCR peak value, and SCR frequency, are calculated in real time. And based on the dynamic change rules of the three indexes, two indexes for evaluating the sudden change of individual emotion and the corresponding judgment thresholds are comprehensively analyzed. If the real-time emotion characterization index is greater than its threshold value, it is considered that the emotion of the person to be detected fluctuates violently in a short time, and the system alarms, otherwise there is no alarm. When collecting the physiological changes of the person to be detected, by evaluating the SDNN index and the SCR index of the sudden change of individual emotion and the corresponding judgment thresholds, the dynamic perception of the physiological changes hidden by the person to be detected through will or behavior is realized, and then the accurate detection of the micro-emotional changes hidden by the person to be detected is realized.

[0026] Specifically, the ECG sensor can be placed on the skin surface of the person to be detected to dynamically perceive the electrocardiogram signal, so as to record the electroactivity change graph generated by each cardiac cycle of the heart from the body surface. The dynamic electrocardiogram signal data is the result of continuously and dynamically collecting the electrocardiogram signal of the person to be detected during the meeting. The galvanic skin response sensor, GSR represents the galvanic skin response, which is a method of measuring skin conductance. Strong emotions will stimulate your sympathetic nervous system, causing the sweat glands to secrete more sweat. For example, painful stimuli, such as acupuncture, will cause a sympathetic response of the sweat glands, increasing sweat secretion. Although this increase is usually small, sweat contains water and electrolytes, thus increasing the conductivity and reducing the skin resistance. These changes in turn affect the GSR. The dynamic galvanic skin response signal data reflects the result of continuously and dynamically collecting the galvanic skin response signal of the person to be detected during the meeting.

[0027] The ECG sensor adopted can obtain a stable electrocardiogram signal that can depict the heart beat change rule of the person to be detected, reflect the activity of the body's autonomic nerves, and at the same time minimize the artifact influence brought by limb movement in the family visit scenario. The galvanic skin response sensor adopted can obtain a galvanic skin response signal that can effectively characterize the physiological and psychological arousal state of an individual under the influence of emotional fluctuations.

[0028] Step S300: By performing preprocessing on the dynamic electrocardiogram signal data and the dynamic skin resistance signal data for a predetermined window length period, respectively obtain the SDNN index and the SCR fusion index;

[0029] Further, as Figure 2 shown, step S300 includes:

[0030] Step S310: Based on the peak detection algorithm, collect adjacent wave peaks of the dynamic electrocardiogram signal data to obtain the peak interval between two adjacent R waves;

[0031] Step S320: Based on the sliding window calculation algorithm, perform dynamic calculation on the peak interval between the two adjacent R waves for the predetermined window length period to determine the SDNN index;

[0032] Step S330: Based on the feature extraction algorithm, extract the SCR component from the dynamic skin resistance signal data to obtain the SCR component;

[0033] Step S340: Use the sliding window calculation algorithm to perform dynamic calculation on the SCR component for the predetermined window length period to determine the SCR peak feature and the SCR frequency feature;

[0034] Step S350: Determine the SCR fusion index by performing feature fusion on the SCR peak feature and the SCR frequency feature.

[0035] Specifically, step S350 includes:

[0036] Step S351: Perform weighted average calculation of the indexes on the SCR peak feature and the SCR frequency feature to obtain the SCR fusion index;

[0037] Step S352: Perform first derivative calculation on the SCR fusion index to obtain the change slope of the fusion index.

[0038] Specifically, after obtaining the dynamic electrocardiogram signal data and the dynamic skin resistance signal data, signal analysis can be performed based on this. First, data preprocessing can be carried out on them, that is, by extracting target features from the collected signals, so as to obtain the target signal features required for analysis, and then the target signal features are further processed. Specifically, based on the peak detection algorithm, adjacent wave peaks of the dynamic electrocardiogram signal data are collected to obtain the peak interval between two adjacent R waves. Since the collected electrocardiogram signal is a fluctuating graph that dynamically reflects the electrocardiogram data of the person to be detected, the peak data in the fluctuating graph can be collected, that is, the point with the largest fluctuation of the electrocardiogram data of the person to be detected is collected. By collecting adjacent wave peaks, the time difference between adjacent wave peaks can be obtained, that is, the peak interval between the two adjacent R waves. Generally, when the peak interval is shorter, it indicates that the emotional fluctuations of the person to be detected are more frequent. The peak detection algorithm is an algorithm for extracting peak data in the fluctuating graph.

[0039] Furthermore, based on the sliding window calculation algorithm, dynamic calculation of the peak interval between the two adjacent R waves for the predetermined window length period can be performed to determine the SDNN index. Among them, the electrocardiogram signal obtains the peak interval between two adjacent R waves through the peak detection algorithm, and in a sliding window calculation manner, the standard deviation of the R-R interval (SDNN) feature within a certain window time is dynamically calculated. The predetermined window length period is the above-mentioned preset certain window time. The so-called sliding window algorithm is similar to the jumping window algorithm. The sliding window (Moving Window) algorithm also controls the traffic volume by restricting the maximum number of cells that can be received within each time window. The difference is that in the sliding window algorithm, the time window does not jump forward, but slides forward once every cell time, and the sliding length is the time of one cell. Among them, the above-mentioned certain window time includes a set of several cell times. SDNN refers to an index of heart rate variability, which refers to the standard deviation of all sinus beats RR intervals (abbreviated as NN intervals), and the unit is microseconds. The larger the SDNN, the greater the heart rate variability. Generally, the younger the age, the larger the SDNN. When the SDNN is lower than the normal value, it indicates that there may be a decrease in parasympathetic nerve activity, that is, a manifestation of a decrease in vagal nerve tone activity. The SDNN index can be used to reflect the heart rate variability of the person to be detected.

[0040] Meanwhile, based on the feature extraction algorithm, the SCR component can be extracted from the dynamic skin resistance signal data to obtain the SCR component. Using the sliding window calculation algorithm, the dynamic calculation of the SCR component for the predetermined window length period is performed to determine the SCR peak feature and the SCR frequency feature. That is, the skin resistance signal is used to obtain the skin conductance response (SCR) component through the feature extraction algorithm in the form of sliding window calculation, and the peak value of SCR within a certain window time and the frequency of SCR appearance within the window time are dynamically calculated. Among them, the GSR signal is mainly composed of the slowly changing basal activity - skin conductance level and the rapidly changing phasic activity - skin conductance response. The phasic response of the skin electrical signal is above the basal level, with a higher change amplitude and faster speed, and is displayed in the form of "GSR burst" or "GSR peak". The phasic response is an instantaneous and relatively fast fluctuation in the skin conductance level and is a physiological and psychological activation state caused by stimulation. When a stimulus is presented to a person, the conductance increases, forming a waveform skin conductance response. The skin conductance response (SCR) refers to phasic sympathetic nerve discharge, which is a physiological and psychological activation state caused by stimulation and is an indicator reflecting short-term brain processes. It can be used as a localization marker for the emotional arousal point generated by external novel event stimuli. SCR is sensitive to specific emotional stimulus events, and the event-related skin conductance responses (ER-SCRs) will occur suddenly between 1 and 5 seconds after the emotional stimulus; the non-specific skin conductance responses (NS-SCRs) occur spontaneously in the human body at a rate of 1 to 3 minutes and are independent of any stimulus. Then the SCR component reflects the physiological and psychological response ability of the person to be tested during the meeting to sensitive events. The SCR peak feature reflects the extreme point of the psychological response of the person to be tested to external sensitive events, and the SCR frequency feature reflects the number of times the extreme point of the response appears.

[0041] After determining the SCR peak feature and the SCR frequency feature, in order to conduct a comprehensive analysis on them, feature fusion can be performed on the SCR peak feature and the SCR frequency feature. Specifically, weighted average calculation of the indicators of the SCR peak feature and the SCR frequency feature is carried out to obtain the SCR fusion indicator. Among them, the SCR fusion indicator reflects the extreme points of the psychological reaction within the person to be detected and the number of times they appear. Furthermore, a first derivative calculation can be performed on the SCR fusion indicator to obtain the change slope of the fusion indicator. The so-called first derivative, a calculus term, the first derivative represents the rate of change of a function, and the most intuitive manifestation lies in the monotonicity theorem of the function. The change slope of the fusion indicator reflects the change slope of the fused indicator. Generally, when the slope is higher, it indicates a faster change. To sum up, weighted average calculation can be performed on the two indicators of the SCR peak indicator and the SCR frequency to obtain the SCR fusion indicator, and by calculating the first derivative of this fusion indicator in real time, the change slope of this fusion indicator can be obtained as the threshold for subsequent judgment of instantaneous large changes in emotions.

[0042] Step S400: Determine whether the SDNN indicator satisfies a decreasing trend;

[0043] Step S500: If the SDNN indicator satisfies a decreasing trend, determine whether the SCR fusion indicator synchronously satisfies an increasing trend;

[0044] Step S600: If the SDNN indicator satisfies a decreasing trend and the SCR fusion indicator synchronously satisfies an increasing trend, trigger the first alarm instruction to alarm the emotional state of the target user.

[0045] Further, as Figure 3 shown, Step S500 includes:

[0046] Step S510: Determine whether the SDNN indicator is lower than the preset indicator standard deviation threshold;

[0047] Step S520: If the SDNN indicator is lower than the preset indicator standard deviation threshold, determine the change characteristics of the change slope of the fusion indicator;

[0048] Step S530: Determine whether the change slope of the fusion indicator is synchronously higher than the preset slope standard deviation threshold;

[0049] Step S540: If the change slope of the fusion indicator is synchronously higher than the preset slope standard deviation threshold, trigger the first alarm instruction to alarm the emotional state of the target user.

[0050] Specifically, after obtaining the SDNN index and the SCR fusion index, it is necessary to judge their changing trends. Specifically, first determine whether the SDNN index is lower than the preset index standard deviation threshold, wherein the preset index standard deviation threshold can be understood as the preset SDNN index standard deviation threshold. When the SCNN index of the person to be detected decreases (with the standard deviation of the SDNN index lower than a certain time interval as the threshold), it means that the sympathetic nerve activity of the person to be detected increases due to tension, pressure, increased emotional arousal, etc., and the body is in an excited state. At the same time, it can also be determined whether the change slope of the fusion index is synchronously higher than the preset slope standard deviation threshold, wherein the preset slope standard deviation threshold can be understood as the slope standard deviation threshold within a preset time interval. At this time, when the change slope of the SCR fusion index increases synchronously (with the slope standard deviation higher than a certain time interval as the threshold), it means that the emotion of the person to be detected reaches a relatively high arousal level at a higher speed, and it is determined that the individual is in a state of violent emotional fluctuations, and the first alarm instruction can be triggered to alarm the emotional state of the person to be detected.

[0051] In summary, the embodiments of the present application have at least the following technical effects:

[0052] 1. Since the person to be tested is easily affected by negative events, during the family meeting activities where emotions are out of control, the ECG sensor and the GSR sensor are placed on the skin surface of the person to be tested to collect the ECG signal and skin resistance signal of the human epidermis. With the ECG signal and skin resistance signal as input, the three physiological characteristic indicators of SDNN, SCR peak and SCR frequency are calculated in real time. Based on the dynamic change rules of the three indicators, two indicators that can evaluate the sudden changes in individual emotions and the corresponding judgment thresholds are obtained through comprehensive analysis. If the real-time emotional representation indicator is greater than its threshold, it is considered that the emotions of the person to be tested have fluctuated violently in a short period of time, and the system will alarm it, otherwise there will be no alarm. When collecting the physiological changes of the person to be tested, the SDNN indicator and SCR indicator that evaluate the sudden changes in individual emotions, as well as the corresponding judgment thresholds, are used to dynamically perceive the physiological changes hidden by the will or behavior of the person to be tested, and then the technical effect of accurately detecting the hidden micro-emotional changes of the person to be tested is achieved.

[0053] 2. The ECG sensor used can obtain stable ECG signals in family meeting scenarios that can depict the heartbeat changes of the person to be tested, reflect the activity of the body's autonomic nervous system, and minimize the artifacts caused by limb movement. The skin resistance sensor used can obtain skin resistance signals that can effectively represent the physiological and psychological arousal state of an individual under the influence of emotional fluctuations.

[0054] Embodiment 2

[0055] Based on the same inventive concept as a method for detecting human emotions based on multimodal physiological signals in the foregoing embodiments, as Figure 4 shown, the present application provides a system for detecting human emotions based on multimodal physiological signals, wherein the system includes:

[0056] An electrocardiogram signal acquisition module, configured to use an electrocardiogram sensor to acquire electrocardiogram signals of a target user to obtain dynamic electrocardiogram signal data;

[0057] A skin resistance signal acquisition module, configured to use a skin resistance sensor to acquire skin resistance signals of a target user to obtain dynamic skin resistance signal data;

[0058] A data preprocessing module, configured to perform preprocessing on the dynamic electrocardiogram signal data and the dynamic skin resistance signal data for a predetermined window length period to respectively obtain an SDNN index and an SCR fusion index;

[0059] An SDNN index judgment module, configured to judge whether the SDNN index meets a decreasing trend;

[0060] An SCR fusion index judgment module, configured to judge whether the SCR fusion index simultaneously meets an increasing trend if the SDNN index meets a decreasing trend;

[0061] A status alarm module, configured to trigger a first alarm instruction to alarm the emotional state of the target user if the SDNN index meets a decreasing trend and the SCR fusion index simultaneously meets an increasing trend.

[0062] Further, the system further includes:

[0063] A peak acquisition unit, configured to perform adjacent peak acquisition on the dynamic electrocardiogram signal data based on a peak detection algorithm to obtain the peak interval between two adjacent R waves;

[0064] A peak interval calculation unit, configured to perform dynamic calculation on the peak interval between the two adjacent R waves for the predetermined window length period based on a sliding window calculation algorithm to determine the SDNN index.

[0065] Further, the system further includes:

[0066] A component extraction unit, configured to perform SCR component extraction on the dynamic skin resistance signal data based on a feature extraction algorithm to obtain SCR components;

[0067] A feature determination unit, which uses the sliding window calculation algorithm to perform dynamic calculation on the SCR component for the predetermined window length period to determine the SCR peak feature and the SCR frequency feature;

[0068] A feature fusion unit, which determines the SCR fusion index by performing feature fusion on the SCR peak feature and the SCR frequency feature;

[0069] Furthermore, the system further includes:

[0070] A weighted calculation unit, which performs weighted average calculation on the SCR peak feature and the SCR frequency feature to obtain the SCR fusion index;

[0071] A reciprocal calculation unit, which performs first derivative calculation on the SCR fusion index to obtain the change slope of the fusion index.

[0072] Furthermore, the system further includes:

[0073] An SDNN index judgment unit, which judges whether the SDNN index is lower than the preset index standard deviation threshold;

[0074] A fusion index judgment unit, which judges the change characteristics of the change slope of the fusion index if the SDNN index is lower than the preset index standard deviation threshold.

[0075] Furthermore, the system further includes:

[0076] A change slope judgment unit, which judges whether the change slope of the fusion index is synchronously higher than the preset slope standard deviation threshold;

[0077] An instruction trigger unit, which triggers the first alarm instruction to alarm the emotional state of the target user if the change slope of the fusion index is synchronously higher than the preset slope standard deviation threshold.

[0078] The present application provides a method for detecting a person's emotion based on multimodal physiological signals. The method includes: placing an electrocardiogram (ECG) sensor and a galvanic skin response (GSR) sensor on the skin surface of the person to be detected, and collecting the ECG signal and the GSR signal on the human epidermis. Taking the ECG signal and the GSR signal as inputs, three physiological characteristic indexes, namely SDNN, SCR peak value, and SCR frequency, are calculated in real time. And based on the dynamic change rules of the three indexes, two indexes for evaluating the sudden change of an individual's emotion and the corresponding judgment thresholds are comprehensively analyzed. If the real-time emotion representation index is greater than its threshold value, it is considered that the emotion of the person to be detected fluctuates violently in a short time, and the system alarms for it. Otherwise, there is no alarm. The technical problem that it is difficult to dynamically perceive the physiological changes hidden by the person to be detected through will or behavior when collecting the physiological changes of the person to be detected, resulting in difficulty in accurately detecting the micro-emotion changes hidden by the person to be detected, is solved. The technical effect of dynamically perceiving the physiological changes hidden by the person to be detected through will or behavior when collecting the physiological changes of the person to be detected, and then accurately detecting the micro-emotion changes hidden by the person to be detected is achieved by evaluating the SDNN index and the SCR index of the sudden change of an individual's emotion and the corresponding judgment thresholds.

[0079] This specification and the drawings are only exemplary descriptions of the present application. If the modifications and variations of the present invention fall within the scope of the present invention and its equivalent technologies, the present invention also intends to include these modifications and variations.

Claims

1. A method for detecting human emotions based on multimodal physiological signals, characterized in that, The method is applied to an emotion detection system, and the system is communicatively connected to an electrocardiogram sensor and a skin resistance sensor. The method includes: Using the electrocardiogram sensor to collect electrocardiogram signals from a target user to obtain dynamic electrocardiogram signal data; Using the skin resistance sensor to collect skin resistance signals from the target user to obtain dynamic skin resistance signal data; Performing preprocessing on the dynamic electrocardiogram signal data and the dynamic skin resistance signal data for a predetermined window length period to respectively obtain an SDNN index and an SCR fusion index; Determining whether the SDNN index meets a decreasing trend; If the SDNN index meets the decreasing trend, determining whether the SCR fusion index synchronously meets an increasing trend; If the SDNN index meets the decreasing trend and the SCR fusion index synchronously meets the increasing trend, triggering a first alarm instruction to alarm the emotional state of the target user; The performing preprocessing for a predetermined window length period includes: Based on a peak detection algorithm, collecting adjacent wave peaks of the dynamic electrocardiogram signal data to obtain the peak interval between two adjacent R waves; Based on a sliding window calculation algorithm, performing dynamic calculation on the peak interval between the two adjacent R waves for the predetermined window length period to determine the SDNN index; Based on a feature extraction algorithm, extracting SCR components from the dynamic skin resistance signal data to obtain SCR components; Using the sliding window calculation algorithm to perform dynamic calculation on the SCR components for the predetermined window length period to determine SCR peak characteristics and SCR frequency characteristics; Determining the SCR fusion index by performing feature fusion on the SCR peak characteristics and the SCR frequency characteristics; Performing weighted average calculation on the SCR peak characteristics and the SCR frequency characteristics to obtain the SCR fusion index; Performing first derivative calculation on the SCR fusion index to obtain the change slope of the fusion index.

2. The method according to claim 1, wherein The method includes: Determining whether the SDNN index is lower than a preset index standard deviation threshold; If the SDNN index is lower than the preset index standard deviation threshold, determining the change characteristics of the change slope of the fusion index.

3. The method according to claim 2, characterized in that, The method includes: Determining whether the change slope of the fusion index is synchronously higher than a preset slope standard deviation threshold; If the change slope of the fusion index is synchronously higher than the preset slope standard deviation threshold, triggering the first alarm instruction to alarm the emotional state of the target user.

4. A personnel emotion detection system based on multimodal physiological signals, characterized in that, The system is used to execute the method according to claims 1 to 3. The system includes: An electrocardiogram signal acquisition module for using an electrocardiogram sensor to collect electrocardiogram signals from a target user to obtain dynamic electrocardiogram signal data; A skin resistance signal acquisition module for using a skin resistance sensor to collect skin resistance signals from the target user to obtain dynamic skin resistance signal data; A data preprocessing module for performing preprocessing on the dynamic electrocardiogram signal data and the dynamic skin resistance signal data for a predetermined window length period to respectively obtain an SDNN index and an SCR fusion index; The SDNN index judgment module is used to judge whether the SDNN index meets the decreasing trend; The SCR fusion index judgment module is used to judge whether the SCR fusion index simultaneously meets the increasing trend if the SDNN index meets the decreasing trend; The status alarm module is used to trigger a first alarm instruction to alarm the emotional state of the target user if the SDNN index meets the decreasing trend and the SCR fusion index simultaneously meets the increasing trend.

Images