Emotion exchange method based on emotion recognition

Abstract

The invention discloses an emotion communication method based on emotion recognition. The method comprises the following steps: acquiring a multi-modal biological signal of a patient; the collected multi-mode biological signals are preprocessed; obtaining a multi-modal fusion characteristic value based on the preprocessed multi-modal biological signal; emotion intensity features are obtained based on the preprocessed multi-mode biological signals; based on the multi-modal fusion feature value, outputting an emotion classification result through a trained classification model; and determining a feedback strategy based on the sentiment classification result and the sentiment intensity feature. According to the emotion communication method based on emotion recognition, through combination of an innovative multi-mode brain-computer interface technology and an emotion recognition technology, multiple biological signals can be synthesized, and more accurate emotion recognition and communication are provided. The technology not only improves the accuracy and interactivity of emotion communication, but also breaks through the limitation of the traditional emotion recognition technology, is especially suitable for patients who cannot speak, and provides a more natural and effective emotion communication means for patients with emotion expression disorder.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and in particular relates to an emotion communication method based on emotion recognition. Background Art

[0002] At present, patients with emotional expression disorders (such as autistic patients, mental disorders, etc.) face difficulties in emotional communication. Existing technical solutions mainly rely on speech recognition, brain-computer interface technology and emotion recognition technology. These technical solutions usually focus on the analysis of language expression or external physiological characteristics.

[0003] The main drawbacks of existing technologies are that the accuracy of emotion recognition is low, especially for patients with limited facial expressions or speech expression. In addition, most solutions rely on only a single signal source and cannot fully understand the patient's emotional state. Summary of the invention

[0004] The present invention provides an emotion communication method based on emotion recognition to solve the above-mentioned technical problems, and specifically adopts the following technical solutions:

[0005] An emotion communication method based on emotion recognition, comprising:

[0006] Collect multimodal biological signals from patients;

[0007] Preprocessing the collected multimodal biological signals;

[0008] Acquire multimodal fusion feature values ​​based on the preprocessed multimodal biological signals;

[0009] Obtain emotion intensity features based on preprocessed multimodal biosignals;

[0010] Based on the multimodal fusion feature values, the sentiment classification results are output through the trained classification model;

[0011] Determine the feedback strategy based on the sentiment classification results and sentiment intensity characteristics.

[0012] Furthermore, the specific method of collecting the multimodal biological signals of the patient is:

[0013] Collect the patient's EEG signals, EMG signals and facial expression data.

[0014] Furthermore, the specific method for preprocessing the collected multimodal biological signals is:

[0015] Noise reduction is performed on EEG signals, EMG signals and facial expression data.

[0016] Furthermore, the specific method for obtaining the multimodal fusion feature value based on the preprocessed multimodal biological signal is:

[0017] Calculate the EEG state characteristic value according to the following formula;

[0018]

[0019] Among them, E EEG is the EEG state characteristic value, PSD i is the power spectral density of each frequency band, a i is the weight coefficient of each frequency band;

[0020] Calculate the electromyographic state characteristic value according to the following formula;

[0021]

[0022] Among them, E EMG is the characteristic value of the electromyographic state, |EMG i | is the amplitude of the electromyographic signal, b i is the weight coefficient of each electromyographic signal;

[0023] Calculate the expression state feature value according to the following formula;

[0024]

[0025] Among them, E Face is the expression state feature value, AU i is the strength of the facial action unit, c i is the weight of each facial action unit in different emotional states;

[0026] The multimodal fusion feature value is calculated by the following formula:

[0027] E fea =w1*E EEG +w2*E EMG +w3*E face

[0028] Among them, E fea is the multimodal fusion feature value after fusion, and w1, w2, and w3 are the corresponding weight coefficients respectively.

[0029] Furthermore, according to the actual situation of the patient, the weight coefficients w1, w2, and w3 are determined to calculate the multimodal fusion feature value;

[0030] When collecting data to train the classification model, the multimodal fusion feature values ​​are calculated by uniformly setting the preset weight coefficients w1, w2, and w3.

[0031] Furthermore, the specific method for obtaining the emotion intensity feature based on the preprocessed multimodal biological signal is:

[0032] The EEG emotion intensity characteristics are calculated according to the following formula;

[0033]

[0034] Among them, I EEG is the EEG emotion intensity feature, P(f i ) for different frequency bands f i The power spectral density of

[0035] Calculate the electromyographic emotion intensity characteristics according to the following formula;

[0036]

[0037] Among them, I EMG EMG is the electromyographic emotion intensity feature. j represents the amplitude of the electromyographic signal at the jth moment, and N is the number of signal sampling points:

[0038] Calculate the expression emotion intensity feature according to the following formula;

[0039]

[0040] Among them, I Face is the feature of expression emotion intensity, AU k represents the strength of the kth facial action unit, W k is the weight value corresponding to the action unit;

[0041] The sentiment intensity feature is calculated using the following formula:

[0042] I total =k1*I EEG +k2*I EMG +k3*I Face

[0043] Among them, I total is the fused emotion intensity feature, k1, k2, and k3 are the corresponding weights respectively.

[0044] Furthermore, the specific method for determining the feedback strategy based on the emotion classification result and the emotion intensity feature is:

[0045] Displaying image content matching the emotion classification result and the emotion intensity feature to the communicator through a display device;

[0046] A tactile indication matching the emotion classification result and the emotion intensity feature is output to the communicator through the tactile device.

[0047] Furthermore, the image content includes one or more of images, colors and dynamic patterns.

[0048] Furthermore, the tactile device is a smart bracelet for the user to wear, and the smart bracelet can output different levels of vibration or different levels of temperature that match the emotion classification result and the emotion intensity characteristics.

[0049] Furthermore, the specific method for determining the feedback strategy based on the emotion classification result and the emotion intensity feature also includes:

[0050] The speech content matching the emotion classification results and emotion intensity characteristics is played to the patient through a speech device.

[0051] The benefit of the present invention lies in the emotional communication method based on emotion recognition, which combines innovative multimodal brain-computer interface technology with emotion recognition technology, and can integrate multiple signals such as EEG, EMG and facial expressions to provide more accurate emotion recognition and communication. This technology not only improves the accuracy and interactivity of emotional communication, but also breaks through the limitations of traditional emotion recognition technology. It is particularly suitable for patients who cannot speak, and provides a more natural and effective means of emotional communication for patients with emotional expression disorders. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 It is a schematic diagram of the emotion communication method based on emotion recognition of the present application. DETAILED DESCRIPTION

[0054] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0055] like Figure 1 The present invention shows an emotion communication method based on emotion recognition, comprising:

[0056] S1: Collect multimodal biological signals of patients.

[0057] Specifically, a high-precision sensor is used to monitor and record the patient's physiological state in real time. In the present application, the specific method of collecting the patient's multimodal biosignals is: collecting the patient's electroencephalogram (EEG) signal, electromyography (EMG) signal and facial expression (Facial Expression) data.

[0058] EEG signals reflect the activity state of the human brain. Spectral analysis is performed on EEG signals to extract the power spectral density (PSD) features of specific frequency bands, such as theta waves (4-8Hz), alpha waves (8-13Hz), etc. These bands correspond to different emotional states.

[0059] EMG signals reflect the electrical activity of muscles. By analyzing the intensity, frequency, and time characteristics of facial or whole-body muscle activity, changes in emotional states can be identified. Common features include the amplitude, frequency, time domain characteristics (such as mean, standard deviation, etc.) and spectrum characteristics of EMG signals, which can reveal changes in muscle tension caused by emotions. Emotional fluctuations (such as pleasure, anxiety, anger, etc.) are usually accompanied by the contraction and relaxation of specific muscle groups. EMG signals help identify and analyze emotional states by capturing these subtle changes.

[0060] Facial expressions are important clues to emotional expression, which can be analyzed through features such as facial action units (AUs), geometric features, and micro-expressions. Facial action units (such as eyebrow raising, mouth corners up, etc.) correspond to specific emotional states, and geometric changes in facial expressions (such as the shape and position of eyes and mouth) reflect the subtle differences in emotions. In addition, information such as the speed and duration of expression changes and the brevity of micro-expressions can further reveal an individual's immediate emotional reactions and potential emotions. Through computer vision and deep learning methods, these features can be efficiently extracted and sentiment analysis can be performed.

[0061] S2: Preprocess the collected multimodal biological signals.

[0062] The specific method for preprocessing the collected multimodal biological signals is: performing noise reduction processing on the EEG signals, EMG signals and facial expression data. It is understandable that other relevant data preprocessing can also be performed on the EEG signals, EMG signals and facial expression data to improve the accuracy of the data.

[0063] S3: Obtain multimodal fusion feature values ​​based on the preprocessed multimodal biological signals.

[0064] In the implementation manner of the present application, the specific method for obtaining the multimodal fusion feature value based on the preprocessed multimodal biological signal is:

[0065] The EEG state eigenvalue refers to the signal eigenvalue related to the emotional state obtained by analyzing the EEG signal. These eigenvalues ​​reflect certain changes in EEG activity under different emotional states (such as anxiety, pleasure, fear, etc.), and are usually the result of frequency domain analysis. For example, the power density values ​​of different frequency bands such as α waves, β waves, and θ waves, the correlation between the changes in these frequency bands and the emotional state is usually quantified through correlation analysis or regression models. Therefore, the eigenvalue can be understood as the mapping relationship between the emotional state and the EEG signal. Specifically, the EEG state eigenvalue is calculated according to the following formula:

[0066]

[0067] Among them, E EEG is the EEG state characteristic value, PSD i is the power spectral density of each frequency band (such as α wave, β wave, θ wave, etc.), a i is the weight coefficient of each frequency band.

[0068] The electromyographic state eigenvalue refers to the eigenvalue related to the emotional state obtained by analyzing the electromyographic signal. These eigenvalues ​​reflect the relationship between muscle activity (such as tension or relaxation of facial muscles) and emotional state. By analyzing the electromyographic signal, it can be determined whether the patient's emotional response triggers muscle contraction or relaxation. The electromyographic emotion-related value is usually calculated by extracting the amplitude, frequency and other characteristics of the electromyographic signal. Specifically, the electromyographic state eigenvalue is calculated according to the following formula:

[0069]

[0070] Among them, E EMG is the characteristic value of the electromyographic state, |EMG i | is the amplitude of the electromyographic signal, b i is the weight coefficient of each EMG signal.

[0071] The expression state eigenvalue refers to the eigenvalue related to the emotional state obtained by analyzing the facial expression image data. This eigenvalue reflects the relationship between the change of facial expression (such as smiling, frowning, drooping corners of the mouth, etc.) and the emotional state. The change of facial expression, especially the activity of facial action units (AUs), is closely related to the type of emotion. The expression state eigenvalue is calculated according to the following formula:

[0072]

[0073] Among them, E Face is the expression state feature value, AU i is the strength of the facial action unit, c iis the weight of each facial action unit in different emotional states.

[0074] The multimodal fusion feature value is calculated by the following formula:

[0075] E fea =w1*E EEG +w2*E EMG +w3*E face

[0076] Among them, E fea is the multimodal fusion feature value after fusion, and w1, w2, and w3 are the corresponding weight coefficients respectively.

[0077] S4: Obtain emotion intensity features based on preprocessed multimodal biosignals.

[0078] In the implementation manner of the present application, the specific method for obtaining the emotion intensity feature based on the preprocessed multimodal biosignal is:

[0079] The emotional state of an individual is reflected by analyzing the energy values ​​of the alpha wave (8-13Hz), beta wave (13-30Hz), theta wave (4-8Hz) and other frequency bands in the EEG signal. Emotional intensity is usually related to the energy changes in these bands. High frequencies (such as beta waves) are usually associated with anxiety or tension, while low frequencies (such as alpha waves) may be associated with relaxation and pleasure. In this application, the EEG emotional intensity feature is calculated according to the following formula:

[0080]

[0081] Among them, I EEG is the EEG emotion intensity feature, P(f i ) for different frequency bands f i The power spectral density of .

[0082] Extract amplitude, frequency and spectrum features from the electromyographic signal to analyze the tension of facial and limb muscles. For example, muscles are relaxed when you are happy, but muscles are more tense when you are anxious or angry, with higher amplitude and frequency. Specifically, the electromyographic emotion intensity feature is calculated according to the following formula:

[0083]

[0084] Among them, I EMG EMG is the electromyographic emotion intensity feature. j represents the amplitude of the electromyographic signal at the jth moment, and N is the number of signal sampling points.

[0085] The intensity of emotion is evaluated based on facial action units (AUs) and the intensity of expression. Each facial action unit is closely related to an aspect of emotion (such as pleasure, anger, surprise, etc.), and the intensity of facial expression can be quantified by the magnitude of the change in facial key points. Specifically, the expression emotion intensity feature is calculated according to the following formula:

[0086]

[0087] Among them, I Face is the feature of expression emotion intensity, AU k represents the strength of the kth facial action unit, W k is the weight value corresponding to the action unit.

[0088] The sentiment intensity feature is calculated using the following formula:

[0089] I total =k1*I EEG +k2*I EMG +k3*I Face

[0090] Among them, I total is the fused emotion intensity feature, k1, k2, and k3 are the corresponding weights respectively.

[0091] S5: Based on the multimodal fusion feature values, the sentiment classification results are output through the trained classification model.

[0092] Specifically, a classification model such as Support Vector Machine (SVM) or Long Short Term Memory Network (LSTM) can be used to categorize the multimodal feature fusion value E fea Classify and accurately identify the emotion category E clas s (such as anxiety, pleasure, fear, etc.).

[0093] When training the classification model, it is necessary to collect data from several subjects to calculate the multimodal feature fusion value E fea , and through several multimodal feature fusion values ​​E fea Train the classification model. In the process of collecting training data, in order to improve efficiency, when calculating the multimodal feature fusion value E fea When , the multimodal fusion feature value E is calculated by unifying the preset weight coefficients wl, w2, w3 (such as the default 1 / 3, 1 / 3, 1 / 3) fea However, it can be understood that the settings of weight coefficients w1, w2, and w3 respectively allocate the EEG state eigenvalue, myoelectric state eigenvalue, and expression state eigenvalue to the multimodal fusion eigenvalue E feaHowever, for different individuals to be judged, in order to improve the accuracy of emotion recognition, it is best to make targeted adjustments based on the individual's physical state, making it different from the default 1 / 3, 1 / 3, 1 / 3. For example, for some patients with emotional disorders who have poor facial expressions, the weight of w3 can be appropriately reduced. For some patients with inflexible limbs, which makes the electromyographic signal inaccurate, the weight of w2 can be reduced, and so on. In this way, for different patients, according to the actual situation of the patient, a weight that is more in line with the patient's true state is set, so that the multimodal fusion feature value E fea The calculation results are more accurate.

[0094] S6: Determine the feedback strategy based on the sentiment classification results and sentiment intensity characteristics.

[0095] In the implementation manner of the present application, the specific method for determining the feedback strategy based on the emotion classification result and the emotion intensity feature is:

[0096] The display device displays image content that matches the emotion classification result and emotion intensity characteristics to the communicator. Specifically, one or more images, colors, or dynamic patterns that match the emotional state are displayed through a screen or light signal. For example, bright and warm tones are displayed in a happy state, and calm blue or dynamically changing light stripes are displayed in an anxious state.

[0097] A tactile indication matching the emotion classification result and the emotion intensity characteristic is output to the communicator through the tactile device. In an embodiment of the present application, the tactile device is a smart bracelet for the user to wear, and the smart bracelet can output different levels of vibration or different levels of temperature matching the emotion classification result and the emotion intensity characteristic.

[0098] In the implementation manner of the present application, the specific method for determining the feedback strategy based on the emotion classification result and the emotion intensity feature further includes:

[0099] The voice device plays voice content that matches the emotion classification result and emotion intensity characteristics to the patient. Appropriate voice information is generated based on the patient's emotion intensity. For example, if the emotion intensity is high and negative (such as anxiety), the system generates calm and soothing voice information.

[0100] Specifically, the relevant decision can be selected through the decision function, and the relevant calculation method is as follows:

[0101] F=(I total , E class )

[0102] Among them, F is the generated feedback form, f() is the decision function for feedback selection, which determines the most appropriate feedback type based on the emotion intensity and category.

[0103] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. An emotion communication method based on emotion recognition, characterized in that: Include: Collect multimodal biological signals from patients; Preprocessing the collected multimodal biological signals; Acquire multimodal fusion feature values ​​based on the preprocessed multimodal biological signals; Obtain emotion intensity features based on preprocessed multimodal biosignals; Based on the multimodal fusion feature values, the sentiment classification results are output through the trained classification model; Determine the feedback strategy based on the sentiment classification results and sentiment intensity characteristics.

2. The emotion communication method based on emotion recognition according to claim 1, characterized in that: Include: The specific method of collecting the multimodal biological signals of the patient is: Collect the patient's EEG signals, EMG signals and facial expression data.

3. The emotion communication method based on emotion recognition according to claim 2, characterized in that: Include: The specific method for preprocessing the collected multimodal biological signals is: Noise reduction is performed on EEG signals, EMG signals and facial expression data.

4. The emotion communication method based on emotion recognition according to claim 2, characterized in that: Include: The specific method for obtaining the multimodal fusion feature value based on the preprocessed multimodal biological signal is: The EEG state characteristic value is calculated according to the following formula: Among them, E EEG is the EEG state characteristic value, PSD i is the power spectral density of each frequency band, a i is the weight coefficient of each frequency band; The characteristic value of the electromyographic state is calculated according to the following formula: Among them, E EMG is the characteristic value of the electromyographic state, |EMG i | is the amplitude of the electromyographic signal, b i is the weight coefficient of each electromyographic signal; The expression state feature value is calculated according to the following formula: Among them, E Face is the expression state feature value, AU i is the strength of the facial action unit, c i is the weight of each facial action unit in different emotional states; The multimodal fusion feature value is calculated by the following formula: HAVE BEEN fea =w1*E EEG +w2*E EMG +w3*E face Among them, E fea is the multimodal fusion feature value after fusion, and w1, w2, and w3 are the corresponding weight coefficients respectively.

5. The emotion communication method based on emotion recognition according to claim 4, characterized in that: Include: According to the actual situation of the patient, the weight coefficients w1, w2, w3 are determined to calculate the multimodal fusion feature value; When collecting data to train the classification model, the multimodal fusion feature values ​​are calculated by uniformly setting the preset weight coefficients w1, w2, and w3.

6. The emotion communication method based on emotion recognition according to claim 2, characterized in that: Include: The specific method for obtaining the emotion intensity feature based on the preprocessed multimodal biological signal is: The EEG emotion intensity feature is calculated according to the following formula: Among them, I EEG is the EEG emotion intensity feature, P(f i ) for different frequency bands f i The power spectral density of The myoelectric emotion intensity characteristics are calculated according to the following formula: Among them, I EMG EMG is the electromyographic emotion intensity feature. j represents the amplitude of the electromyographic signal at the jth moment, and N is the number of signal sampling points; The expression emotion intensity feature is calculated according to the following formula: Among them, I Face is the feature of expression emotion intensity, AU k represents the strength of the kth facial action unit, W k is the weight value corresponding to the action unit; The sentiment intensity feature is calculated using the following formula: I total =k1*I EEG +k2*I EMG +k3*I Face Among them, I total is the fused emotion intensity feature, k1, k2, and k3 are the corresponding weights respectively.

7. The emotion communication method based on emotion recognition according to claim 1, characterized in that: Include: The specific method for determining the feedback strategy based on the emotion classification result and the emotion intensity feature is: Displaying image content matching the emotion classification result and the emotion intensity feature to the communicator through a display device; A tactile indication matching the emotion classification result and the emotion intensity feature is output to the communicator through the tactile device.

8. The emotion communication method based on emotion recognition according to claim 7, characterized in that: Include: The image content includes one or more of an image, a color, and a dynamic pattern.

9. The emotion communication method based on emotion recognition according to claim 7, characterized in that: Include: The tactile device is a smart bracelet for users to wear, and the smart bracelet can output different levels of vibration or different levels of temperature that match the emotion classification results and emotion intensity characteristics.

10. The emotion communication method based on emotion recognition according to claim 7, characterized in that: Include: The specific method for determining the feedback strategy based on the emotion classification result and the emotion intensity feature also includes: The speech content matching the emotion classification results and emotion intensity characteristics is played to the patient through a speech device.

Images