Emotion recognition method based on multi-modal information fusion

Abstract

The invention discloses an emotion recognition method based on multi-modal information fusion. The emotion recognition method comprises the steps that data information and environment information of a to-be-tested person are collected in real time; performing preprocessing and feature extraction on the data information to obtain extracted features; facial expressions and voice texts are recognized through the extracted features by means of a convolutional neural network, and the emotional state of the to-be-tested person is judged in combination with collected environment information and assisted by skin electric signals and electroencephalogram signals. And the automation level and precision of emotion recognition are improved.

Description

Technical Field

[0001] The present invention relates to the field of biomedical engineering, and in particular to an emotion recognition method based on multimodal information fusion. Background Art

[0002] Currently, limited medical resources are an important factor restricting the improvement of medical service quality. Traditional rehabilitation methods often rely on a large amount of medical equipment and human resources, which are costly and difficult to popularize.

[0003] Insufficient medical resources is another major challenge facing the medical field. Emotional rehabilitation is equally important in the rehabilitation process of stroke patients, but it is often difficult to provide adequate emotional support and attention to each patient due to the limited number of medical staff.

[0004] In addition, the low emotional compliance of patients with rehabilitation has become another pain point in rehabilitation care. Due to the long rehabilitation period, high rehabilitation training intensity, and lack of emotional care, some patients develop resistance, disgust, irritability and other mentalities during the rehabilitation process, which is extremely detrimental to the continuity and smoothness of patients' rehabilitation. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide a multimodal information fusion emotion recognition method that overcomes the above problems or at least partially solves the above problems.

[0006] According to one aspect of the present invention, a method for emotion recognition based on multimodal information fusion is provided, the method comprising:

[0007] Collect data information of the test subjects in real time;

[0008] Preprocessing and feature extracting the data information to obtain extracted features;

[0009] The extracted features are used to recognize facial expressions and voice text using a convolutional neural network, and the emotional state of the person being tested is determined by combining environmental information with skin electrical signals and brain electrical signals.

[0010] Optionally, the data information specifically includes: physiological signals, behavioral information and external environment information of the person to be tested.

[0011] Optionally, the physiological signals specifically include: hand skin electrical signals, brain waves and heart rate.

[0012] Optionally, the behavior information specifically includes: facial expressions and voice.

[0013] Optionally, the external environment information specifically includes: real-time weather, indoor and outdoor temperatures, and local time.

[0014] Optionally, the preprocessing specifically includes: filtering, denoising and feature extraction.

[0015] Optionally, features related to the emotional state are extracted from the preprocessed data.

[0016] Optionally, the convolutional neural network specifically includes:

[0017] The convolution operation is expressed as:

[0018] Among them, f is the input image, g is the convolution kernel, * represents the convolution operation, and [u,v] is the position of the output feature map;

[0019] The pooling operation is expressed as:

[0020]

[0021] Among them, p is the output after pooling, f is the input feature map, and max represents the maximum value operation;

[0022] The output of the fully connected layer is calculated by the formula:

[0023] o=W·a+b

[0024] Where o is the output, W is the weight matrix, a is the input feature vector, and b is the bias term.

[0025] The present invention provides an emotion recognition method for multimodal information fusion, which includes: collecting data information of a subject to be tested in real time; using the extracted features to recognize facial expressions and speech text using a convolutional neural network, and combining environmental information with skin electrical signals and brain electrical signals to determine the emotional state of the subject to be tested. The automation level and accuracy of emotion recognition are improved.

[0026] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0028] Figure 1A flowchart of multimodal information collection and processing provided by an embodiment of the present invention;

[0029] Figure 2 A flowchart of emotional state recognition and assessment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0031] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.

[0032] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0033] like Figure 1 and Figure 2 As shown, the present invention provides an emotion recognition system for multimodal information fusion, comprising:

[0034] Data acquisition module: including skin electrical sensors, EEG signal sensors, cameras and microphones. Skin electrical sensors and EEG signal sensors are used to collect the patient's physiological signals in real time, the camera is used to capture the patient's facial expressions, and the microphone is used to collect and store the user's voice information in real time. These sensors, cameras and microphones are all connected to the information processing module to transmit the collected data to the information processing module.

[0035] Information processing module: Receives data from the data acquisition module and performs preprocessing and feature extraction. Preprocessing includes filtering, denoising and other steps to improve the accuracy of the data. Feature extraction is to extract features related to the emotional state from the preprocessed data.

[0036] Emotional analysis module: Based on deep learning algorithms, the extracted features are used for emotional analysis. Convolutional neural network (CNN) is used to recognize facial expressions captured by the camera, long short-term memory network (LSTM) is used to process the voice text collected by the microphone, and combined with skin electrical signals and EEG signals to comprehensively judge the patient's emotional state.

[0037] Rehabilitation plan formulation module: Based on the results of the sentiment analysis module, this module combines rehabilitation medicine theory and expert experience to automatically generate a personalized rehabilitation plan. The rehabilitation plan includes rehabilitation goals, rehabilitation methods, rehabilitation time, etc., aiming to help patients adjust their emotional state and promote the rehabilitation process.

[0038] Voice comfort module: Provides real-time voice comfort according to the patient's emotional state during the rehabilitation process. When the system detects that the patient's emotional state is not good, it automatically plays a soothing voice to help the patient relieve tension, anxiety and other negative emotions.

[0039] 1. Multimodal information collection and processing

[0040] (1) Collection equipment design: Design and optimize multimodal information collection equipment, including physiological signal collection modules (such as hand skin electrical signals, brain waves, heart rate, etc.), behavioral information collection modules (such as facial expressions, voice, etc.), and environmental information collection modules (such as real-time weather, indoor and outdoor temperatures, and local time, etc.).

[0041] (2) Signal processing algorithms: Research and develop signal processing algorithms suitable for multimodal information, including filtering, denoising, feature extraction, etc., to improve the accuracy and reliability of information. The acquisition of user emotions will be based on the sentiment analysis model trained by the Inception model in CNN for simple facial image emotion recognition. If the user inputs the demand through voice or text, the multimodal emotion recognition model based on VGG-16 and BiLSTM will be used to integrate text and image modal information for emotion recognition.

[0042] 2. Multimodal Information Fusion

[0043] (1) Data-level fusion: Fusion is performed at the raw data level. Information from different modalities is aligned and synchronized through algorithms, providing a basis for subsequent feature extraction and emotion recognition.

[0044] (2) Feature-level fusion: After feature extraction, fusion is performed, using machine learning or deep learning algorithms to fuse features of different modalities to form a more comprehensive and accurate description of the patient's emotional state.

[0045] (3) Decision-level fusion: Fusion is performed at the output layer of the emotion recognition model to comprehensively judge the patient’s emotional state based on the recognition results of different modalities and improve recognition accuracy.

[0046] 3. Identification and assessment of emotional states

[0047] (1) Emotion recognition model: Based on the results of multimodal information fusion, an emotion recognition model is constructed to achieve automatic recognition and classification of the patient's emotional state.

[0048] (2) Emotional assessment system: Based on the identification results, an emotional assessment system is established to quantitatively evaluate the patient's emotional state and provide a basis for formulating a rehabilitation plan.

[0049] Specific system construction process:

[0050] By acquiring visual and auditory information, image processing technology based on expression analysis and fusion processing technology of speech semantic recognition, using OpenCV cross-platform computer vision and extracting face images, we can obtain text and image information. Using the long short-term memory classification model based on the attention mechanism and the convolutional neural network based on transfer learning, we can obtain the emotions of the two modes and fuse them to obtain the user's emotions.

[0051] (1) The user’s gender and age information will be obtained using the Yolov5 model for identification.

[0052] (2) The acquisition of user emotions will be based on the sentiment analysis model trained by the Inception model in CNN to perform simple facial image emotion recognition. If the user inputs their needs through voice or text, the multimodal emotion recognition model based on VGG-16 and BiLSTM will be used to integrate text and image modal information for emotion recognition.

[0053] (3) The user’s action information will be divided into normal actions and abnormal actions (such as falling, getting angry), and the recognition model trained by Yolov5 will be used for action recognition.

[0054] (4) Weather, temperature and time information will be obtained through real-time network data.

[0055] (5) First, the state library is constructed with [time, emotion, gender, age, course of disease, whether to fall, going to the toilet] as attributes, and the service library is constructed with [expression, tone, action, rehabilitation service panel, chat, emergency help] as attributes, and the initial case and service information are input. The ReliefF algorithm is used to calculate the weight of the attribute to the service and save the weight.

[0056] (6) For the collected real-time information, we first encode the information according to time, emotion, gender, age, course of illness, whether to fall, use the toilet, etc., and determine whether there is a completely identical case. If so, we use it directly. If not, we use the weight calculated in (5) to calculate the similarity. For each service attribute, we select the service of the case with the highest similarity to generate a new service combination. Then we take this case as a new case and save its status and service.

[0057] (7) For the collected text information, this paper uses regular expressions to analyze the customer’s special service needs and take them into account in the robot’s decision feedback.

[0058] (8) Based on the user's feedback after receiving the service, the service content weight of the service group applied to the user is modified. For example, if the user's score for the "smiley face" behavior in the expression service is not high, the weight of this behavior is reduced. When a similar user is encountered in the future and this case is called again, the expression behavior with a higher weight will be automatically adopted.

[0059] The convolution operation is expressed as:

[0060] Among them, f is the input image, g is the convolution kernel, * represents the convolution operation, and [u,v] is the position of the output feature map;

[0061] The pooling operation is expressed as:

[0062]

[0063] Among them, p is the output after pooling, f is the input feature map, and max represents the maximum value operation;

[0064] The output of the fully connected layer is calculated by the formula:

[0065] o=W·a+b

[0066] Where o is the output, W is the weight matrix, a is the input feature vector, and b is the bias term.

[0067] Beneficial effects: The accuracy of emotion recognition has been improved, and pain points such as limited medical resources, insufficient medical and nursing resources, and low emotional compliance of patients in rehabilitation have been solved through automation and intelligent methods, providing strong support for the emotional rehabilitation of stroke patients.

[0068] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal information fusion emotion recognition method, characterized in that: The emotion recognition method comprises: Collect data information of the test subjects in real time; Preprocessing and feature extracting the data information to obtain extracted features; The extracted features are used to recognize facial expressions and voice text using a convolutional neural network, and the emotional state of the person being tested is determined by combining environmental information with skin electrical signals and brain electrical signals.

2. The emotion recognition method based on multimodal information fusion according to claim 1, characterized in that: The data information specifically includes: physiological signals, behavioral information and external environment information of the person to be tested.

3. The emotion recognition method based on multimodal information fusion according to claim 2, characterized in that: The physiological signals specifically include: hand skin electrical signals, brain waves and heart rate.

4. The emotion recognition method based on multimodal information fusion according to claim 2, characterized in that: The behavior information specifically includes: facial expressions and voice.

5. The emotion recognition method based on multimodal information fusion according to claim 2, characterized in that: The external environment information specifically includes: real-time weather, indoor and outdoor temperatures, and local time.

6. The emotion recognition method based on multimodal information fusion according to claim 1, characterized in that: The preprocessing specifically includes: filtering, denoising and feature extraction.

7. The emotion recognition method based on multimodal information fusion according to claim 1, characterized in that: Features related to emotional states are extracted from the preprocessed data.

8. The emotion recognition method based on multimodal information fusion according to claim 1, characterized in that: The convolutional neural network specifically includes: The convolution operation is expressed as: Among them, f is the input image, g is the convolution kernel, * represents the convolution operation, and [u,v] is the position of the output feature map; The pooling operation is expressed as: Among them, p is the output after pooling, f is the input feature map, and max represents the maximum value operation; The output of the fully connected layer is calculated by the formula: o=W·a+b Where o is the output, W is the weight matrix, a is the input feature vector, and b is the bias term.

Images