A Multimodal Emotion Recognition Method and System for Companion Robots

Abstract

The present invention relates to a multi-modal emotion recognition method and system for companion robots. The method includes separately collecting facial expression pictures, voice signals, and electroencephalogram signals; extracting emotional feature vectors of facial expressions, emotional feature vectors of voices, and feature vectors of electroencephalogram signals; obtaining a weight matrix, multiplying each feature vector by the weight matrix to obtain a fused feature; classifying four common emotions, namely happiness, sadness, calmness, and disgust, through a support vector machine; and predicting emotional scores by evolving emotions into four dimensions of pleasure-tension-excitement-certainty and adopting a multivariate non-linear regression method. Compared with the prior art, the present invention has a closer emotion recognition ability to humans through information fusion; realizes the ability of autonomous evolution and continuous adjustment of emotion discrimination by using the form of dynamically updating weight parameters; and discrete and continuous emotion recognition can achieve a more scientific and in-depth description of emotional changes.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and more particularly to a multimodal emotion recognition method and system for companion robots. Background Art

[0002] Emotional interaction has received great attention in the research of human-machine natural interaction, and emotion recognition is the key to human-machine emotional interaction. The purpose of its research is to enable machines to perceive human emotional states and improve the humanization level of machines. Multimodal emotion recognition technology has broad application prospects and research value in the field of companion robots. By using various sensors carried by the robot, multimodal signals containing potential emotional features such as human facial expressions, behaviors, voices, and physiological signals are obtained. Based on deep learning methods, feature extraction and fusion are realized, and human emotions are analyzed and predicted, enabling companion robots to have stronger emotion recognition and emotion understanding capabilities.

[0003] Currently, emotion recognition devices applicable to companion robots usually conduct a detailed analysis and identification of the changes in waveforms such as physiological characteristics, postures, gestures, and intonations generated by human emotional changes through systems such as chips, videos, and audios, and then deeply understand human emotions and list clear and timely responses. Emotion recognition based on facial expressions usually uses two-dimensional images for analysis and research, that is, geometric methods based on the position features of each organ and protrusions on the face, pixel methods based on facial texture features, and hybrid methods combining the two are used for emotion feature representation. Emotion recognition methods based on speech usually extract and summarize the prosodic information and voice quality features in speech signals, including Mel Frequency Cepstral Coefficients (MFCC), Teager energy operator, etc., and realize emotion classification and recognition through support vector machines or long short-term memory networks. In terms of physiological signals, the signal frequency bands and brain regions most related to emotions, the temporal stability characteristics of electroencephalograms, etc. are used to understand emotions based on traditional machine learning methods and spiking neural networks. Currently, some research integrates features from various behavioral and physiological manifestations in an emotion recognition framework for emotion recognition. For example, inferring the mental state corresponding to behavior from joint head movements and facial expressions, and then reasoning about human emotional expressions. At the same time, there are also discrete emotion classification and recognition by combining facial expressions and speech signals and using the expression subspace between shared modalities.

[0004] Most existing companion robots lack emotion recognition capabilities. Some robots equipped with specific sensors can only perform simple emotion recognition based on a single modality. Emotion recognition technology based solely on facial expressions or voice signals fails to consider the complementary and synergistic effects of emotional expressions captured by different modalities. When corresponding emotional information is interfered with or insufficiently captured, emotion recognition efficiency is often low, failing to meet the application needs of emotional interaction. Furthermore, existing technologies mostly focus on identifying the outward characteristics of human emotional expression, failing to consider the importance of monitoring and detecting physiological signals for emotion recognition. EEG and neural signals can accurately, objectively, and in real time reflect abnormal emotions and changes in psychological state, helping companion robots analyze emotions for different companion groups and achieve precise emotional comfort.

[0005] In recent years, emotion recognition modules on companion robots have often been limited to simple preprocessing of acquired modal information, leading to frequent data loss and errors. Furthermore, multimodal data fusion typically relies on dataset fusion, a complex and cumbersome process without guaranteeing data integrity, resulting in significant waste of data resources. Furthermore, most traditional methods employ discrete emotion recognition strategies, failing to fully account for the continuity and heterogeneity of human emotions. Consequently, emotion recognition performance is often subpar.

[0006] In summary, developing methods based on multimodal feature acquisition and expression, utilizing multimodal emotion representation data of facial expressions, voice and physiological signals, fully expressing the discriminative power of heterogeneous features, overcoming the existing difficulties in single-modal data research, and constructing a multimodal emotion recognition system suitable for companion robots have become urgent problems to be solved by technicians in this research field. Summary of the Invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a multimodal emotion recognition method and system for companion robots that utilizes multimodal emotion representation data of facial expressions, voice and physiological signals to fully express the discriminative power of heterogeneous features.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] A multimodal emotion recognition method for a companion robot, comprising the following steps:

[0010] Collect facial expression pictures, speech signals and EEG signals respectively;

[0011] Extract the emotional feature vector of the facial expression from the facial expression picture, extract the emotional feature vector of the speech from the speech signal, and extract the feature vector of the EEG signal from the EEG signal;

[0012] Obtain a weight matrix, multiply the emotional feature vector of the facial expression, the emotional feature vector of the speech, and the feature vector of the EEG signal by the weight matrix to obtain a fused feature;

[0013] Load the fused feature into a pre-constructed and trained classification model for classification and recognition to obtain multiple discrete emotional label recognition results. The weight matrix is also trained during the training process of the classification model;

[0014] Perform emotional prediction based on the fused feature. This emotional prediction is used to perform data fitting training on the fused feature to obtain a continuous emotional intensity value. The emotional intensity value is divided into multiple emotional dimensions, and the multiple emotional dimensions include pleasantness, tension, excitement, and certainty.

[0015] Further, the extraction of the emotional feature vector of the facial expression specifically includes:

[0016] First, use the Adaboost algorithm to extract the Haar features in the facial expression picture, construct a Haar feature map, then preprocess the Haar feature map through histogram equalization, and then use the uniform pattern LBP algorithm to extract the emotional feature vector of the facial expression.

[0017] Further, the extraction process of the uniform pattern LBP algorithm includes:

[0018] Construct a texture region of size 3*3, with the threshold being the central pixel value of the texture region. Compare the surrounding 8 pixel values with the threshold. If the value is greater than the value of the threshold pixel, the area where the pixel is located is set to 1; if the value is less than the value of the threshold pixel, the area where the pixel is located is set to 0. In the 3*3 texture region, form an 8-bit binary number with the values generated by its adjacent 8 pixel points in a clockwise direction, and count the number of jumps from 0 to 1 or from 1 to 0 in the 8-bit binary number. If the number of jumps is within two, the decimal number corresponding to the binary number is the LBP value of the center of the 3*3 neighborhood; if the number of jumps is greater than 2, take P = 8, then set the LBP value of the center of the region to P + 1 = 9; traverse all pixel points to obtain the LBP values of the entire image, and connect all the LBP values in sequence to form a feature vector, which is the emotional feature vector of the facial expression.

[0019] Further, the extraction of the emotional feature vector of the speech specifically includes:

[0020] First, the speech signal is windowed and smoothed using a Hamming window, and the time domain signal is converted to the frequency domain for subsequent spectral analysis. Then, a high-pass filter is designed to eliminate noise interference from vocal cord pronunciation and perform MFCC feature extraction. The Fourier transformed spectrogram is input into a pre-built and trained convolutional neural network layer to extract the spectrogram features and obtain the emotional feature vector of the speech.

[0021] Furthermore, the extracting of the feature vector of the EEG signal specifically includes:

[0022] Firstly, the EEG signal is preprocessed and denoised, and then the fractal dimension features and multi-scale entropy features are extracted respectively to construct the feature vector of the EEG signal.

[0023] Furthermore, the pre-processing denoising includes:

[0024] The EEG signals of the subject are collected at a fixed sampling frequency, and then the db5 wavelet is selected for multi-layer wavelet decomposition. The wavelet packet coefficients generated by the noise are set to zero using the soft threshold method, and finally the EEG signals are reconstructed.

[0025] Furthermore, the process of extracting the fractal dimension features includes:

[0026] The original sequence is uniformly sampled to obtain K sequences. The variation of each element in the K sequences is calculated to construct a new sequence. The new sequence is fitted to obtain the slope, and the inverse of the slope is taken as the initial FD feature. The EEG signal after preprocessing and denoising is used for window processing, and then the window is used to divide the data into multiple segments. The fractal dimension features are extracted from each segment of the data.

[0027] The multi-scale entropy feature extraction process includes:

[0028] The multi-scale entropy of the EEG signal is calculated and the average multi-scale entropy value of the subject's happy and sad emotions is obtained. Then the multi-scale entropy at the first one or more scales is selected as the feature vector of the EEG signal.

[0029] Furthermore, before the classification and recognition, the method also includes: normalizing the emotional feature vectors of the facial expressions, the emotional feature vectors of the voice, and the feature vectors of the EEG signals respectively to obtain fusion features for classification and recognition; the classification model adopts an SVM classification model, and the kernel function of the SVM classification model is an RBF kernel function; the emotional labels include happiness, sadness, calmness, and disgust.

[0030] Furthermore, the modal attention is calculated through the self-attention mechanism to construct the weight matrix and obtain the fusion weight. The calculation expression of the modal attention is:

[0031]

[0032] Where A is the modality attention, (·) is the matrix multiplication, Θ is the query matrix, Φ is the key matrix, T is the transposed symbol, and d is the embedding dimension;

[0033] The query matrix Θ is constructed by connecting the feature vectors of each modality through the first fully connected layer. The connection formula of the first fully connected layer is y1=w1x+b1. Finally, the feature matrix is output through the activation function to form the query matrix Θ. The query matrix Θ is used to represent the influence of the current modality itself on other modalities.

[0034] The construction process of the bond matrix Φ is as follows: the eigenvectors of each mode are connected through the second fully connected layer. The connection formula of the second fully connected layer is y2=w2x+b2. Finally, the output is output through the activation function to obtain the feature matrix to form the bond matrix Φ; the bond matrix Φ is used to represent the influence of other modes on the current mode itself;

[0035] Add the elements of each row in the modal attention A to get the weight of modality i. The corresponding calculation formula is:

[0036] Ψ i =∑ k a ki

[0037] Where, i is the weight of mode i, a ki is the element in row k and column i in the modal attention A;

[0038] The sum of the weights of all modes is 1, which satisfies:

[0039] ∑ i Ψ i =1.

[0040] The modality attention A is trained together with the classification model training process to adjust the parameters w1, b1, w2 and b2 in the first and second fully connected layers.

[0041] The present invention also provides a multimodal emotion recognition system for a companion robot, comprising:

[0042] Multimodal acquisition module, used to collect facial expression images, voice signals and EEG signals respectively;

[0043] A facial expression-based emotion analysis module, configured to extract an emotion feature vector of the facial expression based on the facial expression image;

[0044] A sentiment analysis module based on a speech signal, configured to extract a speech sentiment feature vector based on the speech signal;

[0045] An electroencephalogram (EEG)-based emotion analysis module for extracting a feature vector of an EEG signal according to the EEG signal;

[0046] A feature fusion module based on a self-attention mechanism for obtaining a weight matrix, multiplying the emotion feature vector of the facial expression, the emotion feature vector of the speech, and the feature vector of the EEG signal by the weight matrix to obtain a fusion feature;

[0047] An identification module based on discrete emotion classification for loading the fusion feature into a pre-constructed and trained classification model for classification and identification to obtain multiple discrete emotion label recognition results, and also for training the weight matrix during the training process of the classification model;

[0048] A prediction module based on continuous emotion for performing emotion prediction according to the fusion feature, where the emotion prediction is used to perform data fitting training on the fusion feature to obtain a continuous emotion intensity value, and the emotion intensity value is divided into multiple emotion dimensions, and the multiple emotion dimensions include pleasantness, tension, excitement, and certainty.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] Different from the previous single-modal emotion recognition system, the present invention fully combines facial expressions, speech signals, and EEG signals for emotion analysis and discrimination. Through multi-modal information fusion, it can enhance the ability to enlarge emotion features and improve the mapping ability of the emotion representation space, enabling the robot to exhibit a more human-like emotion recognition ability.

[0051] At the same time, the multi-modal fusion mode based on the self-attention mechanism designed by the present invention has self-adaptability and flexibility. By combining the advantages of emotion expressions of different heterogeneous modalities and using the form of dynamically updating weight parameters, the robot is enabled to have the ability of autonomous evolution and continuous adjustment of emotion discrimination, which greatly provides a new paradigm for human-robot emotion interaction.

[0052] In addition, the joint processing of the emotion recognition mode based on discrete emotion classification and continuous emotion dimension prediction effectively depicts the multi-modal emotion state, can more scientifically and deeply describe the emotion changes, and has obvious advantages for the robot to widely understand human emotions. The system reduces a large amount of information loss when facing complex non-linear multi-modal information processing, and performs well when processing data with a large modal span. Description of the Drawings

[0053] Figure 1 It is a principle block diagram of a multi-modal emotion recognition system for a companion robot provided in an embodiment of the present invention;

[0054] Figure 2This is a functional block diagram of a facial expression-based emotion analysis module provided in an embodiment of the present invention;

[0055] Figure 3 This is a functional block diagram of a voice signal-based emotion analysis module provided in an embodiment of the present invention;

[0056] Figure 4 This is a functional block diagram of an EEG-based emotion analysis module provided in an embodiment of the present invention;

[0057] Figure 5 This is a functional block diagram of a feature fusion module based on a self-attention mechanism provided in an embodiment of the present invention;

[0058] Figure 6 This is a principle block diagram of a recognition module based on discrete emotion classification provided in an embodiment of the present invention;

[0059] Figure 7 This is a principle block diagram of a continuous emotion-based prediction module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0061] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0062] Example 1

[0063] like Figure 1 As shown, this embodiment provides a multimodal emotion recognition system for a companion robot, including:

[0064] Multimodal acquisition module, used to collect facial expression images, voice signals and EEG signals respectively;

[0065] A facial expression-based emotion analysis module, configured to extract an emotion feature vector of the facial expression based on the facial expression image;

[0066] A sentiment analysis module based on a speech signal, configured to extract a speech sentiment feature vector based on the speech signal;

[0067] An emotional analysis module based on electroencephalogram (EEG) signals, which is used to extract the feature vectors of EEG signals according to the EEG signals;

[0068] A feature fusion module based on the self-attention mechanism, which is used to obtain a weight matrix, multiply the emotional feature vectors of facial expressions, the emotional feature vectors of voices, and the feature vectors of EEG signals by the weight matrix to obtain fused features;

[0069] An identification module based on discrete emotion classification, which is used to load the fused features into a pre-constructed and trained classification model for classification and identification to obtain multiple discrete emotion label recognition results, and the weight matrix is also trained during the training process of the classification model;

[0070] A prediction module based on continuous emotion, which is used to perform emotion prediction according to the fused features. This emotion prediction is used to perform data fitting training on the fused features to obtain continuous emotion intensity values, and the emotion intensity values are divided into multiple emotion dimensions, and the multiple emotion dimensions include pleasantness, tension, excitement, and certainty;

[0071] A display module, which is used to display the results of discrete emotion classification and continuous emotion prediction in real time.

[0072] It can be summarized that the multi-modal information comes from the facial expressions, voice signals, and EEG signals acquired by the companion robot through the acquisition module. In the recognition module, the preprocessing, data feature extraction, and modal output of multi-modal information are realized through different modal recognition units. In the fusion module, the self-attention mechanism is introduced to calculate the attention weight coefficients for different modalities, so as to realize the modal fusion at the feature level. After passing through the emotion classification and prediction module, the multi-modal emotion recognition results are displayed using the display module.

[0073] The following is a specific description of each module.

[0074] 1. Multi-modal acquisition module

[0075] The multi-modal acquisition module includes an emotional acquisition device for facial expressions, an emotional acquisition device for voice signals, an emotional acquisition device for EEG signals, and an emotional acquisition device for physiological signals, etc., and collects multi-modal data of the companion subject targeted by the companion robot through multiple sensors.

[0076] 2. Emotional analysis module based on facial expressions

[0077] Figure 2 For the principle block diagram of the emotional analysis module based on facial expressions provided by the present invention, this module includes four units: face detection, image and processing, feature extraction, and modal output. Based on different units, the specific steps are as follows:

[0078] (1) First, use Adaboost combined with Haar features to achieve object detection. The Adaboost algorithm in it extracts Haar-like features of the face, including rectangular features of the input image. Haar features are a kind of feature reflecting the gray-scale change of the image. The feature template is composed of black and white rectangles, and the Haar feature value of the template is calculated by subtracting the sum of the pixels in the white rectangle area from the sum of the pixels in the black rectangle area. Common features include edge features, line features, center surround features, and diagonal features. There are color differences between the facial features and the face, so Haar features can be used to describe the gray-scale change of the face. To achieve fast calculation, the integral image method is used. The integral image can quickly calculate the sum of the pixels in any rectangular area of the image, so as to quickly calculate the Haar features of the image.

[0079] (2) Image preprocessing can restore the useful information in the image and reduce the irrelevant information in the image. Here, histogram equalization is used to make the histogram distribution of the image become approximately uniform distribution, so as to enhance the contrast of the image.

[0080] (3) In the feature extraction unit, considering the low complexity requirement of the system applicable to companion robots, the uniform pattern LBP algorithm is used to extract facial expression features. By constructing a 3*3 texture area, the threshold is the central pixel value of the texture area, and the surrounding 8 pixel values are compared with the threshold. If the value is greater than the threshold pixel value, the neighborhood is set to 1; if the value is less than the threshold pixel value, the neighborhood is set to 0. In the 3*3 area, the values generated by its adjacent 8 pixel points are combined into an 8-bit binary number in clockwise order. Count the number of jumps from 0 to 1 or from 1 to 0 in the 8-bit binary number. If the number of jumps is within two, the decimal number corresponding to the binary number is the LBP value of the center of the 3*3 neighborhood; if the number of jumps is greater than 2, take P = 8, then the LBP value of the center of the area is set to P + 1 = 9; traverse all pixel points to obtain the LBP values of the entire image, and connect all the LBP values in order to form a feature vector, which is the uniform pattern LBP feature of the entire image.

[0081] (4) After obtaining the emotional feature vector of the facial expression, it is pre-stored and output through the modality output unit.

[0082] 3. Emotional analysis module based on voice signals

[0083] Figure 3 The principle block diagram of the emotional analysis module based on voice signals provided by the present invention mainly includes data preprocessing, MFCC feature extraction, spectrogram feature extraction, and modality output unit. The basic steps of the unit are as follows:

[0084] (1) In order to make the acquired original signal smoother, the signal is windowed and smoothed using a Hamming window, and the time domain signal is converted to the frequency domain for subsequent spectrum analysis.

[0085] (2) In the MFCC feature extraction stage, the speech signal is converted to Mel frequency and Hertz frequency, and then the inverse frequency analysis is performed. The idea of this method is that when people speak, there is other interference caused by the lips and vocal cords. In order to solve this interference, the high-frequency part of the step-length speech signal is suppressed by the pronunciation system and the high-frequency resonance peak is highlighted. For this purpose, a high-pass filter is added. By adding coefficients in the frequency domain, which are positively correlated with the frequency, the amplitude of the high frequency is increased.

[0086] (3) In the spectrogram feature extraction unit, the Fourier transformed spectrogram is input into the convolutional neural network layer to extract spectrogram features. This process includes an input layer, a pooling layer, a convolution layer, and a fully connected layer. The CNN structure for spectrogram feature extraction includes an input layer, two convolution layers, two pooling layers, and a fully connected layer. The input image size is 128*128 pixels. The first convolution layer consists of 64 5*5 convolution kernels. After convolution, the nonlinear unit ReLU activation function is introduced. After convolution, a pooling layer consisting of 2*2 convolution kernels is connected. The purpose of connecting this layer is to reduce computational complexity and extract the main features. The second convolution layer consists of 128 5*5 convolution kernels. Then connect the ReLU activation function, connect a 2*2 pooling layer after the second convolutional layer, and output 128 pixel feature maps of size 32*32 after the second pooling layer. Finally, connect to a fully connected layer consisting of 512 neurons, and finally obtain a 512-dimensional feature vector.

[0087] (4) After obtaining the emotional feature vector of the speech, the modality is pre-stored and output through the modality output unit.

[0088] 4. Emotional analysis module based on EEG signals

[0089] Figure 4 This is a block diagram of the principle of the EEG-based emotion analysis module provided by the present invention. The module mainly includes pre-processing denoising, fractal dimension feature extraction, multi-scale entropy feature extraction, and modal output unit. The specific steps are as follows:

[0090] (1) First, preprocess and denoise the EEG signals. The signals are from the acquisition module, removing the EEG signals of the FP1 channel of the companion. The length is at a sampling rate of 128 Hz, and the signal acquisition time length is 63 s. Remove the first 3 s of baseline time, so the signal time is 60 s, and the total number of sampling points is 7,680. The system selects the db5 wavelet for 5-layer wavelet decomposition, then uses the soft threshold method to set the wavelet packet coefficients generated by noise to zero, and finally completes the reconstruction of the EEG signals. The EEG signals before and after preprocessing can be smoother and are suitable for further processing.

[0091] (2) In the fractal dimension feature extraction part, fix the length of the EEG signal as N, then uniformly sample different K values, then construct new K sequences, and then select the intervals between adjacent two numbers from the K sequences to construct new sequences. Then calculate the lengths of all new sequences, and use the least squares method to fit to obtain the slope. Finally, the opposite of the slope is the required initial FD feature. Next, perform window processing on the EEG signals after wavelet threshold denoising. Use a window with a length of 256 points to segment the data, and the data can be divided into 30 segments. Extract the fractal dimension features for each segment of data, and a 30-dimensional feature vector can be obtained.

[0092] (3) In the multi-scale entropy feature extraction unit, multi-scale entropy analyzes the complexity of the time series from different time scales. To calculate the sample entropy at different time scales, it is necessary to perform coarse-graining processing on the original signal. Coarse-graining is to segment the original signal using non-overlapping windows with a length of i. The sample entropy values obtained at different scales will also be different, and the dimensions of the constructed multi-scale entropy features will also be different. This system selects the EEG signals of the FP1 channel of the companion detector, extracts all the EEG data obtained from their experiments, and determines the samples with a Valence dimension greater than 6 in the self-evaluation as happy emotions, and those less than 4 as sad emotions. Calculate the multi-scale entropy of the EEG signals and find the average multi-scale entropy values of happy and sad emotions in the experiment. Then select the multi-scale entropy at the first few scales as the feature vector of the EEG signals, and a total of 15-dimensional multi-scale entropy features can be obtained.

[0093] After obtaining the fractal dimension features and multi-scale entropy features, perform modal pre-storage and output through the modal output unit.

[0094] (4) After obtaining the fractal dimension features and multi-scale entropy features, perform modal pre-storage and output through the modal output unit.

[0095] 5. Feature Fusion Module Based on Self-Attention Mechanism

[0096] Figure 5This is the principle block diagram of the feature fusion module based on the self-attention mechanism provided by the present invention. The main process of the attention calculation mechanism is information input, calculation of the attention distribution, and calculation of the weighted average of the input information. After extracting multi-modal features such as facial expressions, speech, and EEG signals, effective fusion is carried out. First, a weight matrix is initialized to represent the weight values of each modal feature. During the feature fusion process, the weight values of each modality are multiplied by the corresponding eigenvector, and after multiplication, they are cascaded. And during the training process of the entire system model, this weight matrix will follow the training of the system model, and the corresponding values will also be continuously transformed and adjusted. Compared with the manually fixed weight values, the effect is better.

[0097] The feature fusion module includes the interaction between two types of inputs: α ii They are the self-attention interactions of modality i. α ij Is the attention interaction between modalities, reflecting the influence of modality i on modality j. The calculation of modal attention is as follows:

[0098]

[0099] In the formula, A is the modal attention, (·) is the matrix multiplication, Θ is the query matrix, Φ is the key matrix, T is the transpose symbol, and d is the embedding dimension;

[0100] The construction process of the query matrix Θ is as follows: The feature vectors of each modality are connected through the first fully connected layer. The connection formula of this first fully connected layer is y1 = w1x + b1, and finally, it is output through the activation function to obtain the feature quantity matrix, which constitutes the query matrix Θ;

[0101] The construction process of the key matrix Φ is as follows: The feature vectors of each modality are connected through the second fully connected layer. The connection formula of this second fully connected layer is y2 = w2x + b2, and finally, it is output through the activation function to obtain the feature quantity matrix, which constitutes the key matrix Φ;

[0102] Add the elements of each row in the modal attention A to obtain the weight of modality i. The corresponding calculation formula is:

[0103] Ψ i = ∑ k a ki

[0104] In the formula, Ψ i is the weight of modality i, and a ki is the element in the k-th row and the i-th column of the modal attention A;

[0105] The sum of the weights of all modalities with high weights is 1, that is, it satisfies:

[0106] ∑ i Ψ i=1.

[0107] The modality attention A is trained together with the training process of the SVM classification model to adjust the parameters w1, b1, w2 and b2 in the first fully connected layer and the second fully connected layer.

[0108] 6. Recognition module based on discrete emotion classification

[0109] Figure 6 The principle block diagram of the recognition module based on discrete emotion classification provided by the present invention, after calculating the multimodal feature vector, it is usually necessary to normalize the feature vector. When using data for SVM classification, the normalized data is compared with the original data, not only shortening the training time, but also improving the test accuracy. Data normalization makes the data more compact, which is conducive to the derivation of the optimal classification hyperplane. The system uses svm-scale to scale the data accordingly, scaling the data size to [0,1] or [-1,1]. The purpose of scaling is to prevent a certain feature from being too large or too small, speeding up the calculation speed, and facilitating the training of the model. The system selects the RBF kernel function as the kernel function of the SVM classification algorithm. RBF, as a special case corresponding to nonlinear mapping, can handle nonlinear separable problems and is suitable for processing multidimensional vectors. After model training, the recognition module will correctly output one of the four corresponding emotional labels of happiness, sadness, calmness and disgust.

[0110] 7. Prediction module based on continuous emotion

[0111] Figure 7 This is a principle block diagram of the continuous emotion prediction module provided by the present invention. In the module, the emotion prediction of the continuous dimension is first defined as pleasure-tension-excitement-certainty, and then the emotion prediction is defined as a value range of 0-10 through standard quantization. Then, the multimodal emotion features are trained by data fitting using the multivariate nonlinear regression method. Finally, the module will output the emotion intensity values corresponding to the four different dimensions.

[0112] The multimodal emotion recognition system for companion robots demonstrated in this embodiment effectively solves the problem of emotional interaction and recognition of companion robots. By fully collecting multimodal information such as facial expressions, voice signals, and difficult signals, it combines different recognition modules to achieve preprocessing and feature extraction of heterogeneous data, and further realizes multimodal information fusion at the feature layer based on the self-attention mechanism. Unlike traditional robot-mounted emotion recognition systems, this invention fully combines discrete emotion classification and continuous dimensional emotion prediction, fully depicting the subject's emotional feature space. It can comprehensively obtain recognized emotional feedback from the system and predict the subject's emotional changes, greatly improving the accuracy of emotion recognition.

[0113] Example 2

[0114] This embodiment provides a multi-modal emotion recognition method for a companion robot. This method corresponds to the processing procedures of each module in the multi-modal emotion recognition system in Embodiment 1. The method specifically includes the following steps:

[0115] Collect facial expression images, voice signals, and electroencephalogram (EEG) signals respectively;

[0116] Extract the emotion feature vector of the facial expression according to the facial expression image, extract the emotion feature vector of the voice according to the voice signal, and extract the feature vector of the EEG signal according to the EEG signal;

[0117] Obtain a weight matrix, multiply the emotion feature vector of the facial expression, the emotion feature vector of the voice, and the feature vector of the EEG signal by this weight matrix to obtain a fusion feature;

[0118] Load the fusion feature into a pre-constructed and trained classification model for classification recognition to obtain multiple discrete emotion label recognition results. The weight matrix is also trained during the training process of the classification model;

[0119] Perform emotion prediction according to the fusion feature. This emotion prediction is used to perform data fitting training on the fusion feature to obtain a continuous emotion intensity value. This emotion intensity value is divided into multiple emotion dimensions, and the multiple emotion dimensions include pleasantness, tension, excitement, and certainty.

[0120] The specific process of extracting the emotion feature vector of the facial expression includes:

[0121] First, use the Adaboost algorithm to extract Haar features in the facial expression image, construct a Haar feature map, then preprocess the Haar feature map through histogram equalization, and then use the local binary pattern (LBP) algorithm with uniform pattern to extract the emotion feature vector of the facial expression.

[0122] The extraction process of the LBP algorithm with uniform pattern includes:

[0123] Construct a texture area of size 3*3, with the threshold being the central pixel value of the texture area. Compare the values of the surrounding 8 pixels with the threshold. If the value is greater than the threshold pixel value, the neighborhood is set to 1; if the value is less than the threshold pixel value, the neighborhood is set to 0. In the 3*3 texture area, the values generated by the 8 adjacent pixels are combined into an 8-bit binary number in a clockwise direction. Count the number of transitions from 0 to 1 or from 1 to 0 in the 8-bit binary number. If the number of transitions is less than two, the decimal number corresponding to the binary number is the LBP value of the center of the 3*3 neighborhood. If the number of transitions is greater than 2, take P=8, and set the LBP value of the center of the area to P+1=9. Traverse all pixel points to obtain the LBP value of the entire image, and connect all LBP values in sequence into a feature vector, which is the emotional feature vector of the facial expression.

[0124] The extraction of the emotion feature vector of the speech specifically includes:

[0125] First, the speech signal is windowed and smoothed using a Hamming window, converting the time-domain signal to the frequency domain for subsequent spectral analysis. A high-pass filter is then designed to eliminate vocal noise interference and extract MFCC features. The Fourier-transformed spectrogram is then fed into a pre-built and trained convolutional neural network layer to extract spectrogram features and generate the speech's emotional feature vector.

[0126] The extraction of the feature vector of the EEG signal specifically includes:

[0127] Firstly, the EEG signal is preprocessed and denoised, and then the fractal dimension features and multi-scale entropy features are extracted respectively to construct the feature vector of the EEG signal.

[0128] The pre-processing denoising includes:

[0129] The EEG signals of the subject are collected at a fixed sampling frequency, and then the db5 wavelet is selected for multi-layer wavelet decomposition. The wavelet packet coefficients generated by the noise are set to zero using the soft threshold method, and finally the EEG signals are reconstructed.

[0130] The extraction process of the fractal dimension feature includes:

[0131] After analyzing the original sequence, a new sequence signal is sampled and the EEG signal after preprocessing and denoising is used for window processing. The data is then divided into multiple segments using the window, and the fractal dimension features are extracted based on each segment of the data.

[0132] The multi-scale entropy feature extraction process includes:

[0133] Calculate the multi-scale entropy of the EEG signal and find the average multi-scale entropy values in the happy and sad emotions of the subject. Subsequently, select the multi-scale entropy at the previous or multiple scales as the feature vector of the EEG signal.

[0134] Before the classification and recognition, it further includes: after normalizing the emotional feature vectors of the facial expressions, the emotional feature vectors of the speech, and the feature vectors of the EEG signals respectively, obtaining the fusion features and performing classification and recognition; the classification model uses an SVM classification model, and the kernel function of this SVM classification model is an RBF kernel function; the emotional labels include happy, sad, calm, and disgusted.

[0135] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field of the present invention based on the concept of the present invention through logical analysis, reasoning, or limited experiments on the basis of the prior art should be within the protection scope determined by the claims.

Claims

1. A multimodal emotion recognition method for companion robots, characterized in that, It includes the following steps: Collect facial expression pictures, voice signals, and electroencephalogram (EEG) signals respectively; Extract the emotional feature vector of the facial expression according to the facial expression picture, extract the emotional feature vector of the voice according to the voice signal, and extract the feature vector of the EEG signal according to the EEG signal; Obtain a weight matrix, multiply the emotional feature vector of the facial expression, the emotional feature vector of the voice, and the feature vector of the EEG signal by the weight matrix to obtain a fused feature; Load the fused feature into a pre-constructed and trained classification model for classification and recognition to obtain multiple discrete emotional label recognition results. The weight matrix is also trained during the training process of the classification model; Perform emotional prediction according to the fused feature. This emotional prediction is used to perform data fitting training on the fused feature to obtain a continuous emotional intensity value. This emotional intensity value is divided into multiple emotional dimensions, and the multiple emotional dimensions include pleasantness, tension, excitement, and certainty; Calculate modal attention through the self-attention mechanism, thereby constructing the weight matrix to obtain the fusion weight. The calculation expression of the modal attention is: In the formula, A is the modal attention, (·) is matrix multiplication, Θ is the query matrix, Φ is the key matrix, T is the transpose symbol, and d is the embedding dimension; The construction process of the query matrix Θ is as follows: Connect the feature vectors of each modality through the first fully connected layer. The connection formula of the first fully connected layer is y1 = w1x + b1, and finally output through the activation function to obtain a feature quantity matrix, which constitutes the query matrix Θ; This query matrix Θ is used to represent the influence of the current modality itself on other modalities; The construction process of the key matrix Φ is as follows: Connect the feature vectors of each modality through the second fully connected layer. The connection formula of the second fully connected layer is y2 = w2x + b2, and finally output through the activation function to obtain a feature quantity matrix, which constitutes the key matrix Φ; This key matrix Φ is used to represent the influence of other modalities on the current modality itself; Add the elements of each row in the modal attention A to obtain the weight of modality i. The corresponding calculation formula is: Ψ i = ∑ k a ki where Ψ i is the weight of mode i, and a ki is the element in the k-th row and i-th column of the modal attention A; The sum of the weights of all modalities with high weights is 1, that is, it satisfies: ∑ i Ψ i = 1 The modal attention A is trained jointly with the training process of the classification model to adjust the parameters w1, b1, w2, and b2 in the first fully connected layer and the second fully connected layer.

2. The multimodal emotion recognition method for a companion robot according to claim 1, wherein, The specific process of extracting the emotional feature vector of the facial expression includes: First, use the Adaboost algorithm to extract Haar features in the facial expression picture, construct a Haar feature map, then preprocess the Haar feature map through histogram equalization, and then use the uniform pattern local binary pattern (LBP) algorithm to extract the emotional feature vector of the facial expression.

3. A multimodal emotion recognition method for a companion robot according to claim 2, characterized in that The extraction process of the uniform pattern LBP algorithm includes: Construct a texture region of size 3*3, with the threshold being the central pixel value of the texture region. Compare the surrounding 8 pixel values with the threshold. If the value is greater than the threshold pixel value, the area where the pixel is located is set to 1; if the value is less than the threshold pixel value, the area where the pixel is located is set to 0. In the 3*3 texture region, the values generated by its adjacent 8 pixel points are composed into an 8-bit binary number in a clockwise direction. Count the number of transitions from 0 to 1 or from 1 to 0 in the 8-bit binary number. If the number of transitions is within two, the decimal number corresponding to this binary number is the LBP value of the center of the 3*3 neighborhood; if the number of transitions is greater than 2, take P = 8, and then set the LBP value of the center of this region to P + 1 = 9. Traverse all pixel points to obtain the LBP values of the entire image, and connect all the LBP values in sequence to form a feature vector, which is the emotional feature vector of the facial expression.

4. A multimodal emotion recognition method for a companion robot according to claim 1, characterized in that, The extraction of the emotional feature vector of speech specifically includes: First, perform windowing processing on the speech signal, use a Hamming window for smoothing processing, and transform the time-domain signal to the frequency domain for subsequent spectral analysis; then design a high-pass filter to eliminate the noise interference of vocal cord pronunciation and perform MFCC feature extraction; input the spectrogram after Fourier transform into a pre-constructed and trained convolutional neural network layer to extract spectrogram features and obtain the emotional feature vector of speech.

5. A multimodal emotion recognition method for a companion robot according to claim 1, characterized in that The extraction of the feature vector of electroencephalogram signals specifically includes: First, perform preprocessing and denoising on the electroencephalogram signals, and then extract fractal dimension features and multi-scale entropy features respectively to construct the feature vector of electroencephalogram signals.

6. A multimodal emotion recognition method for a companion robot according to claim 5, characterized in that, The preprocessing and denoising include: Collect the electroencephalogram signals of the subject through a fixed sampling frequency, then select db5 wavelet for multi-level wavelet decomposition, and then use the soft threshold method to set the wavelet packet coefficients generated by noise to zero, and finally complete the reconstruction of the electroencephalogram signals.

7. A multimodal emotion recognition method for a companion robot according to claim 6, characterized in that The extraction process of the fractal dimension features includes: Perform uniform sampling on the original sequence to obtain K sequences, calculate the change amounts of the elements in the K sequences, construct a new sequence, fit this new sequence to obtain the slope, and take the opposite of this slope as the initial FD feature; use the preprocessed and denoised electroencephalogram signals for window processing, and then divide the data into multiple segments with the window, and extract fractal dimension features according to each segment of data respectively; The extraction process of the multi-scale entropy features includes: Calculate the multi-scale entropy of the electroencephalogram signals and find the average multi-scale entropy values in the happy and sad emotions of the subject. Then select the multi-scale entropy at the previous or multiple scales as the feature vector of the electroencephalogram signals.

8. A multimodal emotion recognition method for a companion robot according to claim 1, characterized in that, Before the classification and recognition, it also includes: respectively perform data normalization on the emotional feature vector of the facial expression, the emotional feature vector of speech, and the feature vector of electroencephalogram signals to obtain the fusion features for classification and recognition; the classification model uses an SVM classification model, and the kernel function of this SVM classification model is the RBF kernel function; the emotional labels include happy, sad, calm, and disgust.

9. A system adopting a multimodal emotion recognition method for a companion robot as described in any one of claims 1-8, characterized in that, It includes: A multi-modal acquisition module for respectively acquiring facial expression pictures, speech signals, and electroencephalogram signals; An emotion analysis module based on facial expressions, which is used to extract an emotion feature vector of a facial expression according to the facial expression picture; An emotion analysis module based on voice signals, which is used to extract an emotion feature vector of voice according to the voice signals; An emotion analysis module based on electroencephalogram signals, which is used to extract a feature vector of electroencephalogram signals according to the electroencephalogram signals; A feature fusion module based on self-attention mechanism, which is used to obtain a weight matrix, multiply the emotion feature vector of the facial expression, the emotion feature vector of the voice and the feature vector of the electroencephalogram signals by the weight matrix to obtain a fusion feature; An identification module based on discrete emotion classification, which is used to load the fusion feature into a pre-constructed and trained classification model for classification and identification to obtain multiple discrete emotion label identification results, and the weight matrix is also trained during the training process of the classification model; A prediction module based on continuous emotion, which is used to perform emotion prediction according to the fusion feature. This emotion prediction is used to perform data fitting training on the fusion feature to obtain a continuous emotion intensity value, and the emotion intensity value is divided into multiple emotion dimensions, and the multiple emotion dimensions include pleasantness, tension, excitement and certainty.

Images