Multi-modal emotion recognition system based on global-local space-time semantic alignment

Abstract

The invention discloses a multi-modal emotion recognition system based on global-local space-time semantic alignment, and the system comprises the steps: dividing a multi-modal physiological signal into a plurality of single-modal physiological signals, and then dividing all the single-modal physiological signals into a plurality of batches; obtaining the maximum sequence length of the feature sequence of the single-mode physiological signal fragment in each batch, and by taking the maximum sequence length in each batch as a reference value, supplementing the sequence length of the single-mode physiological signal fragment in each batch to the maximum sequence length, therefore, the lengths of the feature sequences of the single-mode physiological signal fragments in each batch are the same, and the problems that the lengths of the physiological signal data are inconsistent and key information is lost are solved; according to the method, the local time semantic alignment module is provided to align each mode in the time dimension, so that the interpretability and robustness of the model are enhanced; according to the method, the semantic information of each mode is aligned in the spatial dimension, so that the emotion recognition process is optimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of auxiliary diagnosis and treatment of depression, and specifically relates to a multimodal emotion recognition system based on global-local spatiotemporal semantic alignment. Background Art

[0002] Emotion recognition is a hot research area in artificial intelligence. It aims to automatically identify an individual's emotional state by collecting and analyzing a variety of data. Although humans can easily perceive emotions in daily communication, further research and exploration are still needed to enable machines to accurately recognize and simulate human emotional states. Similar to the five senses of humans, the data obtained by machines through various sensors can be classified into different modalities. Generally speaking, the modalities that respond to human emotions can be divided into two categories: (1) overt behavior, such as sound, posture, gesture, facial expression, etc. (2) physiological signals, including electroencephalography (EEG), electrocardiography (ECG), galvanic skin reaction (GSR), electromyography (EMG), eye movement signals (EM), etc. Overt behavior is easy to obtain, but it can be disguised or concealed, which leads to a certain degree of unreliability in reflecting the true emotional state. In contrast, physiological signals directly reflect the activity of the nervous system and are difficult to manipulate and conceal. Therefore, they can more objectively reflect the changes in emotional state caused by stimuli.

[0003] Among various physiological signals, EEG signals are physiological signals that record the electrical activity of the cerebral cortex. Compared with other physiological signals, EEG signals provide information that directly reflects brain activity. The changes in its frequency and amplitude can reflect the synchronous and asynchronous activities of neurons, thereby effectively revealing the individual's cognitive and emotional processing, which is crucial for the understanding of different cognitive and emotional states and has significant advantages in emotion recognition tasks. With the continuous development of multimodal learning, multimodal emotion recognition based on psychophysiological data has become a research hotspot. Multimodal emotion recognition perceives emotions from multiple perspectives, combining subjective and objective factors, making the features more comprehensive, and the information of each modality synergistic and complementary. EEG signals provide information that directly reflects brain activity and can effectively reveal the individual's cognitive and emotional processing. Eye movement signals provide additional insights into visual attention and emotional responses. Other physiological signals such as electromyography (EMG) and skin conductance (GSR) provide physiological indicators related to the activity of the autonomic nervous system and can also be used to indirectly reflect emotional states. By combining these signals, researchers can more accurately understand the response patterns of individuals to different emotional stimuli, thereby improving the accuracy and reliability of emotion recognition.

[0004] Although relevant research on multimodal representation and fusion provides a rich and in-depth theoretical basis, there is still an unresolved problem in the field of multimodal emotion recognition based on EEG and other modalities, namely, how to discover the response delay and semantic association characteristics between different emotion modalities and brain regions from diverse and variable data.

[0005] First, the time and duration of individual reactions to different emotions vary from person to person, which leads to diversity and variability in the data. Existing methods usually divide the signal into small segments of equal length and then input them into deep neural networks. However, this approach ignores the basic characteristics of physiological signals as a special carrier of valuable information, such as the integrity of frequency and phase, and the temporal fluctuations of EEG signal components, which may lead to information loss. In addition, the emotion recognition task depends on understanding and interpreting the entire context, and this method may destroy the coherence of the context. Therefore, the diversity and variability of this data need to be fully considered.

[0006] Secondly, there is a response delay between different emotional modalities. Different physiological signals at the same time may express completely different emotional semantic information, which means that not all time periods in the entire stimulus source are valid. After receiving the stimulus, there is a certain information transmission lag between various parts of the body. This phenomenon can be explained by the temporal alignment relationship between emotional semantic information. Unfortunately, current research generally ignores this phenomenon, resulting in low robustness and interpretability of multimodal models. Therefore, in order to explore the temporal activation order relationship of multimodal signals, it is particularly necessary to perform semantic alignment in the time dimension.

[0007] Finally, different areas of the brain have their own unique functions, and the semantic association characteristics and importance of EEG signals and other physiological signals in each brain area are different. In emotion recognition, brain areas related to emotions and other physiological signals play a vital role, and this phenomenon can be understood as semantic alignment in space. Although some studies have explored from the perspective of brain area space, they only focus on a single modality of EEG signals, resulting in redundant information in brain area space, which will have a negative impact on emotion recognition results. Therefore, to enhance the semantic information of important brain areas, it is also crucial to align semantic information in the spatial dimension.

[0008] In summary, the present invention aims to propose a multimodal emotion recognition model that performs spatiotemporal semantic alignment of multimodal signals to solve the above-mentioned scientific problems. Summary of the invention

[0009] In view of this, an object of the present invention is to provide a multimodal emotion recognition system based on global-local spatiotemporal semantic alignment.

[0010] A multimodal emotion recognition system based on global-local spatiotemporal semantic alignment, comprising:

[0011] (1) Data acquisition and feature extraction unit, including:

[0012] 1) A data acquisition module, used to collect EEG data and at least one other modality of physiological signals, recorded as physiological signals of modality b;

[0013] 2) Feature extraction module, used to extract features from EEG data and physiological signals of modality b to obtain EEG feature data X EEG and the characteristic data X of mode b B ;

[0014] 3) A brain region division module, which is used to divide the cerebral cortex into multiple specific regions and divide the EEG feature data according to the specific regions;

[0015] The global-local multimodal spatiotemporal semantic alignment unit comprises:

[0016] 1) Time information encoding module, used to: convert EEG feature data X EEG and feature data X B Input 1D-CNN network to extract deep features and obtain feature X′ EEG and X′ B , the feature X′ B Do R sublinear mappings and get ″′ with R subsets

[0017] Feature X B , where R represents the number of brain regions; EEG and X B Add the respective position encoding information to obtain the deep feature sequence with position information. and

[0018] 2) Local temporal semantic alignment module, used to:

[0019] Use multi-brain cross-modal attention mechanism to analyze deep feature sequences and Through linear transformation, we can obtain the representation of the two queries respectively. and Key indication and And the value representation and Brain regions correspond to different attention heads;

[0020] Next, we calculate the cross-modal attention of multiple brain regions and obtain the attention weights in two directions as shown below:

[0021]

[0022] Among them, M B The mask representing the mode b; M EEG represents the mask of EEG data; the subscripts B→EEG and EEG→B represent different attention directions respectively;

[0023] In order to retain only the correct alignment information, the alignment matrix of the rth brain region in two directions is obtained and

[0024]

[0025] in, and Represent the alignment matrix Alignment Matrix The element in row i and column j of ; and Respectively represent matrices The element in row i and column j of ;

[0026] The alignment matrix is ​​used to calculate the output of the two modalities for each brain region, which is a weighted sum of the values:

[0027]

[0028] Among them, r represents the rth brain region, r∈(1,R);

[0029] Next, the deep features of the target modality are added to the output of the linear layer through the residual connection operation, so that the alignment in each direction contains data from both modalities; finally, one-dimensional global average pooling is used to summarize in the time dimension. The result obtained is a unidirectional alignment representation of the two modalities in each brain region, defined as follows:

[0030]

[0031] Among them, GAP() represents one-dimensional global average pooling;

[0032] Finally, all brain regions are summarized, and the obtained unidirectional time-aligned representation of the two modalities is expressed as follows:

[0033]

[0034] 3) Local-global spatial semantic alignment module, used to:

[0035] A. First, learn the dynamic attention weights of each brain region: the model learns the local time alignment relationship between the two modalities (EEG modality A and modality B) and obtains a unidirectional time alignment representation and The dynamic attention weights of each brain region are learned through two linear layers. The weight parameter of the first linear layer is The bias parameter is It is used to integrate the unidirectional time alignment representation and use the tanh function for nonlinear activation; the second linear layer calculates the spatial alignment score, and the weight parameter is The bias parameter is Finally, the Softmax function is used for normalization to obtain the spatial alignment weight:

[0036]

[0037] The sum of the weights of all brain regions is 1;

[0038] Similarly, a set of spatial alignment weights reflecting mode b is obtained

[0039]

[0040] B. Secondly, integrate the unidirectional time-aligned representation: Use the learned spatial alignment weights to scale the unidirectional time-aligned representation, that is:

[0041]

[0042]

[0043] The symbol ⊙ represents the Hadamard product. and Splicing

[0044] C. Then, the output is added to the original representation to achieve residual connection, and the output of each brain region is concatenated (concat()) in sequence to form a higher-dimensional representation; finally, it is processed by a linear layer and layer normalization operation (LN()), and the final result is a unidirectional spatiotemporal alignment representation:

[0045]

[0046] Among them, concat() represents concatenation processing; Linear() represents linear layer processing; LN() represents layer normalization operation;

[0047] 4) Global fusion module, used for:

[0048] First, the one-way spatiotemporal aligned representations are concatenated to obtain a fused representation:

[0049]

[0050] Then, a multi-layer perceptron is used to output the score logits of the current sample belonging to C different emotion categories:

[0051] logits=MLP(O fusion )=(ReLU(O fusion W1+b1))W2+b2

[0052] Wherein, MLP stands for multi-layer perceptron; W1 and W2 are weight parameters of the multi-layer perceptron, b1 and b2 are bias parameters of the multi-layer perceptron, and the ReLU activation function is used for nonlinear mapping;

[0053] 5) Optimization module: After obtaining the output logits, the Softmax function is further used to convert it into a predicted probability distribution P.

[0054] P = Softmax(logits)

[0055] The sum of the probabilities of all emotion categories is equal to 1;

[0056] Set the loss function and train and optimize the entire multimodal emotion recognition system by minimizing the value of the loss function:

[0057] (3) The emotion recognition unit uses the trained multimodal emotion recognition system to perform emotion recognition on the input data to obtain the corresponding emotion category.

[0058] Furthermore, the data acquisition and feature extraction unit also includes a data preprocessing module for downsampling and filtering the data.

[0059] Preferably, the mask of mode b is expressed as express; Indicates the padding mask used to make the EEG data consistent with the length of modality b data; represents the future mask of modality b data;

[0060] The mask representation of EEG data is express, Indicates the padding mask used to make the EEG data consistent with the length of modality b data. Represents the future mask of EEG data.

[0061] Preferably, the process of constructing the loss function includes: using a unidirectional spatiotemporal alignment representation and The L2 loss between them is used to constrain the collaborative representation of the two modalities, that is, the square difference of each corresponding element between them is calculated and the average is taken. The specific calculation is as follows:

[0062]

[0063] Among them, Rd represents the depth feature dimension;

[0064] Adding cross entropy loss and L2 regularization, assuming that the total number of samples is N, the total optimization goal is calculated as:

[0065]

[0066] Among them, W is the trainable weight parameter and λ is the regularized weight decay.

[0067] Preferably, the emotion recognition unit further comprises an evaluation test for the trained multimodal emotion recognition system, wherein leave-one-out-subject cross-validation (LOSO-CV) is selected as the evaluation method.

[0068] Preferably, the modality b data includes at least one of electroencephalogram, electrocardiogram, and eye movement.

[0069] Preferably, when the feature extraction module performs feature extraction, it includes obtaining differential entropy for dividing the EEG signal into frequency bands and leads.

[0070] Preferably, when the feature extraction module performs feature extraction, it includes performing z-score normalization processing on the EEG signal and other physiological signals.

[0071] The present invention has the following beneficial effects:

[0072] 1. When solving the problem of inconsistent physiological signal length, existing multimodal emotion recognition methods usually divide physiological signals into small segments of equal length and then input them into deep neural networks, which may cause the loss of key information and destroy contextual coherence. After dividing multimodal physiological signals into multiple single-modal physiological signals, the present invention divides all single-modal physiological signals into multiple batches, obtains the maximum sequence length of the characteristic sequence of the single-modal physiological signal segments in each batch, and uses the maximum sequence length in each batch as a reference value to pad the sequence length of the single-modal physiological signal segments in each batch to the maximum sequence length, so that the length of the characteristic sequence of the single-modal physiological signal segments in each batch is the same, solving the problems of inconsistent physiological signal data length and loss of key information.

[0073] 2. Existing multimodal emotion recognition methods generally ignore the response delay between different modalities, which makes not all time periods in the entire stimulus source valid. The present invention proposes a local temporal semantic alignment module to align each modality in the temporal dimension, which enhances the interpretability and robustness of the model.

[0074] 3. The existing technologies mostly ignore the semantic correlation characteristics and importance of EEG signals and other physiological signals in each brain region, resulting in redundant information in the brain region space, which has a negative impact on emotion recognition results. The present invention aligns the semantic information of each modality in the spatial dimension, thereby optimizing the emotion recognition process. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 Schematic diagram of 62 electrodes divided into 16 brain regions in an embodiment of the present invention;

[0076] Figure 2 This is the structural diagram of the local time semantic alignment module;

[0077] Figure 3 It is the working flow chart of the system of the present invention;

[0078] Figure 4 This is the structural diagram of the local-global spatial semantic alignment module;

[0079] Figure 5 Flowchart of the leave-one-out cross-validation method. DETAILED DESCRIPTION

[0080] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0081] The present invention provides a multimodal emotion recognition system based on global-local spatiotemporal semantic alignment, comprising:

[0082] (1) Data acquisition and feature extraction unit: collects and preprocesses the EEG, ECG, eye movement and other physiological signals of different subjects, and extracts features from multiple single-modality physiological signals;

[0083] (2) Global-local multimodal spatiotemporal semantic alignment unit: The temporal semantic alignment between the EEG features divided into multiple brain regions and other modal features is calculated locally, and the activation degree of each brain region is calculated from a global perspective to achieve spatial semantic alignment between modalities, thus obtaining a representation of spatiotemporal semantic alignment after global fusion.

[0084] (3) Emotion recognition unit: After extracting the spatiotemporal semantic features of different data, Leave One Subject Out Cross Validation (LOSO-CV) is used as an evaluation method to evaluate the effectiveness of the model proposed in the emotion recognition task and verify its generalization performance.

[0085] (1) The data acquisition and feature extraction unit includes the following modules:

[0086] 1) Data acquisition module: Different types of physiological signals require different acquisition methods. The types of physiological signals can be classified through various forms such as the device information and interface information of the physiological signals. For example, when collecting EEG information of the target person, an EEG acquisition device is used. The connection interface between the EEG acquisition device and the terminal device can be determined, and the source of the EEG data transmitted through the connection interface can also be determined. Therefore, the multimodal physiological signals can be divided into multiple single-modal physiological signals through the information of the connection interface.

[0087] 2) Data preprocessing module: Different types of physiological signals require different preprocessing methods. For example, for EEG signals, the sampling frequency of the signal is downsampled from 1000Hz to 200Hz to reduce storage space and calculation, improve the subsequent processing speed, and use a 1-75Hz bandpass filter to process the EEG signal to further remove low-frequency baseline drift and high-frequency noise. Then use a notch filter to filter the 50Hz power frequency signal to obtain a purer EEG signal.

[0088] 3) Feature extraction module: Different types of physiological signals require different feature extraction methods. For example, for EEG signals, DE is a generalization of Shannon entropy on continuous variables, which is often used to measure the total amount of information of continuous random signals, thereby statistically analyzing the uncertainty of the probability density distribution of continuous signals. Differential entropy is calculated for each EEG signal by dividing it into frequency bands and leads:

[0089]

[0090] Finally, in order to eliminate the influence of individual differences, the present invention performed z-score normalization on all physiological characteristics of each subject:

[0091]

[0092] Where X is the original feature sequence of a subject in an experimental day, μ is the mean, σ is the standard deviation, and X′ is the feature sequence after z-score normalization.

[0093] 4) Brain region division module (such as attached Figure 1): According to the anatomical and functional knowledge of the cerebral cortex, the cerebral cortex is divided into several specific areas, such as the frontal lobe, temporal lobe, parietal lobe and occipital lobe, which have unique functions and tasks, such as cognitive function, emotion regulation, perception and motor control. In order to study brain activity and function more accurately, we used the international standard 10-20 system to arrange electrodes on the brain, which ensures that the electrodes are evenly distributed on the surface of the scalp to capture comprehensive information about the brain's electrical activity. According to the standard electrode positions of the 10-20 system, the brain is divided into 16 specific areas, each of which contains a specific set of EEG electrodes, and the positions of these electrodes precisely correspond to specific functional areas of the brain. The EEG modal features are divided according to different 16 brain regions.

[0094] (2) The global-local multimodal spatiotemporal semantic alignment unit includes the following modules:

[0095] 1) Temporal information encoding module: In this module, the EEG feature sequences of multiple brain regions and the feature sequences of another modality (hereinafter referred to as modality b, such as eye movement, skin electricity, etc.) are respectively input into the 1D-CNN network to extract deep features after being filled, and position encoding information is added. This makes the present invention (such as the attached Figure 3 ) can learn the changes in time information in each modality, thereby solving the problem of non-fixed data length. Before input, the original EEG signal needs to be divided into multiple small subsets according to brain regions, and divided into several segments in time together with the modality b signal, extracting the aforementioned features to generate a feature sequence.

[0096] Assuming that a total of R brain regions are divided, the EEG and modality b (B) feature sequences of a sample are as follows:

[0097]

[0098]

[0099] Among them, the characteristic sequence of a single brain region is is the length of the EEG feature sequence (the number of segments), is the characteristic dimension of the current brain area, r≤R. is the length of the feature sequence of mode b (the number of segments), d b is the characteristic dimension of mode b.

[0100] Next, find the maximum sequence length in batch b where sample x is located. Use a specific token PAD to pad all samples in batch b to match the maximum sequence length and get the padding mask

[0101] After padding, the feature sequence of each brain region and the feature sequence of modality b are fed into a point-wise convolutional 1D-CNN network for processing to ensure that they have the same feature dimension. Since the feature dimension of each brain region is different, the 1D-CNN of each brain region is initialized with different parameters:

[0102]

[0103] Therefore, there are the following deep feature sequences of EEG and modality b:

[0104]

[0105] in, is the maximum sequence length in the batch, and d is the dimension of the deep feature. In order to make the deep features of modality b have richer semantic information and match them with various brain regions in the subsequent temporal semantic alignment, the deep feature sequence X′ of modality b is B Perform R linear mappings to ensure that it is mapped into R groups of data, and its deep feature dimension is set to Rd.

[0106] After that, the deep EEG feature sequences of each brain region are spliced ​​together to form Rd-dimensional features, which are consistent with the data of modality b. In order to ensure that the sequence can carry time information, the present invention also introduces position embedding into the EEG and modality b feature sequences. The process can be expressed as:

[0107]

[0108] Finally, and Connect them together to get the deep features with location information As the output of this module.

[0109] 2) Local temporal semantic alignment module: In this module, Multi-Region Cross-Modal Attention (MRCMA) is used to ensure the correct alignment information of each brain region and generate the corresponding sparse alignment matrix, specifically:

[0110] A. First, the deep features According to the number of brain regions R, it can be divided back into The representation of query, key and value is obtained through linear transformation, where the linear transformation parameters of all brain regions are not shared and the brain regions correspond to different attention heads:

[0111]

[0112] The dimension of each head is d. Each attention head is responsible for learning the attention pattern from the query, key, and value of the corresponding brain region, thereby helping the model better capture the sequence correlation information of the local brain region.

[0113] B. Similarly, the present invention also uses the future mask and Ensure that the future time information of the source mode will not be used in the dot product calculation. Add the padding mask and the future mask to get the mask matrix in two directions, which is calculated as follows:

[0114]

[0115] C. Next, we calculate the cross-modal attention of multiple brain regions and obtain the attention weights in two directions as follows:

[0116]

[0117] Among them, B→EEG and EEG→B represent different attention directions. By adding the scaled dot product to the mask matrix, after passing through the Softmax function, the masked part in each row, that is, the position of negative infinity, becomes zero, thus preventing them from participating in subsequent calculations. After masking, the attention weight The actual meaningful part of This means that only the information of the current and past moments of the source modality is retained in the attention calculation, and no information of the future moments is included.

[0118] D. In order to retain only the correct alignment information, an adaptive threshold is set on the attention weight of each brain region to obtain the following alignment matrices in two directions: and

[0119]

[0120]

[0121] in, and Represent the alignment matrix Alignment Matrix The element in row i and column j of ; and Respectively represent matrices The element in row i and column j of ;

[0122] The threshold is set to the current number of rows, so that the threshold size can be adaptively determined according to the sequence length, and only the attention weights above the threshold are retained, which represents the unidirectional alignment information after thresholding. After thresholding, the alignment matrix of each brain region is sparse and different.

[0123] E. Use the alignment matrix to calculate the output of each brain region, which is the weighted sum of the values:

[0124]

[0125] in, Each row vector in represents a segment in the EEG sequence of the current brain region, which is obtained by weighting the alignment information of all segments in the modality b sequence. This means that given a target EEG segment in the current brain region, the alignment matrix determines which modality b segments are the most relevant and important. On the other hand, From the perspective of modality b, each row vector represents a segment in the modality b sequence, which is also obtained by weighting the alignment information of all EEG segments in the current brain region. This means that the alignment matrix of the current brain region determines which EEG segments are the most relevant and important for a given target segment of modality b.

[0126] F. Next, the deep features of the target modality are added to the output of the linear layer through the residual connection operation, so that the alignment in each direction contains data from both modalities. Finally, one-dimensional global average pooling (GAP) is used to summarize in the time dimension. The result obtained in this way is a unidirectional alignment representation of each brain region, defined as follows:

[0127]

[0128] Finally, all brain regions are summarized and the resulting unidirectional time-aligned representation can be expressed as follows:

[0129]

[0130] The entire process of the local time semantic alignment module can be summarized as follows: Figure 2 .

[0131] 3) Local-global spatial semantic alignment module:

[0132] This module is used to integrate the time-aligned information from different brain regions into a unified semantic space to achieve the transition from local to global. Brain region-level spatial attention is used for time-aligned representation, and the dynamic attention weight of each brain region is learned. Through weighted operations, the model focuses on more important brain region information to achieve spatial semantic alignment.

[0133] A. First, the module learns the dynamic attention weights of each brain region. The model learns the local time alignment relationship between the two modalities (EEG modality A and modality B) and obtains a unidirectional time alignment representation. and The dynamic attention weights of each brain region are learned through two linear layers. The weight parameter of the first linear layer is The bias parameter is It is used to integrate the unidirectional temporal alignment representation and use the tanh function for nonlinear activation. The second linear layer calculates the spatial alignment score with a weight parameter of The bias parameter is Finally, the Softmax function is used for normalization to obtain the spatial alignment weight:

[0134]

[0135] The sum of the weights of all brain regions is 1;

[0136] Similarly, a set of spatial alignment weights reflecting mode b can be obtained

[0137]

[0138] B. Secondly, the module integrates the unidirectional time-aligned representation. The unidirectional time-aligned representation is scaled using the learned spatial alignment weights. The Hadamard product (element-wise product) is used here to achieve this, namely:

[0139]

[0140] The symbol ⊙ represents the Hadamard product. and Splicing

[0141] C. The module then adds the output to the original representation to achieve a residual connection. The module adds the scaled representation to the original representation. This residual connection helps preserve the original information and makes the model easier to train and converge.

[0142] D. Next, the module concatenates the outputs of the brain regions in sequence. The module concatenates (concat()) the outputs of each brain region in sequence to form a higher-dimensional representation.

[0143] E. Finally, the module passes through a linear layer processing (Linear()) and performs a layer normalization operation (LN()). The module performs a linear layer processing on the concatenated representation to abstract and integrate the information of all brain regions. The result is then subjected to a layer normalization operation to improve the stability of the model and the training effect.

[0144] Through the above steps, the final result is a one-way spatiotemporal alignment representation:

[0145]

[0146]

[0147] These representations can reflect the dynamic characteristics of different brain regions and have richer and more complex representation capabilities. This process can be summarized as follows Figure 4 .

[0148] 4) Global Fusion Module: After the above three modules, a one-way spatiotemporal alignment representation and Having learned the spatiotemporal alignment relationship between the two modalities, this module leverages this spatiotemporal alignment information to facilitate emotion recognition.

[0149] First, concatenate them to obtain a fused representation

[0150]

[0151] Use a multi-layer perceptron to get the output This is the score of the current sample belonging to C different emotion categories. For the SEED-IV dataset, C = 4, while C = 5 for the SEED-V dataset. This step can be expressed as:

[0152] logits=MLP(O fusion )=(ReLU(O fusion W1+b1))W2+b2

[0153] Among them, W1 and W2 are the weight parameters of the multilayer perceptron, b1 and b2 are the bias parameters of the multilayer perceptron, and the ReLU activation function is used for nonlinear mapping.

[0154] 5) Optimization module: After obtaining the output logits, the Softmax function is further used to convert them into a predicted probability distribution P, where the sum of the probabilities of all emotion categories is equal to 1:

[0155] P = Softmax(logits)

[0156] Using one-way spatiotemporal alignment representation and The L2 loss between them is used to constrain the collaborative representation of the two modalities, that is, the square difference of each corresponding element between them is calculated and the average is taken. The specific calculation is as follows:

[0157]

[0158] Adding cross entropy loss and L2 regularization, assuming that the total number of samples is N, the overall optimization goal of the present invention can be calculated as:

[0159]

[0160] Among them, W is the trainable weight parameter and λ is the regularized weight decay.

[0161] By making the total loss Minimize, thereby training and optimizing the entire multimodal emotion recognition system:

[0162]

[0163] Among them, W * is the weight parameter that minimizes the total loss. The above formula can effectively solve the proposed optimization goal, while accurately identifying different emotion categories, narrowing the distance between the two modalities, thereby achieving a better spatiotemporal alignment effect.

[0164] (3) The emotion recognition unit includes an evaluation test of the trained multimodal emotion recognition system and recognition of input data, specifically:

[0165] The present invention selects Leave One Subject Out Cross Validation (LOSO-CV) as the evaluation method. The process is shown in the attached figure. Figure 5 shown.

[0166] In each cycle of the experiment, one subject was selected as the test set, and the rest of the subjects were used as the training set to ensure that each subject was used as the test set at least once. The purpose of this strategy is to test whether individual differences have an impact on model evaluation.

[0167] The average and standard deviation of the test accuracy of all cycles are taken as evaluation indicators. The accuracy of the i-th fold of cross validation can be expressed as:

[0168]

[0169] Then the expression of average accuracy and standard deviation is:

[0170]

[0171] Among them, K is the number of subjects and also the number of cross-validation folds.

[0172] The EEG and at least one other physiological signal of the subject are input into a trained multimodal emotion recognition system to perform emotion recognition.

[0173] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal emotion recognition system based on global-local spatiotemporal semantic alignment, characterized in that: include: (1) Data acquisition and feature extraction unit, including: 1) A data acquisition module, used to collect EEG data and at least one other modality of physiological signals, recorded as physiological signals of modality b; 2) Feature extraction module, used to extract features from EEG data and physiological signals of modality b to obtain EEG feature data X EEG and the characteristic data X of mode b B ; 3) A brain region division module, which is used to divide the cerebral cortex into multiple specific regions and divide the EEG feature data according to the specific regions; The global-local multimodal spatiotemporal semantic alignment unit comprises: 1) Time information encoding module, used to: convert EEG feature data X EEG and feature data X B Input 1D-CNN network to extract deep features and obtain feature X′ EEG and X′ B , the feature X′ B Do R sublinear mappings to get the feature X′ with R subsets B , where R represents the number of brain regions; EEG and X′ B Add the respective position encoding information to obtain the deep feature sequence with position information. and 2) Local temporal semantic alignment module, used to: Use multi-brain cross-modal attention mechanism to analyze deep feature sequences and Through linear transformation, we can obtain the representation of the two queries respectively. and Key indication and And the value representation and , brain regions correspond to different attention heads; Next, we calculate the cross-modal attention of multiple brain regions and obtain the attention weights in two directions as shown below: Among them, M B The mask representing the mode b; M EEG represents the mask of EEG data; the subscripts B→EEG and EEG→B represent different attention directions respectively; In order to retain only the correct alignment information, the alignment matrix of the rth brain region in two directions is obtained and in, and Represent the alignment matrix Alignment Matrix The element in row i and column j of ; and Respectively represent matrices The element in row i and column j of ; The alignment matrix is ​​used to calculate the output of the two modalities for each brain region, which is a weighted sum of the values: Among them, r represents the rth brain region, r∈(1,R); Next, the deep features of the target modality are added to the output of the linear layer through the residual connection operation, so that the alignment in each direction contains data from both modalities; finally, one-dimensional global average pooling is used to summarize in the time dimension. The result obtained is a unidirectional alignment representation of the two modalities in each brain region, defined as follows: Among them, GAP() represents one-dimensional global average pooling; Finally, all brain regions are summarized, and the obtained unidirectional time-aligned representation of the two modalities is expressed as follows: 3) Local-global spatial semantic alignment module, used to: A. First, learn the dynamic attention weights of each brain region: the model learns the local time alignment relationship between the two modalities and obtains a unidirectional time alignment representation and The dynamic attention weights of each brain region are learned through two linear layers; the weight parameter of the first linear layer is The bias parameter is It is used to integrate the unidirectional time alignment representation and use the tanh function for nonlinear activation; the second linear layer calculates the spatial alignment score, and the weight parameter is The bias parameter is Finally, the Softmax function is used for normalization to obtain the spatial alignment weight: The sum of the weights of all brain regions is 1; Similarly, a set of spatial alignment weights reflecting mode b is obtained B. Secondly, integrate the unidirectional time-aligned representation: Use the learned spatial alignment weights to scale the unidirectional time-aligned representation, that is: The symbol ⊙ represents the Hadamard product. and Splicing C. Then, the output is added to the original representation to realize residual connection, and the output of each brain region is spliced ​​together in order to form a higher-dimensional representation; finally, it is processed by a linear layer and layer normalization operation, and the final result is a unidirectional spatiotemporal alignment representation: Among them, concat() represents concatenation processing; Linear() represents linear layer processing; LN() represents layer normalization operation; 4) Global fusion module, used for: First, the one-way spatiotemporal aligned representations are concatenated to obtain a fused representation: Then, a multi-layer perceptron is used to output the score logits of the current sample belonging to C different emotion categories: logits=MLP(O fusion )=(ReLU(O fusion W1+b1))W2+b2 Wherein, MLP represents multi-layer perceptron; W1 and W2 are weight parameters of the multi-layer perceptron, b1 and b2 are bias parameters of the multi-layer perceptron, and ReLU activation function is used for nonlinear mapping; 5) Optimization module: After obtaining the output logits, the Softmax function is further used to convert it into a predicted probability distribution P. P = Softmax(logits) The sum of the probabilities of all emotion categories is equal to 1; Set the loss function and train and optimize the entire multimodal emotion recognition system by minimizing the value of the loss function: (3) The emotion recognition unit uses the trained multimodal emotion recognition system to perform emotion recognition on the input data to obtain the corresponding emotion category.

2. A multimodal emotion recognition system based on global-local spatiotemporal semantic alignment as claimed in claim 1, characterized in that: The data acquisition and feature extraction unit also includes a data preprocessing module for downsampling and filtering the data.

3. A multimodal emotion recognition system based on global-local spatiotemporal semantic alignment as claimed in claim 1, characterized in that: The mask of modality b is represented as express; Indicates the padding mask used to make the EEG data consistent with the length of modality b data; represents the future mask of modality b data; The mask representation of EEG data is express, Indicates the padding mask used to make the EEG data consistent with the length of modality b data. Represents the future mask of EEG data.

4. A multimodal emotion recognition system based on global-local spatiotemporal semantic alignment as claimed in claim 1, characterized in that: The process of constructing the loss function includes: using a unidirectional spatiotemporal alignment representation and The L2 loss between them is used to constrain the collaborative representation of the two modalities, that is, the square difference of each corresponding element between them is calculated and the average is taken. The specific calculation is as follows: Among them, Rd represents the depth feature dimension; Adding cross entropy loss and L2 regularization, assuming that the total number of samples is N, the total optimization goal is calculated as: Among them, W is the trainable weight parameter and λ is the regularized weight decay.

5. A multimodal emotion recognition system based on global-local spatiotemporal semantic alignment as claimed in claim 1, characterized in that: The emotion recognition unit also includes an evaluation test for the trained multimodal emotion recognition system, wherein leave-one-out-subject cross-validation (LOSO-CV) is selected as the evaluation method.

6. A multimodal emotion recognition system based on global-local spatiotemporal semantic alignment as claimed in claim 1, characterized in that: The modality b data includes at least one of electroencephalogram, electrocardiogram, and eye movement.

7. A multimodal emotion recognition system based on global-local spatiotemporal semantic alignment as claimed in claim 1, characterized in that: When the feature extraction module performs feature extraction, it includes calculating the differential entropy for the EEG signal by dividing the frequency bands and the leads.

8. A multimodal emotion recognition system based on global-local spatiotemporal semantic alignment as claimed in claim 1, characterized in that: When the feature extraction module performs feature extraction, it includes performing z-score normalization processing on the EEG signal and other physiological signals.

Images