A multi-modal collaborative optimization based emotion recognition method

Abstract

The application discloses a kind of based on multi-modal collaborative optimization emotion recognition method, constructs the emotion recognition network of fusion electroencephalogram and face.For electroencephalogram mode, based on electroencephalogram big model, combined with time series convolution, space-time embedding and prefix vector and adapter mixed fine-tuning, obtain high-quality electroencephalogram representation.For face mode, construct AU and non-AU double feature flow, through time series convolution, self-attention and bidirectional cross-feature flow attention realize the deep interaction of emotional clues and context.In single mode alternation iterative training stage, introduce multi-modal complementary constraint and memory consolidation regular term, further promote multi-modal collaborative optimization effect.In multi-modal fusion stage, each modal feature is mapped to the same dimensional space by adapter, and is input into multi-modal Transformer with class token, realize the deep fusion of electroencephalogram and face, generate multi-modal fusion feature for emotion recognition.The application can promote multi-modal collaborative optimization, significantly improve the accuracy and robustness of emotion recognition.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of emotion recognition, and mainly relates to a multi-modal collaborative optimization-based emotion recognition method. BACKGROUND

[0002] Emotion recognition is one of the key technologies in intelligent human-computer interaction systems, and its goal is to endow machines with accurate perception and deep understanding of human emotions. Human emotions are usually expressed as internal physiological state changes and external behavioral manifestations. Among them, electroencephalogram (EEG) signals, as an electrical signal of the central nervous system, can objectively reflect the deep neurophysiological activity of a person when emotions fluctuate; while facial features can intuitively present the emotional state of a person. Therefore, multi-modal collaborative optimization of EEG signals and facial features not only can take advantage of their complementary strengths, effectively overcome the limitations of single modal in information expression, but also can significantly improve the accuracy and robustness of emotion recognition. The multi-modal collaborative optimization emotion recognition technology thus constructed can be used for accurately recognizing emotional changes of patients and has broad application prospects and practical value in patient emotion monitoring scenarios.

[0003] Current recognition methods are generally divided into "EEG signal and facial feature emotion recognition method based on shallow fusion" and "EEG signal and facial feature emotion recognition method based on deep fusion". Specifically, there are:

[0004] 1) EEG signal and facial feature emotion recognition method based on shallow fusion

[0005] Invention patent applications with publication numbers CN119723635A, CN117409396A and CN114366107A, the main technical means of which include: extracting EEG signals and facial features respectively, and then realizing multi-modal fusion by using simple feature splicing or feature addition. This kind of method relies on traditional signal processing methods or shallow network to extract EEG features, which makes it difficult to capture complex EEG spatio-temporal dynamic information; at the same time, simple multi-modal fusion methods also fail to effectively mine the deep association between EEG and facial modalities.

[0006] 2) EEG signal and facial feature emotion recognition method based on deep fusion

[0007] The main technical means of the invention patent applications with publication numbers CN119066559A, CN118779820A and CN117763497A include: through attention mechanism and multi-scale convolutional neural network and other technologies, different modal features are assigned weights in the deep stage of the model to realize dynamic weighting and information interaction. This kind of method is insufficient in mining cross-modal complementary information, and may cause modal competition problem, resulting in low utilization efficiency of modal information and decreased emotion recognition accuracy. SUMMARY

[0008] To solve the problems of limited quality of electroencephalogram features and modal competition in multi-modal joint learning in the prior art, the present application provides a multi-modal collaborative optimization-based emotion recognition method.

[0009] To achieve the above purpose, the solution of the present application is:

[0010] A multi-modal collaborative optimization-based emotion recognition method, comprising:

[0011] Step 1, obtaining a multi-modal data set including electroencephalogram data and facial data;

[0012] Step 2, constructing an emotion recognition network;

[0013] Step 3, dividing the multi-modal data set into a training set and a validation set, training and testing the emotion recognition network to obtain an emotion recognition model;

[0014] Step 4, using the emotion recognition model to realize emotion recognition.

[0015] Preferably, the multi-modal data set obtaining step comprises:

[0016] Step 1.1, collecting electroencephalogram signals and pre-processing to obtain electroencephalogram modal data; wherein the pre-processing includes filtering, down-sampling and normalization;

[0017] Step 1.2, collecting facial videos, extracting expression action units, head posture and gaze direction features frame by frame to obtain facial modal data;

[0018] Step 1.3, dividing the results of steps 1.1 and 1.2 into non-overlapping data segments and performing emotion labeling to form a multi-modal data set.

[0019] Preferably, the emotion recognition network construction step comprises:

[0020] Constructing an electroencephalogram encoder to extract features from the electroencephalogram data to obtain electroencephalogram modal features;

[0021] Constructing a facial encoder to extract features from the facial data to obtain facial modal features;

[0022] The multi-modal interaction fusion module is constructed to fuse the electroencephalogram modal features and the facial modal features to obtain multi-modal fusion features.

[0023] A sentiment classifier is constructed to obtain a corresponding recognition result by taking the multi-modal fusion features as input.

[0024] Preferably, the construction of the electroencephalogram encoder comprises:

[0025] A time series encoder composed of a convolutional layer, a group normalization layer, and a GELU activation function is constructed to obtain an electroencephalogram feature sequence by taking electroencephalogram modal data as input.

[0026] An electroencephalogram Transformer encoder is constructed to obtain electroencephalogram modal features by taking the electroencephalogram feature sequence as input.

[0027] Preferably, the construction of the electroencephalogram Transformer encoder comprises:

[0028] In the self-attention module of the Transformer encoder built in the large model LaBraM, the learnable prefix vector is respectively spliced to the front end of the key and value sequence.

[0029] For the feedforward network, an electroencephalogram perception adapter parallel to the layer normalization and the feedforward network is introduced; wherein the electroencephalogram perception adapter adopts a cascaded structure and is composed of two fully connected layers, which are connected by a ReLU activation function.

[0030] Preferably, after obtaining the electroencephalogram feature sequence, a learnable time embedding and a spatial embedding are introduced and added to the electroencephalogram feature sequence to obtain an enhanced electroencephalogram feature sequence as input of the electroencephalogram Transformer encoder.

[0031] Preferably, the construction of the facial encoder comprises:

[0032] The facial modal data is divided into two feature streams with different semantic levels and in parallel: an AU feature stream and a non-AU feature stream; wherein the AU feature stream is composed of expression action units, and the non-AU feature stream is composed of head posture and gaze direction features.

[0033] The AU feature stream and the non-AU feature stream are respectively input into two independent two-layer time series convolutional networks TCN, and refined feature representations of the AU feature stream and the non-AU feature stream are respectively obtained by the two-layer time series convolutional networks Then a self-attention module is introduced to obtain refined feature representations of the AU feature stream and the non-AU feature stream.

[0034] The refined feature representations of the AU feature stream and the non-AU feature stream are taken as each other as query, key and value, and a bidirectional cross-modal attention module DCA is adopted to obtain facial modal features.

[0035] Preferably, the constructing step of the multi-modal interaction fusion module comprises:

[0036] constructing a lightweight modal adapter for each modality respectively to obtain a corresponding modal feature sequence;

[0037] introducing a learnable class token feature, splicing with all modal feature sequences to form a multi-modal input sequence;

[0038] inputting the multi-modal input sequence into a multi-modal Transformer encoder to obtain multi-modal fusion features.

[0039] Preferably, the training of the emotion recognition network comprises two stages:

[0040] In the first stage, the electroencephalogram encoder and the face encoder are trained using a single-modal alternating iterative training strategy:

[0041] The electroencephalogram modality features and the face modality features are extracted by the electroencephalogram encoder and the face encoder respectively, and on this basis, a modal classifier composed of a self-attention module and a pooling layer is used to perform emotion recognition on the electroencephalogram modality features and the face modality features respectively to obtain the corresponding prediction results;

[0042] During the training process, the electroencephalogram encoder and the face encoder are updated alternately and iteratively, and only the parameters of one of them and the corresponding modal classifier are updated each iteration, while the other remains frozen;

[0043] In the second stage, the parameters of the electroencephalogram encoder and the face encoder are frozen, and the multi-modal interaction fusion module and the emotion classifier are trained.

[0044] Preferably, the loss function of the first stage is wherein, represents a single-modal emotion classification loss, represents a multi-modal complementary constraint term, represents a memory consolidation regularization term, and represents a balance loss function hyperparameter, represents a KL divergence, represents the i-th sample in the multi-modal data set the feature under the current active modality m, represents the output of the encoder corresponding to the current active modality m, represents the prediction result obtained by reasoning under the current active modality m, represents the i-th sample in the multi-modal data set the feature under the frozen modality j, represents the output of the encoder corresponding to the frozen modality j, represents the prediction result obtained by reasoning under the frozen modality j, represents the prediction result obtained by reasoning under all frozen modalities, is a set of modalities, E represents an electroencephalogram modality, and F represents a face modality; and respectively represent the electroencephalogram encoder and the face encoder parameter gradient, represents corresponding emotional category labels, and respectively represent the prediction results under the electroencephalogram modality and the face modality, and respectively represent the emotional classification loss under the electroencephalogram modality and the face modality calculated based on the cross-entropy loss function; the multi-modal emotional classification loss function in the second stage = , wherein, represents the category token representation after multi-modal Transformer fusion, represents an emotional classifier, represents an emotional label, represents a cross-entropy loss function.

[0045] The application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the above-mentioned emotional recognition method.

[0046] The application also provides an electronic device, comprising:

[0047] a memory for storing a computer program;

[0048] a processor for executing the computer program to realize the steps of the above-mentioned emotional recognition method.

[0049] Compared with the prior art, the present application has the following advantages: the present application designs a multi-modal collaborative optimization based emotion recognition network, which effectively improves the emotion recognition performance through high-quality single-modal feature extraction, alternating iterative training and deep cross-modal fusion. For the electroencephalogram modality, based on the pre-trained electroencephalogram large model LaBraM, time series convolution, learnable space-time embedding and Transformer encoder are used to extract space-time enhanced features, and a hybrid fine-tuning strategy combining prefix vectors and electroencephalogram adapters is used to efficiently inject emotional information and realize high-quality electroencephalogram feature representation. For the facial modality, two parallel feature streams of AU feature flow and non-AU feature flow are constructed, multi-level features are extracted through time series convolution network and self-attention module, and deep interaction of emotional clues and context information is realized by means of bidirectional cross-feature flow attention. In order to fully exploit the complementarity between modalities, multi-modal complementary constraints and memory consolidation regularization are introduced in the alternating iterative training stage to avoid catastrophic forgetting and improve multi-modal collaborative effect. In the multi-modal fusion stage, the features of each modality are mapped to the same dimensional feature space through a lightweight adapter, and are input into the multi-modal Transformer encoder together with the learnable class token to realize deep interaction and fusion of electroencephalogram and facial information, thereby generating high-discriminative multi-modal features and completing emotion classification. Finally, a multi-modal collaborative optimization based emotion recognition framework is constructed, which integrates high-quality feature extraction, alternating training and deep fusion of electroencephalogram and facial features, promotes multi-modal collaborative optimization, and significantly improves the accuracy and robustness of emotion recognition. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is a method flowchart of the present application.

[0051] Figure 2 is a training stage schematic diagram of single-modal encoder alternating iteration of the emotion recognition network.

[0052] Figure 3 is a training stage schematic diagram of multi-modal interaction fusion module and emotion classifier of the emotion recognition network.

[0053] Figure 4 is a test stage schematic diagram of the emotion recognition network. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described clearly and completely below, and the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0055] As Figure 1As shown, the specific process of the emotion recognition method based on multi-modal collaborative optimization of the present application is as follows:

[0056] 1. Data preparation stage

[0057] 1.1 Collect multi-modal data sets of electroencephalogram data and facial data.

[0058] The original electroencephalogram signal is preprocessed, and a band-pass filter (0.1-75.0 Hz) is used in turn to remove low-frequency drift and high-frequency noise, and a 50.0 Hz notch filter is used to suppress interference. Then, the signal is uniformly down-sampled to 200 Hz and data normalized to obtain the electroencephalogram modal data. For the original facial video, the OpenFace tool is used to extract expression action units, head posture and gaze direction features frame by frame to form the facial modal data. Then, the preprocessed results of each modal are divided into non-overlapping data segments, and all samples are labeled with emotions to construct a multi-modal data set . Among them, denotes the emotion category label of the th sample , is a set of modalities, denotes the input features of the sample in the modality .

[0059] 2. Model design stage, the model design of two-stage training is as follows:

[0060] 2.1 The overall model is denoted as , including electroencephalogram and facial encoders , , electroencephalogram and facial classifiers , , electroencephalogram and facial adapters , , a multi-modal Transformer encoder and an emotion classifier composed of two layers of fully connected layers . Among them, the electroencephalogram encoder includes a time series encoder , an electroencephalogram Transformer encoder , an electroencephalogram perception adapter ; the facial encoder includes two layers of time series convolutional networks , self-attention modalities and bidirectional cross-modal attention modules . The model input is a multi-modal data set including electroencephalogram modal data and facial modal data . .

[0061] 2.2 Extracting high-quality EEG features.

[0062] First, in order to efficiently extract the temporal information from the EEG signals, the EEG modal data... Input to timing encoder Perform convolution to aggregate temporal information and obtain EEG feature sequences. The timing encoder It consists of convolutional layers, grouped normalization layers, and GELU activation functions in a cascaded manner.

[0063] Secondly, to further enhance the model's ability to perceive temporal and spatial information in EEG feature sequences, a learnable temporal embedding is introduced. and spatial embedding and with EEG characteristic sequences Adding them together yields an enhanced EEG feature sequence that incorporates spatiotemporal contextual information. This process can be represented as: .

[0064] Finally, the enhanced EEG feature sequence will be... The input EEG model LaBraM's built-in Transformer encoder To extract high-quality EEG features, and to efficiently utilize the general knowledge of large EEG models, a Transformer encoder for EEG is used. The above employs a hybrid fine-tuning strategy combining prefix vectors and an adapter. In the self-attention module, the learnable prefix vectors are concatenated to the beginning of the key and value sequences, respectively. This process can be represented as: .in, and These represent the original key and value sequences, respectively. and This represents the corresponding learnable prefix vector. This represents the vector concatenation operation. The corresponding attention mechanism is defined as follows:

[0065] ,

[0066] in, , , It is a learnable parameter matrix. Scaling factor ( Usually get the key (Dimensions). For the feedforward network, an EEG sensing adapter is introduced that runs parallel to layer normalization and the feedforward network. Fine-tune the original output of the feedforward network. The adapter adopts a cascade structure composed of two fully connected layers, connected by a ReLU activation function in the middle, and its calculation process can be represented as: . Wherein, represents the general input features, represents the feedforward network, represents the scaling coefficient, which is used to adjust the influence degree of the adapter output on the overall representation. Through the above mixed fine-tuning strategy, high-quality electroencephalogram features fused with general knowledge are obtained

[0067]

[0068] During the fine-tuning process, the original parameters of the electroencephalogram time series encoder and the Transformer encoder are kept frozen to preserve the general knowledge in the pre-training stage, while the prefix vector and the adapter are trained to efficiently inject emotional information, thereby improving the expression ability of the electroencephalogram features in the emotional task. In summary, the feature extraction process described in step 2.2 is completed by the modules in the electroencephalogram encoder .

[0069] 2.3 Extract multi-level facial features.

[0070] Firstly, the facial modality data is divided into two parallel feature streams with different semantic levels: AU feature stream and non-AU feature stream . Among them, the AU feature stream is composed of expression action unit features, which are used to capture the micro-expression changes (such as mouth up, frown, etc.) that convey emotional information, thereby reflecting the emotional state of the individual; while the non-AU feature stream is composed of head pose and gaze direction features, which provide key context clues for emotional understanding. Secondly, the AU feature stream and the non-AU feature stream are respectively input into two independent two-layer time series convolutional networks to model their respective time series dynamics and capture long-term dependencies. On this basis, a self-attention module is introduced to dynamically adjust the key time step weights of the feature sequence, efficiently extract key emotional information, and obtain refined feature representations and with high discriminability. This process can be formally represented as:

[0071] ,

[0072] wherein, and correspond to the time series convolutional networks of the AU feature stream and the non-AU feature stream, respectively. and ​Self-attention modules corresponding to the AU feature stream and the non-AU feature stream, respectively, and their definitions refer to the attention mechanism formula in step 2.2: . Finally, to achieve the deep fusion and information complementarity of AU features and non-AU features, the refined AU feature stream and the non-AU feature stream are used as queries (Query), keys (Key), and values (Value) to each other, and a bidirectional cross-modal attention module is used to calculate multi-level facial features:

[0073] ,

[0074] wherein, represents a feature concatenation operation. This module realizes mutual guidance and information complementarity between AU features and non-AU features through a bidirectional cross-modal attention mechanism, promoting efficient interaction and fusion of emotional cues and contextual information between the two feature streams. In summary, the feature extraction process described in step 2.3 is completed by the modules within the face encoder .

[0075] 2.4 Single-modal alternating iterative training strategy is adopted.

[0076] To alleviate the modal competition problem in multi-modal learning, a single-modal alternating iterative training strategy is adopted. First, the electroencephalogram encoder of step 2.2 and the face encoder of step 2.3 are used to extract electroencephalogram features and facial features , respectively. On this basis, the modal classifiers and composed of self-attention modules and pooling layers are used to perform emotional recognition on the respective features, obtaining prediction results and . During training, only one modality is activated at each iteration, i.e., only the parameters of the encoder and its corresponding classifier for that modality are updated, while the other modality remains frozen. The two modalities are activated alternately in subsequent iterations, thus realizing single-modal alternating optimization. For any modality , when modality is activated in the th iteration, its single-modal emotional classification loss is calculated, and its encoder parameters are updated:

[0077] ,

[0078] ,

[0079] wherein, represents the general features of modality m, denotes the emotion label, which is independent of the specific sample. and denote the sample under the input feature and the corresponding emotion label of the modality , denotes the cross-entropy loss function, denotes the single-modality emotion classification loss, denotes the learning rate. Through this alternating update mechanism, each modality can effectively avoid inter-modal gradient interference, thereby fully learning discriminative features.

[0080] To further exploit the complementary information between the EEG and facial modalities, when optimizing the current active modality , a multi-modal complementary constraint term based on KL (Kullback-Leibler Divergence) divergence is introduced:

[0081] ,

[0082] wherein denotes the prediction result obtained by all frozen modalities for the sample ; denotes the prediction result of the current active modality. This constraint encourages the active modality to actively fit the outputs of the remaining frozen modalities during the learning process, thereby explicitly improving the complementarity and synergy across modalities.

[0083] During the alternating iterative training process, when optimizing a certain modality (such as the EEG modality), it is often necessary to correct the prediction bias caused by other modalities together. However, this may cause the modality to forget the effective knowledge it has previously learned, resulting in a performance decline on its originally excellent samples. To this end, a memory consolidation regularization term is introduced:

[0084] ,

[0085] wherein and denote the prediction results of the EEG and facial modalities, denotes the corresponding emotion label, and denote the gradient derivation of the encoder parameters of modalities E and F, and denote the emotion classification losses of modalities E and F calculated by the cross-entropy loss function. This regularization term constrains the gradient update direction of the current active modality encoder on the emotion task loss, so that it remains consistent with the previously active modality, thereby effectively alleviating knowledge forgetting during the optimization process and promoting cross-modal feature collaborative optimization.

[0086] 2.5 A multi-modal interaction fusion module is constructed to facilitate cross-modal interaction between EEG and facial modalities and achieve deep fusion of features.

[0087] First, the EEG encoder and the facial encoder trained by alternating iterative training are used to extract EEG features and facial features from the input data . To map different modalities to the same dimensional feature space, a lightweight modal adapter is constructed for each modality , each adapter consisting of three fully connected layers and two activation functions. The specific mapping process can be represented as:

[0088]

[0089] wherein and represent the token-level sequence features of the EEG and facial modalities, respectively. To achieve deep interaction and information fusion between modalities, a learnable class token feature is introduced and concatenated with the processed modality feature sequence to form a unified multi-modal input sequence .

[0090] Then, the sequence is input into the multi-modal Transformer encoder to capture the complex association between the internal physiological state reflected by the EEG signal and the external behavioral state revealed by the facial expression, obtaining a multi-modal fusion feature representation . This feature fully aggregates the most critical emotional discriminative information in each modality and is input into the final emotion classifier for emotion recognition. In the multi-modal interaction fusion stage, the pre-trained encoder and parameters are frozen, and only the modal adapter, multi-modal Transformer, and emotion classifier are optimized. During training, the model is optimized by the multi-modal emotion classification loss based on the cross-entropy loss function. Through the above design, the model can more efficiently realize the interaction and fusion of EEG and facial modalities, thereby improving the accuracy and robustness of emotion recognition.

[0091] 2.6 Model Training is divided into two stages:

[0092] The first stage aims to efficiently train the EEG encoder , the facial encoder to extract the most discriminative high-quality modal features and achieve preliminary modal collaborative optimization. The overall loss function is used in this stage , where represents the single-modal sentiment classification loss, represents the multi-modal complementary constraint term, represents the memory consolidation regularization term, and represents the hyperparameters for balancing the loss function.

[0093] The second stage is based on the first stage, and all modal encoder parameters are frozen. The modal adapter , multi-modal Transformer encoder and sentiment classifier are introduced. The multi-modal sentiment classification loss is used to optimize the newly introduced modules to achieve deep interaction and fusion of different modal features.

[0094] Both stages update the model parameters using the gradient descent method, and the update strategy is as follows:

[0095] ,

[0096] where represents the learning rate. represents the loss function corresponding to the current stage: the first stage uses , and the second stage uses . First, perform steps 2.1, 2.2, 2.3, and 2.4 to complete the training of the first stage, then perform step 2.5 to complete the training of the second stage, until convergence.

[0097] 2.7 The above steps are unified into a two-stage deep neural network framework to achieve the optimization and training of the model.

[0098] 3. Model training stage

[0099] 3.1 Divide the multi-modal data set obtained in step 1.1 into a training set and a validation set ;

[0100] 3.2 Input the training set into the network model designed in step 2 , and use the mini-batch stochastic gradient descent method for model training. There are four loss functions: the first stage has a single-modal sentiment classification loss , a multi-modal complementary constraint term , and a memory consolidation regularization term ; the second stage has a multi-modal emotion classification loss . The training stage simultaneously uses a validation set The model training effect is verified, that is, when the model obtains good emotion recognition results on the validation set , and the performance does not improve in subsequent training iterations, the model training converges, and the training process is stopped;

[0101] 3.3 The final training is completed to obtain a model

[0102] 4. Model testing stage

[0103] 4.1 The input data is a multi-modal test set obtained by processing similar to step 1.1 , from which a test set is divided . The model used in the test stage is the EEG and face encoder , , the EEG and face adapter , , the multi-modal Transformer encoder and the emotion classifier .

[0104] 4.2 Input the test set into the model obtained in step 3.3 to obtain the emotion recognition results of the test set .

[0105] In order to solve the problems of limited quality of EEG features and competition between modalities in multi-modal joint learning in the prior art, the key points of the emotion recognition method based on multi-modal collaborative optimization proposed by the present application are:

[0106] 1) High-quality EEG feature extraction: To efficiently and accurately extract high-quality features from EEG signals, a parameter-efficient fine-tuning method based on the pre-trained EEG large model LaBraM (Large Brain Model) is adopted. First, to fully capture the temporal information of EEG signals, the EEG modality data is input into a temporal encoder, which aggregates information in the time dimension through convolution operations, thereby generating a preliminary EEG feature sequence. The temporal encoder consists of a one-dimensional convolution layer, a group normalization layer, and a GELU activation function. Second, to further enhance the model's spatio-temporal perception ability of EEG features, learnable temporal and spatial embeddings are introduced and added to the preliminary EEG feature sequence, thereby obtaining an enhanced EEG feature sequence that fuses spatial and temporal context information. Then, the enhanced feature sequence is input into the Transformer encoder built-in LaBraM to extract high-quality EEG features. To efficiently utilize the general knowledge of large models, the encoder adopts a hybrid fine-tuning strategy that combines prefix vectors with adapters: in the self-attention module, learnable prefix vectors are concatenated to the front of the key and value sequences to guide attention calculation; brain EEG perception adapters are added in parallel with layer normalization and feedforward networks to fine-tune the original output. The adapter adopts a cascading structure composed of two fully connected layers connected by a ReLU activation function in the middle to improve feature representation capability. During the entire fine-tuning process, the original parameters of the temporal encoder and the Transformer encoder are kept frozen to preserve the general knowledge from the pre-training stage, while the emotion information is efficiently injected through prefix vectors and adapters to enhance the expression ability of EEG features in the emotion task. In summary, the EEG feature extraction process is based on LaBraM, and through the hybrid fine-tuning strategy of prefix vectors and adapters, only a small number of parameters need to be trained to efficiently utilize the general knowledge of pre-trained models, achieving high-quality EEG feature extraction.

[0107] 2) Multi-level facial feature extraction: First, the facial modality data is divided into two feature streams with different semantic levels and in parallel: AU feature stream and non-AU feature stream. The AU feature stream contains expression action unit information, reflecting the emotional state of the individual; the non-AU feature stream contains head pose and gaze direction features, providing key context clues for emotion understanding. Second, the two feature streams are respectively input into two independent layers of temporal convolutional network (TCN) to model their respective temporal dynamics and capture long-term dependencies. On this basis, the self-attention module is introduced to dynamically adjust the weight of the key time step in the sequence, so as to realize efficient extraction of emotional information and obtain discriminative refined feature representation. Finally, in order to realize the deep fusion and information complementation of AU feature stream and non-AU feature stream, the refined AU feature stream and non-AU feature stream are used as query (Query), key (Key) and value (Value) to each other, and the bidirectional cross-modal attention mechanism is used for interactive calculation, so as to realize mutual guidance and mutual complementation between the two feature streams, thereby promoting efficient fusion of emotional clues and context information. Through the above facial feature extraction process, the features of different semantic levels realize deep interaction, generating multi-level facial representation with emotional discriminability and context understanding ability, providing high-quality and information-rich input features for subsequent emotion recognition.

[0108] 3) Single-modal alternating iterative training promotes multi-modal collaborative optimization: First, the EEG encoder and the facial encoder are used to extract their respective features, and the corresponding modal classifier is used for emotion recognition to obtain the preliminary prediction results. Subsequently, only one modality is activated in each iteration, i.e. only the encoder and classifier parameters of the modality are updated, while the other modalities remain frozen and are activated alternately in subsequent iterations, thereby realizing single-modal alternating training. Through this mechanism, each modality can fully learn discriminative features in a relatively independent optimization environment, while effectively avoiding gradient interference between modalities. On this basis, in order to further exploit the complementary information between EEG and facial modalities, multi-modal complementary constraints are introduced to make the activated modality actively fit the prediction results of the frozen modality during optimization, thereby improving the multi-modal collaborative optimization effect. At the same time, in order to further enhance the collaborative optimization and complementary use of cross-modal features, memory consolidation regularization is introduced to constrain the gradient update direction of the activated modality on the task loss to be consistent with the previous modality, thereby preserving the effective knowledge learned by each modality.

[0109] 4) Multi-modal interaction fusion: Firstly, the EEG features and facial features are extracted from the raw data by the EEG encoder and the facial encoder trained alternately. To map different modalities into a unified feature space, a lightweight adapter is constructed for each modality, which consists of three fully connected layers and two activation functions, to convert the modality features into sequence features of the same dimension. Subsequently, a learnable class token is introduced and concatenated with the processed EEG and facial sequence features to form a unified multi-modal input sequence. This sequence is fed into a multi-modal Transformer encoder to dynamically capture the complex correlation between the internal physiological state reflected by EEG and the external behavioral state revealed by facial expression, thereby generating the fused multi-modal feature representation. Through this multi-modal collaborative optimization process, the model can effectively integrate the key emotional discriminative information in each modality, realize deep interaction between modalities, and be used for final emotion classification. In the multi-modal interaction fusion stage, the pre-trained encoders are kept frozen, and only the modality adapters, multi-modal Transformer, and emotion classifier are collaboratively optimized. Through the above process, the model can efficiently fuse EEG and facial information, thereby significantly improving the accuracy and robustness of emotion recognition.

[0110] 5) Method of emotion recognition based on multi-modal collaborative optimization: As shown in Figures 2-4 , the emotion recognition model based on multi-modal collaborative optimization is trained in two stages.

[0111] The first stage is single-modal alternating iterative training: In this stage, only one modality is activated for training at a time, and the parameters of the other modality are kept frozen. Specifically, when the EEG modality is activated, the parameters of the facial encoder are frozen; when the facial modality is activated, the parameters of the EEG encoder are frozen, and the training is alternated. The activated modality each time optimizes its emotion discrimination ability through the calculation of the single-modal emotion classification loss . In this process, each modality learns discriminative features in a relatively independent optimization environment, thereby fully extracting emotional information and effectively avoiding cross-modal gradient interference. On this basis, to further exploit the complementary information between modalities, a multi-modal complementary constraint term is introduced, which actively fits the prediction results of the frozen modality by the activated modality, realizing the complementarity of cross-modal information. At the same time, to avoid possible catastrophic forgetting in the training process, a memory consolidation regularization term is introduced, which constrains the gradient update direction of the activated modality on the task loss to be consistent with the previous modality, preserving the effective knowledge learned by each modality, thereby improving the stability of multi-modal collaborative optimization.

[0112] The second stage is multi-modal fusion: in this stage, the electroencephalogram and the face data are respectively input into the encoder which has completed the first stage training and the parameters are frozen, so as to extract high-quality features of each mode. Then, the light adapter is used to map the features of each mode to the same dimension feature space, and the learnable class token is spliced to form a multi-modal input sequence. The sequence is input into the multi-modal Transformer encoder, and the self-attention mechanism of the encoder dynamically captures the complex correlation between the internal physiological state reflected by the electroencephalogram and the external behavior state revealed by the face, promotes the deep interaction and integration of cross-modal information, and obtains multi-modal fusion features. Finally, the multi-modal fusion features are input into the emotion classifier, and the multi-modal emotion classification loss The model is trained, so as to improve the accuracy and robustness of emotion recognition. Through the above two-stage training strategy, the high-quality single-modal feature extraction, the alternating iterative optimization and the multi-modal deep fusion are fully combined, the complementation and cooperation between the electroencephalogram and the face mode are realized, the role of multi-modal collaborative optimization is fully played, and therefore the accuracy and robustness of emotion recognition are significantly improved.

[0113] Based on the same technical solution, the application further provides an electronic device, which comprises:

[0114] a memory for storing a computer program;

[0115] a processor for executing the computer program to realize the steps of the above-mentioned emotion recognition method based on multi-modal collaborative optimization.

[0116] Based on the same technical solution, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the above-mentioned emotion recognition method based on multi-modal collaborative optimization. The computer readable storage medium can include: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0117] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the computer program can include the processes of the above-mentioned embodiments of the methods. Any reference to memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. The non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. The volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM).

Claims

1. An emotion recognition method based on multimodal collaborative optimization, characterized in that, include: Step 1: Obtain a multimodal dataset including EEG data and facial data; Step 2: Construct an emotion recognition network; Step 3: Divide the multimodal dataset into a training set and a validation set, train and test the emotion recognition network, and obtain the emotion recognition model; The steps for constructing the emotion recognition network include: An EEG encoder was constructed to extract features from EEG data, resulting in EEG modal features; A face encoder is constructed to extract features from face data, thereby obtaining face modal features; A multimodal interaction fusion module is constructed to fuse EEG modal features and facial modal features to obtain multimodal fusion features; Construct an emotion classifier, taking multimodal fusion features as input, to obtain the corresponding recognition results; Training an emotion recognition network involves two phases: In the first stage, a single-modal alternating iterative training strategy was used to train the EEG encoder and the face encoder: EEG modal features and facial modal features were extracted using an EEG encoder and a face encoder, respectively. Based on these, a modal classifier consisting of a self-attention module and a pooling layer was used to perform emotion recognition on the EEG modal features and facial modal features, respectively, and the corresponding prediction results were obtained. During training, the EEG encoder and face encoder are updated alternately and iteratively. Each iteration updates only one of them and the parameters of its corresponding modality classifier, while the other remains frozen. In the second stage, the parameters of the EEG encoder and face encoder are frozen, and the multimodal interaction fusion module and emotion classifier are trained. Loss function in the first stage ,in, This represents the loss in unimodal sentiment classification. This represents the multimodal complementary constraint term. This indicates a regular term for memory consolidation. and The hyperparameters representing the balanced loss function are... Denotes KL divergence, Represents the i-th sample in the multimodal dataset. Features under the current active mode m, This represents the output of the encoder corresponding to the currently active mode m. This represents the prediction result obtained through inference under the current active mode m. Represents the i-th sample in the multimodal dataset. Features under frozen mode j This indicates the encoder output corresponding to the frozen mode j. This represents the prediction result obtained from inference under frozen mode j. This represents the prediction results obtained from all frozen modal inferences. This is a set of modalities, where E represents the EEG modality and F represents the facial modality; and These represent the gradients of the EEG encoder and the face encoder parameters, respectively. This indicates the corresponding sentiment category label. and These represent the prediction results under the EEG modality and the facial modality, respectively. and These represent the emotion classification losses calculated based on the cross-entropy loss function for the EEG modality and the face modality, respectively. Step 4: Use the emotion recognition model to achieve emotion recognition.

2. The method according to claim 1, characterized in that, The steps for obtaining the multimodal dataset include: Step 1.1: Collect EEG signals and perform preprocessing to obtain EEG modal data; the preprocessing includes filtering, downsampling, and normalization. Step 1.2: Collect facial videos, extract facial expression and motion units, head posture and gaze direction features frame by frame to obtain facial modality data; Step 1.3: Divide the results of steps 1.1 and 1.2 into non-overlapping data segments and perform sentiment annotation to form a multimodal dataset.

3. The method according to claim 1, characterized in that, The construction steps of the EEG encoder include: A temporal encoder consisting of convolutional layers, grouping normalization layers, and GELU activation functions was constructed to obtain EEG feature sequences from EEG modal data as input. A Transformer EEG encoder was constructed, which takes EEG feature sequences as input to obtain EEG modal features.

4. The method according to claim 3, characterized in that, The construction steps of the EEG Transformer encoder include: In the self-attention module of the Transformer encoder built into the large model LaBraM, the learnable prefix vectors are concatenated to the front of the key and value sequences respectively; An EEG sensing adapter is introduced that runs parallel to layer normalization and a feedforward network; wherein the EEG sensing adapter consists of two cascaded fully connected layers connected by a ReLU activation function.

5. The method according to claim 3, characterized in that, After obtaining the EEG feature sequence, learnable temporal and spatial embeddings are introduced and added to the EEG feature sequence to obtain an enhanced EEG feature sequence, which serves as the input to the EEG Transformer encoder.

6. The method according to claim 1, characterized in that, The steps for constructing the face encoder include: The facial modality data is divided into two parallel feature streams with different semantic levels: AU feature stream and non-AU feature stream; the AU feature stream consists of facial expression action units, while the non-AU feature stream consists of head pose and gaze direction features. The AU feature stream and the non-AU feature stream are respectively input into two independent temporal convolutional networks (TCNs), and a self-attention module is introduced after each of the two TCNs to obtain the refined feature representations of the AU feature stream and the non-AU feature stream. The refined feature representations of AU feature stream and non-AU feature stream are used as queries, keys and values ​​respectively. A bidirectional cross-modal attention module (DCA) is used to obtain facial modal features.

7. The method according to claim 1, characterized in that, The construction steps of the multimodal interaction fusion module include: A lightweight modality adapter is constructed for each modality to obtain the corresponding modality feature sequence; A learnable category token feature is introduced and concatenated with all modal feature sequences to form a multimodal input sequence; The multimodal input sequence is fed into the multimodal Transformer encoder to obtain multimodal fused features.

Images