Cross-modal emotion recognition model training method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a cross-modal emotion recognition model training method and system, electronic equipment and a storage medium. The method comprises the following steps: inputting initial emotion data of various different modes into a cross-modal emotion recognition model; obtaining a first comparison loss based on the difference between the outputs of the emotion-related encoders of each mode; obtaining a second comparison loss based on the difference between the output of the emotion-related encoder and the output of the emotion-independent encoder of each mode; determining reconstruction loss based on the reconstructed emotion data obtained by the decoder and the initial emotion data; determining a prediction loss for assisting modal alignment based on the historical fusion features and the future features; and pre-training the cross-modal emotion recognition model based on the first comparison loss, the second comparison loss, the reconstruction loss and the prediction loss. According to the embodiment of the invention, emotion data of any mode is needed, and the encoder and the classifier corresponding to the mode can be used for emotion recognition.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition, and in particular to a training method, system, electronic device and storage medium for a cross-modal emotion recognition model. Background Art

[0002] Emotion recognition is an important research direction of BCI (brain-computer interfaces). Since emotion induction experiments usually only require participants to sit and watch emotion induction videos, compared with other brain-computer interface studies, emotion induction experiments are more likely to obtain more physiological signals, such as electroencephalogram (EEG), eye movement (eye), electrocardiogram (ECG), and peripheral physiological signals (PPS). This advantage has greatly promoted the application of multimodal models in the field of emotion recognition. By fusing the features of multiple pairs of modalities, the performance of the fused multi-pair modal model has been significantly improved compared with the unimodal model.

[0003] There are currently two main multimodal emotion recognition models based on EEG and eye movement signals. One is to use complete multimodal data for training and testing, and to fuse features of different modalities into the same feature space through multimodal fusion technology, such as the currently excellent MAET model (Multimodal Adaptive Emotion Transformer). The second method is to explore the cross-modal mapping between multimodal and unimodal features, and to map unimodal features to a feature space containing multimodal information through encoding. For example, a cross-modal method based on generative adversarial learning has been used to convert eye movement signals into EEG signals to establish multimodal information. Among them: 1. The MAET model (Multimodal Adaptive Emotion Transformer) uses a deep recurrent neural network to explore recurrent connections and generate multimodal features from eye movement signals. The model consists of a multi-view encoding module, an adaptive transformer, a hybrid transformer, and several classifiers. It is trained using EEG and eye movement features to enable it to handle multimodal inputs. It is also tested using complete multimodal data.

[0004] 2. The cross-modal method based on generative adversarial learning has a model framework and training process divided into two stages: the first stage is to extract high-level features by encoding and fusing EEG and eye movement signals, using classification loss and reconstruction loss; the second stage uses the encoded eye movement features to generate the corresponding encoded EEG features, and constrains the high-level feature generator through mean square error and discriminator to complete the generation of high-level features. Only eye movement signals are needed in the test stage.

[0005] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art: These methods have obvious limitations in practical applications. In actual scenarios limited by the experimental environment, not all modal data can be obtained. This is especially common because EEG signals are very sensitive to the environment. Any external interference and slight movements of the subject will bring significant noise to the EEG signal, resulting in a decrease in signal quality or even making the data unusable. For EYE signals, although their acquisition process is more convenient than EEG signals, there are still some problems. Acquiring eye movement data is sensitive to distance. The eye tracker must be kept at a certain distance from the subject, and the subject's eyes must be focused on the screen. This requirement may be challenging for the subject during the extended signal collection period. As for ECG and PPS, they have similar problems to EEG signals because they are all collected using similar electrodes. Therefore, in practical situations, it is difficult to obtain high-quality data from multiple modalities at the same time.

[0006] Specific: The first method requires complete multimodal data for both training and testing. However, in actual applications, it is difficult to collect complete multimodal data due to interference from factors such as noise (interference with EEG signals), experimental conditions (lack of certain acquisition equipment), and the subjects themselves (body movements causing disconnection of EEG electrodes), which makes the trained model unusable.

[0007] The second method uses a generative adversarial model to map the features of one modality to another modality. The training of the generator is too difficult, and the generated features are inevitably different from the real features. These defects limit the further development of this field. Summary of the invention

[0008] In order to at least solve the problem that the training of existing technologies is too dependent on complete multimodal data.

[0009] In a first aspect, an embodiment of the present invention provides a method for training a cross-modal emotion recognition model, comprising: Inputting the initial emotion data of multiple different modalities into a cross-modal emotion recognition model, wherein the cross-modal emotion recognition model comprises: a classifier for emotion recognition, an emotion-related encoder for extracting emotion features for each modality, an emotion-independent encoder for extracting emotion-independent features, and a decoder based on a multi-layer perceptron; Narrowing the gap between the outputs of the emotion-related encoders of each modality based on a first contrast loss function to obtain a first contrast loss; The difference between the output of the emotion-related encoder and the output of the emotion-independent encoder of each modality is increased based on the second contrast loss function to obtain a second contrast loss; The decoder of each modality receives the output of the emotion-related encoder and the output of the emotion-independent encoder, and determines the reconstruction loss based on the reconstructed emotion data obtained by the decoder and the initial emotion data; Using a many-to-many comparison prediction coding module, the output of the emotion-related encoder of each modality is multimodally fused to obtain a historical fusion feature, predicting the generalized future features of each modality through the historical fusion feature, and determining a prediction loss for assisting modality alignment based on the historical fusion feature and the future feature; The cross-modal emotion recognition model is pre-trained based on the first contrast loss, the second contrast loss, the reconstruction loss, and the prediction loss until a preset goal is achieved.

[0010] In a second aspect, an embodiment of the present invention provides a training system for a cross-modal emotion recognition model, comprising: A data input module, used to input initial emotion data of multiple different modalities into a cross-modal emotion recognition model, wherein the cross-modal emotion recognition model includes: a classifier for emotion recognition, an emotion-related encoder for extracting emotion features for each modality, an emotion-independent encoder for extracting emotion-independent features, and a decoder based on a multi-layer perceptron; an alignment module, configured to reduce the gap between the outputs of the emotion-related encoders of each modality based on a first contrast loss function to obtain a first contrast loss; An emotion-irrelevant information filtering module, configured to increase the difference between the output of the emotion-related encoder and the output of the emotion-irrelevant encoder of each modality based on a second contrast loss function to obtain a second contrast loss; A reconstruction module, configured for receiving the output of the emotion-related encoder and the output of the emotion-independent encoder by a decoder of each modality, and determining a reconstruction loss based on the reconstructed emotion data obtained by the decoder and the initial emotion data; An auxiliary module, configured to use a many-to-many comparison prediction coding module to perform multimodal fusion on the output of the emotion-related encoder of each modality to obtain a historical fusion feature, predict the generalized future features of each modality through the historical fusion feature, and determine a prediction loss for assisting modality alignment based on the historical fusion feature and the future feature; A pre-training module is used to pre-train the cross-modal emotion recognition model based on the first contrast loss, the second contrast loss, the reconstruction loss and the prediction loss until a preset goal is achieved.

[0011] In a third aspect, an embodiment of the present invention provides a cross-modal emotion recognition model, including: A plurality of emotion encoders and a decoder based on a multi-layer perceptron after contrastive learning, wherein each emotion encoder and the decoder based on the multi-layer perceptron respectively processes emotion data of each different modality, and the emotion encoder comprises: an emotion-related encoder based on a spatiotemporal sparse attention mechanism for extracting emotion features, and an emotion-independent encoder for extracting emotion-independent features; A classifier is used to receive the emotion features output by the emotion-related encoder and output the emotion recognition result.

[0012] In a fourth aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the training method of the cross-modal emotion recognition model of any embodiment of the present invention.

[0013] In a fifth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the program implements the steps of the training method of the cross-modal emotion recognition model of any embodiment of the present invention.

[0014] In a sixth aspect, an embodiment of the present invention provides a computer program product, comprising a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the steps of the training method of the cross-modal emotion recognition model of any embodiment of the present invention are implemented.

[0015] The beneficial effect of the embodiment of the present invention is that the method combines a variety of contrastive learning methods to extract emotion-related features from different modalities and align them to the same feature space, ensuring that the features extracted by the ER encoder of each modality are as similar as possible, so that after subsequent fine-tuning, each modality can share a classifier. Furthermore, the many-to-many contrast prediction coding module can assist in feature alignment and multimodal fusion in cross-modal tasks. This module has very good generalization capabilities and can be transplanted to existing methods to help improve their performance. At the same time, the spatiotemporal sparse attention mechanism of the present method can simultaneously combine the temporal and spatial information of the EEG data, making full use of the stable spatial information and rich temporal information of the EEG data. In this way, it is possible to achieve emotion recognition using the encoder and classifier corresponding to any one modality with only the emotion data of that modality, without the need for complete multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is a flow chart of a training method for a cross-modal emotion recognition model provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of a pre-training structure of a training method for a cross-modal emotion recognition model provided by an embodiment of the present invention; Figure 3 It is a structural schematic diagram of an M2M CPC module of a training method for a cross-modal emotion recognition model provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of an STS attention of a training method of a cross-modal emotion recognition model provided by an embodiment of the present invention; Figure 5 It is a fine-tuning flow chart of a training method for a cross-modal emotion recognition model provided by an embodiment of the present invention; Figure 6 It is a schematic diagram of basic information of five public multimodal emotion data sets of a training method for a cross-modal emotion recognition model provided by an embodiment of the present invention; Figure 7 is a schematic diagram of subject-related accuracy (%) and kappa score (%) for cross-modal and unimodal settings on SEED, SEED-IV and SEED-V datasets for a training method for a cross-modal emotion recognition model provided by an embodiment of the present invention (EEG→EYE represents fine-tuning of EEG and testing of EYE. The best result is shown in bold, and the second best result is shown in underline); Figure 8 It is a schematic diagram of subject-related accuracy (%) and F1 score (%) of cross-modal and unimodal settings on DEAP and DREAMER datasets of a training method for a cross-modal emotion recognition model provided by an embodiment of the present invention; Fig. 9 It is a schematic diagram of the balanced accuracy (%) and kappa score (%) of the multimodal method compared with M2S on SEED, SEEDIV and SEED-V datasets of a training method for a cross-modal emotion recognition model provided by an embodiment of the present invention, wherein (S) indicates that the same encoder is used for unimodal supervised learning; Fig.10It is a schematic diagram of the results of different attention mechanisms on SEED, SEED-IV and SEED-V datasets based on EEG unimodal fine-tuning of a training method for a cross-modal emotion recognition model provided by an embodiment of the present invention; Fig.11 It is a schematic diagram of the change in cross-modal task accuracy of CLIP and ECO-FET models on SEED, SEED-IV and SEED-V datasets after adding an M2M CPC module to a training method for a cross-modal emotion recognition model provided by an embodiment of the present invention; Fig.12 It is a heat map of two cross-modal tasks on the SEED dataset of a training method for a cross-modal emotion recognition model provided by an embodiment of the present invention; Fig.13 It is a schematic diagram of t-SNE dimensionality reduction distribution of training data of two subjects in a SEED dataset after pre-training of a training method for a cross-modal emotion recognition model provided by an embodiment of the present invention; Fig.14 It is a structural schematic diagram of a training system for a cross-modal emotion recognition model provided by an embodiment of the present invention; Fig.15 A schematic diagram of the structure of an embodiment of an electronic device for training a cross-modal emotion recognition model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] like Figure 1 FIG. 1 is a flow chart of a method for training a cross-modal emotion recognition model provided by an embodiment of the present invention, comprising the following steps: S11: inputting the initial emotion data of multiple different modalities into a cross-modal emotion recognition model, wherein the cross-modal emotion recognition model comprises: a classifier for emotion recognition, an emotion-related encoder for extracting emotion features for each modality, an emotion-independent encoder for extracting emotion-independent features, and a decoder based on a multi-layer perceptron; S12: reducing the gap between the outputs of the emotion-related encoders of each modality based on a first contrast loss function to obtain a first contrast loss; S13: increasing the difference between the output of the emotion-related encoder and the output of the emotion-independent encoder of each modality based on a second contrast loss function to obtain a second contrast loss; S14: the decoder of each modality receives the output of the emotion-related encoder and the output of the emotion-independent encoder, and determines the reconstruction loss based on the reconstructed emotion data obtained by the decoder and the initial emotion data; S15: using a many-to-many comparison prediction coding module to perform multimodal fusion on the output of the emotion-related encoder of each modality to obtain a historical fusion feature, predicting the generalized future features of each modality through the historical fusion feature, and determining a prediction loss for assisting modality alignment based on the historical fusion feature and the future feature; S16: Pre-training the cross-modal emotion recognition model based on the first contrast loss, the second contrast loss, the reconstruction loss, and the prediction loss until a preset goal is achieved.

[0020] In this embodiment, in order to ensure the recognition effect, the existing technology usually uses more modal data for training and changes the structure of each modal encoder, etc., but when the modal data is missing, it is still difficult to achieve a good multimodal emotion recognition effect. In order to solve the problem of multimodal emotion recognition when the modality is missing, this method reconstructs the cross-modal emotion recognition model. By using contrastive learning and a specially designed spatiotemporal attention mechanism module, the test results of single-modal data can achieve multimodal effects, thereby greatly reducing the dependence of emotion recognition tasks on multimodal data.

[0021] As an implementation mode, the method develops a cross-modal emotion recognition model, a plurality of emotion encoders and a decoder based on a multi-layer perceptron after contrastive learning, wherein each emotion encoder and the decoder based on the multi-layer perceptron process emotion data of each different modality respectively, and the emotion encoder comprises: an emotion-related encoder based on a spatiotemporal sparse attention mechanism for extracting emotion features, and an emotion-independent encoder for extracting emotion-independent features; A classifier is used to receive the emotion features output by the emotion-related encoder and output the emotion recognition result.

[0022] In general, the cross-modal emotion recognition model of this method can be called the M2S model (Multi-to-Single Emotion Recognition model). Simply put, the model uses a variety of contrastive learning methods and contains two modules with good generalization: M2M CPC (Multi-to-Multi Contrastive Predictive Coding) and STS Attention (Spatial and Temporal-Sparse attention mechanism).

[0023] In the specific use of the model of this method, the model has an emotion-related encoder and an emotion-independent encoder, but only the emotion-related encoder is used for emotion recognition.

[0024] As an implementation mode, the modalities include: electroencephalogram modality, eye movement modality, electrocardiogram modality, and peripheral physiological signal modality; For example, the model of this method has these four modalities, so there are emotion-related encoders for the EEG modality, emotion-related encoders for the eye movement modality, emotion-related encoders for the ECG modality, and peripheral physiological signal modalities.

[0025] In actual emotion recognition, if the data collected is the EEG data of the subject, that is, the input data is "EEG data", only the emotion-related encoder of the EEG modality is used for processing. If the data collected is the EEG data and eye movement data of the subject, the emotion-related encoder of the EEG modality and the emotion-related encoder of the eye movement modality are used for processing respectively. Finally, the emotional features of the output processed by each encoder are input into the same decoder, and the decoder outputs the final emotion recognition result.

[0026] The architecture of the model of this method is shown above. The training of the model is also particularly important. Only by using contrastive learning and spatiotemporal attention mechanism training in this application can the classifier of this application and the encoder of any modality realize cross-modal emotion recognition, reducing the dependence of emotion recognition tasks on multimodal data. Model training is divided into two stages: pre-training and fine-tuning.

[0027] For step S11, in the pre-training stage, initial emotion data of different modalities are received, such as EEG data, eye movement data, ECG data, peripheral physiological signal data, etc. The purpose of the overall framework of this method is to establish a feature space, align the features closely related to emotions in these different modalities in this space, narrow the distribution gap between them, and then use any modality to train a classifier to classify the emotion features in this space. In this way, when testing, only any one modality is needed, and the encoder and classifier corresponding to the modality can be used for emotion recognition without the need for complete multimodal data, reducing the dependence of the emotion recognition task on multimodal data.

[0028] The pre-training phase of the model in this method is as follows Figure 2 As shown in the figure, in simple terms, each modality has an ER (Emotion-Related) encoder, EI (Emotion-Independent) encoder and decoder, which are used to extract emotion-related and emotion-independent features and reconstruct the modality respectively. Among them, the M2M CPC module is used to assist modality alignment and multimodal fusion; the output of the ER encoder is mapped to a feature space, and feature alignment is performed using the contrastive learning loss function.

[0029] In order to more accurately capture the emotion-related information in each modality, this method designs two encoders for each modality. One is the ER (Emotion-Related) encoder based on the transformer, which is used to extract emotion features from the data, and the other is the EI (Emotion-Independent) encoder based on the MLP (Multilayer Perceptron) to extract emotion-independent features. The model architecture of this application has its own emotion-related ER encoder, emotion-independent EI encoder, and MLP decoder for each modality from the diagram in the pre-training phase, and the output of the emotion-related ER encoder of each modality is directly connected to the classifier.

[0030] For step S12, the method does this for the following reasons: In addition to the emotional information contained in each modal data, there must be some factors unrelated to the emotion, and there is no necessary correspondence between these unrelated factors. When aligning the features of different modalities, if these information unrelated to the emotion is introduced, it will inevitably bring difficulties to the alignment operation.

[0031] Therefore, for the output of the ER encoder (i.e., the emotion-related encoder), this method uses the contrast loss function to narrow the distribution gap between them to extract the corresponding information (emotion information) between different modalities. Specifically, for any modality of emotion data, the embedding vector , it will be projected into a new embedding space through the average pooling layer and linear layer of the time window dimension. This method defines the final embedding of the output of the emotion-related encoder of modality X as , where S represents the number of samples, represents the final embedding dimension. There are S positive sample pairs and negative sample pairs.

[0032] The first contrast loss function includes: Contrastive Loss function, and the first contrast loss includes L-Contra loss. Specifically, according to the idea of ​​InfoNCE, it can be regarded as an S-class classification task. This application takes EEG and EYE data as an example using Contrastive Loss function, and the prepared real data GT can be defined as [0,1,...,S-1]. The contrast loss L-Contra is calculated by the output of the real data and the emotion-related encoder as follows:

[0033] For step S13, for the EI encoder, the ER encoder contains several Transformer blocks consisting of feed-forward and layer normalization.

[0034] As an implementation mode, the second contrast loss function includes: a contrast logarithmic ratio upper limit loss function, and the second contrast loss includes: an L-CLUB loss; In this embodiment, in order to minimize the correlation between the features extracted by the two encoders, the CLUB (Contrastive Log-ratio Upper Bound) method is used. Unlike the traditional InfoNCE method referenced in step S12, this application increases the distance between the EI encoder output and the ER encoder output to ensure that the information extracted by the EI encoder is as independent of the emotional features as possible. Given any modality X, according to the variational CLUB term with unknown conditional distribution, the CLUB loss of X is defined as:

[0035] in, and are the outputs of the ER and EI encoders respectively. is a variational distribution with parameter θ, which is used to approximate .

[0036] For step S14, this method adds a decoder to the modality to reconstruct the original data. The decoder receives the output from the ER and EI encoders. Adding reconstruction loss can help the two encoders to better extract data features. The reconstruction loss of modality X is defined as:

[0037] Where DX represents the decoder of mode X. The loss is determined by the error between the reconstructed emotion data calculated by DX and the original modality X data.

[0038] For step S15, in order to make better use of multimodal data, this method proposes an M2M CPC multi-to-multi comparison prediction coding module. The design idea of ​​this module is as follows: changes in emotions usually lead to synchronous changes in multiple modalities, so it is more reasonable to use the fusion features of multiple modalities to predict subsequent features than to use a single modality. M2M CPC accepts and fuses the features of the past moments from multiple modalities, and uses the fused features to predict the features of each modality at the future moment. Taking EEG and EYE data as an example, after passing through their emotion-related encoders, we get the embedding vectors:

[0039] Where T is the length of the time window, S is the number of data samples, and D is the embedding dimension. This method uses two layers of LSTM as the autoregressive model for each modality. Define M as the length of the observation sequence and N as the prediction step size, and we get:

[0040] Where D′ is the hidden dimension of LSTM, M+N≤T. Then connect and to predict the future, with N steps for each modality. This method uses InfoNCE loss to optimize the module. The loss function consists of four parts: the loss of the predicted vector and the real vector in the same modality, and the loss of the predicted vector and the true vector in the other modality. and It is defined as a set of negative samples and one positive sample, where A, B∈{eeg,eye}. Then we get:

[0041] Where [·] represents the cascade operation. Therefore, the complete M2M CPC loss is:

[0042] In general, in order to understand the characteristics between different modalities, this method constructs a new M2M CPC module, such as Figure 3As shown. The traditional CPC module extracts data features by using an autoregressive model to predict the correlation between future embedding vectors and current embedding vectors in the sequence. For human emotional changes, it usually leads to simultaneous changes in multiple physiological signals of the human body. In actual situations, this method can comprehensively judge the current emotional state of the subject by detecting changes in various physiological signals, which is an important reason why traditional multimodal models usually have better performance than unimodal models. Therefore, in the M2M CPC module, this method uses current embedding vectors from multiple modalities to predict future vectors of each modality, such as using past EEG features to predict subsequent EEG features.

[0043] For step S16, considering the losses of the above parts, the final loss function of the pre-training stage is:

[0044] Taking all the above losses into account, this method derives the final loss of the pre-training stage:

[0045] Among them, α′, β′, γ′ and λ′ are corresponding coefficients. The final loss is used to train the classifier in the cross-modal emotion recognition model, the emotion-related encoder for extracting emotion features for each modality, the emotion-independent encoder for extracting emotion-independent features, and the decoder based on the multi-layer perceptron. The pre-training ends when the preset goal is reached, for example, when the final loss is less than the preset threshold.

[0046] As an implementation mode, the emotion-related encoder of the present application uses a spatiotemporal sparse attention mechanism for encoding, which is described in detail. The emotion-related encoder uses a spatiotemporal sparse attention mechanism for encoding, including: selecting the average values ​​of the spatial features and temporal features of the data at the 1st moment, the t-1th moment, and the tth moment in the time window to calculate the Value value and Key value at the tth moment.

[0047] In this embodiment, for EEG and some other signals, this method also designs a special STS (Spatial and Temporal-Sparse, spatiotemporal sparse attention mechanism).

[0048] In actual training, this method uses a variety of signals, including: EEG, ECG, and PPC. These signals have the following in common: they are all collected continuously for a long time through electrodes at certain fixed positions on the human body. Therefore, these signals have strong intrinsic characteristics in both spatial and temporal dimensions. This method proposes a spatiotemporal sparse (STS) attention mechanism.

[0049] This method makes full use of this feature. The structure of STS Attention is as follows Figure 4 As shown. For the data of any time window, in addition to learning the structural information between electrodes, this method also integrates the data of the time dimension. After linear transformation, the average value of the values ​​at time t, t-1 and the initial time is and .

[0050] The reason for this design is as follows: first, the initial data in each time window determines the basic information of the data segment; second, the data at each moment is closely related to the previous moment. So now we have:

[0051] Where ⊕ represents the sum and average. Given the attention dimension Datt, the attention can be calculated as follows:

[0052] As an implementation manner, after the pre-training, the method further includes: Receive unimodal emotion data, call the emotion-related encoder corresponding to the unimodality for processing, and use the classifier to obtain the predicted emotion. Based on the loss of the predicted emotion and the baseline emotion corresponding to the unimodal emotion data, only the classifier of the cross-modal emotion recognition model is trained to update parameters.

[0053] In this embodiment, if Figure 5 As shown in the figure, it is fine-tuning after pre-training. During fine-tuning, only the ER encoder with fixed parameters is used to extract the features of a certain modality and train a classifier that is shared by all modalities. In order to make full use of the information on the spatial and temporal scales of EEG data.

[0054] After pre-training is completed, this method selects data from any modality to train a classifier. In the fine-tuning stage, only the ER encoder and mapping layer corresponding to the modality are used, and the decoder and EI encoder are no longer used. The ER encoder parameters and mapping layer are frozen, that is, they are no longer updated during the training process, and only the classifier parameters are updated. The loss function at this stage is:

[0055] Among them, y is the emotion classification result. Only the decoder is trained by the loss function determined by the predicted emotion classification result and the real emotion.

[0056] During testing, only data from any one modality is needed, and the ER encoder and mapping layer corresponding to the modality, as well as the classifier, are used for emotion recognition. Since the feature space has been well trained, the differences between modalities are eliminated and multimodal information is retained, so single-modal data is sufficient to compare with multimodal results. Compared with some cross-modal work in the prior art that only focuses on the conversion from EEG to eye movement, this method can accept input from any modality.

[0057] It can be seen from this implementation that the method combines a variety of contrastive learning methods to extract emotion-related features from different modalities and align them to the same feature space, ensuring that the features extracted by each modality through the ER encoder are as similar as possible, so that after subsequent fine-tuning, each modality can share a classifier. Furthermore, the many-to-many contrast prediction coding module can assist in feature alignment and multimodal fusion in cross-modal tasks. This module has very good generalization capabilities and can be transplanted to existing methods to help improve their performance. At the same time, the spatiotemporal sparse attention mechanism of this method can simultaneously combine the temporal and spatial information of the EEG data, making full use of the stable spatial information and rich temporal information of the EEG data. In this way, it is possible to achieve emotion recognition using the encoder and classifier corresponding to any one modality with only the emotion data of that modality, without the need for complete multimodal data.

[0058] The method is experimentally illustrated. In the experiment, the method uses five public multimodal emotion datasets: SEED, SEED-IV, SEED-V, DEAP and DREAMER. All five datasets use video stimulation to induce the emotions of the subjects. The basic information of these datasets is shown in Figure 6. The method extracts DE features from EEG signals. Since the EEG signals in the DEAP dataset have been filtered by a 4-75Hz filter, the method only extracts four frequency bands from the EEG signals in DEAP, without the delta band (1-3Hz). For the processing of ECG and PPS signals, 10-dimensional and 48-dimensional features are extracted, respectively. For EYE signals, 33-dimensional statistical features and computational features are used.

[0059] For the division of training and test sets, due to the fixed emotion labels corresponding to each video clip in the SEED series datasets, SEED, SEED-IV and SEED V datasets are divided in the ratio of 9:6, 16:8 and 10:5 respectively. The labels in the DEAP and DREAMER datasets are the scores of the subjects on certain evaluation indicators, including valence, arousal and dominance. This label leads to uneven distribution of data, so this method performs four and three cross-validations on DEAP and DREAMER respectively. The training and testing ratios of each fold are 3:1 and 2:1 respectively.

[0060] This method selects balanced accuracy and kappa score as the classification evaluation indicators for the SEED series datasets. Since a single subject in the DEAP and DREAMER datasets has only one label category in a certain fold, it is difficult to calculate the kappa score for each subject. Therefore, this method calculates the balanced accuracy and F1 score for these two datasets.

[0061] Regarding baseline comparison, this method selects several state-of-the-art cross-modal methods as baselines, including BDAE-regressor, BDAE-cGAN, ECO-FET, CLIP, and Uni-Code. This method uses the same encoder and fine-tuning process as the CLIP model to ensure fairness in comparison. This method also compares some classic and state-of-the-art supervised multimodal methods, including BDAE, ETF (Emotion Transformer Fusion), VigilanceNet, and MAET models to determine whether the model's pre-training and contrastive learning modules play a role in learning and fusing key information between different modalities.

[0062] Compared with the cross-modal method, the cross-modal and unimodal experimental results of the SEED series datasets are shown in Figure 7 The results of DEAP and DREAMER datasets are shown in Figure 8 As shown in the figure, our M2S model outperforms all previous methods in various settings. Both cross-modal and unimodal tasks significantly improve accuracy. In all cross-modal tasks, our model is generally more than 5 percentage points better than the existing suboptimal methods, with the largest improvement being more than 10 percentage points (independent sample t-test, p<0.05). After fine-tuning using one modality, the test performance of unimodal is slightly better than that of cross-modal.

[0063] Regarding the comparison with multimodal methods, for example, the results compared with supervised multimodal methods are as follows Fig. 9 As shown. Since EEG is still the most interesting data in most studies, this method only shows the fine-tuning results of our M2S model on the EEG modality. As shown in the figure, with only one two-layer MLP classifier for fine-tuning, our model outperforms most supervised multimodal methods. At the same time, our method tested the unimodal supervised training of the ER encoder using only EEG. Its accuracy is slightly lower than that using the pre-trained model, but it can still reach the performance of some multimodal methods. The experimental results show that the pre-training process of our method plays an important role, and the introduction of the contrastive learning module also enables the features of different modes to be effectively fused.

[0064] Regarding the impact of different functional losses, e.g. Figure 7As shown in Figure 1, this method conducts ablation experiments on the four loss functions in the model. After removing certain loss functions, the performance of the model will decrease to varying degrees. Removing the loss function of contrastive learning has the most serious impact on cross-modal tasks.

[0065] Regarding the role of the STS attention mechanism, in order to verify the effectiveness of the STS attention mechanism, this method conducted the following experiments on the SEED series datasets: replacing the STS attention mechanism with a conventional multi-head attention mechanism or directly deleting the attention mechanism. Since the STS attention mechanism is only applicable to signals collected by electrodes, this method only presents the results of EEG unimodal fine-tuning. Fig.10 As shown, the STS attention mechanism significantly improves the performance of the model.

[0066] Ablation study of the M2M CPC module,In order to analyze the impact of the M2M CPC module, this method first tests the performance of the existing method after adding this module. Fig.11 As shown. After adding M2M CPC, each modality learns each other's features through predictive coding, achieving feature fusion and improving accuracy. This method also tests the impact of different time window lengths and prediction steps on the final accuracy. Heat maps of two cross-modal tasks are drawn on the SEED dataset. Fig.12 In the same hyperparameter tuning range, when the time window length is fixed, the accuracy of the model generally increases with the increase of the prediction step, and then decreases. When the prediction step length is fixed, a longer time window helps the model perform better.

[0067] This method draws the t-SNE dimensionality reduction distribution map of different modalities after pre-training. Fig.13 The distribution of training data of two subjects in the SEED dataset after pre-training using different models is shown. It can be seen that the model of this method better integrates the features of different modalities while maintaining the temporal structure.

[0068] In summary, this method proposes a cross-modal learning method M2S, which can reduce multimodal dependencies in emotion recognition and achieve multimodal performance using only a single modality. Various experiments were conducted on multiple datasets, and the experimental results showed that the model obtained by this method achieved SOTA performance in cross-modal tasks. In addition, the M2M CPC module proposed in this method can be extended to other models to learn and fuse features of different modalities and help existing models perform better.

[0069] like Fig.14 The figure is a schematic diagram of the structure of a training system for a cross-modal emotion recognition model provided by an embodiment of the present invention. The system can execute the training method for a cross-modal emotion recognition model described in any of the above embodiments and be configured in a terminal.

[0070] A cross-modal emotion recognition model training system 10 provided in this embodiment includes: a data input module 11, an alignment module 12, an emotion-irrelevant information filtering module 13, a reconstruction module 14, an auxiliary module 15 and a pre-training module 16.

[0071] The data input module 11 is used to input the initial emotion data of multiple different modalities into the cross-modal emotion recognition model, wherein the cross-modal emotion recognition model includes: a classifier for emotion recognition, an emotion-related encoder for each modality for extracting emotion features, an emotion-independent encoder for extracting emotion-independent features, and a decoder based on a multi-layer perceptron; the alignment module 12 is used to reduce the gap between the outputs of the emotion-related encoders of each modality based on a first contrast loss function to obtain a first contrast loss; the emotion-independent information filtering module 13 is used to increase the gap between the outputs of the emotion-related encoders and the outputs of the emotion-independent encoders of each modality based on a second contrast loss function to obtain a second contrast loss; the reconstruction module 14 is used to The respective decoders of each modality receive the output of the emotion-related encoder and the output of the emotion-independent encoder, and determine the reconstruction loss based on the reconstructed emotion data obtained by the decoder and the initial emotion data; the auxiliary module 15 is used to use the many-to-many contrast prediction coding module to perform multimodal fusion on the output of the emotion-related encoder of each modality to obtain historical fusion features, predict the generalized future features of each modality through the historical fusion features, and determine the prediction loss for auxiliary modality alignment based on the historical fusion features and the future features; the pre-training module 16 is used to pre-train the cross-modal emotion recognition model based on the first contrast loss, the second contrast loss, the reconstruction loss and the prediction loss until the preset goal is achieved.

[0072] The embodiment of the present invention further provides a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the training method of the cross-modal emotion recognition model in any of the above method embodiments; As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows: Inputting the initial emotion data of multiple different modalities into a cross-modal emotion recognition model, wherein the cross-modal emotion recognition model comprises: a classifier for emotion recognition, an emotion-related encoder for extracting emotion features for each modality, an emotion-independent encoder for extracting emotion-independent features, and a decoder based on a multi-layer perceptron; Narrowing the gap between the outputs of the emotion-related encoders of each modality based on a first contrast loss function to obtain a first contrast loss; The difference between the output of the emotion-related encoder and the output of the emotion-independent encoder of each modality is increased based on the second contrast loss function to obtain a second contrast loss; The decoder of each modality receives the output of the emotion-related encoder and the output of the emotion-independent encoder, and determines the reconstruction loss based on the reconstructed emotion data obtained by the decoder and the initial emotion data; Using a many-to-many comparison prediction coding module, the output of the emotion-related encoder of each modality is multimodally fused to obtain a historical fusion feature, predicting the generalized future features of each modality through the historical fusion feature, and determining a prediction loss for assisting modality alignment based on the historical fusion feature and the future feature; The cross-modal emotion recognition model is pre-trained based on the first contrast loss, the second contrast loss, the reconstruction loss, and the prediction loss until a preset goal is achieved.

[0073] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by the processor, the training method of the cross-modal emotion recognition model in any of the above method embodiments is executed.

[0074] Fig.15 is a schematic diagram of the hardware structure of an electronic device for a training method of a cross-modal emotion recognition model provided by another embodiment of the present application, such as Fig.15 As shown, the device includes: One or more processors 1510 and memory 1520, Fig.15 A processor 1510 is taken as an example. The apparatus of the cross-modal emotion recognition model training method may further include: an input device 1530 and an output device 1540 .

[0075] The processor 1510, the memory 1520, the input device 1530 and the output device 1540 may be connected via a bus or other means. Fig.15 The example of connecting through bus is taken in the following.

[0076] The memory 1520, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the training method of the cross-modal emotion recognition model in the embodiment of the present application. The processor 1510 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 1520, that is, the training method of the cross-modal emotion recognition model in the above method embodiment is implemented.

[0077] The memory 1520 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data, etc. In addition, the memory 1520 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1520 may optionally include a memory remotely arranged relative to the processor 1510, and these remote memories may be connected to the mobile device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0078] The input device 1530 may receive input digital or character information. The output device 1540 may include a display device such as a display screen.

[0079] The one or more modules are stored in the memory 1520, and when executed by the one or more processors 1510, the training method of the cross-modal emotion recognition model in any of the above method embodiments is executed.

[0080] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0081] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0082] An embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the training method of the cross-modal emotion recognition model of any embodiment of the present invention.

[0083] The electronic device of the embodiment of the present application exists in various forms, including but not limited to: (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0084] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDAs, MIDs, and UMPC devices, such as tablet computers.

[0085] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0086] (4) Other electronic devices with data processing functions.

[0087] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or also include elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

[0088] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a cross-modal emotion recognition model, comprising: Inputting the initial emotion data of multiple different modalities into a cross-modal emotion recognition model, wherein the cross-modal emotion recognition model comprises: a classifier for emotion recognition, an emotion-related encoder for extracting emotion features for each modality, an emotion-independent encoder for extracting emotion-independent features, and a decoder based on a multi-layer perceptron; Narrowing the gap between the outputs of the emotion-related encoders of each modality based on a first contrast loss function to obtain a first contrast loss; The difference between the output of the emotion-related encoder and the output of the emotion-independent encoder of each modality is increased based on the second contrast loss function to obtain a second contrast loss; The decoder of each modality receives the output of the emotion-related encoder and the output of the emotion-independent encoder, and determines the reconstruction loss based on the reconstructed emotion data obtained by the decoder and the initial emotion data; Using a many-to-many comparison prediction coding module, the output of the emotion-related encoder of each modality is multimodally fused to obtain a historical fusion feature, predicting the generalized future features of each modality through the historical fusion feature, and determining a prediction loss for assisting modality alignment based on the historical fusion feature and the future feature; The cross-modal emotion recognition model is pre-trained based on the first contrast loss, the second contrast loss, the reconstruction loss, and the prediction loss until a preset goal is achieved.

2. The method according to claim 1, wherein: After the pre-training, the method further comprises: Receive unimodal emotion data, call the emotion-related encoder corresponding to the unimodality for processing, and use the classifier to obtain the predicted emotion. Based on the loss of the predicted emotion and the baseline emotion corresponding to the unimodal emotion data, only the classifier of the cross-modal emotion recognition model is trained to update parameters.

3. The method according to claim 2, wherein: The first contrast loss function includes: a contrastive loss function, the first contrast loss includes an L-Contra loss, the second contrast loss function includes: a contrast logarithmic ratio upper limit loss function, and the second contrast loss includes: an L-CLUB loss; After performing parameter update training only on the classifier of the cross-modal emotion recognition model, the method further includes: The outputs of emotion-related encoders of multiple different modalities share the classifier for emotion recognition.

4. The method according to claim 1, wherein: The modalities include: electroencephalogram modality, eye movement modality, electrocardiogram modality, and peripheral physiological signal modality; The emotion-related encoder uses a spatiotemporal sparse attention mechanism for encoding, including: selecting the average values ​​of the spatial features and temporal features at the 1st moment, the t-1th moment, and the tth moment in the time window to calculate the Value value and the Key value at the tth moment.

5. A cross-modal emotion recognition model, comprising: A plurality of emotion encoders and a decoder based on a multi-layer perceptron after contrastive learning, wherein each emotion encoder and the decoder based on the multi-layer perceptron respectively processes emotion data of each different modality, and the emotion encoder comprises: an emotion-related encoder based on a spatiotemporal sparse attention mechanism for extracting emotion features, and an emotion-independent encoder for extracting emotion-independent features; A classifier is used to receive the emotion features output by the emotion-related encoder and output the emotion recognition result.

6. A training system for a cross-modal emotion recognition model, comprising: A data input module, used to input initial emotion data of multiple different modalities into a cross-modal emotion recognition model, wherein the cross-modal emotion recognition model includes: a classifier for emotion recognition, an emotion-related encoder for extracting emotion features for each modality, an emotion-independent encoder for extracting emotion-independent features, and a decoder based on a multi-layer perceptron; an alignment module, configured to reduce the gap between the outputs of the emotion-related encoders of each modality based on a first contrast loss function to obtain a first contrast loss; An emotion-irrelevant information filtering module, configured to increase the difference between the output of the emotion-related encoder and the output of the emotion-irrelevant encoder of each modality based on a second contrast loss function to obtain a second contrast loss; A reconstruction module, configured for receiving the output of the emotion-related encoder and the output of the emotion-independent encoder by a decoder of each modality, and determining a reconstruction loss based on the reconstructed emotion data obtained by the decoder and the initial emotion data; An auxiliary module, configured to use a many-to-many comparison prediction coding module to perform multimodal fusion on the output of the emotion-related encoder of each modality to obtain a historical fusion feature, predict the generalized future features of each modality through the historical fusion feature, and determine a prediction loss for assisting modality alignment based on the historical fusion feature and the future feature; A pre-training module is used to pre-train the cross-modal emotion recognition model based on the first contrast loss, the second contrast loss, the reconstruction loss and the prediction loss until a preset goal is achieved.

7. A training system for a cross-modal emotion recognition model, further comprising: A fine-tuning module is used to receive unimodal emotion data, call the emotion-related encoder corresponding to the unimodality for processing, and use a classifier to obtain a predicted emotion. Based on the predicted emotion and the loss of the baseline emotion corresponding to the unimodal emotion data, only the classifier of the cross-modal emotion recognition model is trained to update parameters.

8. A storage medium having a computer program product stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 4 are implemented.

9. A computer program product having instructions embedded on a storage medium, wherein the instructions implement the steps of the method according to any one of claims 1 to 4.

10. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 4.

Images