Double-attention cross-modal emotion recognition system based on constrained adversarial network

Abstract

The embodiment of the invention provides a double-attention cross-modal emotion recognition system based on a constrained adversarial network and a model training method of the system. The system comprises a plurality of generators based on multi-head self-attention, a multi-head cross-attention layer, a common joint space, a plurality of decoders corresponding to the generators, a modal discriminator and an emotion classifier. According to the embodiment of the invention, two attention mechanisms are adopted to effectively extract single-mode and multi-mode features. And excellent performance is achieved in the aspect of emotion recognition by interpreting an electroencephalogram and eye movement. And a common joint space is coded through adversarial learning. In the training process, a common joint space is constructed by aligning single-mode information and inter-mode complementary information, so that the distribution difference between different modes is reduced. In the testing process, a single mode is converted into representation of a common joint space, and the performance equivalent to that of multiple modes is achieved in emotion recognition.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition, and in particular to a dual-attention cross-modal emotion recognition system based on a constrained adversarial network and a model training method for the dual-attention cross-modal emotion recognition system based on a constrained adversarial network. Background Art

[0002] In order to more accurately identify human emotions, existing technologies use human EEG and eye movement signals for cross-modal emotion recognition. Specifically, the following methods are usually used: 1. MAET (Multimodal Adaptive Emotion Transformer) model. This model maps unimodal features to a feature space containing multimodal information through encoding. Specifically, it uses a deep recurrent neural network to explore recurrent connections and generate multimodal features from eye movement signals. The model consists of a multi-view encoding module, an adaptive transformer, a hybrid transformer, and several classifiers. During training, it is first trained using EEG and eye movement features to enable it to handle multimodal inputs. Then, the backbone of MAET is frozen, and emotional cue adjustment is introduced, and only the emotional cue and single modality classifiers are adjusted. After training, MAET can use EEG or eye movement or both EEG and eye movement as input to obtain emotion recognition results.

[0003] 2. Recognition framework based on generative adversarial networks. Eye movements with noisy data are used to generate multimodal features through conditional generative adversarial networks (GANs). The framework consists of two parts: the multimodal feature fusion part, which reconstructs and classifies the encoded EEG and eye movement signals so that the multimodal fusion encoder can extract multimodal information related to emotions; the second part is the multimodal feature generation part based on conditional generative adversarial networks, which uses eye movement signals containing noise to generate multimodal signals, and uses discriminator constraints to propose representations that are not related to emotions. In the test phase, only eye movement signals need to be used as input to obtain emotion recognition results.

[0004] 3. A cross-modal method based on generative adversarial learning. The framework of this method includes two training stages. In the first stage, the EEG and eye movement signals are encoded and fused, and the classification loss and reconstruction loss are used to extract high-level features. In the second stage, the encoded eye movement features are used to generate encoded EEG features, and the high-level feature generator is constrained by the mean square error and the discriminator to complete the generation of high-level features. Similarly, in the test stage, only the eye movement signal needs to be used as input to obtain the emotion recognition result.

[0005] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art: Both the multimodal adaptive emotion Transformer model and the recognition framework based on generative adversarial networks attempt to map unimodal information into a multimodal information space. Therefore, a multimodal encoding fusion framework needs to be trained in advance to generate the required multimodal information joint space, and then map the unimodal signal into a multimodal signal. In this process, unimodal information may be missing.

[0006] The cross-modal method based on generative adversarial learning directly maps the features of one modality to another modality, which makes it too difficult to train the generator and is more likely to cause information loss. This method can handle the scenario of decision information recognition, but it is not ideal in the scenario of emotion recognition. Summary of the invention

[0007] In order to at least solve the above-mentioned problems existing in cross-modal emotion recognition in the prior art.

[0008] In the first aspect, the present invention provides a dual-attention cross-modal emotion recognition system based on a constrained adversarial network, characterized in that it includes: multiple generators based on multi-head self-attention, multi-head cross-attention layers, a common joint space, multiple decoders corresponding to the generators, a modality discriminator and an emotion classifier, wherein: The multiple multi-head self-attention based generators are used to receive unimodal data related to the emotion recognition and generate unimodal features; The multi-head cross attention layer is used to generate a multi-modal fusion feature that classifies a plurality of single-modal data related to the emotion recognition into a same dimension; The common joint space is used to complement the features of the multiple single-modal features and the fusion features through adversarial learning coding, reduce the feature distribution differences of each modality, and construct a common feature; The decoder is used to decode the universal features to generate reconstructed unimodal features of unimodal data related to the emotion recognition; The modality identifier is used to identify the modality category corresponding to the general feature; The emotion classifier is used to obtain emotion recognition results based on the reconstructed unimodal features and the common features and using the modal categories for constraints, wherein the generator and the modal discriminator are learned by an adversarial network.

[0009] In a second aspect, an embodiment of the present invention provides a model training method for a dual-attention cross-modal emotion recognition system based on a constrained adversarial network: Inputting the first modality data and the second modality data related to the emotion recognition into the dual-attention cross-modality emotion recognition system, and obtaining the first modality features and the second modality features by using a plurality of multi-head self-attention-based generators, wherein the first modality includes an electroencephalogram and the second modality includes eye movement; Using a multi-head cross attention layer, the first modal features and the second modal features are classified into multi-modal fusion features of the same dimension; Using a common joint space, the first modal feature, the second modal feature, and the fusion feature are complementary to each other through adversarial learning coding, so as to reduce the feature distribution difference of each modality and construct a common feature; Decoding the universal features by using a first modality decoder and a second modality decoder to obtain first modality reconstruction features and second modality reconstruction features, wherein the first modality reconstruction features and the second modality reconstruction features are used to compensate for the single modality information lost due to feature complementation of the adversarial learning coding, and determining the reconstruction loss by using the first modality features, the second modality features, the first modality reconstruction features, and the second modality reconstruction features; Using a modality discriminator to identify that the universal feature is generated by a multi-head self-attention generator of a corresponding modality category among the plurality of multi-head self-attention generators, so as to retain information of the modality category; The reconstructed unimodal features and the universal features are used by an emotion classifier, and the modal category is used for constraint to obtain a predicted emotion recognition result, and a cross entropy loss between the predicted emotion recognition result and the corresponding true emotion label is determined as a classifier loss; Performing adversarial training on the modality discriminator and the generator using the first modality feature, the second modality feature, and the fusion feature to obtain a discriminator adversarial loss and a generator adversarial loss; The reconstruction loss, classifier loss, discriminator adversarial loss and generator adversarial loss are used to determine the total loss, and the dual-attention cross-modal emotion recognition system is trained using the total loss to obtain a trained dual-attention cross-modal emotion recognition model.

[0010] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the model training method for a dual-attention cross-modal emotion recognition system based on a constrained adversarial network according to any embodiment of the present invention.

[0011] In a fourth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of a model training method for a dual-attention cross-modal emotion recognition system based on a constrained adversarial network of any embodiment of the present invention are implemented.

[0012] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the steps of the model training method of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network of any embodiment of the present invention are implemented.

[0013] The beneficial effects of the embodiments of the present invention are: two attention mechanisms are used to effectively extract unimodal and multimodal features. Excellent performance has been achieved in interpreting electroencephalograms and eye movements for emotion recognition. A common joint space is encoded through adversarial learning. During the training process, a common joint space is constructed by aligning unimodal information with complementary information between modalities to reduce the distribution differences between different modalities. During the testing process, the unimodality is converted into a representation of the common joint space, and performance comparable to multimodality is achieved in emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 It is a structural schematic diagram of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network provided by an embodiment of the present invention; Figure 2 It is a structural schematic diagram of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network provided by another embodiment of the present invention; Figure 3 It is a schematic diagram of a feature fusion structure of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network provided by an embodiment of the present invention; Figure 4 It is a flow chart of a model training method of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network provided by an embodiment of the present invention; Figure 5 It is a schematic diagram of adversarial training of a model training method of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network provided by an embodiment of the present invention; Figure 6It is a test flow chart of a model training method of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network provided by an embodiment of the present invention; Figure 7 It is a schematic diagram of a comparison between a model training method of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network and a multimodal model (using a single modality) provided by an embodiment of the present invention; Figure 8 It is a schematic diagram of a model training method of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network provided by an embodiment of the present invention, compared with a single-modal model; Fig. 9 It is a schematic diagram of a model training method of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network provided by an embodiment of the present invention, compared with a multimodal model; Fig.10 It is a schematic diagram of an ablation experiment of a model training method of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network provided by an embodiment of the present invention; Fig.11 It is a confusion matrix diagram of a model training method of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network provided by an embodiment of the present invention; Fig.12 It is a distribution diagram in a public joint space of a model training method for a dual-attention cross-modal emotion recognition system based on a constrained adversarial network provided by an embodiment of the present invention; Fig.13 A schematic diagram of the structure of an embodiment of an electronic device for model training of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] like Figure 1 Shown is a structural schematic diagram of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network provided by an embodiment of the present invention. The system can perform model training and emotion recognition of the dual-attention cross-modal emotion recognition system based on a constrained adversarial network, and is configured in a terminal.

[0018] The present embodiment provides a dual-attention cross-modal emotion recognition system based on a constrained adversarial network, characterized in that it includes: multiple generators based on multi-head self-attention, multi-head cross-attention layers, a common joint space, multiple decoders corresponding to the generators, a modality discriminator and an emotion classifier, wherein: The multiple multi-head self-attention based generators are used to receive unimodal data related to the emotion recognition and generate unimodal features; The multi-head cross attention layer is used to generate a multi-modal fusion feature that classifies a plurality of single-modal data related to the emotion recognition into a same dimension; The common joint space is used to complement the features of the multiple single-modal features and the fusion features through adversarial learning coding, reduce the feature distribution differences of each modality, and construct a common feature; The decoder is used to decode the universal features to generate reconstructed unimodal features of unimodal data related to the emotion recognition; The modality identifier is used to identify the modality category corresponding to the general feature; The emotion classifier is used to obtain emotion recognition results based on the reconstructed unimodal features and the common features and using the modal categories for constraints, wherein the generator and the modal discriminator are learned by an adversarial network.

[0019] In this embodiment, it is considered that in the prior art, more data is used for training emotion recognition, or the structure of the multimodal encoder is changed. However, the essential improvements of the prior art all follow the idea of ​​mapping one modality to multimodality or mapping one modality to another modality. The dual-attention cross-modal emotion recognition system of the present application is different from the improvement idea of ​​the prior art, which is to construct a common joint space with adversarial learning and constraints to align the distribution of the two modalities.

[0020] In order to avoid the loss of unimodal information in cross-modal emotion recognition and ensure the performance in emotion recognition scenarios, the dual-attention cross-modal emotion recognition system of the present application constructs a dual-attention constrained adversarial network. This system can be called CANDA (Constrained Adversarial Network with Dual Attention). In practical applications, it can also be called a dual-attention cross-modal emotion recognition model.

[0021] The overall framework of the dual-attention cross-modal emotion recognition system of the present application aims to establish a joint feature space, combining multiple single modalities related to emotion recognition (for example, single-modal data includes: EEG data obtained by using an EEG acquisition device and eye movement data obtained by using an eye movement acquisition device) and the multimodal features of these multiple single modalities to narrow the distribution gap between them. The purpose of this design is that, through subsequent model training of the dual-attention cross-modal emotion recognition system of the present application, the trained encoder can infer the features in the common joint space of the single modality during the emotion recognition process.

[0022] Figure 1 The figure shows a dual-attention cross-modal emotion recognition system with two types of unimodal information. In the construction of the system framework, other unimodal information besides EEG and eye movement can also be introduced. For example, when there is other unimodal information, the corresponding generator and decoder can be added to extract modal information and complementary information from different unimodal information respectively, and input different unimodal information into the multi-head cross-attention layer through linear transformation. Therefore, the construction of the system framework can be adapted according to the type of unimodal information required, which can be EEG and eye movement signals, or other signals can be introduced, such as breathing signals (when sad, the breathing rate will be short, and when fearful, the breathing rate will slow down), and heart rate (skin electrical signals have been shown to be closely related to people's nervousness and excitement). This is not limited here. In this patent, only the effects of EEG and eye movement signals are discussed.

[0023] The dual-attention cross-modal emotion recognition system of the present application includes: multiple generators based on multi-head self-attention, multi-head cross-attention layers, a common joint space, multiple decoders corresponding to the generators, a modality discriminator and an emotion classifier.

[0024] The modality discriminator is used to identify different modalities, including various types of unimodal and fused multimodal. The dual attention mechanism of this application includes: multi-head self-attention mechanism (extracting unimodal features) and cross-attention mechanism (learning multimodal features), and the unimodal features and multimodal features are complementary in a common joint space.

[0025] Adversarial networks refer to adversarial learning between a generator and a modality discriminator based on multi-head self-attention, and the adversarial network also includes: a decoder and a sentiment classifier, which help to maintain modality-specific and sentiment-related representations, respectively. Adversarial learning makes the representations of unimodal and multimodal similar and contains information from different aspects, so only one modality is sufficient to generate features for testing in a common space.

[0026] In order to further explain the dual-attention cross-modal emotion recognition system based on constrained adversarial network in this application in more detail, Figure 2 A more specific example of the dual-attention cross-modal emotion recognition system based on constrained adversarial network of the present application, wherein the multiple generators based on multi-head self-attention include: an EEG feature generator, an eye movement feature generator; The plurality of single-modal data related to emotion recognition include: electroencephalogram data and eye movement data, and the modality categories include: electroencephalogram, eye movement and fusion categories; The EEG feature generator is used to receive EEG data and generate unimodal EEG features; The eye movement feature generator is used to receive eye movement data and generate unimodal eye movement features; The multi-head cross attention layer is used to classify the electroencephalogram data and the eye movement data into multi-modal fusion features of the same dimension.

[0027] The multi-head self-attention based generator includes a transformer-based encoder, wherein the encoder is used to learn unimodal features from unimodal data related to the emotion recognition, and the encoder includes: position embedding, multi-head self-attention layers, residual connections and layer normalization layers, a feedforward network, and a common transformation layer.

[0028] The dual attention mechanism (Dual Attention) of the dual attention cross-modal emotion recognition system based on constrained adversarial network in this application uses two attention mechanisms to effectively extract unimodal and multimodal features. The multi-head self-attention mechanism in Transformer learns unimodal representations, which can effectively perform emotion recognition from EEG and eye movements. In addition, the multi-head cross-attention layer between EEG and eye movements fuses multimodal information and encodes the complementary characteristics of the common joint space through adversarial learning. The basic principles of the two attention mechanisms can be expressed as: For the multi-head self-attention mechanism, the constrained adversarial network-based dual-attention cross-modal emotion recognition system of this application includes an EEG generator and an eye movement generator. Therefore, there are two transformer-based encoders that learn representations from EEG and eye movements, respectively. Both encoders contain position embedding, multi-head self-attention, and a common transformation layer. Since both are time series, where time information is crucial for decoding emotions, self-attention is performed in the time domain to further explain the emotional state. First, in order to obtain the information of the time series, the position embedding elements are aggregated into the input signal and Figure 2The classification token denoted as cls in is concatenated to the input sequence for classification. Each encoder includes L identical layers consisting of multi-head self-attention and fully connected feed-forward network sublayers in addition to the position embedding layer and the common transformation layer. In addition, each sublayer starts with layer normalization and is contained by residual connections to mitigate internal covariate shift and enhance the stability of the model.

[0029] The h-head and multi-head self-attention are expressed as follows: Since the feature dimensions of EEG and eye movement signals are different, a fully connected network is used as a common transformation layer after multi-head self-attention to unify the dimensions of the two modalities. The distribution of the encoded eye movement and EEG features p(xeye) and p(xeeg) can be expressed as: Since the symmetric structure of the autoencoder can better support reconstruction, in addition to the position embedding layer and the common transformation layer, the same encoder structure as the EEG and eye movement decoders is designed.

[0030] The cross-attention mechanism of the multi-head cross-attention layer aims to not only utilize other modalities, such as EEG, when only one modality is available, but also utilize the multimodal features of EEG and eye movement, which are complementary. To achieve this goal, Figure 3 As shown in Figure 2, it involves extracting multimodal information from EEG and eye movements through a cross-attention mechanism. Subsequently, the fused features are used as a new modality for the modality discriminator in an adversarial learning framework. This helps integrate multimodal fusion signals in a common joint space: Similar to Transformer, we also trained L layers of multi-head cross attention, and added layer normalization and feed-forward networks in each layer. Before calculating the attention score, we used linear transformation to convert the EEG and eye movement feature dimensions into the same dimension, which is one of the hyperparameters of the overall structure during training.

[0031] about Figure 2The EEG decoder, modality discriminator, emotion classifier, and eye movement decoder in the model are shown in Figure 1. The EEG decoder and the eye movement decoder generate reconstructed unimodal features of EEG and eye movement respectively from the common features. The modality discriminator is used to identify the modality category corresponding to the common features; the emotion classifier is used to obtain the emotion recognition result based on the reconstructed unimodal features and the common features, and using the modality category for constraints. The significance of the specific settings and the adversarial learning process will be further explained in the following model training.

[0032] It can be seen from the above implementation that in the structure of the dual-attention cross-modal emotion recognition system based on constrained adversarial network of the present application, two attention mechanisms are used to effectively extract unimodal and multimodal features. The multi-head self-attention mechanism in Transformer learns unimodal representations and achieves outstanding performance in interpreting EEG and eye movements for emotion recognition. In addition, the cross-attention between EEG and eye movements fuses multimodal information and encodes the complementary characteristics of the common joint space through adversarial learning.

[0033] like Figure 4 The flowchart of a model training method of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network provided by an embodiment of the present invention includes the following steps: S11: inputting the first modality data and the second modality data related to the emotion recognition into the dual-attention cross-modality emotion recognition system, and obtaining the first modality feature and the second modality feature by using a plurality of multi-head self-attention-based generators, wherein the first modality includes an electroencephalogram and the second modality includes eye movement; S12: using a multi-head cross attention layer to classify the first modal features and the second modal features into multi-modal fusion features of the same dimension; S13: using a common joint space to complement the first modal feature, the second modal feature, and the fusion feature through adversarial learning encoding, thereby reducing the feature distribution difference of each modality and constructing a common feature; S14: Decoding the universal feature by using a first modality decoder and a second modality decoder to obtain a first modality reconstruction feature and a second modality reconstruction feature, wherein the first modality reconstruction feature and the second modality reconstruction feature are used to compensate for the single modality information lost due to feature complementation of the adversarial learning coding, and the reconstruction loss is determined by the first modality feature, the second modality feature, the first modality reconstruction feature and the second modality reconstruction feature; S15: using a modality discriminator to identify that the universal feature is generated by a multi-head self-attention generator of a corresponding modality category among the plurality of multi-head self-attention generators, so as to retain information of the modality category; S16: using the reconstructed unimodal features and the common features through an emotion classifier and using the modality category for constraints to obtain a predicted emotion recognition result, and determining the cross entropy loss of the predicted emotion recognition result and the corresponding true emotion label as the classifier loss; S17: Perform adversarial training on the modality discriminator and the generator using the first modality feature, the second modality feature, and the fusion feature to obtain a discriminator adversarial loss and a generator adversarial loss; S18: Determine the total loss using the reconstruction loss, classifier loss, discriminator adversarial loss, and generator adversarial loss, and use the total loss to perform model training on the dual-attention cross-modal emotion recognition system to obtain a trained dual-attention cross-modal emotion recognition model.

[0034] In this embodiment, the model training is performed on the constructed dual-attention cross-modal emotion recognition system based on constrained adversarial network. For the sake of convenience, step S11 is illustrated with two modalities: electroencephalogram (EEG) and eye movement. The subjects are asked to wear an EEG acquisition device and an eye movement acquisition device to watch the emotion-inducing material. During the process of watching the emotion-inducing material, the EEG data and eye movement data of the subjects are collected. Since the emotion-inducing material is prepared in advance for training, the real emotion label corresponding to the emotion-inducing material segment is also known in advance.

[0035] The first modal data is EEG data and the second modal data is eye movement data. These two data are input into a dual-attention cross-modal emotion recognition system based on a constrained adversarial network. The EEG modal features and eye movement modal features are obtained based on the EEG generator and the eye movement generator.

[0036] For steps S12-S17, a multi-head cross-attention layer is used to classify the EEG modal features and eye movement modal features into multi-modal fusion features of the same dimension, and a common joint space is used to encode the EEG modal features, eye movement modal features and fusion features through adversarial learning to complement each other, thereby reducing the differences in feature distribution of each modality and constructing common features. The modality category corresponding to the universal feature is identified by the modality discriminator. After determining the modality category of the universal feature, it can be inferred from the modality category which category of multi-head self-attention generator this universal feature is generated by. Specifically, for example, only the eye movement data signal is input, and there are eye movement generators and EEG generators in the model system. At this time, only the eye movement generator is used to process the eye movement data signal, and the processed data is input into the trained common joint space. In the common joint space, the single modality is converted into a universal representation of the common joint space. Then the modality discriminator will determine which modality category generator (eye movement generator? Or EEG generator?) the universal representation is generated by, and the corresponding modality category information (eye movement modality) is recorded. Simply put, for this step, the modality discriminator is used to identify the modality category corresponding to the universal feature. In subsequent training, the modality category information is used to constrain the eye movement modality.

[0037] Regarding adversarial learning, the traditional generative network consists of a generator G and a discriminator D, which compete with each other in the form of a minmax game. The generator G maps the noise samples z drawn from the distribution pz(z) to the generated space pg. The discriminator D initially has the ability to distinguish the generated data from the training data x distributed as Pdata(x). During training, the generator tries to produce more distributions similar to the training data so that the discriminator cannot tell the difference until it maximizes the probability of assigning true labels to both the training data and the generated data. The competition between the discriminator and the generator can be expressed as: like Figure 5 As shown in the figure, in this method, the feature generator (encoder) of EEG and eye movement signals, and the multimodal cross-attention module serve as the generator G. The task of the discriminator D can be set as: classifying eye movement p(EYE) as true, and classifying electroencephalogram p(EEG) and fused modality p(Muse) as false. At the same time, the generator tries to deceive the discriminator to classify EEG and multimodal distribution as true, and eye movement as false. The discriminator and the generator compete with each other until the discriminator can no longer distinguish eye movement from other modalities. In other words, the probability that the discriminator classifies eye movement as true is roughly equal to the probability that the discriminator classifies EEG and fused data as true.

[0038] In different training purposes, the truth or falsehood of the classification of other modalities can also be set to meet various corresponding needs. The generative adversarial structure of this model is a universal structure, and in future experiments it can be adapted to cross-modal feature extraction of other modal information such as EEG and eye movement signals. At this point, the distributions of the three modalities are mapped to a common joint space. In order to simplify the training process, this method defines the loss function as two terms: the discriminator adversarial loss Ldis and the generator adversarial loss Lgen: Among them, w is the parameter of the loss function. Regarding unimodal feature reconstruction and emotion classification, due to the dual attention mechanism of this application, the modal distribution features are complementary, reducing the difference in feature distribution of each modality, and making the distribution of each modality close. However, when the modal distribution becomes closer, the following defects will be caused, and the unique unimodal information may be lost, thereby affecting the performance of emotion recognition. Therefore, it is necessary to use a decoder to reconstruct the signal from the common joint space to the original distribution. As mentioned above, the decoder has the same structure as the encoder to enhance the ability to interpret physiological data. The distribution of reconstructed eye movements is expressed as: The reconstruction loss in this process is expressed as: In addition, each modality contains a sentiment classifier to constrain the representations in the common joint space to be relevant to the sentiment task. Therefore, the features extracted by the encoder and the fused representation are input to the classifier, which is a fully connected network, and the sentiment category is the output. The cross entropy loss is used as the classifier loss: During the training process, the discriminator needs to have the ability to distinguish which distribution the sample comes from before the generator can successfully deceive the discriminator. Therefore, the discriminator is trained first, and the generator is kept frozen after training to deceive the discriminator. Generally speaking, the number of training times for the discriminator is twice that of the generator. In different iterations, the discriminator or generator is trained separately, and this method defines different losses: the discriminator adversarial loss Ldis and the generator adversarial loss Lgen.

[0039] As an implementation manner, determining the total loss using the reconstruction loss, the classifier loss, the discriminator adversarial loss, and the generator adversarial loss includes: The sum of the reconstruction loss, the classifier loss, and the discriminator adversarial loss is determined as the total loss, or the sum of the reconstruction loss, the classifier loss, and the generator adversarial loss is determined as the total loss.

[0040] Specifically, the total loss is: Or: The dual-attention cross-modal emotion recognition system is trained using the total loss, and the training ends when the total loss reaches a preset threshold.

[0041] After the model is trained, the dual-attention cross-modal emotion recognition model can be tested or used in practice. In special scenarios where EEG and eye movement data can be collected, EEG and eye movement data can be input to perform emotion prediction. However, in real-world scenarios, EEG acquisition equipment is often difficult to implement, such as Figure 6 As shown in Figure 1, for practical scenarios where EEG cannot be collected, only the eye movement modality is needed to predict emotions. The eye movement test signal is constructed with a common feature in the common joint space by the eye movement generator, and further passed through the emotion classifier. Since the encoding space is well trained, eliminating the differences between the modalities and retaining the complementary properties, the monocular movement data is sufficient to compare with the multimodal results.

[0042] It can be seen from this implementation that during the training process, a common joint space is constructed by aligning the unimodal information with the complementary information between the modalities to reduce the distribution differences between the different modalities. And during the test process, the unimodality is converted into a representation of the common joint space to achieve performance comparable to that of multimodality. The emotion recognition model trained by this method can achieve the most advanced cross-modal emotion recognition performance. Finally, the visualization of the joint space features shows that the distribution of different modalities is consistent with the ability to discriminate different emotions.

[0043] Experiments were conducted on the dual-attention cross-modal emotion recognition model based on constrained adversarial networks trained by this method, such as Figure 7 The performance of this method model and the popular multimodal model (using single modality) and cross-modal model on the three datasets of SEED, SEED-IV and SEED-V is shown in Figure 2. This method model has achieved the best results. Figure 8 The performance of this method model and the popular single-modal model on the three datasets of SEED, SEED-IV, and SEED-V shows that the improvement effect of this method model is very significant. Fig. 9 The performance of this method model and the popular multimodal model on the three datasets of SEED, SEED-IV, and SEED-V is shown in Figure 2. The effect of this method model on the first two datasets is even better than that of the multimodal model, and the effect on the third dataset is comparable to that of the multimodal model. Fig.10 is the result of the ablation experiment of this method. Fig.11The confusion matrix of the model of this method on the three datasets. It can be seen that negative emotions are relatively difficult to identify on SEED and are often confused with neutral emotions. This trend is consistent with the research on the published SEED dataset. In addition, positive emotion recognition has the highest average accuracy of 98%. On the SEED-IV dataset, neutral and fear emotions are easy to distinguish, while sad and happy emotions are difficult to distinguish. Sad and happy states are often confused with neutral emotions. On the SEED-V dataset, fear, sadness, and neutral emotions are relatively easy to distinguish, with similar accuracy, while disgust is the most difficult to distinguish, with an accuracy of only 54%.

[0044] This method uses t-SNE to visualize and compare the original features in the SEED-V dataset with the encoded EEG and eye movement features, as shown in Fig.12 As shown in the figure, Disgust is disgust, Fear is fear, Sad is sadness, Neutral is neutral, Happy is happiness, EEG is electroencephalogram data, and Eye movement is eye movement data. It can be seen from the upper part that the features of different emotional states are entangled, while the distribution of dots and stars is separated. The gap between the two modes can be clearly seen. In contrast, in the lower part, dots and stars of the same color are clustered together, proving that the distribution difference has weakened in the common joint space. At the same time, dots and stars are well clustered together by different colors, which means that the features of the same emotional state are clustered together. Therefore, emotions are distinguishable in the common joint space, which shows that the emotion classifier plays an effective role in restricting the encoder to generate emotion-related features.

[0045] The embodiment of the present invention further provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions can execute the model training method of the dual-attention cross-modal emotion recognition system based on the constrained adversarial network in any of the above method embodiments; As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows: Inputting the first modality data and the second modality data related to the emotion recognition into the dual-attention cross-modality emotion recognition system, and obtaining the first modality features and the second modality features by using a plurality of multi-head self-attention-based generators, wherein the first modality includes an electroencephalogram and the second modality includes eye movement; Using a multi-head cross attention layer, the first modal features and the second modal features are classified into multi-modal fusion features of the same dimension; Using a common joint space, the first modal feature, the second modal feature, and the fusion feature are complementary to each other through adversarial learning coding, so as to reduce the feature distribution difference of each modality and construct a common feature; Decoding the universal features by using a first modality decoder and a second modality decoder to obtain first modality reconstruction features and second modality reconstruction features, wherein the first modality reconstruction features and the second modality reconstruction features are used to compensate for the single modality information lost due to feature complementation of the adversarial learning coding, and determining the reconstruction loss by using the first modality features, the second modality features, the first modality reconstruction features, and the second modality reconstruction features; Using a modality discriminator to identify that the universal feature is generated by a multi-head self-attention generator of a corresponding modality category among the plurality of multi-head self-attention generators, so as to retain information of the modality category; The reconstructed unimodal features and the universal features are used by an emotion classifier, and the modal category is used for constraint to obtain a predicted emotion recognition result, and a cross entropy loss between the predicted emotion recognition result and the corresponding true emotion label is determined as a classifier loss; Performing adversarial training on the modality discriminator and the generator using the first modality feature, the second modality feature, and the fusion feature to obtain a discriminator adversarial loss and a generator adversarial loss; The reconstruction loss, classifier loss, discriminator adversarial loss and generator adversarial loss are used to determine the total loss, and the dual-attention cross-modal emotion recognition system is trained using the total loss to obtain a trained dual-attention cross-modal emotion recognition model.

[0046] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by the processor, the model training method of the dual-attention cross-modal emotion recognition system based on the constrained adversarial network in any of the above method embodiments is executed.

[0047] Fig.13 is a schematic diagram of the hardware structure of an electronic device for a model training method of a dual-attention cross-modal emotion recognition system based on a constrained adversarial network provided in another embodiment of the present application, such as Fig.13 As shown, the device includes: One or more processors 1310 and memory 1320, Fig.13A processor 1310 is taken as an example. The device for the model training method of the dual-attention cross-modal emotion recognition system based on the constrained adversarial network may also include: an input device 1330 and an output device 1340.

[0048] The processor 1310, the memory 1320, the input device 1330 and the output device 1340 may be connected via a bus or other means. Fig.13 The example of connecting through bus is taken in the following.

[0049] The memory 1320, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the model training method for the dual-attention cross-modal emotion recognition system based on the constrained adversarial network in the embodiment of the present application. The processor 1310 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 1320, that is, the model training method for the dual-attention cross-modal emotion recognition system based on the constrained adversarial network in the above method embodiment is realized.

[0050] The memory 1320 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data, etc. In addition, the memory 1320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1320 may optionally include a memory remotely arranged relative to the processor 1310, and these remote memories may be connected to the mobile device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0051] The input device 1330 can receive input digital or character information. The output device 1340 can include a display device such as a display screen.

[0052] The one or more modules are stored in the memory 1320, and when executed by the one or more processors 1310, the model training method of the dual-attention cross-modal emotion recognition system based on the constrained adversarial network in any of the above method embodiments is executed.

[0053] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0054] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0055] An embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the model training method for a dual-attention cross-modal emotion recognition system based on a constrained adversarial network of any embodiment of the present invention.

[0056] The electronic device of the embodiment of the present application exists in various forms, including but not limited to: (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0057] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDAs, MIDs, and UMPC devices, such as tablet computers.

[0058] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0059] (4) Other electronic devices with data processing functions.

[0060] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or also include elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

[0061] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0062] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dual-attention cross-modal emotion recognition system based on constrained adversarial networks, characterized in that: include: A plurality of multi-head self-attention based generators, a multi-head cross-attention layer, a common joint space, a plurality of decoders corresponding to the generators, a modality discriminator and a sentiment classifier, wherein: The multiple multi-head self-attention based generators are used to receive unimodal data related to the emotion recognition and generate unimodal features; The multi-head cross attention layer is used to generate a multi-modal fusion feature that classifies a plurality of single-modal data related to the emotion recognition into a same dimension; The common joint space is used to complement the features of the multiple single-modal features and the fusion features through adversarial learning coding, reduce the feature distribution differences of each modality, and construct a common feature; The decoder is used to decode the universal features to generate reconstructed unimodal features of unimodal data related to the emotion recognition; The modality identifier is used to identify the modality category corresponding to the general feature; The emotion classifier is used to obtain emotion recognition results based on the reconstructed unimodal features and the common features and using the modal categories for constraints, wherein the generator and the modal discriminator are learned by an adversarial network.

2. The system according to claim 1, wherein: The multiple generators based on multi-head self-attention include: an electroencephalogram feature generator and an eye movement feature generator; The plurality of single-modal data related to emotion recognition include: electroencephalogram data and eye movement data, and the modality categories include: electroencephalogram, eye movement and fusion categories; The EEG feature generator is used to receive EEG data and generate unimodal EEG features; The eye movement feature generator is used to receive eye movement data and generate unimodal eye movement features; The multi-head cross attention layer is used to classify the electroencephalogram data and the eye movement data into multi-modal fusion features of the same dimension.

3. The method according to claim 1, wherein: The multi-head self-attention based generator includes a transformer-based encoder, wherein the encoder is used to learn unimodal features from unimodal data related to the emotion recognition, and the encoder includes: position embedding, multi-head self-attention layers, residual connections and layer normalization layers, a feedforward network, and a common transformation layer.

4. A model training method for a dual-attention cross-modal emotion recognition system based on constrained adversarial networks: The first modality data and the second modality data related to the emotion recognition are input into the dual-attention cross-modality emotion recognition system, and the first modality features and the second modality features are obtained by using multiple multi-head self-attention-based generators, wherein: The first modality includes EEG, and the second modality includes eye movements; Using a multi-head cross attention layer, the first modal features and the second modal features are classified into multi-modal fusion features of the same dimension; Using a common joint space, the first modal feature, the second modal feature, and the fusion feature are complementary to each other through adversarial learning coding, so as to reduce the feature distribution difference of each modality and construct a common feature; Decoding the universal features by using a first modality decoder and a second modality decoder to obtain first modality reconstruction features and second modality reconstruction features, wherein the first modality reconstruction features and the second modality reconstruction features are used to compensate for the single modality information lost due to feature complementation of the adversarial learning coding, and determining the reconstruction loss by using the first modality features, the second modality features, the first modality reconstruction features, and the second modality reconstruction features; Using a modality discriminator to identify that the universal feature is generated by a multi-headed self-attention generator of a corresponding modality category among the plurality of multi-headed self-attention generators, so as to retain information of the modality category; The reconstructed unimodal features and the universal features are used by an emotion classifier, and the modal category is used for constraint to obtain a predicted emotion recognition result, and a cross entropy loss between the predicted emotion recognition result and the corresponding true emotion label is determined as a classifier loss; Performing adversarial training on the modality discriminator and the generator using the first modality feature, the second modality feature, and the fusion feature to obtain a discriminator adversarial loss and a generator adversarial loss; The reconstruction loss, classifier loss, discriminator adversarial loss and generator adversarial loss are used to determine the total loss, and the dual-attention cross-modal emotion recognition system is trained using the total loss to obtain a trained dual-attention cross-modal emotion recognition model.

5. The method according to claim 4, wherein: After obtaining the trained dual-attention cross-modal emotion recognition model, the method includes: Inputting data related to emotion recognition into a dual-attention cross-modal emotion recognition model, wherein the data includes unimodal data and multimodal data; When the input is the unimodal data, constructing a common feature in a common joint space through a generator corresponding to the unimodal data; An emotion classifier is used to determine the emotion recognition result corresponding to the common feature.

6. The method according to claim 4, wherein: Determining the total loss using the reconstruction loss, the classifier loss, the discriminator adversarial loss, and the generator adversarial loss includes: The sum of the reconstruction loss, the classifier loss, and the discriminator adversarial loss is determined as the total loss, or the sum of the reconstruction loss, the classifier loss, and the generator adversarial loss is determined as the total loss.

7. The method according to claim 4, wherein: Model training for the dual-attention cross-modal emotion recognition system includes: when the total loss reaches a preset threshold, the training ends.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 4 to 7 are implemented.

9. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 4 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 4 to 7 are implemented.

Images