A multimodal emotion recognition method, system, device and medium

Abstract

The present invention discloses a multimodal emotion recognition method, system, device and medium, including a virtual reality scene construction module, a signal acquisition module, a feature extraction module, a feature reconstruction module and a multimodal emotion recognition model; the subject wearing the VR device is induced to have emotions through a VR emotion picture library, the physiological signals of the subject under the virtual reality device are collected, and feature extraction is performed, and feature reconstruction is performed on each module through transfer learning, and emotion classification is finally achieved through multimodal feature fusion. The present invention provides a multimodal emotion recognition system based on attention mechanism and transfer learning in a virtual reality induced state, which utilizes the advantages of different physiological signals, explores and mines their role in emotion recognition, and enhances the generalization ability of the emotion recognition system with the help of transfer learning and attention mechanism, which not only has a good development prospect in practical applications, but also creates a new model of emotion recognition system.

Description

Technical Field

[0001] The present invention relates to the field of human-machine signal recognition, and in particular to a multimodal emotion recognition method, system, device and medium. Background Art

[0002] Emotions originate from the psychological level and are one of the important characteristics of human beings. There are three common research methods for emotion classification: emotion recognition based on facial expressions or voice; emotion recognition based on peripheral physiological signals; and emotion recognition based on central nervous system signals. Traditional detection indicators such as electroencephalogram (EEG) have the characteristics of high temporal resolution, low spatial resolution, and poor positioning ability. In a virtual reality environment, compared with EEG signals, emotion recognition based on eye movement signals is more convenient, with strong anti-interference and more convenient acquisition mode. Functional Near-Infrared Spectroscopy (fNIRS), as an emerging new brain function detection technology, has the characteristics of low temporal resolution, high spatial resolution, low requirements for the measurement environment, strong anti-interference and anti-electromagnetic capabilities, and easy to carry.

[0003] Multimodal brain function detection methods can obtain multi-dimensional information of brain activity, which is convenient for describing brain neural activities from different directions. Therefore, the emotion recognition system that combines three modes of EEG signals, eye movement signals and functional near-infrared spectroscopy technology has a high research value.

[0004] There are differences in physiology and psychology between different individuals. For the same emotion inducer, the emotions induced by different individuals may not be exactly the same. Even if different individuals have the same emotion, the physiological signals they produce may be different. Therefore, it is a better solution to build an emotion recognition system with strong generalization ability through transfer learning.

[0005] With the continuous development of virtual reality, metaverse, human-computer interaction, deep learning and other fields, multimodal sentiment analysis with generalization ability based on VR devices has broad application prospects and important significance. Summary of the invention

[0006] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, an object of the present invention is to provide a multimodal emotion recognition method, system, device and medium.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] A multimodal emotion recognition method, comprising:

[0009] Build a VR panoramic emotional picture library and data set, and label the data in the data set with emotions;

[0010] The user wears a VR device and uses pictures from the VR panoramic emotional picture library for stimulation to collect EEG signals, eye movement signals, and fNIRS signals;

[0011] Use sparse autoencoders to process EEG signals, eye movement signals, and fNIRS signals respectively, extract the features of the output layer, and use the output layer features as the input layer for further processing to obtain the EEG signals, eye movement signals, and fNIRS signal features of each layer;

[0012] Through inter-modal transfer function learning, feature reconstruction is performed to obtain the reconstruction features of each modality;

[0013] The modal reconstruction features, namely the eye movement signal reconstruction features, the EEG signal reconstruction features and the fNIRS signal reconstruction features, are input into the residual multi-head self-attention module, and the three modalities of eye movement modality, EEG modality and fNIRS modality are combined in pairs. By introducing the multi-head mutual attention module of the conversation multi-head attention mechanism, cascade splicing is used for feature fusion, and then global average pooling processing is performed to classify and output the emotion label.

[0014] Furthermore, the EEG signal, eye movement signal and fNIRS signal characteristics of each layer are obtained, specifically including:

[0015] The encoding process, specifically the process of learning the function h;

[0016] The decoding process, specifically reconstructing the input signal by minimizing the reconstruction error;

[0017] The features of EEG signals, eye movement signals and fNIRS signals of each layer of the sparse autoencoder are obtained.

[0018] Furthermore, the feature reconstruction is performed through inter-modal transfer function learning to obtain the reconstruction features of each modality, specifically:

[0019] Combine the three modalities of EEG modality, eye movement modality and fNIRS modality in pairs, project the modalities into a common space, learn the transfer function in the space, obtain the projection matrix of α and β, and use the matrix as the transfer function of the corresponding modality combination;

[0020] The migration function is used to obtain the migration features of the i-th layer of each modality, and further obtain the output of each layer of the sparse autoencoder;

[0021] Fine-tune weights through back-propagation BP algorithm Until convergence. After emotional feature transfer learning, the reconstructed features of each modality can be obtained, and the i-th layer with better feature performance is selected as the emotional feature. At this time, the EEG emotional feature H is obtained. E , Eye movement emotion characteristics H Y and fNIRS emotional features F .

[0022] Furthermore, the modal reconstruction features, i.e., the eye movement signal reconstruction features, the EEG signal reconstruction features, and the fNIRS signal reconstruction features, are input into the residual multi-head self-attention module, and the three modalities of the eye movement modality, the EEG modality, and the fNIRS modality are combined in pairs. By introducing the multi-head mutual attention module of the conversation multi-head attention mechanism, cascade splicing is used for feature fusion, and then the emotion label is classified and output after global average pooling processing, specifically:

[0023] The residual multi-head self-attention module includes a residual module, a linear transformation module, a scaling dot product attention mechanism module and a splicing module. The sentiment features of any modality are linearly transformed to obtain the query vector matrix Q, the key value vector matrix K, and the output value vector matrix V. The matrix is ​​passed through the scaling dot product attention mechanism, and the output is finally spliced ​​to obtain the output of any modality.

[0024] After the multi-head self-attention module is added to the residual network, the output of each modality is combined with the shallow feature matrix of the modality, that is, the features of the corresponding modality, so that the EEG emotion features, eye movement emotion features and fNIRS emotion features are obtained as the output of the residual multi-head self-attention module;

[0025] The conversational multi-head mutual attention module includes a linear transformation module, a scaled dot product attention mechanism module processed by the conversation mechanism, and a splicing module. Unlike the self-attention module, it can process the signal features of the two modules at the same time.

[0026] The modalities are combined in pairs, and the residual multi-head self-attention outputs are linearly transformed to obtain the query vector matrix Q, key value vector matrix K, and output value vector matrix V of the two modalities. The scaled dot product attention mechanism of the conversation mechanism is introduced to splice the results of the two modalities to obtain the multi-head mutual attention output. Finally, the results of the pairwise combination splicing are spliced ​​again to obtain the multi-modal output;

[0027] Cascade splicing is used for feature fusion, and the multimodal fused emotional features are obtained through global average pooling. Finally, the emotion category is output through the classifier discriminator.

[0028] Furthermore, the KSLPCCA method is used to project the modes into a common space.

[0029] Furthermore, the data in the data set are emotionally labeled, specifically: the data set is screened for pleasantness, the data induced by negative emotion pictures are labeled as negative emotion, with a label value of 0, the data induced by neutral emotion pictures are labeled as neutral emotion, with a label value of 1, and the data induced by positive emotion pictures are labeled as positive emotion, with a label value of 2.

[0030] Furthermore, the EEG signal, eye movement signal and fNIRS signal also include a bandpass filtering process to reduce noise.

[0031] An emotion recognition system, used to implement the multimodal emotion recognition method, comprises:

[0032] Virtual reality scene construction module: With the help of the International Affective Picture System (IAPS), panoramic pictures of different emotions are collected, including the same number of negative emotion pictures, neutral emotion pictures and positive emotion pictures, to form a VR panoramic emotion picture library. And with the help of Unity, virtual reality is built as the scene in the induced state;

[0033] Signal acquisition module: including EEG acquisition equipment, eye movement acquisition equipment and fNIRS acquisition equipment, which are used to collect EEG signals, eye movement signals and fNIRS signals respectively;

[0034] Feature extraction module: Use sparse autoencoders to process EEG, eye movement, and fNIRS signals to obtain their features;

[0035] Feature reconstruction module: feature reconstruction is performed through inter-modal transfer function learning, and the input signal of each modality is used as input to obtain its corresponding reconstructed features;

[0036] Multimodal emotion recognition model: Process the features of different modalities, and combine the three modalities in pairs by adding the multi-head self-attention module of the residual network. By introducing the multi-head mutual attention module of the conversation multi-head attention mechanism, feature fusion is performed through cascade splicing, and the emotion labels are classified and output after global average pooling.

[0037] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the multimodal emotion recognition method is implemented when the processor executes the computer program.

[0038] A storage medium stores a computer program, which implements the multimodal emotion recognition method when executed by a processor.

[0039] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0040] (1) In the induction of virtual reality, the International Affective Picture System (IAPS) is used to collect the same number of different types of panoramic pictures of emotions to form a VR panoramic emotional picture library, and virtual reality is built as the scene in the induced state with the help of Unity, and the SAM table is used to score the pleasure and arousal, and physiological signals are collected to form a data set applied to this system, and the data set is screened for pleasure and set labels. In general, the triggering of emotional responses by common emotion inducers depends on perceptual stimuli, especially visual cues, sounds, and sometimes touch and smell, and visual stimulation has a stronger impact on emotions than hearing and touch. The present invention emphasizes the visual impact of virtual reality in a virtual reality system with the help of Unity, and can have stronger real-time and interactivity than traditional emotion inducers.

[0041] (2) The present invention proposes to use machine learning methods to extract features. Traditional manual feature extraction and expression rely more on human experience, and the process is cumbersome and time-consuming. Using machine learning methods to extract features can improve the ability of prediction, recognition and classification.

[0042] (3) The present invention proposes a feature reconstruction method based on transfer learning. Different individuals stimulated by the same emotional element may produce different emotions, and the physiological signals they produce may also be different. Therefore, emotion recognition is greatly affected by individual factors. The feature reconstruction method based on transfer learning proposed in the present invention can enable the emotion recognition system to have a stronger generalization ability, and therefore proposes an emotion recognition system based on transfer learning. The advantage is that when the physiological signal of a certain modality of an individual is weak or missing, the feature can be reconstructed by the method provided by the present invention to enhance the generalization ability of the system.

[0043] The present invention proposes a feature fusion algorithm based on the attention mechanism, specifically based on the residual multi-head self-attention mechanism and the multi-head mutual attention mechanism. The traditional feature fusion method often directly cascades all the features. This makes the features of each modality have the same weight in affecting the emotion recognition classification. In actual situations, the degree of influence of each modality in emotional expression should be different. The attention mechanism can give different weights to different modalities, which can improve the effect of emotion recognition. At the same time, adding a residual network can also solve problems such as gradient explosion and disappearance that may exist during the training process. In the multi-head mutual attention module, a conversational multi-head attention is proposed, a linear mapping of multi-head attention is introduced, and information exchange between multiple attention mechanisms is increased. This method can further improve the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a work flow chart of the present invention;

[0045] Figure 2It is a diagram of the feature reconstruction process based on transfer learning of the present invention;

[0046] Figure 3 It is a structural diagram of an emotion recognition model based on a residual multi-head self-attention module and a multi-head mutual attention module of the present invention;

[0047] Figure 4 It is a schematic diagram of the signal acquisition module of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below in conjunction with examples, but the embodiments of the present invention are not limited thereto.

[0049] Figure 1-Figure 4 As shown, a fNIRS emotion recognition system and method based on attention mechanism and transfer learning under virtual reality induced state is suitable for emotion recognition tasks of virtual reality equipment, EEG acquisition equipment, fNIRS acquisition equipment and eye movement acquisition equipment, and mainly includes a virtual reality scene building module, a signal acquisition module, a feature extraction module, a feature reconstruction module, and a multimodal emotion recognition model based on residual multi-head self-attention mechanism and multi-head mutual attention mechanism.

[0050] Virtual reality scene construction module: The virtual reality emotional picture library uses the International Emotional Picture System (IAPS) to collect panoramic pictures of different emotions, including the same number of negative emotional pictures, neutral emotional pictures and positive emotional pictures, and induces emotions in users through VR equipment, and uses the SAM table to score the pleasure and arousal. The score is considered qualified if it is 80% of the standard score of the selected picture, and physiological signals are collected to form a data set applied to this system. The data set is screened for pleasure, and the data induced by negative emotional pictures is labeled as negative emotions with a label value of 0, the data induced by neutral emotional pictures is labeled as neutral emotions with a label value of 1, and the data induced by positive emotional pictures is labeled as positive emotions with a label value of 2.

[0051] Signal acquisition modules, such as Figure 4 As shown, when the user is induced by the virtual reality emotion gallery, the signal acquisition module transmits a corresponding signal to the signal acquisition device.

[0052] First, the user wears VR, EEG acquisition equipment, eye movement acquisition equipment and fNIRS acquisition equipment to collect EEG signals, eye movement signals and fNIRS signals in real time.

[0053] The EEG signals are continuously collected by the EEG acquisition device, and at the same time, the EEG signals are band-pass filtered to remove the ECG signals and EMG signals.

[0054] The eye movement signal is continuously collected by the eye movement acquisition device, and the eye movement signal is band-pass filtered to reduce noise.

[0055] The fNIRS signal was continuously collected by the fNIRS acquisition device, and the fNIRS signal was band-pass filtered to smooth the signal and reduce noise.

[0056] Feature extraction module: The input signals of the three modalities are input into the feature extraction module containing the sparse autoencoder. After encoding and decoding, the features of the EEG, eye movement, and fNIRS signals of each layer of the sparse autoencoder can be obtained, which are recorded as:

[0057] Specifically:

[0058] The encoding process, that is, the process of learning the function h, is as follows:

[0059] h(x)=f(Wx+b) (1)

[0060] f(z)=1 / (1+exp(-z)) (2)

[0061] Where f() is a nonlinear activation function, W is a weight matrix, and b is an offset vector

[0062] The decoding process, that is, reconstructing the input signal by minimizing the reconstruction error, is as follows:

[0063]

[0064]

[0065]

[0066] Where p is the size of the sparsity level and β is the weight of the sparsity penalty term.

[0067] Get the features of EEG, eye movement, and fNIRS signals of each layer of the sparse autoencoder:

[0068]

[0069] in They are the inputs of EEG, eye movement and fNIRS respectively.

[0070] The above obtained Considered as the input of the next layer, the features of each layer are obtained in turn:

[0071]

[0072] Where W i→i+1 is the weight between the i-th layer and the i+1-th layer, and g() is the activation function

[0073] Feature reconstruction module: feature reconstruction is performed through inter-modal transfer function learning. The signals of each modality are input, and the reconstructed features of each modality can be obtained through the learning of the transfer function. The EEG features are reconstructed through eye movement features and fNIRS features. At this time, the EEG reconstruction features are The fNIRS features are reconstructed through eye movement features and EEG features. At this time, the fNIRS reconstructed features are: The eye movement features are reconstructed through EEG features and fNIRS features. At this time, the reconstructed features of brain eye movement are: Select the i-th layer with better feature performance. The value of i can be selected through the performance of multiple experiments. Therefore, the EEG emotion feature H is now obtained. E , Eye movement emotion characteristics H Y , fNIRS emotional features F .

[0074] The feature reconstruction is performed through inter-modal transfer function learning, and the reconstruction features of each modality can be obtained, which are specifically:

[0075] The EEG modality, eye movement modality, and fNIRS modality are combined in pairs, and the kernel supervised local canonical correlation analysis (KSLPCCA) method is used to project the modalities into a common space, and their transfer functions are learned in this space. Taking the EEG emotion features and blood oxygen emotion features as examples:

[0076] The objective function of KSLPCCA is

[0077]

[0078] Can be transformed into a generalized characteristic equation for solution

[0079]

[0080] Among them, F E is the feature set H E The k-nearest neighbor graph within the class, F F is the feature set H F The intra-class k-nearest neighbor graph, Yes H E The diagonal matrix of columns and elements, Yes H F The diagonal matrix of columns and elements and the projection vector are denoted as: W = H E α、U=H F β, K E =(H E ) T H E Representative feature set H E The kernel matrix, KF =(H F ) T H F Representative feature set H F The kernel matrix, ξ E , F represents the regularization factor.

[0081] And satisfy the following relationship

[0082]

[0083] KSLPCCA can effectively extract feature information, and project the EEG and fNIRS feature information into the shared space through W and U to obtain the projection matrix of α and β. The matrix is ​​used as the coefficient to obtain the transfer function between the EEG and fNIRS modalities.

[0084] After obtaining the projection space α and β, use them as coefficients through H E Obtain the transfer function, and then use the transfer function to obtain the adjusted fNIRS feature H' F , the formula is as follows:

[0085] H' F =T E→F (H E )=H E αβ T (11)

[0086] Similarly, there are:

[0087]

[0088]

[0089] H” F =T γ→F (H Y )=H Y γβ T (14)

[0090] According to the above formulas #(11)-#(14), the migration function can be used to obtain the migration characteristics of the i-th layer of each mode:

[0091]

[0092] Get the output of each layer of the sparse autoencoder:

[0093]

[0094] Fine-tune weights through back-propagation BP algorithm Until convergence. After emotional feature transfer learning, the reconstruction features of each modality can be obtained. The i-th layer with better feature performance is selected as the emotional feature. At this time, the EEG emotional feature H is obtained. E , Eye movement emotion characteristics H Y , fNIRS emotional features F .

[0095] Multimodal emotion recognition model: Process the features of different modalities, and combine the three modalities in pairs by adding the multi-head self-attention module of the residual network. By introducing the multi-head mutual attention module of the conversation multi-head attention mechanism, feature fusion is performed through cascade splicing, and the emotion labels are classified and output after global average pooling.

[0096] Specifically:

[0097] Taking EEG modality as an example, the query vector matrix Q, key value vector matrix K, and output value vector matrix V are obtained through linear transformation;

[0098]

[0099]

[0100]

[0101] in are respectively trainable linear transformation parameter matrices.

[0102] Then, by scaling the dot product attention mechanism, we get the output A of the multi-head self-attention module. E :

[0103]

[0104] A E =Concat(H1,...,H n ) (twenty one)

[0105] in, is the output of the ith self-attention module of the EEG multi-head self-attention module, A E is the complete output of the EEG emotion feature after the multi-head self-attention module. n is the number of sub-attention modules in the residual multi-head self-attention module, and d is the output dimension of the trainable linear transformation parameter matrix.

[0106] Similarly, we get the complete output of eye movement features and fNIRS emotion features after the multi-head self-attention module A Y , A F ;

[0107] After adding the residual network, the complete output of EEG, eye movement, and fNIRS modalities are combined with the shallow feature matrix E to obtain the output R of the residual multi-head self-attention module for EEG emotion features, eye movement emotion features, and fNIRS emotion features. E , R Y , R F ;

[0108] By using a parameter matrix to re-integrate the multi-head attention into multiple mixed attentions, we get the conversational multi-head attention.

[0109] The modalities are combined in pairs, and the residual multi-head self-attention output R is obtained E , R Y , R F The multi-head mutual attention module that introduces the multi-head attention mechanism of conversation obtains the multi-head mutual attention output. Taking EEG and fNIRS modalities as an example, EEG and fNIRS obtain the output A of the multi-head mutual attention module through the multi-head mutual attention mechanism. RE_F :

[0110]

[0111]

[0112]

[0113]

[0114]

[0115]

[0116]

[0117]

[0118] Similarly, the EEG and eye movement emotion features can be obtained through the output of the multi-head mutual attention module A RE_Y , eye movement and fNIRS emotion features are output through the multi-head mutual attention module A RU_F .

[0119] Finally, cascade splicing is used for feature fusion, and the multimodal fused emotional features are obtained through global average pooling. Finally, the emotion recognition probability is output through the classifier.

[0120] At present, most of the emotion recognition methods based on physiological signals are based on single modality such as EEG signals, and there are few multimodal emotion recognition studies based on functional near-infrared spectroscopy technology, EEG signals, and eye movement signals. The present invention proposes a multimodal emotion recognition system based on attention mechanism and transfer learning in the induced state of virtual reality, which takes advantage of different physiological signals, explores and mines their role in emotion recognition, and uses transfer learning to take into account the differences between individuals and the possibility that the signals of the modality may be poor or missing, so as to enhance the generalization ability of the system, and further enhance the generalization ability of the system through the attention mechanism, which is conducive to the universality and applicability of the model. This not only has a good development prospect in practical applications, but also creates a new model of emotion recognition system.

[0121] The present invention proposes a feature extraction method of a sparse autoencoder based on machine learning, which can realize feature extraction of EEG signals, eye movement signals, and functional near-infrared spectroscopy signals, and can avoid the problem that manual feature extraction is overly dependent on human experience. This method can improve the ability of prediction, recognition, and classification, and reduce the tediousness of feature extraction.

[0122] The present invention proposes a feature reconstruction method based on transfer learning, which can obtain the reconstructed features of each modality through the interaction of three modalities and with the help of transfer functions, so as to solve the problem of different physiological signals generated by different individuals as much as possible, enhance the generalization ability of the emotion recognition system, and improve the emotion recognition system's ability to handle individual differences.

[0123] The present invention proposes a multimodal feature fusion method based on residual multi-head self-attention and multi-head mutual attention. The attention mechanism can effectively combine the feature information between different modalities, can better combine EEG features, eye movement features and fNIRS features, and adds a residual network, which can reduce the problems such as gradient explosion that may exist during training. In the multi-head mutual attention module, a conversational multi-head attention is proposed, a linear mapping of multi-head attention is introduced, and information exchange between multiple attention mechanisms is increased. This method can further improve the accuracy of emotion recognition.

[0124] Another embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the multimodal emotion recognition method when executing the computer program.

[0125] Another embodiment of the present invention provides a storage medium having a computer program stored thereon, wherein the computer program implements the multimodal emotion recognition method when executed by a processor.

[0126] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A multimodal emotion recognition method, characterized in that: include: Build a VR panoramic emotional picture library and data set, and label the data in the data set with emotions; The user wears a VR device and uses pictures from the VR panoramic emotional picture library for stimulation to collect EEG signals, eye movement signals, and fNIRS signals; Use sparse autoencoders to process EEG signals, eye movement signals, and fNIRS signals respectively, extract the features of the output layer, and use the output layer features as the input layer for further processing to obtain the EEG signals, eye movement signals, and fNIRS signal features of each layer; Through inter-modal transfer function learning, feature reconstruction is performed to obtain the reconstruction features of each modality; The modal reconstruction features, i.e., the eye movement signal reconstruction features, the EEG signal reconstruction features, and the fNIRS signal reconstruction features, are input into the residual multi-head self-attention module, and the three modalities, namely, the eye movement modality, the EEG modality, and the fNIRS modality, are combined in pairs. By introducing the multi-head mutual attention module of the conversation multi-head attention mechanism, cascade splicing is used for feature fusion, and then the emotion label is classified and output after global average pooling processing; The feature reconstruction is performed through inter-modal transfer function learning to obtain the reconstruction features of each modality, specifically: Combine the three modalities of EEG modality, eye movement modality and fNIRS modality in pairs, project the modalities into a common space, learn the transfer function in the space, obtain the projection matrix of α and β, and use the matrix as the transfer function of the corresponding modality combination; The migration function is used to obtain the migration features of the i-th layer of each modality, and further obtain the output of each layer of the sparse autoencoder; Fine-tune weights through back-propagation BP algorithm Until convergence, after emotional feature transfer learning, the reconstructed features of each modality can be obtained, and the i-th layer with better feature performance is selected as the emotional feature. At this time, the EEG emotional feature H is obtained E , Eye movement emotion characteristics H Y and fNIRS emotional features F .

2. The multimodal emotion recognition method according to claim 1, characterized in that: The features of each layer are obtained, specifically including: The encoding process, specifically the process of learning the function h; The decoding process, specifically reconstructing the input signal by minimizing the reconstruction error; The features of EEG signals, eye movement signals and fNIRS signals of each layer of the sparse autoencoder are obtained.

3. A multimodal emotion recognition method according to claim 1, characterized in that: The modality reconstruction features, i.e., the eye movement signal reconstruction features, the EEG signal reconstruction features, and the fNIRS signal reconstruction features, are input into the residual multi-head self-attention module, and the three modalities of the eye movement modality, the EEG modality, and the fNIRS modality are combined in pairs, and the multi-head mutual attention module of the conversation multi-head attention mechanism is introduced, and the cascade splicing is used for feature fusion, and then the emotion label is classified and output after global average pooling processing, specifically: The residual multi-head self-attention module includes a residual module, a linear transformation module, a scaling dot product attention mechanism module and a splicing module. The sentiment features of any modality are linearly transformed to obtain the query vector matrix Q, the key value vector matrix K, and the output value vector matrix V. The matrix is ​​passed through the scaling dot product attention mechanism, and the output is finally spliced ​​to obtain the output of any modality. After the multi-head self-attention module is added to the residual network, the output of each modality is combined with the shallow feature matrix of the modality, that is, the features of the corresponding modality, so that the EEG emotion features, eye movement emotion features and fNIRS emotion features are obtained as the output of the residual multi-head self-attention module; The conversational multi-head mutual attention module includes a linear transformation module, a scaled dot product attention mechanism module processed by the conversation mechanism, and a splicing module. Unlike the self-attention module, it can process the signal features of the two modules at the same time. The modalities are combined in pairs, and the residual multi-head self-attention outputs are linearly transformed to obtain the query vector matrix Q, key value vector matrix K, and output value vector matrix V of the two modalities. The scaled dot product attention mechanism of the conversation mechanism is introduced to splice the results of the two modalities to obtain the multi-head mutual attention output. Finally, the results of the pairwise combination splicing are spliced ​​again to obtain the multi-modal output; Cascade splicing is used for feature fusion, and the multimodal fused emotional features are obtained through global average pooling processing, and finally the emotion category is output by the classifier discriminator.

4. A multimodal emotion recognition method according to claim 1, characterized in that: The KSLPCCA method is used to project the modes into a common space.

5. A multimodal emotion recognition method according to claim 1, characterized in that: The emotional labeling of the data in the data set is specifically as follows: the data set is screened for pleasantness, the data induced by negative emotional pictures are labeled as negative emotions, with a label value of 0, the data induced by neutral emotional pictures are labeled as neutral emotions, with a label value of 1, and the data induced by positive emotional pictures are labeled as positive emotions, with a label value of 2.

6. The multimodal emotion recognition method according to claim 1, characterized in that: The EEG signal, eye movement signal and fNIRS signal also include a bandpass filtering process to reduce noise.

7. An emotion recognition system, characterized in that: The multimodal emotion recognition method for implementing any one of claims 1 to 6 comprises: Virtual reality scene construction module: With the help of the International Affective Picture System (IAPS), panoramic pictures of different emotions are collected, including the same number of negative emotion pictures, neutral emotion pictures and positive emotion pictures, to form a VR panoramic emotion picture library, and virtual reality is built as the scene in the induced state with the help of Unity; Signal acquisition module: including EEG acquisition equipment, eye movement acquisition equipment and fNIRS acquisition equipment, which are used to collect EEG signals, eye movement signals and fNIRS signals respectively; Feature extraction module: Use sparse autoencoders to process EEG, eye movement, and fNIRS signals to obtain their features; Feature reconstruction module: feature reconstruction is performed through inter-modal transfer function learning, and the input signal of each modality is used as input to obtain its corresponding reconstructed features; Multimodal emotion recognition model: Process the features of different modalities, and combine the three modalities in pairs by adding the multi-head self-attention module of the residual network. By introducing the multi-head mutual attention module of the conversation multi-head attention mechanism, feature fusion is performed through cascade splicing, and the emotion labels are classified and output after global average pooling.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multimodal emotion recognition method according to any one of claims 1 to 6 is implemented.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal emotion recognition method according to any one of claims 1 to 6 is implemented.

Images