Multi-modal emotion recognition method and device based on hierarchical interactive alignment network

Abstract

The invention discloses a multi-modal emotion recognition method and device based on a hierarchical interaction alignment network, and relates to the field of data processing, and the method comprises the steps: constructing a multi-modal emotion recognition model, and carrying out the training through the alignment of a hierarchical representation distribution alignment layer, and obtaining a trained multi-modal emotion recognition model; electroencephalogram data and eye movement data pairs of one to-be-recognized person in the target domain crowd are obtained and input into a trained multi-modal emotion recognition model, and final cross-modal features corresponding to the to-be-recognized person are obtained through a feature extraction module and a layered self-adaptive interactive attention module in sequence; and inputting the final cross-modal feature corresponding to the to-be-identified person and the final cross-modal feature corresponding to the target domain data into a few-sample learning module to obtain a probability value of the to-be-identified person belonging to each emotion category, and selecting the emotion category corresponding to the maximum probability value as a predicted emotion category of the to-be-identified person. According to the method, the problems of large individual difference and limited sample quantity in multi-modal emotion recognition are solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a multimodal emotion recognition method and device based on a hierarchical interactive alignment network. Background Art

[0002] With the deepening of neuroscience research and the increasing demand for understanding human emotions, multimodal emotion recognition (MER) has gradually become a key research direction to promote the development of human-computer interaction and affective computing. Emotion plays a vital role in human cognitive processes and behavioral regulation, involving individual subjective experience and complex regulation of the central and peripheral nervous systems. Therefore, assessing emotional states by capturing and analyzing physiological signals can provide a more objective basis. However, most current emotion recognition methods rely heavily on single-modal signal sources. This limitation makes comprehensive analysis and accurate recognition of emotions challenging. Therefore, multimodal emotion recognition (MER) methods that integrate multiple physiological data have received increasing attention.

[0003] EEG signals are widely used in detecting human emotions due to their non-invasiveness, economy and high sampling rate. They can directly reflect the neural activity during the emotion induction process. To this end, many researchers have developed a series of emotion recognition models based on EEG data, such as multi-channel convolutional neural networks, bidirectional long short-term memory networks (BiLSTM) and graph-based networks. In addition, eye movement (EM) signals, as one of the physiological signals of the peripheral nervous system, can characterize the gaze behavior and pupil changes of individuals in different emotional states, thereby providing supplementary information for the MER task.

[0004] Although MER has made significant progress, it still faces many challenges, mainly in the following aspects: First, there is variability in the distribution of individual samples. Multimodal physiological signals are non-stationary and show significant differences between different individuals. Second, labeled samples are scarce. It is costly to obtain data with emotion labels, and existing MER models have difficulties in extracting emotion information using limited labeled samples. Summary of the invention

[0005] The purpose of this application is to propose a multimodal emotion recognition method and device based on a hierarchical interactive alignment network to address the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a multimodal emotion recognition method based on a hierarchical interaction alignment network, comprising the following steps:

[0007] Construct a multimodal emotion recognition model and a hierarchical representation distribution alignment layer. The multimodal emotion recognition model includes a feature extraction module, a hierarchical adaptive interactive attention module, and a few-shot learning module. The feature extraction module includes a brain electronic network and an eye movement sub-network. The hierarchical adaptive interactive attention module includes several layers of dynamic interactive attention layers connected in sequence.

[0008] Source domain data and target domain data are obtained, the source domain data includes EEG data and eye movement data pairs of the source domain population and their corresponding real emotion category labels, and the target domain data includes EEG data and eye movement data pairs of the target domain population; the EEG data and eye movement data pairs in the source domain data and the EEG data and eye movement data pairs in the target domain data are respectively input into the multimodal emotion recognition model, and the EEG depth features and eye movement depth features corresponding to the source domain data and the EEG depth features and eye movement depth features corresponding to the target domain data are first extracted through the brain electronic network and the eye movement sub-network in the feature extraction module; the EEG depth features and eye movement depth features corresponding to the source domain data and the EEG depth features and eye movement depth features corresponding to the target domain data are respectively input into the hierarchical adaptive interactive attention module, and after each layer of dynamic interactive attention layer, the cross-modal features of each layer corresponding to the source domain data and the cross-modal features of each layer corresponding to the target domain data are output. The final cross-modal features corresponding to the source domain data and the final cross-modal features corresponding to the target domain data; the cross-modal features of each layer corresponding to the source domain data and the cross-modal features of each layer corresponding to the target domain data are input into the hierarchical representation distribution alignment layer, and the multi-layer maximum average difference loss function is calculated; the support set and the query set are extracted from the source domain data, and the final cross-modal features corresponding to the support set and the final cross-modal features corresponding to the query set are determined according to the final cross-modal features corresponding to the source domain data and input into the few-shot learning module to obtain the probability value of each query sample in the query set belonging to each emotion category, and the cross-entropy loss function is calculated according to the probability value of each query sample in the query set belonging to each emotion category and its corresponding true emotion category label; the total loss function is constructed based on the multi-layer maximum average difference loss function and the cross-entropy loss function, and the multi-modal emotion recognition model is trained using the total loss function to obtain a trained multi-modal emotion recognition model;

[0009] Obtain the EEG data and eye movement data pair of one of the persons to be identified in the target domain population and input them into the trained multimodal emotion recognition model. The final cross-modal features corresponding to the person to be identified are obtained through the feature extraction module and the hierarchical adaptive interactive attention module in sequence. The final cross-modal features corresponding to the person to be identified and the final cross-modal features corresponding to the target domain data are input into the few-sample learning module to obtain the probability value that the person to be identified belongs to each emotion category, and select the emotion category corresponding to the maximum probability value as the predicted emotion category of the person to be identified.

[0010] Preferably, the brain electronic network includes a residual block and a hybrid attention module connected in sequence, and the eye movement sub-network adopts a DenseNet structure;

[0011] The EEG data in the EEG data and eye movement data pair is input into the brain electronic network to obtain the corresponding EEG depth features, as shown in the following formula:

[0012] ;

[0013] in, represents EEG data, represents the EEG depth feature, represents the brain electronic network;

[0014] The eye movement data in the EEG data and eye movement data pair is input into the eye movement sub-network to obtain the corresponding eye movement depth features, as shown in the following formula:

[0015] ;

[0016] in, represents eye movement data, represents the eye movement depth feature, Represents the eye movement subnetwork.

[0017] Preferably, each dynamic interactive attention layer in the hierarchical adaptive interactive attention module includes an attention module, a gating module, a splicing layer and a multi-layer perceptron;

[0018] The cross-modal features of the i-1th layer output by the dynamic interaction attention layer of the i-1th layer are input into the dynamic interaction attention layer of the i-th layer, and are linearly transformed in the attention module to obtain the query matrix, as shown in the following formula:

[0019] ;

[0020] in, , represents the cross-modal features of the i-1th layer, represents the query matrix of the i-th layer, represents the query weight matrix of the i-th layer. When i=1, the cross-modal features output by the dynamic interactive attention layer of the 0th layer EEG depth feature ;

[0021] The eye movement depth feature is input into the dynamic interaction attention layer of the i-th layer. After linear transformation in the attention module, the key matrix and value matrix are obtained as shown in the following formula:

[0022] ;

[0023] ;

[0024] in, and Represent the key weight matrix of the i-th layer and the value weight matrix of the i-th layer respectively; and denote the key matrix and value matrix of the i-th layer respectively, Indicates the eye movement depth feature;

[0025] The following formula is used to calculate the attention feature of the i-th layer:

[0026] ;

[0027] in, (⋅) means function, d represents the feature dimension, T represents the transposed matrix, Represents the attention feature of the i-th layer;

[0028] The attention features of the i-th layer are input into the gating module, and the channel weight of the i-th layer is calculated through the gated attention mechanism, as shown in the following formula:

[0029] ;

[0030] Among them, σ (⋅) represents the sigmoid activation function, represents the gating weight, represents the channel weight of the i-th layer;

[0031] The attention features of the i-th layer are gated and selected through the channel weights of the i-th layer to obtain the selected features of the i-th layer, as shown in the following formula:

[0032] ;

[0033] in, represents the selected features of the i-th layer, ⊙ represents channel-by-channel multiplication;

[0034] The selected features of the i-th layer are concatenated with the cross-modal features of the i-1-th layer and input into the multi-layer perceptron to obtain the fused features of the i-th layer, as shown in the following formula:

[0035] ;

[0036] Among them, [⋅;⋅] represents the feature concatenation operation, Represents the function corresponding to the multilayer perceptron, represents the fusion features of the i-th layer;

[0037] The fusion features of the i-th layer are connected with the cross-modal features of the i-1-th layer through residual connection to obtain the cross-modal features of the i-th layer, as shown in the following formula:

[0038] ;

[0039] in, Represents the cross-modal features of the i-th layer.

[0040] Preferably, in the hierarchical representation distribution alignment layer, the maximum average difference loss function is used to measure the inter-domain differences of the cross-modal features of each layer, as shown in the following formula:

[0041] ;

[0042] in, is the cross-modal feature of the i-th layer corresponding to one of the samples in the source domain data, for A collection of represents the cross-modal feature of the i-th layer corresponding to one of the samples in the target domain data, for The set of cross-modal features of each layer corresponding to the source domain data and the cross-modal features of each layer corresponding to the target domain data, represents the MMD loss of the i-th layer, Indicates that the feature points are transformed into a reproducing kernel Hilbert space The mapping function in represents the reproducing kernel Hilbert space The square of the norm of ;

[0043] Introduce a hierarchical weighting mechanism based on logarithmic decay:

[0044] ;

[0045] Secondly, represents the alignment weight of the i-th layer, and Represent the weight base parameter and logarithmic growth parameter respectively;

[0046] The multi-layer maximum average difference loss function is calculated using the following formula:

[0047] ;

[0048] Where I represents the total number of dynamic interactive attention layers, Represents the multi-layer maximum average discrepancy loss function.

[0049] Preferably, the cross-modal features of the first layer output by the dynamic interactive attention layer of the last layer are used as the final cross-modal features; in the reasoning process of the trained multimodal emotion recognition model, the person to be identified is used as a query sample in the query set, and the support set is extracted from the target domain data;

[0050] In the few-shot learning module, the prototype vector of each emotion category is calculated based on the final cross-modal features corresponding to the support set, as shown in the following formula:

[0051] ;

[0052] in, represents one of the support samples belonging to the cth sentiment category in the support set, Indicates one of the support samples belonging to the cth sentiment category in the support set The corresponding final cross-modal features, represents the set of support samples belonging to the cth sentiment category in all support sets, Represents the prototype vector of the cth emotion category, c=1,2,…,N;

[0053] The probability value of each query sample in the query set belonging to each emotion category is calculated based on the final cross-modal features corresponding to the query set and the prototype vector of each emotion category, as shown in the following formula:

[0054] ;

[0055] in, represents one of the query samples in the query set, Represents one of the query samples in the query set The corresponding final cross-modal features, represents the prototype vector of the u-th emotion category, Represents the prototype vector of the vth emotion category, u=1,2,…,N, v=1,2,…,N, Represents one of the query samples in the query set The probability value of belonging to the u-th emotion category, represents the predicted sentiment category, represents the calculation function of Euclidean distance, Represents an exponential function with base e.

[0056] Preferably, the cross entropy loss function is calculated using the following formula:

[0057] ;

[0058] in, represents the cross entropy loss function, Represents one of the query samples in the query set The one-hot encoding of the true sentiment category label. If one of the query samples in the query set The true emotion category label is u, then ,otherwise ;

[0059] The total loss function is calculated using the following formula:

[0060] ;

[0061] in, represents the multi-layer maximum average difference loss function, represents the total loss function, Represents the balance weight.

[0062] In a second aspect, the present invention provides a multimodal emotion recognition device based on a hierarchical interactive alignment network, comprising:

[0063] A model building module is configured to build a multimodal emotion recognition model and a hierarchical representation distribution alignment layer, the multimodal emotion recognition model includes a feature extraction module, a hierarchical adaptive interactive attention module and a few-shot learning module, the feature extraction module includes a brain electronic network and an eye movement sub-network, and the hierarchical adaptive interactive attention module includes a plurality of layers of dynamic interactive attention layers connected in sequence;

[0064] The training module is configured to obtain source domain data and target domain data, wherein the source domain data includes EEG data and eye movement data pairs of the source domain population and their corresponding real emotion category labels, and the target domain data includes EEG data and eye movement data pairs of the target domain population; the EEG data and eye movement data pairs in the source domain data and the EEG data and eye movement data pairs in the target domain data are respectively input into the multimodal emotion recognition model, and the EEG depth features and eye movement depth features corresponding to the source domain data and the EEG depth features and eye movement depth features corresponding to the target domain data are first extracted through the brain electronic network and the eye movement sub-network in the feature extraction module; the EEG depth features and eye movement depth features corresponding to the source domain data and the EEG depth features and eye movement depth features corresponding to the target domain data are respectively input into the hierarchical adaptive interactive attention module, and after each layer of dynamic interactive attention layer, the cross-modal features of each layer corresponding to the source domain data and the cross-modal features of each layer corresponding to the target domain data are output. Features and the final cross-modal features corresponding to the source domain data and the final cross-modal features corresponding to the target domain data; the cross-modal features of each layer corresponding to the source domain data and the cross-modal features of each layer corresponding to the target domain data are input into the hierarchical representation distribution alignment layer, and the multi-layer maximum average difference loss function is calculated; the support set and the query set are extracted from the source domain data, and the final cross-modal features corresponding to the support set and the final cross-modal features corresponding to the query set are determined according to the final cross-modal features corresponding to the source domain data and input into the few-shot learning module to obtain the probability value of each query sample in the query set belonging to each emotion category, and the cross-entropy loss function is calculated according to the probability value of each query sample in the query set belonging to each emotion category and its corresponding true emotion category label; the total loss function is constructed based on the multi-layer maximum average difference loss function and the cross-entropy loss function, and the multimodal emotion recognition model is trained using the total loss function to obtain a trained multimodal emotion recognition model;

[0065] The prediction module is configured to obtain the EEG data and eye movement data pair of one of the persons to be identified in the target domain population and input them into the trained multimodal emotion recognition model, and then pass through the feature extraction module and the hierarchical adaptive interactive attention module in sequence to obtain the final cross-modal features corresponding to the person to be identified. The final cross-modal features corresponding to the person to be identified and the final cross-modal features corresponding to the target domain data are input into the few-sample learning module to obtain the probability value that the person to be identified belongs to each emotion category, and the emotion category corresponding to the maximum probability value is selected as the predicted emotion category of the person to be identified.

[0066] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0067] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0068] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] (1) The multimodal emotion recognition method based on hierarchical interactive alignment network proposed in this paper makes full use of the complementary features between electroencephalogram (EEG) data and eye movement (EM) data. Through the hierarchical adaptive interactive attention module and cross-domain optimization strategy, the features of the two modal data are fused and mapped into a common feature space, thereby achieving effective recognition of complex emotional states and significantly improving the performance of multimodal emotion recognition under few-sample conditions.

[0071] (2) The multimodal emotion recognition method based on hierarchical interactive alignment network proposed in the present invention effectively captures multi-dimensional emotion features through the dynamic interactive attention layer in the hierarchical adaptive interactive attention module, integrates the cross-modal information of EEG data and eye movement data, and uses the hierarchical representation distribution alignment layer to eliminate inter-domain differences and transform complementary features into a common feature space. It significantly improves the accuracy of multimodal emotion recognition under the condition of few samples and can effectively deal with the problems of large individual differences and limited sample number in multimodal emotion recognition.

[0072] (3) The multimodal emotion recognition method based on hierarchical interaction alignment network proposed in this paper can dynamically capture the optimal interaction representation between modalities, adaptively reduce the difference in modal features, and improve the adaptability of the model under different data distributions, providing an efficient and robust solution for multimodal few-sample emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0074] Figure 1 A schematic diagram of a process of a multimodal emotion recognition method based on a hierarchical interactive alignment network according to an embodiment of the present application;

[0075] Figure 2A schematic diagram of a multimodal emotion recognition model and a hierarchical representation distribution alignment layer of a multimodal emotion recognition method based on a hierarchical interactive alignment network according to an embodiment of the present application;

[0076] Figure 3 A schematic diagram of a brain electronic network of a multimodal emotion recognition method based on a hierarchical interactive alignment network according to an embodiment of the present application;

[0077] Figure 4 A schematic diagram of an eye movement sub-network of a multimodal emotion recognition method based on a hierarchical interaction alignment network according to an embodiment of the present application;

[0078] Figure 5 A schematic diagram of a hierarchical adaptive interactive attention module of a multimodal emotion recognition method based on a hierarchical interactive alignment network according to an embodiment of the present application;

[0079] Figure 6 A schematic diagram of a multimodal emotion recognition device based on a hierarchical interactive alignment network according to an embodiment of the present application;

[0080] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] Figure 1 A multimodal emotion recognition method based on a hierarchical interactive alignment network provided by an embodiment of the present application is shown, comprising the following steps:

[0083] S1, construct a multimodal emotion recognition model and a hierarchical representation distribution alignment layer. The multimodal emotion recognition model includes a feature extraction module, a hierarchical adaptive interactive attention module and a few-shot learning module. The feature extraction module includes a brain electronic network and an eye movement sub-network. The hierarchical adaptive interactive attention module includes several layers of dynamic interactive attention layers connected in sequence.

[0084] Specifically, refer to Figure 2, the embodiment of the present application constructs a multimodal emotion recognition model and a hierarchical representation distribution alignment (HRDA) layer, wherein the hierarchical representation distribution alignment layer is used in the training process of the multimodal emotion recognition model, and the two can constitute a hierarchical interactive alignment network, and the hierarchical representation distribution alignment layer is not used in the reasoning process. The multimodal emotion recognition model includes a feature extraction module, a hierarchical adaptive interactive attention (HAIA) module, and a few-shot learning (FSL) module, wherein the feature extraction module includes a brain electronic network for extracting EEG deep features and an eye movement sub-network for extracting eye movement deep features. Specifically, refer to Figure 3 and Figure 4 , the EEG data is extracted through the brain electronic network that combines the residual block and the hybrid attention (CBAM) module, and the eye movement data is extracted using the eye movement sub-network of the DenseNet network architecture. Among them, the residual block, the hybrid attention module and the DenseNet network architecture are all existing modules or networks, and their specific structures and calculation processes are not repeated here. The hierarchical adaptive interactive attention module includes several layers of dynamic interactive attention (DIA) layers connected in sequence. The dynamic interactive attention layer fuses the EEG deep features and the eye movement deep features to extract cross-modal features. In the training process of the multimodal emotion recognition model, the hierarchical representation distribution alignment layer is used to align the hierarchical cross-modal feature distribution of the source domain data and the target domain data to narrow the difference between domains. Finally, the prototype vector metric learning based on each emotion category is performed through the few-shot learning module to achieve emotion classification.

[0085] S2, obtain source domain data and target domain data, the source domain data includes the EEG data and eye movement data pairs of the source domain population and their corresponding real emotion category labels, and the target domain data includes the EEG data and eye movement data pairs of the target domain population; input the EEG data and eye movement data pairs in the source domain data and the EEG data and eye movement data pairs in the target domain data into the multimodal emotion recognition model respectively, first extract the EEG depth features and eye movement depth features corresponding to the source domain data and the EEG depth features and eye movement depth features corresponding to the target domain data through the brain electronic network and eye movement sub-network in the feature extraction module respectively; the EEG depth features and eye movement depth features corresponding to the source domain data and the EEG depth features and eye movement depth features corresponding to the target domain data are input into the hierarchical adaptive interactive attention module respectively, and after each layer of dynamic interactive attention layer, the cross-modal features of each layer corresponding to the source domain data and the cross-modal features of each layer corresponding to the target domain data are output. and the final cross-modal features corresponding to the source domain data and the final cross-modal features corresponding to the target domain data; the cross-modal features of each layer corresponding to the source domain data and the cross-modal features of each layer corresponding to the target domain data are input into the hierarchical representation distribution alignment layer, and the multi-layer maximum average difference loss function is calculated; the support set and the query set are extracted from the source domain data, and the final cross-modal features corresponding to the support set and the final cross-modal features corresponding to the query set are determined according to the final cross-modal features corresponding to the source domain data and input into the few-shot learning module to obtain the probability value of each query sample in the query set belonging to each emotion category, and the cross-entropy loss function is calculated according to the probability value of each query sample in the query set belonging to each emotion category and its corresponding true emotion category label; the total loss function is constructed based on the multi-layer maximum average difference loss function and the cross-entropy loss function, and the multimodal emotion recognition model is trained using the total loss function to obtain a trained multimodal emotion recognition model.

[0086] In a specific embodiment, in the hierarchical representation distribution alignment layer, the maximum average difference loss function is used to measure the inter-domain differences of the cross-modal features of each layer, as shown in the following formula:

[0087] ;

[0088] in, is the cross-modal feature of the i-th layer corresponding to one of the samples in the source domain data, for A collection of represents the cross-modal feature of the i-th layer corresponding to one of the samples in the target domain data, for The set of cross-modal features of each layer corresponding to the source domain data and the cross-modal features of each layer corresponding to the target domain data, represents the MMD loss of the i-th layer, Indicates that the feature points are transformed into a reproducing kernel Hilbert space The mapping function in represents the reproducing kernel Hilbert space The square of the norm of ;

[0089] Introduce a hierarchical weighting mechanism based on logarithmic decay:

[0090] ;

[0091] Secondly, represents the alignment weight of the i-th layer, and Represent the weight base parameter and logarithmic growth parameter respectively;

[0092] The multi-layer maximum average difference loss function is calculated using the following formula:

[0093] ;

[0094] Where I represents the total number of dynamic interactive attention layers, Represents the multi-layer maximum average discrepancy loss function.

[0095] In a specific embodiment, the cross entropy loss function is calculated using the following formula:

[0096] ;

[0097] in, represents the cross entropy loss function, Represents one of the query samples in the query set The one-hot encoding of the true sentiment category label. If one of the query samples in the query set The true emotion category label is u, then ,otherwise ;

[0098] The total loss function is calculated using the following formula:

[0099] ;

[0100] in, represents the multi-layer maximum average difference loss function, represents the total loss function, Represents the balance weight.

[0101] Specifically, the source domain data containing pairs of EEG data and eye movement data will contain pairs of EEG data and eye movement data. and target domain data ,in and Respectively represent the first The EEG data and eye movement data of samples are collected and form a set of EEG data and eye movement data pairs in the source domain data. Indicates the first The true sentiment category labels of samples, and Respectively represent the first The EEG data and eye movement data of samples are collected and form a set of EEG data and eye movement data pairs in the target domain data. and are the total number of samples of the source domain population and the target domain population, respectively. During the training process, the EEG data and eye movement data pairs in the source domain data and the EEG data and eye movement data pairs in the target domain data are respectively input into the multimodal emotion recognition model. With the help of the feature extraction module in the hierarchical interactive alignment network, the dynamic interactive attention layer of the hierarchical adaptive interactive attention module, the hierarchical representation distribution alignment layer and the few-shot learning module, the extraction of EEG deep features and eye movement deep features, the fusion of EEG deep features and eye movement deep features, the reduction of inter-domain distribution differences and the emotion recognition classification based on prototype vector metric learning of emotion categories are completed in sequence.

[0102] The cross-modal features of each layer are input into the hierarchical representation distribution alignment layer for feature distribution alignment. In the hierarchical representation distribution alignment layer, the maximum mean difference (MMD) loss function is used to reduce the inter-domain differences of multi-layer interactive modal information. In order to increase the alignment weight of deep features in the loss function, a hierarchical weighting mechanism based on logarithmic decay is introduced. In the calculation formula of the alignment weight of each layer, the parameter Determines the baseline size of the alignment weight, parameter Affects the growth rate of the logarithmic function. Finally, the multi-layer maximum average difference loss function is calculated.

[0103] In the few-shot learning module, the prototype vector of each emotion category is first calculated based on the cross-modal features corresponding to the support set extracted from the source domain data, and then the query samples in the query set extracted from the source domain data are classified. Finally, the cross entropy loss function is used as the classification loss function. The total loss function is constructed by combining the cross entropy loss function (emotion classification supervision) with the multi-layer maximum average difference loss function (domain alignment constraint). The parameters of the multimodal emotion recognition model are optimized by gradient descent, and the total loss function is minimized to obtain a trained multimodal emotion recognition model with optimized generalization performance.

[0104] S3, obtain the EEG data and eye movement data pair of one of the persons to be identified in the target domain population and input them into the trained multimodal emotion recognition model, and pass them through the feature extraction module and the hierarchical adaptive interactive attention module in sequence to obtain the final cross-modal features corresponding to the person to be identified. The final cross-modal features corresponding to the person to be identified and the final cross-modal features corresponding to the target domain data are input into the few-sample learning module to obtain the probability value that the person to be identified belongs to each emotion category, and select the emotion category corresponding to the maximum probability value as the predicted emotion category of the person to be identified.

[0105] In a specific embodiment, the brain electronic network includes a residual block and a hybrid attention module connected in sequence, and the eye movement sub-network adopts a DenseNet structure;

[0106] The EEG data in the EEG data and eye movement data pair is input into the brain electronic network to obtain the corresponding EEG depth features, as shown in the following formula:

[0107] ;

[0108] in, represents EEG data, represents the EEG depth feature, represents the brain electronic network;

[0109] The eye movement data in the EEG data and eye movement data pair is input into the eye movement sub-network to obtain the corresponding eye movement depth features, as shown in the following formula:

[0110] ;

[0111] in, represents eye movement data, represents the eye movement depth feature, represents the eye movement subnetwork.

[0112] Specifically, the trained multimodal emotion recognition model is deployed. In the inference stage, the EEG data and eye movement data of one of the persons to be identified in the target domain population are input into the trained multimodal emotion recognition model, wherein the EEG data is passed through a brain electronic network combining residual blocks and hybrid attention modules to extract EEG deep features; the eye movement data is passed through an eye movement sub-network and the DenseNet network architecture is used to extract eye movement deep features.

[0113] In a specific embodiment, each dynamic interactive attention layer in the hierarchical adaptive interactive attention module includes an attention module, a gating module, a splicing layer, and a multi-layer perceptron;

[0114] The cross-modal features of the i-1th layer output by the dynamic interaction attention layer of the i-1th layer are input into the dynamic interaction attention layer of the i-th layer, and are linearly transformed in the attention module to obtain the query matrix, as shown in the following formula:

[0115] ;

[0116] in, , represents the cross-modal features of the i-1th layer, represents the query matrix of the i-th layer, represents the query weight matrix of the i-th layer. When i=1, the cross-modal features output by the dynamic interactive attention layer of the 0th layer EEG depth feature ;

[0117] The eye movement depth feature is input into the dynamic interaction attention layer of the i-th layer. After linear transformation in the attention module, the key matrix and value matrix are obtained as shown in the following formula:

[0118] ;

[0119] ;

[0120] in, and Represent the key weight matrix of the i-th layer and the value weight matrix of the i-th layer respectively; and denote the key matrix and value matrix of the i-th layer respectively, Indicates the eye movement depth feature;

[0121] The following formula is used to calculate the attention feature of the i-th layer:

[0122] ;

[0123] in, (⋅) means function, d represents the feature dimension, T represents the transposed matrix, represents the attention feature of the i-th layer;

[0124] The attention features of the i-th layer are input into the gating module, and the channel weight of the i-th layer is calculated through the gated attention mechanism, as shown in the following formula:

[0125] ;

[0126] Among them, σ (⋅) represents the sigmoid activation function, represents the gating weight, represents the channel weight of the i-th layer;

[0127] The attention features of the i-th layer are gated and selected through the channel weights of the i-th layer to obtain the selected features of the i-th layer, as shown in the following formula:

[0128] ;

[0129] in, represents the selected features of the i-th layer, ⊙ represents channel-by-channel multiplication;

[0130] The selected features of the i-th layer are concatenated with the cross-modal features of the i-1-th layer and input into the multi-layer perceptron to obtain the fused features of the i-th layer, as shown in the following formula:

[0131] ;

[0132] Among them, [⋅;⋅] represents the feature concatenation operation, Represents the function corresponding to the multilayer perceptron, represents the fusion features of the i-th layer;

[0133] The fusion features of the i-th layer are connected with the cross-modal features of the i-1-th layer through residual connection to obtain the cross-modal features of the i-th layer, as shown in the following formula:

[0134] ;

[0135] in, Represents the cross-modal features of the i-th layer.

[0136] Specifically, through the collaborative processing of EEG deep features and eye movement deep features by multiple modules, emotion recognition and classification under few-sample conditions are realized, which includes: using the dynamic interactive attention layer in the hierarchical adaptive interactive attention module to fuse EEG and eye movement data with interactive attention; Figure 5In the first layer of dynamic interactive attention layer, the EEG deep features are input into the attention module, and the EEG deep features are linearly projected using the linear layer to generate the query matrix. The eye movement deep features are linearly projected using the linear layer and mapped to the feature space of the EEG deep features to generate the key matrix and value matrix. The attention features of the first layer are calculated through the query matrix, key matrix and value matrix. The channel weights of the first layer are calculated through the gated attention (GA) mechanism and the attention features of the first layer. The attention features of the first layer are gated and selected using the channel weights of the first layer to obtain the selected features of the first layer. The selected features of the first layer and the EEG deep features are spliced ​​and input into the multi-layer perceptron (MLP) to obtain the fused features of the first layer. The fused features of the first layer are then connected with the EEG deep features through residual connection to obtain the cross-modal features of the first layer. Since the dynamic interactive attention layer retains the original EEG deep features through residual connections, the dynamic interactive attention layer can be stacked into a multi-layer structure to enhance the multimodal interaction capability. After stacking, the EEG deep features input to the dynamic interactive attention layer of the next layer are replaced with the cross-modal features output by the dynamic interactive attention layer of the previous layer. In other words, the cross-modal features output by the dynamic interactive attention layer of the previous layer are input into the dynamic interactive attention layer of the next layer, and the attention features of the next layer are first calculated through the attention module. The channel weights of the next layer are calculated through the gated attention (GA) mechanism and the attention features of the next layer. The attention features of the next layer are gated and selected using the channel weights of the next layer to obtain the selected features of the next layer. The selected features of the next layer and the cross-modal features output by the dynamic interactive attention layer of the previous layer are spliced ​​and input into the multi-layer perceptron (MLP) to obtain the fused features of the next layer. The fused features of the next layer are then connected with the cross-modal features output by the dynamic interactive attention layer of the previous layer through residual connections to obtain the cross-modal features of the next layer. Finally, the cross-modal features of the last layer are used as the final cross-modal features.

[0137] In a specific embodiment, the cross-modal features of the first layer output by the dynamic interactive attention layer of the last layer are used as the final cross-modal features; in the reasoning process of the trained multimodal emotion recognition model, the person to be identified is used as a query sample in the query set, and the support set is extracted from the target domain data;

[0138] In the few-shot learning module, the prototype vector of each emotion category is calculated based on the final cross-modal features corresponding to the support set, as shown in the following formula:

[0139] ;

[0140] in, represents one of the support samples belonging to the cth sentiment category in the support set, Indicates one of the support samples belonging to the cth sentiment category in the support set The corresponding final cross-modal features, represents the set of support samples belonging to the cth sentiment category in all support sets, Represents the prototype vector of the cth emotion category, c=1,2,…,N;

[0141] The probability value of each query sample in the query set belonging to each emotion category is calculated based on the final cross-modal features corresponding to the query set and the prototype vector of each emotion category, as shown in the following formula:

[0142] ;

[0143] in, represents one of the query samples in the query set, Represents one of the query samples in the query set The corresponding final cross-modal features, represents the prototype vector of the u-th emotion category, Represents the prototype vector of the vth emotion category, u=1,2,…,N, v=1,2,…,N, Represents one of the query samples in the query set The probability value of belonging to the u-th emotion category, represents the predicted sentiment category, represents the calculation function of Euclidean distance, Represents an exponential function with base e.

[0144] Specifically, during the reasoning process, a support set is extracted from the target domain data, and the person to be identified is used as a query sample in the query set to calculate the probability value of the person to be identified belonging to each emotion category in the few-sample learning module. Finally, the emotion category with the largest probability value is selected as the predicted emotion category corresponding to the person to be identified, and finally the emotion recognition classification is completed.

[0145] Data and Processing: In a cross-subject experiment, the embodiments of the present application used two multimodal emotion datasets with electroencephalogram (EEG) data and eye movement (EM) data: SEED and SEED-FRA.

[0146] SEED: This study used a multimodal version of the dataset, which contains EEG signals from 12 participants and corresponding emotion labels. Each participant was asked to annotate their emotional response after watching a series of videos. The dataset contains three basic emotion categories: happy, sad, and angry. The EEG data is recorded by 62 channels with a sampling rate of 1000 Hz. In order to obtain eye movement data synchronized with the EEG data, SMI eye tracking glasses were used to collect the corresponding eye movement data.

[0147] SEED-FRA: This study recruited 8 native French subjects to participate in the experiment, which was conducted three times, with 21 stimulus material clips each time.

[0148] In the experiment, the embodiment of the present application adopts the leave-one-out cross-validation method to evaluate the performance. That is, one subject is alternately selected as the target domain population, and the remaining subjects are combined into a source domain population to evaluate the classification accuracy. All samples of the source domain population are used as the training set. At the same time, k labeled samples are randomly selected from each emotion category of the target domain population, that is, a total of N × k samples are selected as the training set, and 300 samples are selected from the remaining samples as the validation set, and the remaining samples are used as the test set. In the meta-training stage, the support set and query set of each emotion category are extracted from the source domain training set, and a small number of labeled samples in the target domain are only used to calculate the MMD loss. In the meta-validation and meta-testing stages, N * k labeled samples of the training set of the target domain are used as the support set, and the query set is extracted from the validation set or the test set, respectively.

[0149] For the setting of N categories and k samples, M is set to 3, and k is set to 1, 5, 10, and 20. The embodiment of the present application sets the number of dynamic interactive attention layers to 3. The Adam optimizer is used to minimize the loss function. The learning rate is set to 1e-4. The λ in the dynamic adjustment loss function is λ = , where p is the ratio of the number of training rounds currently in progress to the total number of training rounds, and θ is fixed to 10 during the training process. An early stopping scheme is used to save training time. 50 rounds of training are performed, and the number of tasks in each round is set to 20.

[0150] Through experiments on SEED and SEED-FRA datasets, the trained multimodal emotion recognition model (denoted as HIA-Net) proposed in the embodiment of the present application is systematically compared with existing emotion recognition methods. Table 1 shows the performance of HIA-Net in multimodal emotion recognition tasks. The embodiment of the present application uses accuracy as the evaluation metric and bolds the best results under different tasks to clearly show the performance differences of each model. Based on the reported experimental results, the following observations are summarized:

[0151] (1) HIA-Net showed the best emotion recognition results in most cases. HIA-Net consistently outperformed other comparison models in different datasets and different k-sample settings. In the SEED dataset, the average emotion recognition accuracy of HIA-Net under different k-sample conditions was 86.60%, 95.85%, 98.34%, and 99.00%, respectively; while on the SEED-FRA dataset, the average emotion recognition accuracy of HIA-Net was 73.66%, 82.73%, 91.17%, and 93.57%, respectively. This series of results shows that HIA-Net has excellent ability in effectively combining EEG and EM modal information, significantly reducing individual differences between subjects, and adaptively eliminating differences between multiple layers of interactive modal information, thereby achieving higher accuracy in emotion recognition tasks.

[0152] By comparing the multimodal emotion recognition results of HIA-Net and SDA-FSL, it is found that HIA-Net is significantly better than SDA-FSL in most cases. This shows that the hierarchical adaptive interaction attention module in HIA-Net can effectively combine the information of EEG and EM modalities to fit the optimal fusion representation of modal interaction features.

[0153] Table 1:

[0154]

[0155] Further references Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a multimodal emotion recognition device based on a hierarchical interactive alignment network. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0156] The embodiment of the present application provides a multimodal emotion recognition device based on a hierarchical interactive alignment network, comprising:

[0157] A model building module 1 is configured to build a multimodal emotion recognition model and a hierarchical representation distribution alignment layer, the multimodal emotion recognition model includes a feature extraction module, a hierarchical adaptive interactive attention module and a few-shot learning module, the feature extraction module includes a brain electronic network and an eye movement sub-network, and the hierarchical adaptive interactive attention module includes a plurality of layers of dynamic interactive attention layers connected in sequence;

[0158] Training module 2 is configured to obtain source domain data and target domain data, the source domain data includes EEG data and eye movement data pairs of the source domain population and their corresponding real emotion category labels, and the target domain data includes EEG data and eye movement data pairs of the target domain population; the EEG data and eye movement data pairs in the source domain data and the EEG data and eye movement data pairs in the target domain data are respectively input into the multimodal emotion recognition model, and the EEG depth features and eye movement depth features corresponding to the source domain data and the EEG depth features and eye movement depth features corresponding to the target domain data are first extracted through the brain electronic network and the eye movement sub-network in the feature extraction module; the EEG depth features and eye movement depth features corresponding to the source domain data and the EEG depth features and eye movement depth features corresponding to the target domain data are respectively input into the hierarchical adaptive interactive attention module, and after each layer of dynamic interactive attention layer, the cross-modal features of each layer corresponding to the source domain data and the cross-modal features of each layer corresponding to the target domain data are output. The cross-modal features of each layer corresponding to the source domain data and the cross-modal features of each layer corresponding to the target domain data are input into the hierarchical representation distribution alignment layer to calculate the multi-layer maximum average difference loss function; the support set and the query set are extracted from the source domain data, and the final cross-modal features corresponding to the support set and the final cross-modal features corresponding to the query set are determined according to the final cross-modal features corresponding to the source domain data and input into the few-shot learning module to obtain the probability value of each query sample in the query set belonging to each emotion category, and the cross-entropy loss function is calculated according to the probability value of each query sample in the query set belonging to each emotion category and its corresponding true emotion category label; the total loss function is constructed based on the multi-layer maximum average difference loss function and the cross-entropy loss function, and the multi-modal emotion recognition model is trained using the total loss function to obtain the trained multi-modal emotion recognition model;

[0159] The prediction module 3 is configured to obtain the EEG data and eye movement data pair of one of the persons to be identified in the target domain population and input them into the trained multimodal emotion recognition model, and then pass through the feature extraction module and the hierarchical adaptive interactive attention module in sequence to obtain the final cross-modal features corresponding to the person to be identified. The final cross-modal features corresponding to the person to be identified and the final cross-modal features corresponding to the target domain data are input into the few-sample learning module to obtain the probability value that the person to be identified belongs to each emotion category, and the emotion category corresponding to the maximum probability value is selected as the predicted emotion category of the person to be identified.

[0160] Figure 7 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 7As shown, the electronic device of this embodiment includes: a processor 701 and a memory 702; wherein the memory 702 is used to store computer-executable instructions; the processor 701 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0161] Optionally, the memory 702 may be independent or integrated with the processor 701 .

[0162] When the memory 702 is independently provided, the electronic device further includes a bus 703 for connecting the memory 702 and the processor 701 .

[0163] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 701 executes the computer execution instructions, the above method is implemented.

[0164] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 701, the above method is implemented.

[0165] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0166] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0167] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0168] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 701 to perform some steps of the methods of various embodiments of the present application.

[0169] It should be understood that the processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 701 may be any conventional processor 701, etc. The steps of the method disclosed in the invention may be directly embodied in the execution of the hardware processor 701, or may be executed by a combination of hardware and software modules in the processor 701.

[0170] The memory 702 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0171] The bus 703 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 703 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 703 in the drawings of the present application is not limited to only one bus 703 or one type of bus 703.

[0172] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0173] An exemplary storage medium is coupled to the processor 701, so that the processor 701 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 701. The processor 701 and the storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as ASIC). Of course, the processor 701 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0174] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal emotion recognition method based on hierarchical interaction alignment network, characterized in that: The following steps are involved: Constructing a multimodal emotion recognition model and a hierarchical representation distribution alignment layer, wherein the multimodal emotion recognition model includes a feature extraction module, a hierarchical adaptive interactive attention module and a few-shot learning module, wherein the feature extraction module includes a brain electronic network and an eye movement subnetwork, and the hierarchical adaptive interactive attention module includes a plurality of sequentially connected dynamic interactive attention layers; Acquire source domain data and target domain data, wherein the source domain data includes EEG data and eye movement data pairs of source domain people and their corresponding real emotion category labels, and the target domain data includes EEG data and eye movement data pairs of target domain people; input the EEG data and eye movement data pairs in the source domain data and the EEG data and eye movement data pairs in the target domain data into the multimodal emotion recognition model respectively, first extract the EEG depth features and eye movement depth features corresponding to the source domain data and the EEG depth features and eye movement depth features corresponding to the target domain data through the brain electron network and eye movement sub-network in the feature extraction module respectively; input the EEG depth features and eye movement depth features corresponding to the source domain data and the EEG depth features and eye movement depth features corresponding to the target domain data into the hierarchical adaptive interactive attention module respectively, and output the cross-modal features of each layer corresponding to the source domain data and the cross-modal features of each layer corresponding to the target domain data, as well as the final cross-modal features corresponding to the source domain data and the final cross-modal features corresponding to the target domain data through each layer of dynamic interactive attention layer; The cross-modal features of each layer corresponding to the source domain data and the cross-modal features of each layer corresponding to the target domain data are input into the hierarchical representation distribution alignment layer, and a multi-layer maximum average difference loss function is calculated; a support set and a query set are extracted from the source domain data, and the final cross-modal features corresponding to the support set and the final cross-modal features corresponding to the query set are determined according to the final cross-modal features corresponding to the source domain data and input into the few-sample learning module to obtain the probability value of each query sample in the query set belonging to each emotion category, and a cross-entropy loss function is calculated according to the probability value of each query sample in the query set belonging to each emotion category and its corresponding true emotion category label; a total loss function is constructed based on the multi-layer maximum average difference loss function and the cross-entropy loss function, and the multi-modal emotion recognition model is trained using the total loss function to obtain a trained multi-modal emotion recognition model; Obtain a pair of EEG data and eye movement data of one of the persons to be identified in the target domain population and input them into the trained multimodal emotion recognition model; sequentially pass through the feature extraction module and the hierarchical adaptive interactive attention module to obtain the final cross-modal features corresponding to the person to be identified; the final cross-modal features corresponding to the person to be identified and the final cross-modal features corresponding to the target domain data are input into the few-sample learning module to obtain the probability value that the person to be identified belongs to each emotion category; and select the emotion category corresponding to the maximum probability value as the predicted emotion category of the person to be identified.

2. The multimodal emotion recognition method based on hierarchical interaction alignment network according to claim 1 is characterized in that: The brain electronic network includes a residual block and a hybrid attention module connected in sequence, and the eye movement sub-network adopts a DenseNet structure; The EEG data in the EEG data and eye movement data pair is input into the brain electronic network to obtain the corresponding EEG depth feature, as shown in the following formula: ； in, represents EEG data, represents the EEG depth feature, represents the brain electronic network; The eye movement data in the pair of EEG data and eye movement data is input into the eye movement sub-network to obtain the corresponding eye movement depth feature, as shown in the following formula: ； in, represents eye movement data, represents the eye movement depth feature, Represents the eye movement subnetwork.

3. The multimodal emotion recognition method based on hierarchical interaction alignment network according to claim 1 is characterized in that: Each dynamic interactive attention layer in the hierarchical adaptive interactive attention module includes an attention module, a gating module, a splicing layer and a multi-layer perceptron; The cross-modal features of the i-1th layer output by the dynamic interaction attention layer of the i-1th layer are input into the dynamic interaction attention layer of the i-th layer, and a query matrix is ​​obtained by linear transformation in the attention module, as shown in the following formula: ； in, , represents the cross-modal features of the i-1th layer, represents the query matrix of the i-th layer, represents the query weight matrix of the i-th layer. When i=1, the cross-modal features output by the dynamic interactive attention layer of the 0th layer EEG depth feature ; The eye movement depth feature is input into the dynamic interaction attention layer of the i-th layer, and is linearly transformed in the attention module to obtain the key matrix and value matrix, as shown in the following formula: ； ； in, and Represent the key weight matrix of the i-th layer and the value weight matrix of the i-th layer respectively; and denote the key matrix and value matrix of the i-th layer respectively, Indicates the eye movement depth feature; The following formula is used to calculate the attention feature of the i-th layer: ； in, (⋅) means function, d represents the feature dimension, T represents the transposed matrix, Represents the attention feature of the i-th layer; The attention features of the i-th layer are input into the gating module, and the channel weight of the i-th layer is calculated through the gated attention mechanism, as shown in the following formula: ； Among them, σ (⋅) represents the sigmoid activation function, represents the gating weight, represents the channel weight of the i-th layer; The attention features of the i-th layer are gated and selected through the channel weights of the i-th layer to obtain the selected features of the i-th layer, as shown in the following formula: ； in, represents the selected features of the i-th layer, ⊙ represents channel-by-channel multiplication; The selected features of the i-th layer are concatenated with the cross-modal features of the i-1-th layer and input into the multi-layer perceptron to obtain the fusion features of the i-th layer, as shown in the following formula: ； Among them, [⋅;⋅] represents the feature concatenation operation, Represents the function corresponding to the multilayer perceptron, represents the fusion features of the i-th layer; The fusion features of the i-th layer are connected with the cross-modal features of the i-1-th layer through residual connection to obtain the cross-modal features of the i-th layer, as shown in the following formula: ； in, Represents the cross-modal features of the i-th layer.

4. The multimodal emotion recognition method based on hierarchical interaction alignment network according to claim 1 is characterized in that: In the hierarchical representation distribution alignment layer, the maximum average difference loss function is used to measure the inter-domain differences of the cross-modal features of each layer, as shown in the following formula: ； in, is the cross-modal feature of the i-th layer corresponding to one of the samples in the source domain data, for A collection of represents the cross-modal feature of the i-th layer corresponding to one of the samples in the target domain data, for The set of cross-modal features of each layer corresponding to the source domain data and the cross-modal features of each layer corresponding to the target domain data, represents the MMD loss of the i-th layer, Represents the transformation of feature points into a reproducing kernel Hilbert space The mapping function in represents the reproducing kernel Hilbert space The square of the norm of ; Introduce a hierarchical weighting mechanism based on logarithmic decay: ； Secondly, represents the alignment weight of the i-th layer, and Represent the weight base parameter and logarithmic growth parameter respectively; The multi-layer maximum average difference loss function is calculated using the following formula: ； Where I represents the total number of dynamic interactive attention layers, Represents the multi-layer maximum average discrepancy loss function.

5. The multimodal emotion recognition method based on hierarchical interaction alignment network according to claim 1 is characterized in that: The cross-modal features of the Ith layer output by the last layer of dynamic interaction attention layer are used as the final cross-modal features; In the reasoning process of the trained multimodal emotion recognition model, the person to be recognized is used as a query sample in a query set, and a support set is extracted from the target domain data; In the few-shot learning module, the prototype vector of each emotion category is calculated based on the final cross-modal features corresponding to the support set, as shown in the following formula: ； in, represents one of the support samples belonging to the cth sentiment category in the support set, Indicates one of the support samples in the support set that belongs to the cth sentiment category The corresponding final cross-modal features, represents the set of support samples belonging to the cth sentiment category in all support sets, Represents the prototype vector of the cth emotion category, c=1,2,…,N; The probability value of each query sample in the query set belonging to each emotion category is calculated according to the final cross-modal features corresponding to the query set and the prototype vector of each emotion category, as shown in the following formula: ； in, represents one of the query samples in the query set, Represents one of the query samples in the query set The corresponding final cross-modal features, represents the prototype vector of the u-th emotion category, Represents the prototype vector of the vth emotion category, u=1,2,…,N, v=1,2,…,N, Represents one of the query samples in the query set The probability value of belonging to the u-th emotion category, represents the predicted sentiment category, represents the calculation function of Euclidean distance, Represents an exponential function with base e.

6. The multimodal emotion recognition method based on hierarchical interaction alignment network according to claim 5 is characterized in that: The cross entropy loss function is calculated using the following formula: ； in, represents the cross entropy loss function, Represents one of the query samples in the query set The one-hot encoding of the true sentiment category label. If one of the query samples in the query set The true emotion category label is u, then ,otherwise ; The total loss function is calculated using the following formula: ； in, represents the multi-layer maximum average difference loss function, represents the total loss function, Represents the balance weight.

7. A multimodal emotion recognition device based on a hierarchical interactive alignment network, characterized in that: include: A model building module, configured to build a multimodal emotion recognition model and a hierarchical representation distribution alignment layer, wherein the multimodal emotion recognition model includes a feature extraction module, a hierarchical adaptive interactive attention module and a few-shot learning module, wherein the feature extraction module includes a brain electronic network and an eye movement subnetwork, and the hierarchical adaptive interactive attention module includes a plurality of sequentially connected dynamic interactive attention layers; A training module is configured to obtain source domain data and target domain data, wherein the source domain data includes EEG data and eye movement data pairs of a source domain population and their corresponding real emotion category labels, and the target domain data includes EEG data and eye movement data pairs of a target domain population; the EEG data and eye movement data pairs in the source domain data and the EEG data and eye movement data pairs in the target domain data are respectively input into the multimodal emotion recognition model, and the EEG depth features and eye movement depth features corresponding to the source domain data and the EEG depth features and eye movement depth features corresponding to the target domain data are first extracted through the brain electron network and the eye movement sub-network in the feature extraction module; the EEG depth features and eye movement depth features corresponding to the source domain data and the EEG depth features and eye movement depth features corresponding to the target domain data are respectively input into the hierarchical adaptive interactive attention module, and after each layer of dynamic interactive attention layer, the cross-modal features of each layer corresponding to the source domain data and the cross-modal features of each layer corresponding to the target domain data as well as the final cross-modal features corresponding to the source domain data and the final cross-modal features corresponding to the target domain data are output; The cross-modal features of each layer corresponding to the source domain data and the cross-modal features of each layer corresponding to the target domain data are input into the hierarchical representation distribution alignment layer, and a multi-layer maximum average difference loss function is calculated; a support set and a query set are extracted from the source domain data, and the final cross-modal features corresponding to the support set and the final cross-modal features corresponding to the query set are determined according to the final cross-modal features corresponding to the source domain data and input into the few-sample learning module to obtain the probability value of each query sample in the query set belonging to each emotion category, and a cross-entropy loss function is calculated according to the probability value of each query sample in the query set belonging to each emotion category and its corresponding true emotion category label; a total loss function is constructed based on the multi-layer maximum average difference loss function and the cross-entropy loss function, and the multi-modal emotion recognition model is trained using the total loss function to obtain a trained multi-modal emotion recognition model; The prediction module is configured to obtain the EEG data and eye movement data pair of one of the persons to be identified in the target domain population and input them into the trained multimodal emotion recognition model, and successively pass through the feature extraction module and the hierarchical adaptive interactive attention module to obtain the final cross-modal features corresponding to the person to be identified. The final cross-modal features corresponding to the person to be identified and the final cross-modal features corresponding to the target domain data are input into the few-sample learning module to obtain the probability value that the person to be identified belongs to each emotion category, and select the emotion category corresponding to the maximum probability value as the predicted emotion category of the person to be identified.

8. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images