Multimodal few-shot emotion recognition method and device for modality missing

Abstract

This invention discloses a method and apparatus for multimodal few-shot emotion recognition tackling modality loss, relating to the field of emotion computing technology. The method includes: S1, acquiring EEG and eye-tracking data; S2, constructing a multimodal few-shot emotion recognition model including a feature extraction module, a missing modality reconstruction module, an adaptive fusion module for uncertainty perception, and a few-shot classification module; S3, training the multimodal few-shot emotion recognition model using the EEG and eye-tracking data to obtain a trained multimodal few-shot emotion recognition model; and S4, performing emotion recognition using the trained multimodal few-shot emotion recognition model. This invention constructs a multimodal few-shot emotion recognition model that significantly improves the accuracy, robustness, and generalization ability of multimodal emotion recognition in complex real-world scenarios such as modality loss, scarce labeled samples, and cross-subject distribution shifts.

Description

Technical Field

[0001] This invention relates to the field of emotion computing technology, and in particular to a method and apparatus for multimodal few-sample emotion recognition in the face of modality loss. Background Technology

[0002] As research into the mechanisms of human emotion deepens, multimodal emotion recognition (MER) has become an important direction in emotion computing. By integrating multi-source physiological and behavioral signals, MER can more objectively characterize emotional states. Specifically, electroencephalogram (EEG) signals reflect neural dynamics with millisecond-level temporal resolution, while eye movement (EM) signals provide complementary cues from overt behavioral aspects such as fixation, saccades, and pupillary responses. The natural complementarity between these two in terms of temporal dynamics and behavioral representation makes EEG+EM a common combination for physiologically signal-driven MER.

[0003] Despite significant progress in MER (Modal Recognition and Modulation) technology, it still faces multiple constraints in practical applications. First, physiological data itself is non-stationary and exhibits significant individual differences. Combined with factors such as the acquisition environment, subject status, and temporal drift, distribution bias is easily introduced, leading to insufficient cross-subject generalization ability of the model. Second, the modal presence in actual acquisition is unstable. Factors such as poor electrode contact, motion and blink artifacts, occlusion, and changes in illumination can cause some modalities to degenerate or be missing, causing the signal-to-noise ratio to fluctuate over time. Many existing fusion strategies assume that all modalities are complete and equally reliable, lacking explicit characterization and transmission mechanisms for input uncertainty. When missing or noisy conditions exist, the fusion process cannot adaptively suppress unreliable information, thus affecting the final discrimination performance. Therefore, how to perform uncertainty-aware fusion under conditions of modal missingness and improve the model's robustness to cross-subject and scene changes has become a pressing issue in the current MER field. Summary of the Invention

[0004] To address the above problems, this invention proposes a multimodal few-shot emotion recognition method and apparatus for modality missing. By constructing a multimodal few-shot emotion recognition model including a feature extraction module, a missing modality reconstruction module, an adaptive fusion module for uncertainty perception, and a few-shot classification module, and jointly utilizing modal information from EEG and eye movement, when modality missing is detected, an alternative representation of the missing modality is generated based on the present modality. In real-world scenarios with incomplete modalities and scarce annotations, this significantly improves recognition accuracy and robustness.

[0005] On the one hand, a multimodal few-sample emotion recognition method for modality-deficient data collection involves the following steps:

[0006] S1, acquire EEG data and eye movement data;

[0007] S2, construct a multimodal few-shot sentiment recognition model including a feature extraction module, a missing modality reconstruction module, an adaptive fusion module for uncertainty perception, and a few-shot classification module;

[0008] The feature extraction module extracts features from EEG data to obtain EEG data depth features; and extracts features from eye movement data to obtain eye movement data depth features.

[0009] The missing modality reconstruction module performs modality missing detection on EEG data or eye movement data. If eye movement data is missing, it generates a substitute representation based on the EEG data and uses the substitute features as the depth features of the eye movement data.

[0010] The adaptive fusion module for uncertainty perception fuses the deep features of EEG data and the deep features of eye-tracking data to obtain fused features;

[0011] The small sample classification module performs emotion category discrimination on the fused features and outputs the identified emotion category;

[0012] S3, use EEG data and eye-tracking data to train the multimodal few-shot emotion recognition model to obtain the trained multimodal few-shot emotion recognition model;

[0013] S4 uses a trained multimodal few-shot emotion recognition model for emotion recognition.

[0014] Preferably, the feature extraction module is implemented as follows:

[0015] EEG data is input into a convolutional feature extraction network that includes residual propagation paths and attention mechanisms. The convolutional feature extraction network performs residual aggregation between the main branch and the shortcut branch, and applies attention to adaptively weight the features in the channel and spatial dimensions. The output of the convolutional feature extraction network is normalized and downsampled, then globally aggregated and linearly mapped to a unified dimension to obtain deep features of EEG modalities.

[0016] Eye-tracking data is input into a densely connected temporal convolutional network structure; the temporal convolutional network structure achieves feature reuse and efficient gradient transfer through cross-layer connections within the same stage; the temporal convolutional network structure sets transition units between stages to complete channel compression and temporal downsampling; the output of the temporal convolutional network structure is globally aggregated and linearly mapped to a unified dimension to output eye-tracking modality depth features.

[0017] Preferably, the missing mode reconstruction module is implemented as follows:

[0018] Modal missing data detection is performed on the presence status of EEG or eye-tracking data. Data missing in either EEG or eye-tracking data is recorded as missing modal data, and data in the other modality is recorded as present modal data. Correspondingly, the features extracted from the present modal data by the feature extraction module are recorded as present modal depth features.

[0019] Using the presence modality depth feature as the query, retrieve the K nearest neighbor sample index set from the feature database. The feature library consists of paired sample features of previously acquired EEG data and eye-tracking data.

[0020] Calculate the similarity between the depth features of the present modality and the K nearest neighbor samples, and calculate the normalized weight of each nearest neighbor sample based on the similarity.

[0021] The prior vector of the missing mode is calculated based on the normalized weights and is expressed as follows:

[0022] ;

[0023] in, The prior vector representing the missing mode; Represents the set of nearest neighbor sample indices In the context of the depth features of the present modality with the i-th nearest neighbor sample Missing modal features of paired samples; Indicates the normalized weights;

[0024] The feature vector of the field modality depth features and prior vectors of missing modes By applying linear projection and dividing each token into d-dimensional source tokens, two token sequences are obtained. These sequences are then concatenated to obtain a concatenated token sequence. ;

[0025] Introducing a learnable query matrix as the query for multi-head attention, concatenating the token sequence... As keys and values, they interact through multi-layer conditional blocks containing multi-head cross-attention and feedforward networks to obtain a refined final query representation;

[0026] The final query representations are aggregated and processed through two readout heads to obtain the reconstructed vectors. With internal confidence level c;

[0027] Gated fusion yields alternative representations of the missing modes, expressed as:

[0028] ;

[0029] in, Alternative representations of missing modes; Indicates the fusion weights; This represents the preset minimum fusion weight; Indicates internal confidence level. ; This indicates taking the maximum value; This represents the reconstructed vector.

[0030] Preferably, the uncertainty-aware adaptive fusion module is implemented as follows:

[0031] The input q-mode features are output to the p-mode space, and the channel-by-channel uncertainty is calculated, expressed as:

[0032] ;

[0033] in, This indicates uncertainty on a per-channel basis; Represents a frozen random prior network; This represents a trainable prediction network; This represents the q-mode features of the input; p represents any modality of EEG or eye-tracking data. Another modality besides; Represents the square of the L2 norm;

[0034] right Channel-by-channel normalization yields the uncertainty graph Dimensions and Alignment;

[0035] Applying the uncertainty map to the modal features and generating weighted features via a fusion processor is represented as follows:

[0036] The element-wise weighted average is calculated as follows:

[0037] ;

[0038] ;

[0039] in, Indicates weighted features; The fusion unit consists of two layers of linear mappings and nonlinear activations. Indicates will and cascade; This indicates element-wise multiplication; This represents the feature after element-wise weighting;

[0040] The weighted features of the two modalities are pre-fused and summed element by element, as follows:

[0041] ;

[0042] in, Represents the pre-fusion vector; This represents the weighted features corresponding to the EEG data; This represents the weighted features corresponding to the eye-tracking data;

[0043] The fused modes are calculated using channel gating and are represented as follows:

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] in, Indicates fused modes; Representation layer normalization; This represents the sigmoid activation function. Indicates the gating weight; Indicates the gating bias parameter; Represents the channel gating vector; Indicates the gain term; This represents a multilayer perceptron; Indicates cascading; This represents the concatenated vector.

[0049] Preferably, the small sample classification module is implemented as follows: dividing the samples into a support set and a query set; for each sentiment category in the target domain, averaging the fusion features of all samples in its support set to generate a prototype vector for that category; calculating the distance between the fusion features of the samples in the query set and the prototypes of each sentiment category; and negatively mapping the distance to a category probability distribution using a temperature-scaled softmax function.

[0050] Preferably, the joint loss function of the multimodal few-shot sentiment recognition model during training includes classification loss, domain alignment loss, and reconstruction loss; the reconstruction loss supervises the output quality of the missing modality reconstruction module; the domain alignment loss is used to reduce the distribution difference between the source domain and the target domain in the feature space; the classification loss is used to measure the inconsistency between the predicted sentiment category probability of the query sample and the true label; the joint loss function is expressed as:

[0051] ;

[0052] in, Represents the joint loss function; and The weighting parameters represent the balancing loss; Indicates classification loss; Indicates the losses incurred during reconstruction; Indicates the domain alignment loss.

[0053] Preferably, the multimodal few-shot emotion recognition model further includes a domain alignment module, which implements the domain alignment loss. The domain alignment module is configured to: calculate the distribution difference between source domain samples and target domain samples at multiple feature levels; the distribution difference is measured by the maximum mean difference to constrain the distribution consistency of the source domain and the target domain in the common feature space; assign different weights to the distribution difference terms at each level, with the weights of deeper layers being higher than those of shallower layers; use the sum of weighted distribution differences as the domain alignment loss, and gradually increase the proportion of this loss in the total loss during training.

[0054] Preferably, the reconstruction loss is expressed as:

[0055] ;

[0056] in, To reconstruct the vector, Indicates the corresponding truth features, Indicates hyperparameters, ; This represents the set of missing samples for the target modality obtained by sampling according to the missing parameters during training. Indicates the size of the set; Represents the square of the L2 norm; This represents the cosine similarity.

[0057] Preferably, the classification loss employs a cross-entropy loss function based on temperature-scaled prototype probabilities.

[0058] On the other hand, multimodal few-shot emotion recognition devices for modality-deficient applications include the following:

[0059] The EEG and eye-tracking data acquisition module is used to acquire EEG and eye-tracking data;

[0060] A multimodal few-shot sentiment recognition model building module is used to construct a multimodal few-shot sentiment recognition model that includes a feature extraction module, a missing modality reconstruction module, an adaptive fusion module for uncertainty perception, and a few-shot classification module.

[0061] The feature extraction module extracts features from EEG data to obtain EEG data depth features; and extracts features from eye movement data to obtain eye movement data depth features.

[0062] The missing modality reconstruction module performs modality missing detection on EEG data or eye movement data. If eye movement data is missing, it generates a substitute representation based on the EEG data and uses the substitute features as the depth features of the eye movement data.

[0063] The adaptive fusion module for uncertainty perception fuses the deep features of EEG data and the deep features of eye-tracking data to obtain fused features;

[0064] The small sample classification module performs emotion category discrimination on the fused features and outputs the identified emotion category;

[0065] The multimodal few-shot emotion recognition model training module is used to train the multimodal few-shot emotion recognition model using EEG data and eye-tracking data to obtain a trained multimodal few-shot emotion recognition model.

[0066] The emotion recognition module is used to perform emotion recognition using a pre-trained multimodal few-shot emotion recognition model.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] (1) When a missing modality is detected, the missing modality reconstruction module of the present invention uses the present modality as the query, combines the nearest neighbor retrieval prior and conditional cross attention to generate alternative representations, and through confidence gating and prior adaptive fusion, it is significantly better than traditional methods such as zero filling, avoiding cross-modal semantic mismatch and feature distribution shift.

[0069] (2) The uncertainty-aware adaptive fusion module of the present invention outputs dimension-wise uncertainty through a random prior network, forms a confidence map through monotonic mapping, performs element-wise weighting on single-modal features, and then obtains a unified representation through residual cross-modal fusion; this mechanism automatically suppresses unreliable information and improves fusion stability when noise increases, reconstruction participation is affected, or the quality of one modality is unstable.

[0070] (3) This invention combines EEG and eye movement modal information to generate alternative representations of missing modalities when modal absence is detected, based on the present modality. Subsequently, single-modal features are weighted by confidence based on uncertainty estimation, and a unified representation is obtained through cross-modal residual fusion. Under small sample settings, prototype metric learning is used to complete the classification, and a domain alignment strategy is introduced to alleviate cross-individual distribution differences, thereby achieving effective recognition of complex emotional states. In real-world scenarios with incomplete modalities and scarce annotations, the recognition accuracy and robustness are significantly improved, overcoming the shortcomings of existing multimodal emotion recognition in terms of insufficient robustness under modal absence, cross-individual bias, and small sample conditions. Attached Figure Description

[0071] The present invention will now be described in further detail with reference to the accompanying drawings;

[0072] Figure 1 This is a flowchart of a multimodal few-sample emotion recognition method for modality-deficient data according to an embodiment of the present invention;

[0073] Figure 2 This is a schematic diagram illustrating the steps of the multimodal few-sample emotion recognition method for modality-deficient methods according to an embodiment of the present invention.

[0074] Figure 3 This is a schematic diagram of the multimodal few-shot emotion recognition model framework for a multimodal few-shot emotion recognition method for modality missing data, as described in an embodiment of the present invention.

[0075] Figure 4 This is a schematic diagram of the EEG feature extraction network for a multimodal few-sample emotion recognition method for modality-deficient conditions according to an embodiment of the present invention.

[0076] Figure 5 This is a schematic diagram of the eye-tracking feature extraction network for a multimodal few-sample emotion recognition method for modality-deficient conditions according to an embodiment of the present invention.

[0077] Figure 6 This is a schematic diagram of the missing modality reconstruction module in the multimodal few-sample emotion recognition method for modality loss according to an embodiment of the present invention;

[0078] Figure 7 This is a schematic diagram illustrating the training process of the multimodal few-shot emotion recognition method for modality-deficient methods according to an embodiment of the present invention.

[0079] Figure 8 This is a schematic diagram of the model structure of the multimodal few-sample emotion recognition method for modality-deficient methods according to an embodiment of the present invention;

[0080] Figure 9 This is a structural block diagram of a multimodal few-sample emotion recognition device for modality-deficient applications according to an embodiment of the present invention. Detailed Implementation

[0081] The present invention will be further described below through specific embodiments.

[0082] like Figure 1 and Figure 2 As shown, the specific steps of the multimodal few-shot emotion recognition method for modality missing features are as follows:

[0083] S1 acquires EEG and eye-tracking data.

[0084] S2, construct a multimodal few-shot emotion recognition model including a feature extraction module, a missing modality reconstruction module, an adaptive fusion module for uncertainty perception, and a few-shot classification module.

[0085] Multimodal few-shot emotion recognition models, such as Figure 3 As shown. (See also...) Figure 4 and Figure 5 As shown, the feature extraction module includes an EEG feature extraction network and an eye-tracking feature extraction network, which are used to extract the depth features of EEG data and the depth features of eye-tracking data, respectively. Specifically, it includes:

[0086] EEG data is input into a convolutional feature extraction network with residual propagation paths and attention mechanisms to obtain deep features of the EEG data:

[0087] ;

[0088] Eye-tracking data is input into a densely connected temporal convolutional network structure to obtain the depth features of the eye-tracking data:

[0089] ;

[0090] If a missing mode exists, proceed to the Missing Mode Reconstruction (MMR) module to reconstruct modal features; otherwise, skip this module.

[0091] See the missing mode reconstruction module. Figure 6 As shown, the details are as follows:

[0092] First, the depth feature vector extracted from the presence modality is denoted as... ,by To retrieve the K nearest neighbor samples from the feature database, based on their cosine similarity to the query. Calculate the weights:

[0093] ;

[0094] in, The source modal features of the i-th sample in the feature library, For To retrieve the index set of K nearest neighbor samples retrieved from the database based on similarity, This is the temperature coefficient.

[0095] Therefore, we can obtain the retrieval prior for the target modality:

[0096] ;

[0097] in, To and Modal features of paired samples.

[0098] Next, the feature vectors of the present modes will be... and the prior vector obtained by retrieval Projecting and segmenting into d-dimensional source tokens, that is... and And cascade them:

[0099] ;

[0100] Subsequently, L-layer conditional blocks are stacked, and conditional information is read from T within each layer using multi-head cross-attention:

[0101]

[0102] And after refinement by a feedforward network, we obtain:

[0103] ;

[0104] ;

[0105] Finally, the final query representations are aggregated and the reconstructed vectors are obtained through the two readout heads. And the internal confidence level c. And the final reconstructed state is obtained through gating fusion:

[0106] ;

[0107] And calculate the reconstruction loss:

[0108] ;

[0109] in, To reconstruct the vector, To correspond to the true value features, , This is the set of samples with missing modalities.

[0110] An adaptive fusion module for uncertainty perception enables cross-modal feature fusion of EEG and eye-tracking signals, as detailed below:

[0111] First, construct a random prior network. With prediction networks A random prior network is constructed, where p and q are the EEG and EM modes, respectively. The parameters are frozen, randomly initialized, and remain unchanged. The parameters are used in the training. Therefore, the input q-mode feature vector... Calculate channel-by-channel uncertainty:

[0112] ;

[0113] Subsequently, Channel-by-channel normalization yields .

[0114] Next, the uncertainty estimate is applied to the modal features and weighted features are generated by the fusion unit, and the element-wise weighting is calculated:

[0115] ;

[0116] Will After cascading, the input to the fusion processor yields weighted features:

[0117] ;

[0118] The MLP consists of two layers of linear mapping and nonlinear activation.

[0119] Then, taking the two weighted modes obtained above, sum the features of the two modes element by element:

[0120] ;

[0121] Perform feature cascading:

[0122] ;

[0123] Perform nonlinear mapping and set channel gating:

[0124] ;

[0125] in, Represents the sigmoid activation function. This indicates the gating weight.

[0126] Computational fusion mode:

[0127] ;

[0128] in, , Multilayer perceptron .

[0129] The FSL module uses a prototype-based metric learning method to complete sentiment recognition and classification, as detailed below:

[0130] First, the support set based on the target domain category C. The feature calculation of the sample class prototype vector:

[0131] ;

[0132] in, For sample weights, For function mapping, Indicates sample Its characteristics.

[0133] Then, the samples are classified, for the query samples. Calculate the probability that it belongs to each category:

[0134] ;

[0135] in, This represents the distance metric, i.e., in the context of query features. With prototype The Euclidean distance between them is calculated. This is a temperature parameter used to adjust the sharpness of the distribution.

[0136] The cross-entropy loss function based on temperature-scaled prototype probability is used as the classification loss function, defined as follows:

[0137] ;

[0138] Where Q is the total number of categories, and represents the true category of the sample, which is determined by the temperature scaling metric mentioned above.

[0139] This embodiment also includes a domain alignment module, which minimizes the distribution differences between domains during model training, as detailed below:

[0140] In the domain alignment module, the maximum average difference loss function is used to reduce the inter-domain differences in multi-level interaction modal information, defined as follows:

[0141] ;

[0142] Here, represents the features from the source domain that are in the common feature space, and represents the features obtained from the target domain. These features are mapped to a reproducing kernel Hilbert space via a mapping function.

[0143] To increase the alignment weights of deep features in the loss function, an exponential decay strategy is introduced for hierarchical weight allocation:

[0144] ;

[0145] in, This represents the weight of the feature at the k-th layer.

[0146] The final total domain alignment loss is the sum of the layer losses:

[0147] ;

[0148] in, It is a scheduling function that is monotonically non-decreasing as the training progresses. This indicates the total number of floors.

[0149] By combining the reconstruction loss function, cross-entropy loss function, and maximum mean difference loss function, a joint loss function is obtained. This joint optimization model reduces the loss value, resulting in a high-performance multimodal few-sample sentiment recognition solution that can address the modality missing problem, as follows:

[0150] ;

[0151] in, and This is used to balance these losses.

[0152] S3 uses EEG data and eye-tracking data to train a multimodal few-shot emotion recognition model, resulting in a well-trained multimodal few-shot emotion recognition model.

[0153] See Figure 7 As shown, the training process in this embodiment adopts a phased course training mode, which completes the training by enabling or freezing different modules and loss terms in stages, including the following stages:

[0154] W1: Uncertainty estimation warm-up stage: Only the prediction branch of the random prior network is updated, while other parts are frozen, so that the dimension-wise uncertainty of the output under the condition of "opposite modal input" is stably converged, forming a confidence map that can be used for weighting and gating.

[0155] W2: During the reconstruction warm-up phase, under the condition of missing modes in the batch simulation, MMR is enabled to complete the reconstruction supervision by using the present modes as conditions, including retrieval priors, conditional cross-attention, and gating fusion, thereby optimizing the reconstruction loss.

[0156] W3: The overall union phase, which executes the complete forward pass in the aforementioned fixed order.

[0157] S4 uses a trained multimodal few-shot emotion recognition model for emotion recognition.

[0158] In summary, see Figure 8As shown, this invention provides a multimodal few-shot emotion recognition method for modality missing features. A deep neural network containing specific modules is constructed. Source domain data and target domain data, including EEG and eye-tracking data pairs, are input into the network model. When a modality is detected to be missing in the current sample, a missing modality reconstruction (MMR) module generates a substitute representation of the missing modality based on the present modality. An uncertainty-aware adaptive fusion module (UCMI) dynamically modulates single-modality weighting and cross-modal fusion to obtain a fused representation. A domain alignment constraint is then applied to the fused representation using a DAM layer to reduce the distribution difference between the source and target domains. Finally, a few-shot classification (FSL) module uses class prototype-based metric learning to determine the emotion category of the fused representation and outputs the recognition result. The model is trained by simulating modality missing features to improve robustness to real-world missing scenarios. The loss value for emotion recognition is calculated by combining cross-entropy loss, distribution alignment loss, and reconstruction loss. During training, model parameters are continuously optimized to reduce the total loss, resulting in a stable network model. Finally, the model is used to solve the emotion recognition problem in the case of modality missing.

[0159] like Figure 9 As shown, the present invention also discloses a multimodal few-shot emotion recognition device for modality-deficient data, comprising:

[0160] The EEG data and eye-tracking data acquisition module 901 is used to acquire EEG data and eye-tracking data;

[0161] The multimodal few-shot sentiment recognition model building module 902 is used to build a multimodal few-shot sentiment recognition model that includes a feature extraction module, a missing modality reconstruction module, an adaptive fusion module for uncertainty perception, and a few-shot classification module.

[0162] The feature extraction module extracts features from EEG data to obtain EEG data depth features; and extracts features from eye movement data to obtain eye movement data depth features.

[0163] The missing modality reconstruction module performs modality missing detection on EEG data or eye movement data. If eye movement data is missing, it generates a substitute representation based on the EEG data and uses the substitute features as the depth features of the eye movement data.

[0164] The adaptive fusion module for uncertainty perception fuses the deep features of EEG data and the deep features of eye-tracking data to obtain fused features;

[0165] The small sample classification module performs emotion category discrimination on the fused features and outputs the identified emotion category;

[0166] The multimodal few-shot emotion recognition model training module 903 is used to train the multimodal few-shot emotion recognition model using EEG data and eye-tracking data to obtain a trained multimodal few-shot emotion recognition model.

[0167] The emotion recognition module 904 is used to perform emotion recognition using a trained multimodal few-shot emotion recognition model.

[0168] The specific implementation of the multimodal few-shot emotion recognition device for modality missing is the same as that of the multimodal few-shot emotion recognition method for modality missing, and will not be described again in this embodiment.

[0169] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A multi-modal small sample emotion recognition method for missing modalities, characterized in that, The method comprises the following steps: S1, acquiring electroencephalogram data and eye movement data; S2, constructing a multi-modal small sample emotion recognition model comprising a feature extraction module, a missing modality reconstruction module, an uncertainty-aware adaptive fusion module, and a small sample classification module; The feature extraction module extracts features from the electroencephalogram data to obtain electroencephalogram data deep features and extracts features from the eye movement data to obtain eye movement data deep features; The missing modality reconstruction module detects modality missing of the electroencephalogram data or the eye movement data, generates a substitute feature based on the electroencephalogram data if the eye movement data is missing, takes the substitute feature as the eye movement data deep feature, generates a substitute feature based on the eye movement data if the electroencephalogram data is missing, and takes the substitute feature as the electroencephalogram data deep feature; The uncertainty-aware adaptive fusion module fuses the electroencephalogram data deep features and the eye movement data deep features to obtain fused features; The small sample classification module discriminates the fused features to output a recognized emotion category; S3, training the multi-modal small sample emotion recognition model using the electroencephalogram data and the eye movement data to obtain a trained multi-modal small sample emotion recognition model; S4, performing emotion recognition using the trained multi-modal small sample emotion recognition model; The missing modality reconstruction module is implemented as follows: The presence of the electroencephalogram data or the eye movement data is detected for modality missing, and any missing modality data of the electroencephalogram data or the eye movement data is recorded as missing modality data, and the data of the other modality is recorded as present modality data; accordingly, the features extracted from the present modality data by the feature extraction module are recorded as present modality deep features; Retrieving K neighbor sample index set in feature library with on-site modal depth feature as query ; the feature library is composed of paired sample features of the collected electroencephalogram data and eye movement data in advance The similarity of the present modality deep features and K neighbor samples is calculated, and the normalized weight of each neighbor sample is calculated according to the similarity; The prior vector of the missing modality is calculated according to the normalized weight; The feature vector of the on-site modal deep feature And the prior vector of the missing modal Two token sequences are obtained by linear projection and cutting into d-dimensional source tokens respectively, and are concatenated to obtain a concatenated token sequence ; Introduce a learnable query matrix as the query of multi-head attention to cascade token sequences As keys and values, interact through multi-layer conditional blocks containing multi-head cross-attention and feed-forward networks to obtain refined final query representations; The final query representation is aggregated and processed by two readout heads to obtain the reconstructed vector with the internal confidence c; The substitute feature of the missing modality is obtained by gating fusion; The uncertainty-aware adaptive fusion module is implemented as follows: The input q-modality features are output to a p-modality space and the channel-wise uncertainty is calculated; normalization of channel uncertainty per channel to get uncertainty graph, dimension and alignment; The uncertainty map is applied to the modality features and the weighted features are generated by a fusioner; The element-wise weighting is calculated; The weighted features of the two modalities are element-wise pre-fused and summed; The fusion modality is calculated by channel gating.

2. The multi-modal small sample emotion recognition method for missing modalities according to claim 1, characterized in that, The feature extraction module is implemented as follows: The electroencephalogram data is input into a convolutional feature extraction network comprising a residual propagation path and an attention mechanism; the convolutional feature extraction network performs residual aggregation between a main branch and a shortcut branch, and applies attention to the features in the channel and spatial dimensions for adaptive weighting; after the output of the convolutional feature extraction network is normalized and down-sampled, it is globally aggregated and linearly mapped to a unified dimension to obtain electroencephalogram modality deep features; The eye movement data is input into a densely connected time sequence convolution network structure; the time sequence convolution network structure realizes feature multiplexing and gradient efficient transmission through cross-layer connection within the same stage; the time sequence convolution network structure is provided with a transition unit between stages to complete channel compression and time scale downsampling; the output of the time sequence convolution network structure is globally aggregated and linearly mapped to a unified dimension, and an eye movement modality deep feature is output.

3. The multi-modal small sample emotion recognition method for missing modalities according to claim 1, characterized in that, The missing modality reconstruction module is specifically implemented as follows: The presence of the electroencephalogram data or the eye movement data is detected for modality missing detection, and any missing modality data of the electroencephalogram data or the eye movement data is recorded as missing modality data, and the data of the other modality is recorded as present modality data; accordingly, the feature extracted by the feature extraction module from the present modality data is recorded as present modality deep feature; Retrieving K neighbor sample index set in feature library with on-site modal depth feature as query ; the feature library is composed of paired sample features of the collected electroencephalogram data and eye movement data in advance The similarity between the present modality deep feature and K nearest neighbor samples is calculated, and the normalized weight of each nearest neighbor sample is calculated according to the similarity; The prior vector of the missing modality is calculated according to the normalized weight, and is expressed as: ； wherein, represents a prior vector for missing modalities; represents a set of indices of the k nearest neighbors in the presence modalities of the i-th nearest neighbor sample missing modalities features of the same sample pair; represents a normalization weight; The feature vector of the on-site modal deep feature And the prior vector of the missing modal Two token sequences are obtained by linear projection and cutting into d-dimensional source tokens respectively, and are concatenated to obtain a concatenated token sequence ; Introduce a learnable query matrix as the query of multi-head attention to cascade token sequences As keys and values, interact through multi-layer conditional blocks containing multi-head cross-attention and feed-forward networks to obtain refined final query representations; The final query representation is aggregated and processed by two readout heads to obtain the reconstructed vector with the internal confidence c; The gating fusion is performed to obtain the substitute feature of the missing modality, and is expressed as: ； wherein, represents a surrogate for the missing modality; represents a fusion weight; represents a preset minimum fusion weight; represents an internal confidence, ; represents taking the maximum; represents a reconstructed vector.

4. The multi-modal small sample emotion recognition method for missing modalities according to claim 1, characterized in that, The adaptive fusion module with uncertainty perception is specifically implemented as follows: The input q modality feature is output to the p modality space, and the channel-by-channel uncertainty is calculated, and is expressed as: ； wherein, represents per-channel uncertainty; represents a frozen random prior network; represents a trainable prediction network; represents input q-modal features; represents any modality of electroencephalographic data or eye movement data, p represents another modality other than q; represents the square of the L2 norm; To per-channel normalization yields an uncertainty map , dimensions are aligned with each other; The uncertainty map is applied to the modality feature, and the weighted feature is generated through a fusion device; The element-by-element weighting is calculated, and is expressed as: ； ； wherein, represents a weighted feature; represents a fusioner, consisting of two linear mappings and a nonlinear activation; represents a concatenation of and ; represents an element-wise multiplication; represents an element-wise weighted feature; The weighted features of the two modalities are element-by-element pre-fused and summed, and are expressed as: ； wherein, represents a pre-fusion vector; represents a weighted feature corresponding to the electroencephalogram data; represents a weighted feature corresponding to the eye movement data; The fusion modality is calculated through channel gating, and is expressed as: ； ； ； ； wherein, denotes a fusion modality; denotes layer normalization; denotes a sigmoid activation function, denotes a gating weight; denotes a gating bias parameter; denotes a channel gating vector; denotes a gain term; denotes a multi-layer perceptron; denotes concatenation; denotes a concatenated vector.

5. The multi-modal small sample emotion recognition method for missing modalities according to claim 1, characterized in that, The small sample classification module is specifically implemented as follows: the samples are divided into a support set and a query set; for each emotion category in the target domain, the fusion features of all samples in the support set are averaged to generate a prototype vector of the category; the distance between the fusion feature of a sample in the query set and the prototype of each emotion category is calculated; the distance is taken negatively and mapped to a category probability distribution through a temperature scaling softmax function.

6. The multi-modal small sample emotion recognition method for missing modalities according to claim 1, characterized in that, The joint loss function of the multi-modality small sample emotion recognition model during training includes a classification loss, a domain alignment loss and a reconstruction loss; the reconstruction loss supervises the output quality of the missing modality reconstruction module; the domain alignment loss is used to reduce the distribution difference between the source domain and the target domain in the feature space; the classification loss is used to measure the inconsistency between the emotion category prediction probability of the query sample and the true label; the joint loss function is expressed as: ； wherein, represents a joint loss function; and represents a weight parameter balancing the losses; represents a classification loss; represents a reconstruction loss; represents a domain alignment loss.

7. The multi-modal small sample emotion recognition method for missing modalities according to claim 6, characterized in that, The multi-modality small sample emotion recognition model further includes a domain alignment module, and the domain alignment loss is realized through the domain alignment module; the domain alignment module is configured to: calculate the distribution difference between the source domain sample and the target domain sample at multiple feature levels respectively; the distribution difference adopts a maximum mean difference measure, which is used to constrain the distribution consistency of the source domain and the target domain in the public feature space; different weights are assigned to the distribution difference items at different levels, and the weight of the deep layer of the network is higher than that of the shallow layer; the sum of the weighted distribution differences is taken as the domain alignment loss, and the proportion of the loss in the total loss is gradually increased in the training process.

8. The multi-modal small sample emotion recognition method for missing modalities according to claim 6, characterized in that, The reconstruction loss is expressed as: ； wherein, is a reconstructed vector, denotes a corresponding ground truth feature, denotes a hyperparameter, ; denotes a set of target modality missing samples obtained by sampling with missing setting during training; denotes a set size; denotes a square of L2 norm; denotes a cosine similarity.

9. The multi-modal small sample emotion recognition method for missing modalities according to claim 6, characterized in that, The classification loss adopts a cross-entropy loss function based on temperature scaling prototype probability.

10. A modal missing oriented multi-modal small sample sentiment recognition device, characterized in that, It includes the following: An electroencephalogram data and eye movement data acquisition module is configured to acquire electroencephalogram data and eye movement data; A multi-modal small sample emotion recognition model construction module is configured to construct a multi-modal small sample emotion recognition model including a feature extraction module, a missing modality reconstruction module, an uncertainty-aware adaptive fusion module, and a small sample classification module; The feature extraction module is configured to extract features from the electroencephalogram data to obtain electroencephalogram data deep features and extract features from the eye movement data to obtain eye movement data deep features; The missing modality reconstruction module is configured to detect missing modalities of the electroencephalogram data or the eye movement data, generate substitute features based on the electroencephalogram data if the eye movement data is missing, and use the substitute features as the eye movement data deep features, or generate substitute features based on the eye movement data if the electroencephalogram data is missing, and use the substitute features as the electroencephalogram data deep features; The uncertainty-aware adaptive fusion module is configured to fuse the electroencephalogram data deep features and the eye movement data deep features to obtain fused features; The small sample classification module is configured to determine the emotion category of the fused features and output the recognized emotion category. A multi-modal small sample emotion recognition model training module is configured to train the multi-modal small sample emotion recognition model using the electroencephalogram data and the eye movement data to obtain a trained multi-modal small sample emotion recognition model. An emotion recognition module is configured to use the trained multi-modal small sample emotion recognition model to recognize emotions. The missing modality reconstruction module is implemented as follows: The presence of the electroencephalogram data or the eye movement data is detected for missing modality detection, and any missing modality data of the electroencephalogram data or the eye movement data is recorded as missing modality data, and the data of the other modality is recorded as present modality data; accordingly, the features extracted by the feature extraction module from the present modality data are recorded as present modality deep features; Retrieving K neighbor sample index set in feature library with on-site modal depth feature as query ; the feature library is composed of paired sample features of the collected electroencephalogram data and eye movement data in advance The similarity between the present modality deep features and K nearest neighbor samples is calculated, and the normalized weight of each nearest neighbor sample is calculated based on the similarity; The prior vector of the missing modality is calculated based on the normalized weight; The feature vector of the on-site modal deep feature And the prior vector of the missing modal Two token sequences are obtained by linear projection and cutting into d-dimensional source tokens respectively, and are concatenated to obtain a concatenated token sequence ; Introducing a learnable query matrix as the query of multi-head attention, concatenating the token sequence As key and value, interacting through multi-layer conditional blocks containing multi-head cross-attention and feed-forward networks, resulting in refined final query representation; The final query representation is aggregated and processed by two readout heads to obtain the reconstructed vector with the internal confidence c; The substitute features of the missing modality are obtained through gated fusion; The uncertainty-aware adaptive fusion module is implemented as follows: The input q-modality features are output to the p-modality space and the channel-wise uncertainty is calculated; normalization of channel uncertainty per channel to get uncertainty graph, dimension and alignment; The uncertainty map is applied to the modality features and the weighted features are generated through a fusion device; The element-wise weighting is calculated; The weighted features of the two modalities are element-wise pre-fused and summed; The fused modality is calculated through channel gating.

Images