Multi-modal small sample emotion recognition method and device for modal missing

Abstract

The invention discloses a mode-missing-oriented multi-mode small sample emotion recognition method and device, and relates to the technical field of emotion computation.The method comprises the steps that S1, electroencephalogram data and eye movement data are obtained; s2, constructing a multi-modal small sample emotion recognition model comprising a feature extraction module, a missing modal reconstruction module, an uncertainty perception adaptive fusion module and a small sample classification module; s3, training the multi-modal small sample emotion recognition model by using the electroencephalogram data and the eye movement data to obtain a trained multi-modal small sample emotion recognition model; and S4, performing emotion recognition by using the trained multi-modal small sample emotion recognition model. According to the method, the multi-modal small sample emotion recognition model is constructed, and the technical effect of remarkably improving the accuracy, robustness and generalization ability of multi-modal emotion recognition in complex reality scenes such as modal deficiency, labeled sample scarcity and cross-subject distribution offset is achieved.

Description

Technical Field

[0001] This invention relates to the field of emotion computing technology, and in particular to a method and apparatus for multimodal few-sample emotion recognition in the face of modality loss. Background Technology

[0002] As research into the mechanisms of human emotion deepens, multimodal emotion recognition (MER) has become an important direction in emotion computing. By integrating multi-source physiological and behavioral signals, MER can more objectively characterize emotional states. Specifically, electroencephalogram (EEG) signals reflect neural dynamics with millisecond-level temporal resolution, while eye movement (EM) signals provide complementary cues from overt behavioral aspects such as fixation, saccades, and pupillary responses. The natural complementarity between these two in terms of temporal dynamics and behavioral representation makes EEG+EM a common combination for physiologically signal-driven MER.

[0003] Despite significant progress in MER (Modal Recognition and Modulation) technology, it still faces multiple constraints in practical applications. First, physiological data itself is non-stationary and exhibits significant individual differences. Combined with factors such as the acquisition environment, subject status, and temporal drift, distribution bias is easily introduced, leading to insufficient cross-subject generalization ability of the model. Second, the modal presence in actual acquisition is unstable. Factors such as poor electrode contact, motion and blink artifacts, occlusion, and changes in illumination can cause some modalities to degenerate or be missing, causing the signal-to-noise ratio to fluctuate over time. Many existing fusion strategies assume that all modalities are complete and equally reliable, lacking explicit characterization and transmission mechanisms for input uncertainty. When missing or noisy conditions exist, the fusion process cannot adaptively suppress unreliable information, thus affecting the final discrimination performance. Therefore, how to perform uncertainty-aware fusion under conditions of modal missingness and improve the model's robustness to cross-subject and scene changes has become a pressing issue in the current MER field. Summary of the Invention

[0004] To address the above problems, this invention proposes a multimodal few-shot emotion recognition method and apparatus for modality missing. By constructing a multimodal few-shot emotion recognition model including a feature extraction module, a missing modality reconstruction module, an adaptive fusion module for uncertainty perception, and a few-shot classification module, and jointly utilizing modal information from EEG and eye movement, when modality missing is detected, an alternative representation of the missing modality is generated based on the present modality. In real-world scenarios with incomplete modalities and scarce annotations, this significantly improves recognition accuracy and robustness.

[0005] On the one hand, a multimodal few-sample sentiment recognition method for modality-deficient data collection involves the following steps:

[0006] S1, acquire EEG data and eye movement data;

[0007] S2, construct a multimodal few-shot sentiment recognition model including a feature extraction module, a missing modality reconstruction module, an adaptive fusion module for uncertainty perception, and a few-shot classification module;

[0008] The feature extraction module extracts features from EEG data to obtain EEG data depth features; and extracts features from eye movement data to obtain eye movement data depth features.

[0009] The missing modality reconstruction module performs modality missing detection on EEG data or eye movement data. If eye movement data is missing, it generates a substitute representation based on the EEG data and uses the substitute features as the depth features of the eye movement data.

[0010] The adaptive fusion module for uncertainty perception fuses the deep features of EEG data and the deep features of eye-tracking data to obtain fused features;

[0011] The small sample classification module performs emotion category discrimination on the fused features and outputs the identified emotion category;

[0012] S3, use EEG data and eye-tracking data to train the multimodal few-shot emotion recognition model to obtain the trained multimodal few-shot emotion recognition model;

[0013] S4 uses a trained multimodal few-shot emotion recognition model for emotion recognition.

[0014] Preferably, the feature extraction module is implemented as follows:

[0015] EEG data is input into a convolutional feature extraction network that includes residual propagation paths and attention mechanisms. The convolutional feature extraction network performs residual aggregation between the main branch and the shortcut branch, and applies attention to adaptively weight the features in the channel and spatial dimensions. The output of the convolutional feature extraction network is normalized and downsampled, then globally aggregated and linearly mapped to a unified dimension to obtain deep features of EEG modalities.

[0016] Eye-tracking data is input into a densely connected temporal convolutional network structure; the temporal convolutional network structure achieves feature reuse and efficient gradient transfer through cross-layer connections within the same stage; the temporal convolutional network structure sets transition units between stages to complete channel compression and temporal downsampling; the output of the temporal convolutional network structure is globally aggregated and linearly mapped to a unified dimension to output eye-tracking modality depth features.

[0017] Preferably, the missing mode reconstruction module is implemented as follows:

[0018] Modal missing data detection is performed on the presence status of EEG or eye-tracking data. Data missing in either EEG or eye-tracking data is recorded as missing modal data, and data in the other modality is recorded as present modal data. Correspondingly, the features extracted from the present modal data by the feature extraction module are recorded as present modal depth features.

[0019] Using the presence modality depth feature as the query, retrieve the K nearest neighbor sample index set from the feature database. The feature library consists of paired sample features of previously acquired EEG data and eye-tracking data.

[0020] Calculate the similarity between the depth features of the present modality and the K nearest neighbor samples, and calculate the normalized weight of each nearest neighbor sample based on the similarity.

[0021] The prior vector of the missing mode is calculated based on the normalized weights and is expressed as follows:

[0022] ;

[0023] in, The prior vector representing the missing mode; Represents the set of nearest neighbor sample indices In the context of the depth features of the present modality with the i-th nearest neighbor sample Missing modal features of paired samples; Indicates the normalized weights;

[0024] The feature vector of the field modality depth features and prior vectors of missing modes By applying linear projection and dividing each token into d-dimensional source tokens, two token sequences are obtained. These sequences are then concatenated to obtain a concatenated token sequence. ;

[0025] Introducing a learnable query matrix as the query for multi-head attention, concatenating the token sequence... As keys and values, they interact through multi-layer conditional blocks containing multi-head cross-attention and feedforward networks to obtain a refined final query representation;

[0026] The final query representations are aggregated and processed through two readout heads to obtain the reconstructed vectors. With internal confidence level c;

[0027] Gated fusion yields alternative representations of the missing modes, expressed as:

[0028] ;

[0029] in, Alternative representations of missing modes; Indicates the fusion weights; This represents the preset minimum fusion weight; Indicates the internal confidence level. ; This indicates taking the maximum value; This represents the reconstructed vector.

[0030] Preferably, the uncertainty-aware adaptive fusion module is implemented as follows:

[0031] The input q-mode features are output to the p-mode space, and the channel-by-channel uncertainty is calculated, expressed as:

[0032] ;

[0033] in, This indicates uncertainty on a per-channel basis; Represents a frozen random prior network; This represents a trainable prediction network; This represents the q-mode features of the input; p represents any modality of EEG or eye-tracking data. Another modality besides; Represents the square of the L2 norm;

[0034] right Channel-by-channel normalization yields the uncertainty graph Dimensions and Alignment;

[0035] Applying the uncertainty map to the modal features and generating weighted features via a fusion processor is represented as follows:

[0036] The element-wise weighted average is calculated as follows:

[0037] ;

[0038] ;

[0039] in, Indicates weighted features; The fusion unit consists of two layers of linear mappings and nonlinear activations. Indicates will and cascade; This indicates element-wise multiplication; This represents the feature after element-wise weighting;

[0040] The weighted features of the two modalities are pre-fused and summed element by element, as follows:

[0041] ;

[0042] in, Represents the pre-fusion vector; This represents the weighted features corresponding to the EEG data; This represents the weighted features corresponding to the eye-tracking data;

[0043] The fused modes are calculated using channel gating and are represented as follows:

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] in, Indicates fused modes; Representation layer normalization; This represents the sigmoid activation function. Indicates the gating weight; Indicates the gating bias parameter; Represents the channel gating vector; Indicates the gain term; Represents a multilayer perceptron; Indicates cascading; This represents the concatenated vector.

[0049] Preferably, the small sample classification module is implemented as follows: dividing the samples into a support set and a query set; for each sentiment category in the target domain, averaging the fusion features of all samples in its support set to generate a prototype vector for that category; calculating the distance between the fusion features of the samples in the query set and the prototypes of each sentiment category; and negatively mapping the distance to a category probability distribution using a temperature-scaled softmax function.

[0050] Preferably, the joint loss function of the multimodal few-shot sentiment recognition model during training includes classification loss, domain alignment loss, and reconstruction loss; the reconstruction loss supervises the output quality of the missing modality reconstruction module; the domain alignment loss is used to reduce the distribution difference between the source domain and the target domain in the feature space; the classification loss is used to measure the inconsistency between the predicted sentiment category probability of the query sample and the true label; the joint loss function is expressed as:

[0051] ;

[0052] in, Represents the joint loss function; and The weighting parameters represent the balancing loss; Indicates classification loss; Indicates the losses incurred during reconstruction; Indicates domain alignment loss.

[0053] Preferably, the multimodal few-shot emotion recognition model further includes a domain alignment module, which implements the domain alignment loss. The domain alignment module is configured to: calculate the distribution difference between source domain samples and target domain samples at multiple feature levels; the distribution difference is measured by the maximum mean difference to constrain the distribution consistency of the source domain and the target domain in the common feature space; assign different weights to the distribution difference terms at each level, with the weights of deeper layers in the network being higher than those of shallower layers; use the sum of weighted distribution differences as the domain alignment loss, and gradually increase the proportion of this loss in the total loss during training.

[0054] Preferably, the reconstruction loss is expressed as:

[0055] ;

[0056] in, To reconstruct the vector, Indicates the corresponding truth features, Indicates hyperparameters, ; This represents the set of missing samples for the target modality obtained by sampling according to the missing parameters during training. Indicates the size of the set; Represents the square of the L2 norm; This represents the cosine similarity.

[0057] Preferably, the classification loss employs a cross-entropy loss function based on temperature-scaled prototype probabilities.

[0058] On the other hand, multimodal few-shot emotion recognition devices for modality-deficient applications include the following:

[0059] The EEG and eye-tracking data acquisition module is used to acquire EEG and eye-tracking data;

[0060] A multimodal few-shot sentiment recognition model building module is used to construct a multimodal few-shot sentiment recognition model that includes a feature extraction module, a missing modality reconstruction module, an adaptive fusion module for uncertainty perception, and a few-shot classification module.

[0061] The feature extraction module extracts features from EEG data to obtain EEG data depth features; and extracts features from eye movement data to obtain eye movement data depth features.

[0062] The missing modality reconstruction module performs modality missing detection on EEG data or eye movement data. If eye movement data is missing, it generates a substitute representation based on the EEG data and uses the substitute features as the depth features of the eye movement data.

[0063] The adaptive fusion module for uncertainty perception fuses the deep features of EEG data and the deep features of eye-tracking data to obtain fused features;

[0064] The small sample classification module performs emotion category discrimination on the fused features and outputs the identified emotion category;

[0065] The multimodal few-shot emotion recognition model training module is used to train the multimodal few-shot emotion recognition model using EEG data and eye-tracking data to obtain a trained multimodal few-shot emotion recognition model.

[0066] The emotion recognition module is used to perform emotion recognition using a pre-trained multimodal few-shot emotion recognition model.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] (1) When a missing modality is detected, the missing modality reconstruction module of the present invention uses the present modality as the query, combines the nearest neighbor retrieval prior and conditional cross attention to generate alternative representations, and through confidence gating and prior adaptive fusion, it is significantly better than traditional methods such as zero filling, avoiding cross-modal semantic mismatch and feature distribution shift.

[0069] (2) The uncertainty-aware adaptive fusion module of the present invention outputs dimension-wise uncertainty through a random prior network, forms a confidence map through monotonic mapping, performs element-wise weighting on single-modal features, and then obtains a unified representation through residual cross-modal fusion; this mechanism automatically suppresses unreliable information and improves fusion stability when noise increases, reconstruction participation is affected, or the quality of one modality is unstable.

[0070] (3) This invention combines EEG and eye movement modal information to generate alternative representations of missing modalities when modal absence is detected, based on the present modality. Subsequently, single-modal features are weighted by confidence based on uncertainty estimation, and a unified representation is obtained through cross-modal residual fusion. Under small sample settings, prototype metric learning is used to complete the classification, and a domain alignment strategy is introduced to alleviate cross-individual distribution differences, thereby achieving effective recognition of complex emotional states. In real-world scenarios with incomplete modalities and scarce annotations, the recognition accuracy and robustness are significantly improved, overcoming the shortcomings of existing multimodal emotion recognition in terms of insufficient robustness under modal absence, cross-individual bias, and small sample conditions. Attached Figure Description

[0071] The present invention will now be described in further detail with reference to the accompanying drawings;

[0072] Figure 1 This is a flowchart of a multimodal few-sample emotion recognition method for modality-deficient data according to an embodiment of the present invention;

[0073] Figure 2 This is a schematic diagram illustrating the steps of the multimodal few-sample emotion recognition method for modality-deficient methods according to an embodiment of the present invention.

[0074] Figure 3 This is a schematic diagram of the multimodal few-shot emotion recognition model framework for a multimodal few-shot emotion recognition method for modality missing data, as described in an embodiment of the present invention.

[0075] Figure 4 This is a schematic diagram of the EEG feature extraction network for a multimodal few-sample emotion recognition method for modality-deficient conditions according to an embodiment of the present invention.

[0076] Figure 5 This is a schematic diagram of the eye-tracking feature extraction network for a multimodal few-sample emotion recognition method for modality-deficient conditions according to an embodiment of the present invention.

[0077] Figure 6 This is a schematic diagram of the missing modality reconstruction module in the multimodal few-sample emotion recognition method for modality loss according to an embodiment of the present invention;

[0078] Figure 7 This is a schematic diagram illustrating the training process of the multimodal few-shot emotion recognition method for modality-deficient methods according to an embodiment of the present invention.

[0079] Figure 8 This is a schematic diagram of the model structure of the multimodal few-sample emotion recognition method for modality-deficient methods according to an embodiment of the present invention;

[0080] Figure 9 This is a structural block diagram of a multimodal few-sample emotion recognition device for modality-deficient applications according to an embodiment of the present invention. Detailed Implementation

[0081] The present invention will be further described below through specific embodiments.

[0082] like Figure 1 and Figure 2 As shown, the specific steps of the multimodal few-shot emotion recognition method for modality missing features are as follows:

[0083] S1 acquires EEG and eye-tracking data.

[0084] S2, construct a multimodal few-shot emotion recognition model including a feature extraction module, a missing modality reconstruction module, an adaptive fusion module for uncertainty perception, and a few-shot classification module.

[0085] Multimodal few-shot emotion recognition models, such as Figure 3 As shown. (See also...) Figure 4 and Figure 5 As shown, the feature extraction module includes an EEG feature extraction network and an eye-tracking feature extraction network, which are used to extract the depth features of EEG data and the depth features of eye-tracking data, respectively. Specifically, it includes:

[0086] EEG data is input into a convolutional feature extraction network with residual propagation paths and attention mechanisms to obtain deep features of the EEG data:

[0087] ;

[0088] Eye-tracking data is input into a densely connected temporal convolutional network structure to obtain the depth features of the eye-tracking data:

[0089] ;

[0090] If a missing mode exists, proceed to the Missing Mode Reconstruction (MMR) module to reconstruct modal features; otherwise, skip this module.

[0091] See the missing mode reconstruction module. Figure 6 As shown, the details are as follows:

[0092] First, the depth feature vector extracted from the presence modality is denoted as... ,by To retrieve the K nearest neighbor samples from the feature database, based on their cosine similarity to the query. Calculate the weights:

[0093] ;

[0094] in, The source modal features of the i-th sample in the feature library, For To retrieve the index set of K nearest neighbor samples retrieved from the database based on similarity, This is the temperature coefficient.

[0095] Therefore, we can obtain the retrieval prior for the target modality:

[0096] ;

[0097] in, To and Modal features of paired samples.

[0098] Next, the feature vectors of the present modes will be... and the retrieved prior vector Projecting and segmenting into d-dimensional source tokens, that is... and And cascade them:

[0099] ;

[0100] Subsequently, L-layer conditional blocks are stacked, and conditional information is read from T within each layer using multi-head cross-attention:

[0101]

[0102] And after refinement by a feedforward network, we obtain:

[0103] ;

[0104] ;

[0105] Finally, the final query representations are aggregated and the reconstructed vectors are obtained through the two readout heads. And the internal confidence level c. And the final reconstructed state is obtained through gating fusion:

[0106] ;

[0107] And calculate the reconstruction loss:

[0108] ;

[0109] in, To reconstruct the vector, To correspond to the true value features, , This is the set of samples with missing modalities.

[0110] An adaptive fusion module for uncertainty perception enables cross-modal feature fusion of EEG and eye-tracking signals, as detailed below:

[0111] First, construct a random prior network. With prediction networks A random prior network is constructed, where p and q are the EEG and EM modes, respectively. The parameters are frozen, randomly initialized, and remain unchanged. The parameters are used in the training. Therefore, the input q-mode feature vector... Calculate channel-by-channel uncertainty:

[0112] ;

[0113] Subsequently, Channel-by-channel normalization yields .

[0114] Next, the uncertainty estimate is applied to the modal features and weighted features are generated by the fusion unit, and the element-wise weighting is calculated:

[0115] ;

[0116] Will After cascading, the input to the fusion processor yields weighted features:

[0117] ;

[0118] The MLP consists of two layers of linear mapping and nonlinear activation.

[0119] Then, taking the two weighted modes obtained above, sum the features of the two modes element by element:

[0120] ;

[0121] Perform feature cascading:

[0122] ;

[0123] Perform nonlinear mapping and set channel gating:

[0124] ;

[0125] in, Represents the sigmoid activation function. This indicates the gating weight.

[0126] Computational fusion mode:

[0127] ;

[0128] in, , Multilayer perceptron .

[0129] The FSL module uses a prototype-based metric learning method to complete sentiment recognition and classification, as detailed below:

[0130] First, the support set based on the target domain category C. The feature calculation of the sample class prototype vector:

[0131] ;

[0132] in, For sample weights, For function mapping, Indicates sample Its characteristics.

[0133] Then, the samples are classified, for the query samples. Calculate the probability that it belongs to each category:

[0134] ;

[0135] in, This represents the distance metric, i.e., in the context of query features. With prototype The Euclidean distance between them; This is a temperature parameter used to adjust the sharpness of the distribution.

[0136] The cross-entropy loss function based on temperature-scaled prototype probability is used as the classification loss function, defined as follows:

[0137] ;

[0138] Where Q is the total number of categories, and represents the true category of the sample, which is determined by the temperature scaling metric mentioned above.

[0139] This embodiment also includes a domain alignment module, which minimizes the distribution differences between domains during model training, as detailed below:

[0140] In the domain alignment module, the maximum average difference loss function is used to reduce the inter-domain differences in multi-level interaction modal information, defined as follows:

[0141] ;

[0142] Here, represents the features from the source domain that are in the common feature space, and represents the features obtained from the target domain. These features are mapped to a reproducing kernel Hilbert space via a mapping function.

[0143] To increase the alignment weights of deep features in the loss function, an exponential decay strategy is introduced for hierarchical weight allocation:

[0144] ;

[0145] in, This represents the weight of the feature at the k-th layer.

[0146] The final total domain alignment loss is the sum of the losses for each layer:

[0147] ;

[0148] in, It is a scheduling function that is monotonically non-decreasing as the training progresses. This indicates the total number of floors.

[0149] By combining the reconstruction loss function, cross-entropy loss function, and maximum mean difference loss function, a joint loss function is obtained. This joint optimization model reduces the loss value, resulting in a high-performance multimodal few-sample sentiment recognition solution that can address the modality missing problem, as follows:

[0150] ;

[0151] in, and This is used to balance these losses.

[0152] S3 uses EEG data and eye-tracking data to train a multimodal few-shot emotion recognition model, resulting in a well-trained multimodal few-shot emotion recognition model.

[0153] See Figure 7 As shown, the training process in this embodiment adopts a phased course training mode, which completes the training by enabling or freezing different modules and loss terms in stages, including the following stages:

[0154] W1: Uncertainty estimation warm-up stage: Only the prediction branch of the random prior network is updated, while other parts are frozen, so that the dimension-wise uncertainty of the output under the condition of "opposite modal input" is stably converged, forming a confidence map that can be used for weighting and gating.

[0155] W2: During the reconstruction warm-up phase, under the condition of missing modes in the batch simulation, MMR is enabled to complete the reconstruction supervision by using the present modes as conditions, including retrieval priors, conditional cross-attention, and gating fusion, thereby optimizing the reconstruction loss.

[0156] W3: The overall union phase, which executes the complete forward pass in the aforementioned fixed order.

[0157] S4 uses a trained multimodal few-shot emotion recognition model for emotion recognition.

[0158] In summary, see Figure 8As shown, this invention provides a multimodal few-shot emotion recognition method for modality missing features. A deep neural network containing specific modules is constructed. Source domain data and target domain data, including EEG and eye-tracking data pairs, are input into the network model. When a modality is detected to be missing in the current sample, a missing modality reconstruction (MMR) module generates a substitute representation of the missing modality based on the present modality. An uncertainty-aware adaptive fusion module (UCMI) dynamically modulates single-modality weighting and cross-modal fusion to obtain a fused representation. A domain alignment constraint is then applied to the fused representation using a DAM layer to reduce the distribution difference between the source and target domains. Finally, a few-shot classification (FSL) module uses class prototype-based metric learning to determine the emotion category of the fused representation and outputs the recognition result. The model is trained by simulating modality missing features to improve robustness to real-world missing scenarios. The loss value for emotion recognition is calculated by combining cross-entropy loss, distribution alignment loss, and reconstruction loss. During training, model parameters are continuously optimized to reduce the total loss, resulting in a stable network model. Finally, the model is used to solve the emotion recognition problem in the case of modality missing.

[0159] like Figure 9 As shown, the present invention also discloses a multimodal few-shot emotion recognition device for modality-deficient data, comprising:

[0160] The EEG data and eye-tracking data acquisition module 901 is used to acquire EEG data and eye-tracking data;

[0161] The multimodal few-shot sentiment recognition model building module 902 is used to build a multimodal few-shot sentiment recognition model that includes a feature extraction module, a missing modality reconstruction module, an adaptive fusion module for uncertainty perception, and a few-shot classification module.

[0162] The feature extraction module extracts features from EEG data to obtain EEG data depth features; and extracts features from eye movement data to obtain eye movement data depth features.

[0163] The missing modality reconstruction module performs modality missing detection on EEG data or eye movement data. If eye movement data is missing, it generates a substitute representation based on the EEG data and uses the substitute features as the depth features of the eye movement data.

[0164] The adaptive fusion module for uncertainty perception fuses the deep features of EEG data and the deep features of eye-tracking data to obtain fused features;

[0165] The small sample classification module performs emotion category discrimination on the fused features and outputs the identified emotion category;

[0166] The multimodal few-shot emotion recognition model training module 903 is used to train the multimodal few-shot emotion recognition model using EEG data and eye-tracking data to obtain a trained multimodal few-shot emotion recognition model.

[0167] The emotion recognition module 904 is used to perform emotion recognition using a trained multimodal few-shot emotion recognition model.

[0168] The specific implementation of the multimodal few-shot emotion recognition device for modality missing is the same as that of the multimodal few-shot emotion recognition method for modality missing, and will not be described again in this embodiment.

[0169] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A multimodal few-sample emotion recognition method for modality-deficient scenarios, characterized in that, Includes the following steps: S1, acquire EEG data and eye movement data; S2, construct a multimodal few-shot sentiment recognition model including a feature extraction module, a missing modality reconstruction module, an adaptive fusion module for uncertainty perception, and a few-shot classification module; The feature extraction module extracts features from EEG data to obtain EEG data depth features; and extracts features from eye movement data to obtain eye movement data depth features. The missing modality reconstruction module performs modality missing detection on EEG data or eye movement data. If eye movement data is missing, it generates a substitute representation based on the EEG data and uses the substitute features as the depth features of the eye movement data. The adaptive fusion module for uncertainty perception fuses the deep features of EEG data and the deep features of eye-tracking data to obtain fused features; The small sample classification module performs emotion category discrimination on the fused features and outputs the identified emotion category; S3, use EEG data and eye-tracking data to train the multimodal few-shot emotion recognition model to obtain the trained multimodal few-shot emotion recognition model; S4 uses a trained multimodal few-shot emotion recognition model for emotion recognition.

2. The multimodal few-sample emotion recognition method for modality-deficient data as described in claim 1, characterized in that, The feature extraction module is implemented as follows: EEG data is input into a convolutional feature extraction network that includes residual propagation paths and attention mechanisms. The convolutional feature extraction network performs residual aggregation between the main branch and the shortcut branch, and applies attention to adaptively weight the features in the channel and spatial dimensions. The output of the convolutional feature extraction network is normalized and downsampled, then globally aggregated and linearly mapped to a unified dimension to obtain deep features of EEG modalities. Eye-tracking data is input into a densely connected temporal convolutional network structure; the temporal convolutional network structure achieves feature reuse and efficient gradient transfer through cross-layer connections within the same stage; the temporal convolutional network structure sets transition units between stages to complete channel compression and temporal downsampling; the output of the temporal convolutional network structure is globally aggregated and linearly mapped to a unified dimension to output eye-tracking modality depth features.

3. The multimodal few-sample emotion recognition method for modality-deficient data as described in claim 1, characterized in that, The missing mode reconstruction module is implemented as follows: Modal missing data detection is performed on the presence status of EEG or eye-tracking data. Data missing in either EEG or eye-tracking data is recorded as missing modal data, and data in the other modality is recorded as present modal data. Correspondingly, the features extracted from the present modal data by the feature extraction module are recorded as present modal depth features. Using the presence modality depth feature as the query, retrieve the K nearest neighbor sample index set from the feature database. The feature library consists of paired sample features of previously acquired EEG data and eye-tracking data. Calculate the similarity between the depth features of the present modality and the K nearest neighbor samples, and calculate the normalized weight of each nearest neighbor sample based on the similarity. The prior vector of the missing mode is calculated based on the normalized weights and is expressed as follows: ； in, The prior vector representing the missing mode; Represents the set of nearest neighbor sample indices In the context of the depth features of the present modality with the i-th nearest neighbor sample Missing modal features of paired samples; Indicates the normalized weights; The feature vector of the field modality depth features and prior vectors of missing modes By applying linear projection and dividing each token into d-dimensional source tokens, two token sequences are obtained. These sequences are then concatenated to obtain a concatenated token sequence. ; Introducing a learnable query matrix as the query for multi-head attention, concatenating the token sequence... As keys and values, they interact through multi-layer conditional blocks containing multi-head cross-attention and feedforward networks to obtain a refined final query representation; The final query representations are aggregated and processed through two readout heads to obtain the reconstructed vectors. With internal confidence level c; Gated fusion yields alternative representations of the missing modes, expressed as: ； in, Alternative representations of missing modes; Indicates the fusion weights; This represents the preset minimum fusion weight; Indicates the internal confidence level. ; This indicates taking the maximum value; This represents the reconstructed vector.

4. The multimodal few-sample emotion recognition method for modality-deficient data as described in claim 1, characterized in that, The uncertainty-aware adaptive fusion module is implemented as follows: The input q-mode features are output to the p-mode space, and the channel-by-channel uncertainty is calculated, expressed as: ； in, This indicates uncertainty on a per-channel basis; Represents a frozen random prior network; This represents a trainable prediction network; This represents the q-mode features of the input; p represents any modality of EEG or eye-tracking data. Another modality besides; Represents the square of the L2 norm; right Channel-by-channel normalization yields the uncertainty graph Dimensions and Alignment; Applying the uncertainty map to the modal features and generating weighted features via a fusion processor is represented as follows: The element-wise weighted average is calculated as follows: ； ； in, Indicates weighted features; The fusion unit consists of two layers of linear mappings and nonlinear activations. Indicates will and cascade; This indicates element-wise multiplication; This represents the feature after element-wise weighting; The weighted features of the two modalities are pre-fused and summed element by element, as follows: ； in, Represents the pre-fusion vector; This represents the weighted features corresponding to the EEG data; This represents the weighted features corresponding to the eye-tracking data; The fused modes are calculated using channel gating and are represented as follows: ； ； ； ； in, Indicates fused modes; Representation layer normalization; This represents the sigmoid activation function. Indicates the gating weight; Indicates the gating bias parameter; Represents the channel gating vector; Indicates the gain term; Represents a multilayer perceptron; Indicates cascading; This represents the concatenated vector.

5. The multimodal few-sample emotion recognition method for modality-deficient data as described in claim 1, characterized in that, The small sample classification module is implemented as follows: the samples are divided into a support set and a query set; for each sentiment category in the target domain, the fusion features of all samples in its support set are averaged to generate the prototype vector of that category; the distance between the fusion features of the samples in the query set and the prototypes of each sentiment category is calculated; the distance is negative and mapped to a category probability distribution using a temperature-scaled softmax function.

6. The multimodal few-sample emotion recognition method for modality-deficient data as described in claim 1, characterized in that, The joint loss function of the multimodal few-sample sentiment recognition model during training includes classification loss, domain alignment loss, and reconstruction loss; the reconstruction loss supervises the output quality of the missing modality reconstruction module; the domain alignment loss is used to reduce the distribution difference between the source domain and the target domain in the feature space; the classification loss is used to measure the inconsistency between the predicted sentiment category probability of the query sample and the true label; the joint loss function is expressed as: ； in, Represents the joint loss function; and The weighting parameters represent the balancing loss; Indicates classification loss; Indicates the losses incurred during reconstruction; Indicates domain alignment loss.

7. The multimodal few-sample emotion recognition method for modality-deficient data as described in claim 6, characterized in that, The multimodal few-shot emotion recognition model also includes a domain alignment module, which implements the domain alignment loss. The domain alignment module is configured to: calculate the distribution difference between source domain samples and target domain samples at multiple feature levels; the distribution difference is measured by the maximum mean difference to constrain the distribution consistency of the source domain and the target domain in the common feature space; assign different weights to the distribution difference terms at each level, with the weights of deeper layers in the network being higher than those of shallower layers; use the sum of the weighted distribution differences as the domain alignment loss, and gradually increase the proportion of this loss in the total loss during training.

8. The multimodal few-sample emotion recognition method for modality-deficient conditions according to claim 6, characterized in that, The reconstruction loss is expressed as: ； in, To reconstruct the vector, Indicates the corresponding truth features, Indicates hyperparameters, ; This represents the set of missing samples for the target modality obtained by sampling according to the missing parameters during training. Indicates the size of the set; Represents the square of the L2 norm; This represents the cosine similarity.

9. The multimodal few-sample emotion recognition method for modality-deficient data as described in claim 6, characterized in that, The classification loss employs a cross-entropy loss function based on temperature-scaled prototype probabilities.

10. A multimodal few-sample emotion recognition device for modality-deficient conditions, characterized in that, Including the following: The EEG and eye-tracking data acquisition module is used to acquire EEG and eye-tracking data; A multimodal few-shot sentiment recognition model building module is used to construct a multimodal few-shot sentiment recognition model that includes a feature extraction module, a missing modality reconstruction module, an adaptive fusion module for uncertainty perception, and a few-shot classification module. The feature extraction module extracts features from EEG data to obtain EEG data depth features; and extracts features from eye movement data to obtain eye movement data depth features. The missing modality reconstruction module performs modality missing detection on EEG data or eye movement data. If eye movement data is missing, it generates a substitute representation based on the EEG data and uses the substitute features as the depth features of the eye movement data. The adaptive fusion module for uncertainty perception fuses the deep features of EEG data and the deep features of eye-tracking data to obtain fused features; The small sample classification module performs emotion category discrimination on the fused features and outputs the identified emotion category; The multimodal few-shot emotion recognition model training module is used to train the multimodal few-shot emotion recognition model using EEG data and eye-tracking data to obtain a trained multimodal few-shot emotion recognition model. The emotion recognition module is used to perform emotion recognition using a pre-trained multimodal few-shot emotion recognition model.

Images