A multi-modal signal fusion emotion recognition method

Abstract

The application discloses a multi-modal signal fusion emotion recognition method, relates to the technical field of emotion recognition, and comprises the following steps: acquiring electroencephalogram signals and peripheral physiological signals to form multi-modal signals, and performing preprocessing such as downsampling, baseline correction and band-pass filtering on the multi-modal signals; based on the preprocessed signals, differential entropy features and power spectral density features are extracted; the extracted features are mapped into a standardized spatial grid to generate a feature tensor with a unified structure; an adaptive weighting is performed on the mapped feature tensor by using a frequency band fusion attention mechanism, a frequency band weight is generated through global average pooling and a full connection layer; the weighted features are input into a parameterized super-matrix full connection layer, and feature compression and modeling are performed through matrix decomposition and reconstruction; a multi-task learning framework is adopted, and the classification results of emotion valence and arousal are simultaneously output based on the compressed features, and the multi-task cooperation is optimized through a shared feature layer and a dynamic loss weight.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of emotion recognition, and in particular to a multi-modal signal fusion emotion recognition method. BACKGROUND

[0002] Emotion recognition technology has broad application prospects in military medicine, human-computer interaction and other fields, but existing models have deep defects, which restrict their actual deployment effect. First, the core contradiction is that the multi-modal physiological signal fusion efficiency is low, which leads to insufficient model lightweight and limited recognition accuracy. For example, single-modal methods cannot fully capture the complexity of emotions, while multi-modal fusion causes excessive consumption of computing resources due to feature redundancy and large parameters. In addition, the scene constraints are severe, and the model needs to run in high and low temperature fluctuation, electromagnetic interference and high load real-time monitoring environments such as open battlefields, requiring low delay and stability. However, due to high computational complexity, existing technologies are difficult to maintain reliable performance on resource-constrained devices. At the same time, the subject's appeal is diverse and conflicting, researchers focus on explainability and lightweight design, and existing models cannot balance these needs, resulting in insufficient user satisfaction. Qualitative defects such as insufficient feature fusion, low utilization rate of key frequency band information and weak model generalization ability further exacerbate the imbalance between performance and resource utilization. These problems form a logical closed loop, highlighting the fundamental limitations of existing technologies, and innovative solutions are needed to break through the trade-off between lightweight and accuracy. SUMMARY

[0003] The present application provides a multi-modal signal fusion emotion recognition method, which solves the problems of low multi-modal signal fusion efficiency, high computational redundancy and insufficient model lightweight in existing emotion recognition technology, and achieves the technical effects of high-precision emotion recognition, low computational cost, and enhanced generalization ability and explainability.

[0004] The present application provides a multi-modal signal fusion emotion recognition method, which includes:

[0005] S1: Obtain electroencephalogram signals and peripheral physiological signals to form multi-modal signals, and perform preprocessing such as downsampling, baseline correction and bandpass filtering on the multi-modal signals. Feature extraction based on differential entropy and power spectral density is performed on the preprocessed signals;

[0006] S2: Map the extracted features to a standardized spatial grid according to the international 10-20 system to generate a feature tensor with a unified structure. Use a frequency band fusion attention mechanism to adaptively weight the mapped feature tensor, and generate frequency band weights through global average pooling and a fully connected layer;

[0007] S3: Input the weighted features into a parameterized super-matrix fully connected layer, and perform feature compression and modeling through matrix decomposition and reconstruction;

[0008] S4: Using a multi-task learning framework, the classification results of emotion valence and arousal are output based on compressed features, and the multi-task cooperation is optimized through shared feature layers and dynamic loss weights.

[0009] Further, the preprocessing uses a 0.5-second time window for signal slicing, and a band-pass filter of 4-45Hz and 0-75Hz frequency bands is applied according to the electroencephalogram signal and the peripheral physiological signal respectively, wherein the electroencephalogram signal uses a 4-45Hz filter, and the peripheral physiological signal sets the filter passband according to the type; the filter passband includes a 0-2.4Hz filter for skin conductance response, a 0-2.4Hz filter for respiratory rate, a 0.01-0.5Hz filter for blood volume pulse, a 0-0.2Hz filter for body temperature, a 0.5-10Hz filter for electrooculogram, and a 0.5-50Hz filter for electromyogram.

[0010] Further, the feature extraction includes: for each 0.5-second time window of the physiological signal, independently calculating the differential entropy feature and the power spectral density feature of each channel; wherein the differential entropy quantifies the overall activity level of the signal by calculating the signal energy within the time window, and the power spectral density analyzes the energy distribution of the signal in different frequency bands through Fourier transform; the extracted features are mapped to the standardized spatial grid of the international 10-20 system to generate a tensor with the shape of frequency band number x 8 x 9; during grid mapping, the feature values are filled into an 8-row 9-column matrix according to the electrode topology structure, and the missing channel positions are supplemented with zero values.

[0011] Further, the frequency band fusion attention mechanism includes: taking the tensor as input, wherein, is a real set, is the height of the input matrix, is the width of the matrix, is the number of frequency bands; for each frequency band , a global average pooling operation is performed to calculate the average value of the features of all spatial positions in the frequency band, and the frequency band statistics are obtained:

[0012] ,

[0013] wherein, is the frequency band statistics, is the feature value of the frequency band at position , and the statistics of all frequency bands form a vector ;

[0014] The frequency band statistics vector is processed sequentially through two fully connected layers. The first fully connected layer uses a linear rectified function and introduces a compression ratio to reduce the dimensionality. The second fully connected layer uses a logistic function to restore the original dimensionality and outputs the weight coefficients for each frequency band, with the weight coefficients ranging from 0 to 1. The generated weight coefficients are then multiplied element-wise with the original feature tensor to achieve recalibration of the frequency band features, thereby strengthening the key frequency bands related to emotion recognition and suppressing redundant or noisy frequency bands.

[0015] Furthermore, the multimodal signal includes: splicing features of EEG signals and at least one peripheral physiological signal to form a multimodal input matrix; adaptively weighting the feature tensor using a frequency band fusion attention mechanism, obtaining statistics for each frequency band through global average pooling, and generating frequency band weight coefficients using a fully connected layer; multiplying the weight coefficients element-wise with the feature tensor to achieve weighted fusion, thereby strengthening key frequency band features and suppressing redundant information.

[0016] Furthermore, the parameterized hypermatrix fully connected layer decomposes the weight matrix of the traditional fully connected layer into a hypermatrix and a structure matrix using nonnegative matrix factorization (NMF). The rank parameter of the hypermatrix is ​​dynamically adjusted during training: initially set to 4, and gradually increased to 8 based on gradient sparsity to balance parameter efficiency and feature representation capability. A kernel norm regularization term for the weight matrix is ​​introduced into the model loss function to suppress overfitting and improve generalization ability by constraining the low-rank property of the weight matrix. The weight matrix is ​​reconstructed through tensor product operations.

[0017] ,

[0018] in, The reconstructed weight matrix, For the hypermatrix of the first A small block, The first of the structure matrix A small block, For tensor product operations, For hyperparameters;

[0019] The formula for parameter compression is:

[0020] ,

[0021] in, To determine the total number of parameters in a parameterized hypermatrix fully connected layer, Let be the rank parameter of the hypermatrix. The parameters of the hypermatrix. The number of parameters in the structure matrix. To output feature dimensions, The input feature dimension.

[0022] Furthermore, the multi-task learning framework shares a feature extraction layer and uses a weighted loss function to simultaneously optimize the emotion valence and arousal classification tasks, with the loss weights dynamically adjusted according to the task difficulty.

[0023] The dynamic adjustment includes: calculating the loss weight in real time based on the validation set accuracy. The formula for calculating the loss weight for the emotion valence task is as follows:

[0024] ,

[0025] in, For the loss weight of the emotional valence task, and These represent the emotional valence and arousal task accuracy on the validation set during the current training cycle;

[0026] The formula for calculating the loss weight of the wakefulness task is:

[0027] ,

[0028] in, The loss weight for wakefulness tasks;

[0029] The total loss of the weighted loss function is:

[0030] ,

[0031] in, The total loss is the weighted loss function. and For cross-entropy loss, For the frequency band feature covariance constraint term, the weights are... The value is set to 0.01 to reduce feature redundancy.

[0032] Furthermore, the training process of the method employs a multi-stage optimization strategy, including:

[0033] An adaptive moment estimation optimizer with an initial learning rate of 0.001 is used, combined with early stopping to monitor the validation set loss. Training is terminated when the loss does not decrease for 20 consecutive rounds to prevent overfitting. Overfitting is suppressed by random dropout layers and batch normalization to improve the model's generalization ability. The weight coefficients in the multi-task loss function are dynamically adjusted based on the accuracy difference between the emotion valence task and the arousal task on the validation set. A phased training approach is adopted, first training the convolutional backbone network, and then jointly optimizing the fully connected layers and the multi-task classifier to improve feature extraction and classification performance in stages.

[0034] Furthermore, the method also includes ablation experiments to verify the synergistic effect of the band fusion attention mechanism, the parameterized hypermatrix fully connected layer, and the multi-task learning framework; the ablation experiments include the following steps:

[0035] The baseline model is built based on convolutional neural networks and traditional fully connected layers, without attention mechanisms and multi-task learning modules;

[0036] Multiple variant models were constructed, including adding a frequency band fusion attention mechanism to the baseline model, adding a parameterized hypermatrix fully connected layer, adding a multi-task learning framework, and adding module combinations.

[0037] All models were trained using the same hyperparameters and with 5-fold cross-validation to split the training and validation sets. The loss function, accuracy, and F1 score curves of the training and validation sets were tracked in real time to ensure model convergence stability. The accuracy and F1 score of the emotion valence and arousal classification tasks were recorded.

[0038] A two-tailed paired t-test was used to compare the performance differences between the baseline model and the variant model, calculate the change in accuracy and the change in F1 score, and use a probability value less than 0.05 as the statistical significance threshold; the optimal module combination was selected based on the significance results.

[0039] Furthermore, the method also includes using the SHAP visualization method to quantify the contribution of frequency band features, calculating the marginal contribution of each frequency band to sentiment classification prediction based on the Shapley value, and calculating the SHAP value by integrating the spatial location features in the input tensor using a gradient interpreter.

[0040] ,

[0041] in, For frequency band i in spatial location ( The SHAP value on ) For the mathematical expectation, on the background sample set Reference sample in Take the average. For the model prediction function, For interpolation points, For reference sample The j-th eigenvalue, For input samples The j-th eigenvalue corresponds to the spatial position in the input tensor. , For integration weighting terms, These are the weighting coefficients used for integral calculation;

[0042] By aggregating the SHAP values ​​of each frequency band across spatial locations, a global contribution ranking is obtained, using the following formula:

[0043] ,

[0044] in, Let be the average SHAP value of the i-th frequency band. The SHAP value for the i-th frequency band. This is the standardized spatial grid size for the international 10-20 system;

[0045] Feature selection is optimized based on contribution ranking, and frequency bands with contribution below a preset threshold are masked to reduce computational complexity and enhance model interpretability.

[0046] This application employs a parameterized hypermatrix multi-task multimodal emotion recognition model (PH-MTM) based on the fusion of electroencephalography (EEG) and peripheral physiological signals (PPS). The model's effectiveness has been validated on the DEAP and SEED datasets, specifically including:

[0047] Input Module (STIM): Maps the frequency domain features of multi-channel EEG and PPS onto the 8×9 spatial grid of the international 10-20 system to form a feature tensor with a unified structure.

[0048] Band Fusion Attention Mechanism (BFAM): Employs an improved SE attention module to adaptively weight each band, enhancing key emotion-related frequency bands.

[0049] Parametric Efficient Feature Modeling Unit (PEF): Introduces a parametric hypermatrix fully connected layer (PHMDense) to compress the number of parameters through non-negative matrix decomposition, thereby improving computational efficiency.

[0050] Multi-task learning framework: It shares a feature extraction layer, outputs classification results of sentiment valence and arousal, and optimizes task collaboration through dynamic loss weights.

[0051] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0052] By employing a parameterized hypermatrix multi-task multimodal deep learning model that fuses EEG and peripheral physiological signals, high-precision emotion recognition, low computational cost, enhanced model generalization ability, and improved interpretability were achieved. Attached Figure Description

[0053] Figure 1 This is a flowchart of an emotion recognition method based on multimodal signal fusion in an embodiment of the present invention;

[0054] Figure 2 This is a feature extraction diagram of an emotion recognition method based on multimodal signal fusion in an embodiment of the present invention;

[0055] Figure 3 This is a standardized spatial frequency input module diagram of an emotion recognition method based on multimodal signal fusion in an embodiment of the present invention;

[0056] Figure 4 This is a diagram illustrating the frequency band fusion attention mechanism module architecture of a multimodal signal fusion emotion recognition method according to an embodiment of the present invention.

[0057] Figure 5 This is a diagram of the efficient feature modeling unit architecture for a multimodal signal fusion emotion recognition method according to an embodiment of the present invention.

[0058] Figure 6 This is a comparison diagram of single-task learning and multi-task learning structures of an emotion recognition method based on multimodal signal fusion in an embodiment of the present invention.

[0059] Figure 7 This is a graph showing the loss function, accuracy, and F1 score of the last fold of the DEAP dataset for an emotion recognition method based on multimodal signal fusion, as described in an embodiment of the present invention.

[0060] Figure 8 This is a graph showing the loss function, accuracy, and F1 score of the last fold of the SEED dataset for an emotion recognition method based on multimodal signal fusion, as described in an embodiment of the present invention.

[0061] Figure 9 Figure (a) is a visualization of the classification results of a multimodal signal fusion emotion recognition method in an embodiment of the present invention. Figure (b) is a visualization of the emotion binary classification effect of the emotion valence task of the dataset, Figure (c) is a visualization of the emotion binary classification effect of the arousal task of the DEAP dataset, and Figure (d) is a visualization of the emotion ternary classification effect of the SEED dataset.

[0062] Figure 10 Figure 1 shows a visualization of the three-class classification features of the SEED dataset for a multimodal signal fusion emotion recognition method according to an embodiment of the present invention. Figure 2 shows the average SHAP value ranking of the SEED dataset by frequency band, Figure 3 shows the SHAP value distribution beehive plot of the SEED dataset under positive emotion, Figure 4 shows the SHAP value distribution beehive plot of the SEED dataset under neutral emotion, and Figure 5 shows the SHAP value distribution beehive plot of the SEED dataset under negative emotion.

[0063] Figure 11 This is a summary diagram of the arousal SHAP of an emotion recognition method based on multimodal signal fusion in an embodiment of the present invention;

[0064] Figure 12 This is a summary diagram of the valence of a multimodal signal fusion-based emotion recognition method (SHAP) in an embodiment of the present invention. Detailed Implementation

[0065] To facilitate understanding of the present invention, a more complete description of this application will be given below with reference to the accompanying drawings, which illustrate preferred embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to enable a more thorough and complete understanding of the disclosure of the present invention.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention; the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0067] Example 1: As Figure 1 As shown, an emotion recognition method based on multimodal signal fusion is proposed.

[0068] S1: Acquire EEG signals and peripheral physiological signals to form a multimodal signal, and perform preprocessing such as downsampling, baseline correction and bandpass filtering on the multimodal signal; based on the preprocessed signal, extract differential entropy features and power spectral density features;

[0069] Specifically, multimodal signals were acquired by acquiring electroencephalogram (EEG) signals and peripheral physiological signals (PPS). The data came from the SEED dataset (Shanghai Jiao Tong University Emotion EEG Dataset) and the DEAP dataset (Emotion Analysis Database Based on Physiological Signals).

[0070] The original signal is downsampled to a standard frequency to reduce the amount of data while maintaining signal quality. For example, for the DEAP dataset, the EEG signal is downsampled from 512Hz to 128Hz; for the SEED dataset, the EEG signal is downsampled from 1000Hz to 200Hz.

[0071] The preprocessing uses a 0.5-second time window for signal slicing, and applies bandpass filters in the 4-45Hz and 0-75Hz frequency bands to the EEG and peripheral physiological signals, respectively. The EEG signal uses a 4-45Hz filter, and the peripheral physiological signal has its filter passband set according to its type. The filter passbands include a 0-2.4Hz filter for skin conductance response, a 0-2.4Hz filter for respiratory rate, a 0.01-0.5Hz filter for blood volume pulse, a 0-0.2Hz filter for body temperature, a 0.5-10Hz filter for electrooculography, and a 0.5-50Hz filter for electromyography.

[0072] Specifically, a 0.5-second time window was used for data segmentation and baseline correction. Taking the DEAP dataset as an example, the signal acquisition included 3 seconds of baseline data. The baseline data was divided into 6 parts using a 0.5-second window, and the mean of these 6 parts was calculated to correct the entire signal, thus eliminating the influence of baseline drift. The experimental data was also processed using a 0.5-second window to ensure signal stability.

[0073] Filters with specific frequency bands are applied to remove noise and irrelevant frequency components. The DEAP dataset uses a 4-45Hz bandpass filter, while the SEED dataset uses a 0-75Hz bandpass filter for EEG signals. For the PPS signals in the DEAP dataset, the filter passbands are set according to signal type, including: Gestational Skin Response (GSR): 0-2.4Hz; Respiratory Rate (RESP): 0-2.4Hz; Blood Volume Pulse (BVP): 0.01-0.5Hz; Body Temperature (TEMP): 0-0.2Hz; Electrooculography (EOG): 0.5-10Hz; Electromyography (EMG): 0.5-50Hz. The filtered signals more accurately reflect autonomic nervous system activity.

[0074] The feature extraction includes: independently calculating the differential entropy feature and power spectral density feature of each channel for physiological signals within each 0.5-second time window; wherein, the differential entropy quantifies the overall activity level of the signal by calculating the signal energy within the time window, and the power spectral density analyzes the energy distribution of the signal in different frequency bands through Fourier transform; the extracted features are mapped to the standardized spatial grid of the international 10-20 system to generate a tensor with a shape of frequency band number × 8 × 9; during grid mapping, the feature values ​​are filled into an 8-row, 9-column matrix according to the electrode topology, and missing channel positions are filled with zero values.

[0075] Specifically, a 0.5-second time window is uniformly used for signal slicing, which can accurately capture rapid changes in emotional fluctuations and suppress short-term noise, ensuring the accuracy of feature extraction.

[0076] Differential entropy (DE) and power spectral density (PSD) features were extracted for each 0.5-second time window based on baseline and experimental data. For each channel within each 0.5-second time window, the differential entropy feature was calculated to quantify the overall activity level and reflect the intensity of emotional responses by calculating signal energy. The differential entropy feature effectively captures the temporal energy distribution of the signal and is suitable for emotional state classification. Fourier transform analysis was used to analyze the energy distribution of the signal in different frequency bands, extracting power spectral density features. Power spectral density reveals the relationship between the frequency components of physiological signals and emotional states.

[0077] The extracted differential entropy and power spectral density features are mapped onto the standardized spatial grid of the international 10-20 system, generating a feature tensor with a unified structure. The mapping process, based on the electrode topology, fills the eigenvalues ​​into an 8x9 matrix (shape: number of frequency bands × 8 × 9), padding missing channels with zero values ​​to preserve spatial consistency. Figure 2 As shown, feature alignment across different datasets is ensured, supporting cross-dataset validation.

[0078] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages:

[0079] This application constructs multimodal signals by acquiring EEG and peripheral physiological signals, and preprocesses these signals through downsampling, baseline correction, and bandpass filtering. Differential entropy and power spectral density features are extracted from the preprocessed signals, and these features are finally mapped onto a standardized spatial grid of the international 10-20 system to generate a feature tensor with a unified structure. This achieves efficient standardization of multimodal signals, effectively reducing data volume while maintaining signal quality. Noise and irrelevant frequency components are removed through specific frequency band filters, and emotional fluctuations are accurately captured using 0.5-second time window slices. Signal activity levels and frequency band energy distributions are quantified based on differential entropy and power spectral density features. Mapping features to an 8×9 grid structure ensures feature alignment between datasets with different electrode layouts, supports cross-dataset validation, and thus improves the accuracy, robustness, and generalization ability of emotion recognition.

[0080] Example 2: Example 1 achieved preprocessing, feature extraction, and normalization mapping of multimodal signals, but it still suffers from insufficient feature fusion and low utilization of key frequency band information. This example further explains and illustrates the content of Example 1.

[0081] S2: The extracted features are mapped to a standardized spatial grid according to the international 10-20 system to generate a feature tensor with a unified structure; the mapped feature tensor is adaptively weighted using a frequency band fusion attention mechanism, and frequency band weights are generated through global average pooling and fully connected layers;

[0082] Specifically, a standardized spatial frequency input module (STIM) is constructed to achieve spatial alignment and structural uniformity of multimodal and multi-band physiological signals. For example... Figure 3As shown, the first column of the third row is the original electrode layout projected onto the international 10-20 system in two dimensions; the second column of the third row is the mapped 8×9 two-dimensional mesh structure; the first column of the second row is the initial feature matrix after frequency domain feature extraction; the second column of the second row is the spatial feature tensor after sparsification; the first column of the first row is the visualization of frequency band features; and the second column of the first row is the comprehensive feature map after frequency band fusion. The first layer extracts frequency domain features from the original multi-channel EEG and PPS, calculating the DE and PSD features in the four frequency bands respectively. The model extracts four frequency band feature vectors for each physiological signal, ultimately forming a feature map with a dimension of... The frequency domain feature matrix is ​​denoted by C, where C represents the number of modes and F=4 represents the number of frequency bands. The second layer, based on the 10-20 international electrode layout standard, maps the frequency domain features to a fixed-structure two-dimensional spatial topological mesh, generating a shape of... The sparse spatial feature tensor is defined, where 8 and 9 represent the number of rows and columns of the spatial matrix, respectively. The third layer fuses the feature representations of all frequency bands to generate a tensor of uniform shape, which is used as the input to the backbone network of the model.

[0083] The standard Squeeze-and-Excitation (SE) module is applied to 8×9 matrices of different frequency bands obtained from DE and PSD extractions of different physiological signals. Adaptive weighting of the features of each frequency band is called Band Fusion Attention (BFAM). This module, through weighted processing of each frequency band, enables the model to more accurately capture the differences and correlations between frequency bands across different modalities, further enhancing the model's perception and representation capabilities.

[0084] Let the input signal tensor be ,in, For the set of real numbers, The height of the input matrix. The width of the matrix. The frequency band number. The characteristic representation of each signal is as follows: ,in, These represent the height and width of the matrix, respectively. The number of frequency bands is used to obtain statistics through global average pooling. The calculation formula is as follows:

[0085] ,

[0086] in, For frequency band statistics, For position ( ) frequency band The eigenvalues, and the statistics of all frequency bands form a vector. , For the frequency band statistics vector, all Form a vector .

[0087] The frequency band statistics vector is processed sequentially through two fully connected layers. The first fully connected layer uses a linear rectified function and introduces a compression ratio to reduce the dimensionality. The second fully connected layer uses a logistic function to restore the original dimensionality and outputs the weight coefficients for each frequency band, with the weight coefficients ranging from 0 to 1. The generated weight coefficients are then multiplied element-wise with the original feature tensor to achieve recalibration of the frequency band features, thereby strengthening the key frequency bands related to emotion recognition and suppressing redundant or noisy frequency bands.

[0088] The multimodal signal includes: splicing features of EEG signals and at least one peripheral physiological signal to form a multimodal input matrix; adaptively weighting the feature tensor using a frequency band fusion attention mechanism, obtaining statistics for each frequency band through global average pooling, and generating frequency band weight coefficients using a fully connected layer; multiplying the weight coefficients element-wise with the feature tensor to achieve weighted fusion, thereby strengthening key frequency band features and suppressing redundant information.

[0089] Specifically, an SE attention module is used to adaptively model the importance of each mode and frequency band. First, the statistics for each frequency band are sequentially passed through two fully connected layers:

[0090] ,

[0091] ,

[0092] in, The output vector is an intermediate feature representation after compression and activation. and These are the weight matrices used for compression and restoration, respectively. For compression ratio, Number of frequency bands; for Activation function It is the Sigmoid activation function. The weighting coefficients for all frequency bands satisfy the following conditions: ∈[0,1].

[0093] By weighting and fusing the original features, we obtain the fused weighted feature map. The calculation formula is as follows:

[0094] ,

[0095] in, For the new feature tensor after weighted fusion at position ( The value at ) These are the weighting coefficients for the C-th frequency band;

[0096] BFAM, through weighted fusion, effectively enhances the feature contributions of frequency bands highly correlated with the emotion dimension, while suppressing redundant and noisy information. The resulting fused feature map retains key information across modalities and frequency bands. The module architecture is as follows: Figure 4 As shown.

[0097] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages:

[0098] This application employs a standardized spatial frequency input module to map multimodal physiological signal features onto a unified spatial grid according to the international 10-20 system, generating structured tensors. Combined with a frequency band fusion attention mechanism, it utilizes global average pooling and fully connected layers to adaptively allocate frequency band weights, thereby enhancing key emotion-related frequency bands and suppressing redundant noise bands. This significantly improves the accuracy and robustness of emotion recognition. Through parameterized matrix factorization and dynamic weight optimization, it substantially reduces computational complexity while maintaining high accuracy, enhancing the model's generalization ability and interpretability across datasets and in real-time applications, providing an efficient and reliable solution for multimodal emotion recognition.

[0099] Example 3: Example 2 improved the accuracy and computational efficiency of emotion recognition by using feature mapping of multimodal signals and frequency band fusion attention mechanism for adaptive weighting; however, it suffered from weak generalization ability and poor interpretability. This example further supplements the content of Example 2.

[0100] S3: Input the weighted features into the parameterized hypermatrix fully connected layer, and perform feature compression and modeling through matrix decomposition and reconstruction;

[0101] The parameterized hypermatrix fully connected layer decomposes the weight matrix of the traditional fully connected layer into a hypermatrix and a structure matrix using nonnegative matrix factorization. The rank parameter of the hypermatrix is ​​dynamically adjusted during training: the initial rank parameter is set to 4 and gradually increased to 8 according to gradient sparsity to balance parameter efficiency and feature representation ability. A kernel norm regularization term of the weight matrix is ​​introduced into the model loss function to suppress overfitting and improve generalization ability by constraining the low-rank property of the weight matrix.

[0102] Specifically, to improve the expressive power and operational efficiency of physiological signal emotion recognition models, this application proposes a Parametric Efficient Feature Modeling Unit (PEF) with a core consisting of a multi-layer CNN, a parameterized hypermatrix custom fully connected layer (PHMDense), and a multi-task learning branch. Its overall structure is as follows: Figure 5 As shown.

[0103] As shown in Table 1, the hyperparameters in the table were selected by combining prior knowledge, dataset characteristics, and common practices in existing literature. Regarding the setting of hyperparameters such as the number of convolutional kernels, output dimension, maximum number of training epochs, batch size, and initial learning rate, this application has made reasonable adjustments based on previous literature experience and model structure characteristics to ensure the comparability of experimental results across different datasets.

[0104]

[0105] Table 1

[0106] This application introduces the concept of nonnegative matrix factorization (NMF) from linear algebra, decomposing the weight matrix in a traditional fully connected layer (Dense layer) into two smaller tensors. and Then, element-wise multiplication is performed to model the model, and a parameterized hypermatrix fully connected layer (PHMDense) is constructed to reduce the number of parameters and improve computational efficiency.

[0107] Let the input tensor ,in It is the number of samples. It is the input feature dimension. The output tensor of the PHMDense layer is ,in This refers to the dimension of the output features, and the dimension of the final weights is... The PHMDense layer first divides the input and output feature dimensions into blocks according to the hyperparameter n, where n is equivalent to the rank in NMF, but typically can only be an even number such as 2, 4, 6, or 8 that is divisible by the number of input and output features. Hypermatrix Structure matrix is ​​used to model the relationships between input features. Used to divide input and output features into blocks:

[0108] ,

[0109] in, The reconstructed weight matrix, For the hypermatrix of the first A small block, The first of the structure matrix A small block, For tensor product operations, For hyperparameters;

[0110] By input tensor With the constructed weight matrix Multiply to get the output. ,Right now:

[0111] ,

[0112] in, This is a bias term used to optimize the training process.

[0113] The ReLU activation function is applied to the output of the PHMDense layer to enhance the model's nonlinear representation capability. The output after activation is as follows:

[0114] ,

[0115] in, The final activation output of applying the ReLU activation function, For the output tensor of the fully connected layer, ( ) represents the ReLU activation function.

[0116] In the PHMDense layer, the number of parameters is reduced using the idea of ​​matrix factorization. Assume the hypermatrix... The dimension is n×n×n, and the input and output features are multiplied by n sub-blocks (i.e., I / n and O / n). The number of parameters in the PHMDense layer is:

[0117] ,

[0118] in, To determine the total number of parameters in a parameterized hypermatrix fully connected layer, Let be the rank parameter of the hypermatrix. The parameters of the hypermatrix. The number of parameters in the structure matrix. To output feature dimensions, The input feature dimension.

[0119] In practical applications, n is usually a small constant. Through this compression mechanism, the number of parameters in a PHMDense layer is typically lower than that of a traditional Dense layer.

[0120] S4: A multi-task learning framework is adopted, which simultaneously outputs the classification results of emotional valence and arousal based on the compressed features. Multi-task collaboration is optimized by sharing feature layers and dynamic loss weights.

[0121] Specifically, in emotion recognition research based on physiological signals, it is typically necessary to predict two dimensions of emotion: valence and arousal. Since both classification tasks use the same physiological signals as input, traditional single-task learning methods require training independent models for each task. This not only leads to redundancy in feature extraction and model deployment but also makes it difficult to uncover the potential relationships between them. Multi-task learning can utilize shared inputs and features during a single training process to jointly model Valence and Arousal, thereby effectively improving model efficiency. A structural comparison of single-task learning and multi-task learning is provided below. Figure 6 As shown.

[0122] A hard parameter sharing mechanism from multi-task learning is introduced into the model. The model first extracts shared spatiotemporal features through four layers of convolutional-normalized-activation units. These layers perform stepwise feature extraction on the input feature tensor, capturing global representations across frequency bands and modalities. After the shared layer output, the feature map is converted into a high-dimensional feature vector through global average pooling and flattening operations, serving as the common input for the multi-task branches. Each task branch consists of two parallel fully connected layers and a Softmax (normalized exponent) classifier, corresponding to binary classification of valence and arousal, respectively.

[0123] The loss weights for the emotion valence task are calculated in real-time based on the validation set accuracy. The formula for calculating the loss weights is as follows:

[0124] ,

[0125] in, For the loss weight of the emotional valence task, and These represent the emotional valence and arousal task accuracy on the validation set during the current training cycle;

[0126] The formula for calculating the loss weight of the wakefulness task is:

[0127] ,

[0128] in, The loss weight for wakefulness tasks;

[0129] The total loss of the weighted loss function is:

[0130] ,

[0131] in, The total loss is the weighted loss function. and For cross-entropy loss, For the frequency band feature covariance constraint term, the weights are... The weight is set to 0.01 to reduce feature redundancy. Dynamic weight adjustment ensures that the model prioritizes tasks with lower accuracy, mitigating task imbalance.

[0132] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages:

[0133] This application achieves efficient feature compression and multi-dimensional collaborative optimization of emotion recognition models by introducing a parameterized hypermatrix fully connected layer (PHMDense) and a multi-task learning framework. The PHMDense layer, based on non-negative matrix factorization, decomposes the weight matrix of a traditional fully connected layer into a hypermatrix and a structure matrix. The rank parameter of the hypermatrix is ​​dynamically adjusted during training. The weight matrix is ​​reconstructed through tensor product, and a parameter compression formula is used to reduce computational costs, suppressing overfitting while maintaining feature expressiveness. Building upon this, the multi-task learning framework shares a feature extraction layer and employs a dynamic loss weight mechanism to simultaneously optimize emotion valence and arousal classification tasks: the loss weights are adjusted in real-time based on the validation set accuracy, and the total loss function incorporates a covariance constraint term to reduce feature redundancy. This improves the accuracy, generalization ability, and interpretability of emotion recognition, providing a lightweight solution for real-time applications in resource-constrained scenarios.

[0134] Example 4: Example 3 implemented feature compression and modeling through a parameterized hypermatrix fully connected layer, and simultaneously optimized the emotion valence and arousal classification tasks using a multi-task learning framework, improving the computational efficiency and classification performance of emotion recognition; however, the decision-making process lacked transparency, making it difficult to intuitively understand the contribution of key features. This example further supplements and explains the content of Example 3.

[0135] The method also includes using the SHAP visualization method to quantify the contribution of frequency band features, such as... Figure 9 As shown, (a) and (b) represent the binary classification results of emotion valence and arousal tasks on the DEAP dataset, respectively; (c) represents the tri-class classification results of emotion on the SEED dataset. The marginal contribution of each frequency band to the emotion classification prediction is calculated based on the Shapley value. The Shapley value is calculated by integrating the spatial location features in the input tensor using a gradient interpreter.

[0136] ,

[0137] in, For frequency band i in spatial location ( The SHAP value on ) For the mathematical expectation, on the background sample set Reference sample in Take the average. For the model prediction function, For interpolation points, For reference sample The j-th eigenvalue, For input samples The j-th eigenvalue corresponds to the spatial position in the input tensor. , For integration weighting terms, These are the weighting coefficients used for integral calculation;

[0138] By aggregating the SHAP values ​​of each frequency band across spatial locations, a global contribution ranking is obtained, using the following formula:

[0139] ,

[0140] in, Let be the average SHAP value of the i-th frequency band. The SHAP value for the i-th frequency band. This is the standardized spatial grid size for the international 10-20 system;

[0141] Feature selection is optimized based on contribution ranking, and frequency bands with contributions below a preset threshold are masked to reduce computational complexity and enhance model interpretability. Through cross-sample aggregation, the absolute values ​​of the SHAP values ​​of different frequency bands in each sample are summed and averaged to obtain the global importance of each frequency band.

[0142] ,

[0143] in, Let be the absolute value of the average SHAP of the i-th frequency band. Let be the absolute SHAP value of the i-th frequency band, and N represent all test set samples. Finally, sort the results of different frequency bands in descending order to obtain the global contribution ranking of different frequency bands. The frequency bands include δ (1-4Hz), θ (4-8Hz), α wave (8-14Hz), β (14-31Hz), and γ (>31Hz) bands, among which the γ band shows a significant contribution in the emotion classification task.

[0144] To visually present the above sorting results, this application further provides a SHAP visualization analysis chart, such as... Figure 10 As shown in the figure, this figure is based on the SEED dataset's three-class classification task. By calculating the average absolute value of the SHAP values ​​for each frequency band, it clearly demonstrates the differences in the contribution of different frequency bands to emotion classification. Among them, (a) is the ranking of the average SHAP values ​​of the frequency bands, and (b), (c), and (d) are beehive plots of the SHAP value distribution under positive, neutral, and negative emotions, respectively, confirming the key role of the γ frequency band in emotion recognition.

[0145] The training process of the method employs a multi-stage optimization strategy, including:

[0146] An adaptive moment estimation optimizer with an initial learning rate of 0.001 is used, combined with early stopping to monitor the validation set loss. Training is terminated when the loss does not decrease for 20 consecutive rounds to prevent overfitting. Overfitting is suppressed by random dropout layers and batch normalization to improve the model's generalization ability. The weight coefficients in the multi-task loss function are dynamically adjusted based on the accuracy difference between the emotion valence task and the arousal task on the validation set. A phased training approach is adopted, first training the convolutional backbone network, and then jointly optimizing the fully connected layers and the multi-task classifier to improve feature extraction and classification performance in stages.

[0147] Specifically, an adaptive moment estimation optimizer (Adam) with an initial learning rate of 0.001 was used. The learning rate employed an exponential decay strategy, decreasing by a factor of 0.1 every 50 training epochs to smooth the convergence process. The batch size was set to 64, the maximum number of training epochs was set to 400, and the number of frequency bands per physiological signal was set to 8 for the DEAP dataset and 10 for the SEED dataset to ensure model consistency across the DEAP and SEED datasets.

[0148] Early stopping is used to monitor the validation set loss. Training is terminated when the loss fails to decrease for 20 consecutive rounds to prevent overfitting. The validation set is partitioned using 5-fold cross-validation, with a training-to-validation ratio of 0.85:0.15. The loss function, accuracy, and F1 score curves are tracked in real time. Figure 7 The DEAP dataset shown has three curves and Figure 8 The three curves shown in the SEED dataset are: V Task and ATask represent the valence and arousal tasks of the training set, respectively; Validation V Task and Validation A Task represent the valence and arousal tasks of the validation set, respectively; Loss, Accuracy, and F1 represent the three-class classification curves of the training set, respectively; Validation Loss, Validation Accuracy, and Validation F1 represent the three-class classification curves of the validation set, ensuring the model's convergence stability.

[0149] Overfitting is suppressed by using a random dropout layer and batch normalization. A dropout layer is added after the convolutional layer with a dropout rate of 0.5; a batchNorm layer is applied after each convolutional module to normalize the features and improve the model's generalization ability.

[0150] Based on the accuracy differences between the emotion valence and arousal tasks on the validation set, the weight coefficients in the multi-task loss function are dynamically adjusted.

[0151] A phased training approach is adopted: Phase 1: The convolutional backbone network (including four CNN layers) is trained separately with a fixed learning rate of 0.001. Only the parameters of the feature extraction layer are optimized to capture spatiotemporal features across frequency bands and modalities. Phase 2: The fully connected layer (PHMDense layer) and the multi-task classifier are jointly optimized with the learning rate adjusted to 0.0001. The shared feature layer and task-specific parameters are fine-tuned to improve classification performance. This strategy improves feature extraction and classification capabilities in stages, avoiding task conflicts.

[0152] During training, the loss, accuracy, and F1 score curves of the training and validation sets were recorded in real time. A paired-tailed t-test was used to analyze the significance of the experimental results (probability value < 0.05) to ensure stable and reliable model performance. The SHAP visualization method was employed to quantify the contribution of frequency band features, enhancing the model's interpretability.

[0153] The method further includes ablation experiments to verify the synergistic effect of band-fused attention mechanism (BFAM), parameterized hypermatrix fully connected layer (PHMDense), and multi-task learning framework in the PH-MTM model; the ablation experiments include the following steps:

[0154] A baseline model is constructed, which consists of a multi-layer convolutional neural network (CNN), a traditional fully connected layer (Dense layer), and a single-task learning module, without attention mechanism and multi-task learning component. The baseline model serves as the benchmark for performance comparison.

[0155] Multiple variant models were constructed, including: adding a band fusion attention mechanism to the baseline model (denoted as baseline+SE), adding a parameterized hypermatrix fully connected layer (denoted as baseline+PHMDense), adding a multi-task learning framework (denoted as baseline+Multi-Mask), and adding module combinations (such as SE+PHMDense, SE+Multi-Mask, etc.). All models used the same hyperparameter configuration to ensure experimental comparability.

[0156] The experiment employed 5-fold cross-validation to split the training and validation sets (training to validation ratio 0.85:0.15). The loss function, accuracy, and F1 score curves of the training and validation sets were tracked in real time to monitor model convergence stability. The accuracy (Acc) and F1 score (F1) of the emotion valence and arousal classification tasks were recorded, as shown in Table 2: the baseline model achieved an accuracy of 95.31±0.21% and an F1 score of 95.26±0.21% on the Valence task; and an accuracy of 95.64±0.08% and an F1 score of 95.55±0.08% on the Arousal task.

[0157]

[0158] Table 2

[0159] Although adding the SE, PHMDense, and Multi-Mask modules individually to the baseline model did not significantly improve performance on their own, this does not negate the existence of synergistic effects between the modules. In practice, different modules often complement each other in terms of feature representation, attention mechanisms, and multi-scale information fusion. When they work together, the overall performance improvement exceeds the sum of their individual contributions. To quantify this potential synergistic effect, we conducted paired-tailed t-tests on the final model PH-MTM with four models: baseline, baseline+SE, baseline+PHMDense, and baseline+Multi-Mask, as shown in Table 3. * in the table represents a statistically significant improvement (p<0.05), A represents module PHMDense, B represents module Multi-Mask, and C represents module SE. For example, when both the SE and Multi-Mask modules are added simultaneously, the ΔAcc for the Valence task is 0.58±0.23 (p=0.005), showing a significant synergistic gain. This indicates that there is complementarity between the modules, and the SE attention mechanism can amplify the fine-grained features captured by multi-task learning.

[0160]

[0161] Table 3

[0162] Based on the significance results, the optimal module combination (i.e., the complete PH-MTM model) was selected, which achieved the best performance with the fewest parameters (0.82M), verifying the effectiveness of multimodal fusion. This ablation experiment not only quantified the contribution of each module but also provided a basis for lightweight model design, enhancing interpretability.

[0163] As shown in Tables 4 and 5, the accuracy and F1 score of the PH-MTM model on the DEAP and SEED datasets are compared with other state-of-the-art methods. All studies employed a subject-dependent K-fold cross-validation protocol.

[0164]

[0165] Table 4

[0166]

[0167] Table 5

[0168] Table 5 shows that the EEG-based frequency band × 8 × 9 grid structure fusion method is significantly effective in emotion classification. Combining the PH-MTM model with the frequency band fusion strategy not only improves the accuracy of emotion recognition tasks but also enhances the model's stability. Compared with recent studies using the SEED dataset, it achieves an accuracy of 94.47 ± 0.14% on the three-class classification task, surpassing existing methods and fully demonstrating the potential of frequency band fusion technology in emotion recognition.

[0169] Five-fold cross-validation was used to evaluate the classification performance of different peripheral physiological signals (PPS) for emotion recognition. Experimental results show that among single PPS signals, EOG signals have the best emotion classification performance, second only to EEG, followed by EMG and GSR. Under the condition of fusing all peripheral physiological signals, the model's recognition accuracy and F1 score are further improved, surpassing the classification performance of EEG, as shown in Table 6.

[0170]

[0171] Table 6

[0172] Analysis of combinations of EEG signals with various PPS (EOG, EMG, GSR, RESP, BVP, TEMP, and PPS) revealed performance differences across different emotion classification tasks, as shown in Table 7. The EEG+EMG combination demonstrated the best performance in terms of accuracy and F1 score in emotion assessment, particularly in the Arousal task, where both accuracy and F1 score reached 99.77%. In contrast, the EEG+TEMP combination performed slightly worse, although its accuracy and F1 score remained at a high level (99.66% and 99.65%, respectively).

[0173]

[0174] Table 7

[0175] Table 7 shows that the combination of EEG and EMG performs best in the DEAP dataset. Visualizing the feature contributions of each frequency band of the EEG / EMG combination to emotion classification reveals the role of each frequency band in emotion recognition, as well as the contributions of EEG and EMG to the model's predictions. Figure 11 , Figure 12 As shown, in both valence and arousal classification tasks, EEG serves as the primary modality for emotion classification, while EMG acts as an auxiliary modality. This result is consistent with research in the field of emotion recognition, where the vast majority of studies focus on EEG features because EEG effectively reflects an individual's emotional state. In contrast, EMG primarily reflects muscle activity; although it has some influence on emotional state, it typically serves as an auxiliary signal in emotion classification tasks, contributing less. Furthermore, the gamma and beta bands in the DE and PSD features of EEG make major contributions to emotion classification tasks. The high-frequency bands of EMG contribute more to emotion classification than the low-frequency bands. SHAP visualization results demonstrate that the gamma and beta bands of EEG have a significant contribution to emotion classification compared to other bands, a result consistent with most research in the field of emotion recognition.

[0176] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages:

[0177] This application employs the SHAP visualization method to quantify the contribution of frequency band features. Based on the Shapley value, the marginal contribution of each frequency band to emotion classification prediction is calculated. The spatial location features in the input tensor are integrated using a gradient interpreter to obtain the SHAP value of each frequency band. The contribution is then aggregated across spatial locations to generate a global contribution ranking, thereby achieving accurate identification and visualization of key emotion-related frequency bands. This method effectively enhances the interpretability of the model, allowing researchers to intuitively understand the importance of different frequency bands in emotion recognition. At the same time, by filtering out redundant frequency bands with contributions below a preset threshold, feature selection is optimized, computational complexity is reduced, and the efficiency of the model in real-time applications is improved, providing technical support for the reliable deployment of multimodal emotion recognition systems.

[0178] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal signal fusion emotion recognition method, characterized in that, include: S1: Acquire EEG signals and peripheral physiological signals to form a multimodal signal, and perform preprocessing such as downsampling, baseline correction and bandpass filtering on the multimodal signal; extract features of differential entropy and power spectral density based on the preprocessed signal; S2: The extracted features are mapped to a standardized spatial grid according to the international 10-20 system to generate a feature tensor with a unified structure; the mapped feature tensor is adaptively weighted using a frequency band fusion attention mechanism, and frequency band weights are generated through global average pooling and fully connected layers; S3: Input the weighted features into the parameterized hypermatrix fully connected layer, and perform feature compression and modeling through matrix decomposition and reconstruction; S4: A multi-task learning framework is adopted, which simultaneously outputs the classification results of emotional valence and arousal based on the compressed features. Multi-task collaboration is optimized by sharing feature layers and dynamic loss weights.

2. The emotion recognition method based on multimodal signal fusion as described in claim 1, characterized in that, The preprocessing uses a 0.5-second time window for signal slicing, and applies bandpass filters in the 4-45Hz and 0-75Hz frequency bands to the EEG and peripheral physiological signals, respectively. The EEG signal uses a 4-45Hz filter, and the peripheral physiological signal has its filter passband set according to its type. The filter passbands include a 0-2.4Hz filter for skin conductance response, a 0-2.4Hz filter for respiratory rate, a 0.01-0.5Hz filter for blood volume pulse, a 0-0.2Hz filter for body temperature, a 0.5-10Hz filter for electrooculography, and a 0.5-50Hz filter for electromyography.

3. The emotion recognition method based on multimodal signal fusion as described in claim 1, characterized in that, The feature extraction includes: independently calculating the differential entropy feature and power spectral density feature of each channel for physiological signals within each 0.5-second time window; wherein, the differential entropy quantifies the overall activity level of the signal by calculating the signal energy within the time window, and the power spectral density analyzes the energy distribution of the signal in different frequency bands through Fourier transform; the extracted features are mapped to the standardized spatial grid of the international 10-20 system to generate a tensor with a shape of frequency band number × 8 × 9; during grid mapping, the feature values ​​are filled into an 8-row, 9-column matrix according to the electrode topology, and missing channel positions are filled with zero values.

4. The emotion recognition method based on multimodal signal fusion as described in claim 1, characterized in that, The frequency band fusion attention mechanism includes: [the following is a list of tensors] As input, where For the set of real numbers, The height of the input matrix. The width of the matrix. For each frequency band; Perform global average pooling to calculate the mean of features across all spatial locations in the frequency band, yielding the frequency band statistics: ， in, For frequency band statistics, For position ( ) frequency band The eigenvalues, and the statistics of all frequency bands form a vector. ; The frequency band statistics vector is processed sequentially through two fully connected layers. The first fully connected layer uses a linear rectified function and introduces a compression ratio to reduce the dimensionality. The second fully connected layer uses a logistic function to restore the original dimensionality and outputs the weight coefficients for each frequency band, with the weight coefficients ranging from 0 to 1. The generated weight coefficients are then multiplied element-wise with the original feature tensor to achieve recalibration of the frequency band features, thereby strengthening the key frequency bands related to emotion recognition and suppressing redundant or noisy frequency bands.

5. The emotion recognition method based on multimodal signal fusion as described in claim 1, characterized in that, The multimodal signal includes: splicing features of EEG signals and at least one peripheral physiological signal to form a multimodal input matrix; adaptively weighting the feature tensor using a frequency band fusion attention mechanism, obtaining statistics for each frequency band through global average pooling, and generating frequency band weight coefficients using a fully connected layer; multiplying the weight coefficients element-wise with the feature tensor to achieve weighted fusion, thereby strengthening key frequency band features and suppressing redundant information.

6. The emotion recognition method based on multimodal signal fusion as described in claim 1, characterized in that, The parameterized hypermatrix fully connected layer decomposes the weight matrix of a traditional fully connected layer into a hypermatrix and a structure matrix using nonnegative matrix factorization (NMF). The rank parameter of the hypermatrix is ​​dynamically adjusted during training: initially set to 4, and gradually increased to 8 based on gradient sparsity to balance parameter efficiency and feature representation capability. A kernel norm regularization term for the weight matrix is ​​introduced into the model loss function to suppress overfitting and improve generalization ability by constraining the low-rank property of the weight matrix. The weight matrix is ​​reconstructed through tensor product operations. ， in, The reconstructed weight matrix, For the hypermatrix of the first A small block, The first of the structure matrix A small block, For tensor product operations, For hyperparameters; The formula for parameter compression is: ， in, To determine the total number of parameters in a parameterized hypermatrix fully connected layer, Let be the rank parameter of the hypermatrix. The parameters of the hypermatrix. The number of parameters in the structure matrix. To output feature dimensions, The input feature dimension.

7. The emotion recognition method based on multimodal signal fusion as described in claim 1, characterized in that, The multi-task learning framework shares a feature extraction layer and uses a weighted loss function to simultaneously optimize emotion valence and arousal classification tasks, with the loss weights dynamically adjusted according to the task difficulty. The dynamic adjustment includes: calculating the loss weight in real time based on the validation set accuracy. The formula for calculating the loss weight for the emotion valence task is as follows: ， in, For the loss weight of the emotional valence task, and These represent the emotional valence and arousal task accuracy on the validation set during the current training cycle; The formula for calculating the loss weight of the wakefulness task is: ， in, The loss weight for wakefulness tasks; The total loss of the weighted loss function is: ， in, The total loss is the weighted loss function. and For cross-entropy loss, For the frequency band feature covariance constraint term, the weights are... The value is set to 0.01 to reduce feature redundancy.

8. The emotion recognition method based on multimodal signal fusion as described in claim 1, characterized in that, The training process of the method employs a multi-stage optimization strategy, including: An adaptive moment estimation optimizer with an initial learning rate of 0.001 is used, combined with early stopping to monitor the validation set loss. Training is terminated when the loss does not decrease for 20 consecutive rounds to prevent overfitting. Overfitting is suppressed by random dropout layers and batch normalization to improve the model's generalization ability. The weight coefficients in the multi-task loss function are dynamically adjusted based on the accuracy difference between the emotion valence task and the arousal task on the validation set. A phased training approach is adopted, first training the convolutional backbone network, and then jointly optimizing the fully connected layers and the multi-task classifier to improve feature extraction and classification performance in stages.

9. The emotion recognition method based on multimodal signal fusion as described in claim 1, characterized in that, The method further includes ablation experiments to verify the synergistic effect of the band fusion attention mechanism, the parameterized hypermatrix fully connected layer, and the multi-task learning framework; the ablation experiments include the following steps: The baseline model is built based on convolutional neural networks and traditional fully connected layers, without attention mechanisms and multi-task learning modules; Multiple variant models were constructed, including adding a frequency band fusion attention mechanism to the baseline model, adding a parameterized hypermatrix fully connected layer, adding a multi-task learning framework, and adding module combinations. All models were trained using the same hyperparameters and with 5-fold cross-validation to split the training and validation sets. The loss function, accuracy, and F1 score curves of the training and validation sets were tracked in real time to ensure model convergence stability. The accuracy and F1 score of the emotion valence and arousal classification tasks were recorded. A two-tailed paired t-test was used to compare the performance differences between the baseline model and the variant model, calculate the change in accuracy and the change in F1 score, and use a probability value less than 0.05 as the statistical significance threshold; the optimal module combination was selected based on the significance results.

10. The emotion recognition method based on multimodal signal fusion as described in claim 1, characterized in that, The method also includes using the SHAP visualization method to quantify the contribution of frequency band features, calculating the marginal contribution of each frequency band to sentiment classification prediction based on the Shapley value, and calculating the SHAP value by integrating the spatial location features in the input tensor through a gradient interpreter. ， in, For frequency band i in spatial location ( The SHAP value on ) For the mathematical expectation, on the background sample set Reference sample in Take the average. For the model prediction function, For interpolation points, For reference sample The j-th eigenvalue, For input samples The j-th eigenvalue corresponds to the spatial position in the input tensor. , For integration weighting terms, These are the weighting coefficients used for integral calculation; By aggregating the SHAP values ​​of each frequency band across spatial locations, a global contribution ranking is obtained, using the following formula: ， in, Let be the average SHAP value of the i-th frequency band. The SHAP value for the i-th frequency band. This is the standardized spatial grid size for the international 10-20 system; Feature selection is optimized based on contribution ranking, and frequency bands with contribution below a preset threshold are masked to reduce computational complexity and enhance model interpretability.

Images