A multi-modal physiological data fusion driven mental stress classification method

Abstract

The application discloses a multi-modal physiological data fusion driven mental stress classification method, and belongs to the technical field of physiological signal processing and health monitoring. The method comprises the following steps: pre-processing electrodermal activity, body temperature, acceleration, heart rate and blood oxygen saturation signals; adopting an NSGA-II algorithm to adaptively optimize multi-modal fusion weights; realizing data-level fusion through weighted average; extracting statistical features and eliminating redundant features based on correlation analysis; and finally classifying by using a convolutional neural network with an enhanced multi-head attention mechanism. Through multi-scale fusion and adaptive weight optimization, the application improves classification accuracy and robustness, and is suitable for wearable devices, mental health monitoring and other fields.

Description

Technical Field

[0001] This invention relates to the field of physiological signal processing and health monitoring technology, specifically to a method for automatically classifying mental stress states by integrating multimodal physiological data such as skin electrical activity, body temperature, acceleration, heart rate, and blood oxygen saturation, and through adaptive weight optimization and multi-scale feature fusion. Background Technology

[0002] Mental Stress Classification (MSC), as a key technology for monitoring individual stress levels based on physiological signals, has significant application value in occupational health management, clinical monitoring, and wearable devices. Existing technologies mainly fall into two categories:

[0003] One type of method is based on a single physiological signal, such as using only single modal data like electrical skin activity (EDA), heart rate (HR), or heart rate variability (HRV). The limitation of this type of method is that mental stress is a multidimensional physiological and psychological phenomenon; a single physiological signal contains limited information and cannot comprehensively reflect the stress state, easily leading to the loss of key information and affecting classification performance.

[0004] Another type is based on multimodal physiological signals, which integrates multiple physiological signals (such as EDA, temperature, acceleration, heart rate, blood oxygen saturation, etc.) at the feature level and then inputs them into machine learning or deep learning models. Existing methods mainly adopt feature-level fusion strategies, that is, simply splicing multimodal features after feature extraction. Although this method can combine multiple signals, it has the following technical problems: (1) It cannot make full use of the complementarity of different physiological modal data at the data level; (2) The scale difference between different modal data is large, and direct fusion is prone to information redundancy; (3) The fusion weight usually depends on human experience to set, lacks an adaptive optimization mechanism, and is difficult to adapt to the data characteristics of different individuals and scenarios.

[0005] In recent years, although some studies have attempted to use deep learning methods for multimodal fusion, most are still limited to simple concatenation or weighting at the feature level, failing to achieve multi-scale deep fusion at both the data and feature levels. For example, Chinese invention patent CN202510028504.2 discloses a multimodal mental stress assessment method and system based on machine learning. This method collects EEG signals and heart rate variability signals, uses a CNN-LSTM model to extract deep features from the EEG spectrogram and heart rate variability time series respectively, obtains deep feature vectors of the two modalities, and then concatenates them to form a multimodal feature representation. Finally, a support vector machine (SVM) is used for classification. This method utilizes the complementarity of multimodal signals to some extent, but still has the following significant technical problems:

[0006] 1. Incomplete physiological modality coverage and limited information dimension: This method only uses two physiological signals—electroencephalogram (EEG) and heart rate variability—ignoring other valuable physiological responses under mental stress, such as key physiological indicators like sympathetic nerve excitability reflected by electrical skin activity (EDA), changes in body temperature (TEMP), alterations in motor behavior (acceleration), and fluctuations in blood oxygen saturation (SpO2). This modality deficiency leads to an incomplete representation of stress states and makes it difficult to capture the multi-systemic characteristics of stress responses.

[0007] 2. Fixed fusion weights and lack of adaptive optimization: This technology uses a simple feature vector concatenation method in the feature fusion stage, with fixed contribution weights for each modality. It cannot dynamically adjust these weights based on signal quality, individual differences, or real-time status. When a modality's signal is affected by noise or its acquisition quality deteriorates, the fixed weight allocation introduces invalid information. Conversely, it cannot adaptively enhance the influence of modalities with strong discriminative power, thus limiting the robustness and accuracy of the fusion effect.

[0008] 3. Limited fusion level and lack of multi-scale complementarity: Existing technologies only perform post-processing stitching at the feature level, lacking a data-level adaptive fusion process. Data-level fusion can integrate complementary information from different modalities at the original signal level, forming more representative fused data; while single feature-level fusion cannot fully utilize low-level relevant information from different data sources, resulting in both information loss and redundancy.

[0009] 4. Lack of consideration for feature redundancy, resulting in low computational efficiency: Directly concatenating feature vectors without correlation analysis and redundancy removal can lead to redundant information storage and wasted computational resources when features extracted from different modalities are highly correlated. This increases the computational overhead of model training and inference and may cause overfitting.

[0010] 5. Single optimization objective, resulting in model performance bias: This method optimizes only the final classification accuracy during model training, without comprehensively considering multi-objective collaborative optimization such as precision and recall during the fusion stage. This single-objective optimization is prone to bias in imbalanced datasets, failing to guarantee the balance between positive and negative sample recognition.

[0011] 6. Traditional classifier architecture with limited ability to capture feature associations: The traditional SVM classifier lacks advanced architectures such as attention mechanisms to dynamically capture long-range dependencies and importance differences between features, which limits the model's ability to express deep patterns of complex physiological signals.

[0012] The aforementioned technical issues mean that there is still significant room for improvement in the accuracy, robustness, adaptability, and computational efficiency of this method in classifying mental stress. Summary of the Invention

[0013] The purpose of this invention is to provide a mental stress classification method with high classification accuracy and robustness, driven by multimodal physiological data fusion, adaptive weight optimization, and multi-scale feature fusion, in order to solve the above-mentioned problems.

[0014] The present invention provides a multimodal physiological data fusion-driven method for classifying mental stress, comprising the following steps:

[0015] Step 1, Multimodal physiological data preprocessing: Acquire raw signals from multiple physiological sensors, including electrical activity of the skin (EDA), temperature (TEMP), triaxial acceleration, heart rate (HR), and blood oxygen saturation (SpO2) signals; calculate the amplitude of the triaxial acceleration data as a new feature; use linear interpolation to resample all signals to a uniform time scale (5-second interval) and integrate them into a unified data structure;

[0016] Step 2, Adaptive optimization of multi-scale fusion weights based on NSGA-II: Using the accuracy and recall of mental stress classification as multi-objective optimization functions, the fast non-dominated sorting genetic algorithm II (NSGA-II) is used to adaptively optimize the fusion weights of multi-modal data to obtain the Pareto optimal weight set.

[0017] Step 3, Data-level fusion based on weighted average method: Using the optimized weights as input, the weighted average method is used to perform data-level fusion of physiological data from different modalities, generating fused multidimensional data.

[0018] Step 4, Statistical Feature Extraction and Redundancy Elimination: Extract five statistical features from the fused data: mean, median, standard deviation, maximum and minimum values, to construct an initial feature set; use Pearson correlation coefficient to analyze the correlation between features, and eliminate highly redundant features with an absolute correlation coefficient greater than 0.92 to obtain an optimized feature set.

[0019] Step 5, Convolutional Neural Network Classification Based on Enhanced Multi-Head Attention Mechanism: Construct an MHA-1DCNN model containing an input layer, two one-dimensional convolutional hidden layers, a multi-head attention layer, a fully connected layer, and an output layer; input the optimized feature set into the model, extract local temporal features through one-dimensional convolution, capture long-range dependencies between features through the multi-head attention mechanism, and finally output the mental stress state classification result.

[0020] The above-mentioned multimodal physiological data fusion-driven mental stress classification method includes at least four state categories: physical stress, emotional stress, cognitive stress, and relaxation state.

[0021] Compared with existing technologies, this invention has significant advantages. As can be seen from the above technical solution, this invention integrates multimodal physiological data such as skin electrical activity, body temperature, acceleration, heart rate, and blood oxygen saturation, and achieves automatic classification of mental stress states through adaptive weight optimization and multi-scale feature fusion, resulting in high classification accuracy and strong robustness. Specific advantages are as follows.

[0022] 1. Significantly improved classification performance: Experimental results on the NEURO dataset show that the method of this invention achieves a classification accuracy of 94.49%, a recall of 94.12%, a precision of 93.20%, and an F1 score of 93.64%, which is about 3.3 percentage points higher than the best existing methods; on the WESAD dataset, it achieves an accuracy of 97.36%, which is significantly better than traditional machine learning and deep learning methods.

[0023] 2. Advantages of multi-scale feature fusion: By combining data-level weighted average fusion with feature-level statistical fusion, the multi-scale fusion strategy fully utilizes the complementary information of different physiological modalities at the data and feature levels, overcoming the limitations of single feature-level fusion.

[0024] 3. Adaptive weight optimization: The NSGA-II algorithm is used to perform multi-objective adaptive optimization of the fusion weights. The optimal weight combination can be generated based on the data characteristics without manual intervention, which improves the adaptability of the method to different individuals and collection environments.

[0025] 4. Effective elimination of redundant information: By eliminating highly correlated redundant features through correlation analysis, the computational complexity of the model is reduced while ensuring classification performance, thus improving running efficiency.

[0026] 5. Good stability and generalization: Ablation experiments show that accelerometer and heart rate signals contribute the most to classification performance among various modal physiological signals; Pareto optimal weights outperform random weights on different machine learning models, proving the generalization ability of the weight optimization method; the variance of multiple independent experimental results is less than 0.1, showing good robustness.

[0027] This invention can be applied to wearable devices, mental health monitoring systems, occupational stress management, and clinical auxiliary diagnosis. By deploying it on wearable devices such as smart bracelets and smartwatches, and combining it with cloud computing resources, it can achieve real-time mental stress monitoring and personalized stress management, showing broad industrialization prospects. Attached Figure Description

[0028] Figure 1 This is a network architecture diagram of the present invention;

[0029] Figure 2 A flowchart for the Multi-Objective Weighted Adaptive Optimization (MSWA) module;

[0030] Figure 3 The flowchart is for the Multi-Scale Statistical Feature Fusion Module (SFF-WAM);

[0031] Figure 4 This is a diagram of the architecture of a multi-head attention-enhanced convolutional neural network (MHA-1DCNN). Detailed Implementation

[0032] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0033] Example 1:

[0034] See Figure 1 A multimodal physiological data fusion-driven method for classifying mental stress includes the following steps:

[0035] Step 1, Preprocessing of multimodal physiological data:

[0036] Acquire raw time-series signals S from five physiological sensors ori , where s1 to s n These correspond to EDA, TEMP, acceleration, HR, and SpO2 signals, respectively. The spatial amplitude of the triaxial acceleration data is calculated as follows: This was added to the dataset as an independent mode. Linear interpolation was used to resample all signals at fixed 5-second intervals, unifying the sampling frequency, and then formatting the data into a structured data format.

[0037] Step 2, Adaptive optimization of multi-scale fusion weights based on NSGA-II:

[0038] The NSGA-II algorithm is used for multi-objective optimization, with the objective functions being classification accuracy F1 and recall F2.

[0039]

[0040] TP represents a true positive, TN represents a true negative, FP represents a false positive, and FN represents a false negative.

[0041] The NSGA-II algorithm parameters are set as follows: population size 50, parent generation 20, offspring generation 50, crossover probability 0.7, mutation probability 0.2, and maximum number of generations 20. The algorithm initializes the parent population P. a And empty archive population P b During the iteration process, superior individuals are selected through fast non-dominated sorting and crowding calculation. Offspring are generated through crossover and mutation. After merging parent and offspring, an elitist strategy is used to select the next generation of parents. After the iteration ends, individuals with a Pareto rank of 1 are selected as the optimal weight output, resulting in a fusion weight that satisfies multi-objective balance. ;

[0042] Step 3, Data-level fusion based on weighted average method:

[0043] A weighted average method was used to fuse the preprocessed multimodal data. For each 5-second time window, the mean (mean) of each modality was calculated. According to the formula Calculate the fusion value M, where The weights obtained in step 2 are optimized; through this weighted averaging operation, the original 5-dimensional physiological signals are fused into more representative 6-dimensional data.

[0044] Step 4, Statistical Feature Extraction and Redundancy Elimination:

[0045] For the fused 6-dimensional data, the mean, median, standard deviation, maximum, and minimum values ​​for each dimension were calculated, generating a total of 30 statistical features. The Pearson correlation coefficient was used to analyze the correlation between the features; the correlation coefficient formula is as follows: When | | When the correlation coefficient is ≥ 0.92, it is considered a strong correlation, and redundant features are removed. In this embodiment, the correlation coefficients of all sensor features are less than 0.92, and all 30 features are retained to form the final feature set K;

[0046] Step 5, classification using a convolutional neural network based on enhanced multi-head attention mechanism:

[0047] A 7-layer MHA-1DCNN model is constructed, with the network structure as follows: Input layer → First hidden layer (1D convolution + ReLU activation + pooling) → Second hidden layer (1D convolution + ReLU activation + pooling) → Multi-head attention layer → Fully connected layer → Dropout layer (dropout rate 0.5) → Output layer. The formula for 1D convolution is: Where x is the given input sequence, w is the convolution kernel, and y is the output sequence; the specific network parameter configuration of the MHA-1DCNN model is as follows:

[0048] Input layer: Accepts feature sequences of shape (batch_size, 30, 1), where 30 is the number of features;

[0049] First hidden layer: 1D convolution with 128 kernels, kernel size 2, stride 2, padding=1; ReLU activation function; followed by 1D convolution with 128 kernels, kernel size 2, stride 2, padding=1.

[0050] Second hidden layer: one-dimensional convolution with 384 kernels, kernel size 2, stride 2, padding=1; ReLU activation function; followed by one-dimensional convolution with 384 kernels, kernel size 2, stride 2, padding=1.

[0051] Multi-head attention layer: 8 attention heads, feature dimension 128, dropout rate 0.1;

[0052] Fully connected layer: 128 neurons, ReLU activated;

[0053] Dropout layer: Dropout rate of 0.5, used to prevent overfitting;

[0054] Output layer: The number of neurons equals the number of stress categories (4 categories in the NEURO dataset and 3 categories in the WESAD dataset), and the Softmax activation function outputs the category probabilities.

[0055] The multi-head attention mechanism uses scaled dot product attention: Where Q is the query, K and V are the key and value respectively, and d k To determine the dimension of the query vector, multi-head attention is achieved by concatenating the outputs of k attention heads and multiplying them by a weight matrix. The final representation is obtained as follows: , .

[0056] The model was trained using the Adam optimizer with a learning rate of 0.001 and a loss function of Label Smoothing CrossEntropy, with 1500 iterations.

[0057] Step 6, output the classification results:

[0058] The test set is input into the trained MHA-1DCNN model, which outputs a probability distribution of mental stress states. The class with the highest probability is taken as the final classification result S. real .

[0059] Example 2:

[0060] A multimodal physiological data fusion-driven method for classifying mental stress includes the following steps:

[0061] Step 1, Multimodal physiological data preprocessing: Same as in Example 1;

[0062] Step 2, Adaptive optimization of multi-scale fusion weights based on NSGA-II: Same as in Example 1, the weight optimization process based on NSGA-II is as follows:

[0063] 1) Initialize the population P aEach individual is encoded as a 5-dimensional weight vector. The weights are randomly initialized and normalized;

[0064] 2) For each individual, perform the complete process from step 3 to step 5 (data fusion, feature extraction, model training), and calculate its accuracy F1 and recall F2 on the validation set as fitness values;

[0065] 3) For the current population P a And archived population P b The merged set is then sorted using a fast non-dominated sort and hierarchically ordered by Pareto rank.

[0066] 4) Within the same Pareto level, calculate the crowding distance for each individual and prioritize individuals with high crowding to maintain population diversity;

[0067] 5) Select individuals with a Pareto level of 1 and store them in the next generation archive population P. b+1 If the archive population is not full, select individuals of level 2 and level 3 in sequence until the capacity limit is reached.

[0068] 6) Select superior individuals from the archived population using a tournament selection method to enter the mating pool;

[0069] 7) Simulated binary crossover (SBX) and polynomial mutation are performed on individuals in the mating pool to produce offspring populations;

[0070] 8) Merge the offspring with the parent generation and use an elite selection strategy to select the next generation of parent population P. a+1 ;

[0071] 9) Repeat steps 2-8 until the maximum number of generations 20 is reached, and finally select the optimal weight combination from the Pareto front.

[0072] Step 3, Data-level fusion based on weighted average method: Same as in Example 1;

[0073] Step 4, Statistical Feature Extraction and Redundancy Elimination: Same as Example 1;

[0074] Step 5, Convolutional Neural Network Classification Based on Enhanced Multi-Head Attention Mechanism: Same as Example 1;

[0075] Step 6, classification result output: same as in Example 1.

[0076] Example 3:

[0077] A multimodal physiological data fusion-driven method for classifying mental stress includes the following steps:

[0078] Step 1, Multimodal physiological data preprocessing: Same as in Example 1;

[0079] Step 2, Adaptive optimization of multi-scale fusion weights based on NSGA-II: Same as Example 2;

[0080] Step 3, Data-level fusion based on weighted average method: Same as Example 1

[0081] Step 4, Statistical Feature Extraction and Redundancy Elimination: Same as in Example 1, except that the weighted average-based feature fusion method SFF-WAM is:

[0082] 1) Data fusion: Fusion of preprocessed multimodal data S pre The fused data S is generated according to the weighted average formula in step 3 of Example 1. fusion The dimensions were reduced from 5 to 6.

[0083] 2) Statistical feature extraction: For S fusion Data sequence of each dimension Calculate its mean , median Standard deviation Maximum value and minimum value ;

[0084] 3) Feature concatenation: The five statistical features of each dimension are concatenated using the Concat operation to form the feature set K for that dimension. i = { , , , , };

[0085] 4) Construction of the complete feature set: Take the union of the feature sets of all dimensions to obtain K = ∪K i There are a total of 30 features;

[0086] 5) Correlation analysis: Calculate the Pearson correlation coefficient between each pair of features in the feature set and construct a 30×30 correlation matrix; if the absolute value of the correlation coefficient between any two features is greater than 0.92, then remove the feature with the lower information entropy. In this embodiment, all correlation coefficients are less than the threshold, so all features are retained.

[0087] Step 5, Convolutional Neural Network Classification Based on Enhanced Multi-Head Attention Mechanism: Same as Example 1;

[0088] Step 6, classification result output: same as in Example 1.

[0089] Experimental example:

[0090] To verify the effectiveness of this invention, experiments were conducted on two publicly available datasets, NEURO and WESAD. The NEURO dataset contains four stress states: physical stress, emotional stress, cognitive stress, and relaxation; the WESAD dataset contains three states: neutral, stressed, and pleasant.

[0091] Experimental hardware environment: Dell Tower 7920 server, Intel Xeon Gold 218 processor, NVIDIA V100 graphics card (32GB VRAM), Ubuntu 20.04 operating system, PyTorch deep learning framework. Hyperparameter configuration: 1500 iterations, learning rate 0.001, Adam optimizer, Label Smoothing Cross Entropy loss function.

[0092] The comparative experiments included traditional machine learning algorithms such as KNN, SVM, random forest, and logistic regression, as well as deep learning models such as CNN, LSTM, and CNN-LSTM. Furthermore, comparisons were made with advanced methods proposed in recent years, such as MTFDFF-NN, LSTM-1DCNN, LibEER, BERT-XGBoost, PSO-SVR, and DEMA.

[0093] Experimental results show that the proposed MSC-MFF method achieves an accuracy of 94.49% on the NEURO dataset and 97.36% on the WESAD dataset, significantly outperforming the comparative methods in all evaluation metrics. Ablation experiments confirm that removing accelerometer features leads to a 7.12 percentage point decrease in accuracy, validating the importance of each modal signal; removing either the MSWA or SFF-WAM module results in performance degradation, demonstrating the synergistic effect of the components of this invention.

Claims

1. A method for classifying mental stress driven by multimodal physiological data fusion, characterized in that, Includes the following steps: (1) Multimodal physiological data preprocessing: acquire raw signals of skin electrical activity, body temperature, triaxial acceleration, heart rate and blood oxygen saturation, calculate the amplitude of triaxial acceleration, and integrate all signals into structured data after resampling to a unified time scale; (2) Adaptive optimization of multi-scale fusion weights based on NSGA-II: Using classification accuracy and recall as multi-objective optimization functions, the NSGA-II algorithm is used to adaptively optimize the fusion weights of multi-modal data to obtain the Pareto optimal weight set; (3) Data-level fusion based on weighted average method: The optimized weights are used to perform weighted average fusion of multimodal physiological data to generate fused multidimensional data; (4) Statistical feature extraction and redundancy elimination: Five statistical features, namely mean, median, standard deviation, maximum and minimum, are extracted from the fused data to construct an initial feature set; highly redundant features are eliminated through Pearson correlation coefficient analysis to obtain an optimized feature set; (5) Convolutional Neural Network Classification Based on Enhanced Multi-Head Attention Mechanism: The optimized feature set is input into the MHA-1DCNN model, local temporal features are extracted through one-dimensional convolution, long-range dependencies between features are captured through multi-head attention mechanism, and the mental stress state classification result is output.

2. The method for classifying mental stress driven by multimodal physiological data fusion according to claim 1, characterized in that, In step (2), the parameters of the NSGA-II algorithm include: population size 50, parent generation 20, offspring generation 50, crossover probability 0.7, mutation probability 0.2, and maximum number of generations 20.

3. The method for classifying mental stress driven by multimodal physiological data fusion according to claim 1, characterized in that, In step (3), the weighted average fusion formula is: , where w i For the optimized weights of the i-th mode, This represents the mean of the mode within the time window.

4. The method for classifying mental stress driven by multimodal physiological data fusion according to claim 1, characterized in that, In step (4), the threshold for redundancy features is that the absolute value of the Pearson correlation coefficient is greater than 0.

92.

5. The method for classifying mental stress driven by multimodal physiological data fusion according to claim 1, characterized in that, In step (5), the MHA-1DCNN model sequentially includes an input layer, two one-dimensional convolutional hidden layers, a multi-head attention layer, a fully connected layer, a Dropout layer, and an output layer.

6. The method for classifying mental stress driven by multimodal physiological data fusion according to claim 5, characterized in that, The multi-head attention layer contains 8 attention heads with a feature dimension of 128.

7. The method for classifying mental stress driven by multimodal physiological data fusion according to claim 1, characterized in that, The classification results include at least four state categories: physical stress, emotional stress, cognitive stress, and relaxation.

Images