A machine learning-based multi-modal stress evaluation method and system

Abstract

The application relates to the fields of biological signal processing and emotion computing, and discloses a multi-modal mental stress evaluation method and system based on machine learning, which comprises the following steps: collecting brain wave signals and heart rate variability signals of a user; using a CNN-LSTM model to respectively perform multi-modal deep feature extraction on a frequency spectrum diagram of the brain wave signals and a time sequence of the heart rate variability signals, so as to obtain a brain wave signal deep feature vector and a heart rate variability signal deep feature vector; splicing the brain wave signal deep feature vector and the heart rate variability signal deep feature vector to form a high-dimensional multi-modal feature vector representation; and using a support vector machine to classify the multi-modal feature representation and output a stress state discrimination result. The application can realize accurate prediction of a real-time stress state of a user, and solves the technical problem of low detection accuracy of existing stress detection methods.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of biosignal processing and affective computing, and particularly relates to a multi-modal mental stress assessment method and system based on machine learning. BACKGROUND

[0002] Electroencephalogram (EEG) signals and Heart Rate Variability (HRV) signals are two widely used physiological indicators for assessing individual mental state and stress level. EEG signals reflect the electrical activity of brain neurons, and by measuring the potential changes on the scalp, the activity patterns of the brain under different psychological states can be captured. HRV signals reflect the activity of the autonomic nervous system by measuring the changes in heart rate intervals, and are closely related to individual stress level, emotional state, etc.

[0003] Traditional stress detection methods mainly rely on manually extracting features from EEG and HRV signals, such as the power of certain frequency bands of EEG, time and frequency domain features of HRV, etc. However, these methods have obvious limitations. First, a single feature often cannot fully reflect the complex spatiotemporal characteristics of EEG and HRV signals, as both of these signals contain rich time-frequency information and dynamic change patterns. Second, simply fusing these features cannot effectively capture the nonlinear relationship between EEG and HRV signals, thus limiting the accuracy and reliability of stress detection. Existing machine learning-based stress detection methods mostly focus on single modal signals (such as using only EEG or HRV), or simply fuse multi-modal features, failing to fully utilize the complementarity and correlation between multi-modal signals. SUMMARY

[0004] In order to overcome the deficiencies of the prior art, the present application aims to provide a multi-modal mental stress assessment method and system based on machine learning, specifically a user stress state detection method and system combining Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and Support Vector Machine (SVM). The method uses CNN-LSTM to perform multi-modal deep feature extraction on EEG and HRV signals, and then uses SVM to classify the fused features, thus achieving accurate prediction of user real-time stress state. This solves the technical problem that existing stress detection methods cannot fully reflect the complex spatiotemporal characteristics and nonlinear relationship of EEG and HRV signals, resulting in low detection accuracy.

[0005] The purpose of the present application can be achieved by the following technical solutions:

[0006] In a first aspect, the present application provides a multi-modal mental stress evaluation method based on machine learning, comprising:

[0007] collecting electroencephalogram signals and heart rate variability signals of a user;

[0008] extracting multi-modal deep features from a spectrogram of the electroencephalogram signals and a time series of the heart rate variability signals using a CNN-LSTM model, to obtain an electroencephalogram signal deep feature vector and a heart rate variability signal deep feature vector;

[0009] concatenating the electroencephalogram signal deep feature vector and the heart rate variability signal deep feature vector to form a high-dimensional multi-modal feature vector representation;

[0010] classifying the multi-modal feature representation using a support vector machine (SVM) to output a stress state discrimination result.

[0011] Preferably, the method further comprises:

[0012] preprocessing and preliminarily extracting features from the collected electroencephalogram signals and heart rate variability signals, comprising:

[0013] after band-pass filtering and artifact removal of the collected electroencephalogram signals, obtaining a time-frequency matrix of the electroencephalogram signals, i.e., a spectrogram of the electroencephalogram signals, using short-time Fourier transform (STFT) or wavelet transform;

[0014] obtaining an R-R interval sequence of the collected heart rate variability signals, and after normalization and filtering of the R-R interval sequence, forming a one-dimensional time series.

[0015] Preferably, the multi-modal deep feature extraction from the spectrogram of the electroencephalogram signals using the CNN-LSTM model to obtain the electroencephalogram signal deep feature vector comprises:

[0016] extracting convolutional features from the spectrogram of the electroencephalogram signals using a convolutional neural network (CNN) to obtain local time-frequency features of the electroencephalogram signals;

[0017] modeling the extracted local time-frequency features using a long short-term memory network (LSTM) to capture dynamic patterns of the electroencephalogram signals over time, and outputting the electroencephalogram signal deep feature vector.

[0018] Preferably, the multi-modal deep feature extraction from the time series of the heart rate variability signals using the CNN-LSTM model to obtain the heart rate variability signal deep feature vector comprises:

[0019] The time sequence of the heart rate variability signal is input into a one-dimensional convolutional neural network (1D-CNN) to extract local time sequence pattern features.

[0020] The extracted local time sequence pattern features are time sequence modeled using a long short-term memory network (LSTM) to output a heart rate variability signal deep feature vector.

[0021] Preferably, the convolutional feature extraction of the spectrogram of the brain wave signal using a convolutional neural network (CNN) to obtain the local time-frequency features of the brain wave signal comprises:

[0022] The time-frequency features of the spectrogram of the brain wave signal are extracted using different sizes of two-dimensional convolution kernels, and the feature dimension is reduced through a max-pooling layer to output a brain wave signal convolution feature map, i.e., a local local time-frequency feature.

[0023] Preferably, the input of the time sequence of the heart rate variability signal into a one-dimensional convolutional neural network (1D-CNN) to extract local time sequence pattern features comprises:

[0024] The time sequence of the heart rate variability signal is input into a one-dimensional convolutional neural network (1D-CNN), and a more representative feature sequence is obtained through max-pooling down-sampling to obtain a heart rate variability signal convolution feature sequence, i.e., a local time sequence pattern feature of the heart rate variability signal.

[0025] Preferably, the concatenation of the brain wave signal deep feature vector and the heart rate variability signal deep feature vector to form a high-dimensional multi-modal feature vector representation comprises:

[0026] The brain wave signal deep feature vector and the heart rate variability signal deep feature vector are respectively standardized and feature dimension aligned;

[0027] The standardized and dimension-aligned brain wave signal deep feature vector and heart rate variability signal deep feature vector are concatenated in a predetermined order to form a multi-modal feature vector.

[0028] Preferably, the classification of the multi-modal feature representation using a support vector machine (SVM) to output a stress state discrimination result comprises:

[0029] The high-dimensional multi-modal feature vector representation is classified using a pre-trained SVM model to output a stress state prediction result, comprising: the SVM model finds an optimal classification hyperplane in a high-dimensional feature space to distinguish between stress states and normal states and output a stress state label, the stress state label comprising "stress" or "non-stress".

[0030] Preferably, the method further comprises:

[0031] transmit the stress state discrimination result to a health management system, display the stress state of the user in real time and perform long-term stress data monitoring analysis, including:

[0032] When the real-time monitoring shows that the stress state label is "stress", the health management system automatically triggers an alarm and reminds the user to rest or relax;

[0033] The health management system performs monitoring analysis according to the long-term stress data of the user, analyzes the stress pattern of the user, and assists in formulating personalized relaxation or intervention strategies.

[0034] In a second aspect, the present application also provides a multi-modal mental stress evaluation system based on machine learning, including:

[0035] A data acquisition module acquires the brain wave signal and the heart rate variability signal of the user;

[0036] A feature extraction module is configured to use a CNN-LSTM model to perform multi-modal deep feature extraction on the frequency spectrum of the brain wave signal and the time sequence of the heart rate variability signal, respectively, to obtain a brain wave signal deep feature vector and a heart rate variability signal deep feature vector;

[0037] A feature fusion module is configured to splice the brain wave signal deep feature vector and the heart rate variability signal deep feature vector to form a high-dimensional multi-modal feature vector representation;

[0038] A stress prediction module is configured to use a support vector machine (SVM) to classify the multi-modal feature representation and output a stress state discrimination result.

[0039] Compared with the prior art, the present application has the following beneficial effects:

[0040] (1) By combining convolutional neural network (CNN), long short-term memory network (LSTM) and support vector machine (SVM), the present application can more comprehensively capture the complex spatiotemporal characteristics and nonlinear relationship of EEG and HRV signals. CNN is used to extract local time-frequency features, LSTM is used to model the dynamic evolution of time series, and SVM is used to classify the fused multi-modal features, thereby realizing accurate prediction of the real-time stress state of the user and improving the accuracy and reliability of stress detection.

[0041] (2) Compared with the traditional pressure detection method, it often only focuses on a single mode signal or simply fuses multi-modal features, and fails to fully utilize the complementarity and correlation between multi-modal signals. The application extracts deep features of EEG and HRV signals through a CNN-LSTM model, and splices the extracted feature vectors to form a high-dimensional multi-modal feature vector representation, thereby more effectively utilizing the complementarity and correlation of multi-modal signals.

[0042] (3) The pressure state discrimination result of the application is transmitted to the health management system, which can display the user's pressure state in real time and perform long-term pressure data monitoring and analysis. When the real-time monitoring of the pressure state is performed, the health management system can automatically trigger an alarm and remind the user to rest or relax, thereby helping the user to timely manage his / her own pressure state.

[0043] (4) Through long-term monitoring and analysis of the user's pressure data, the health management system can analyze the user's pressure pattern and assist in formulating personalized relaxation or intervention strategies, thereby better meeting the user's individual needs.

[0044] In summary, the application improves the accuracy and reliability of pressure detection by combining machine learning and multi-modal signal processing technology, realizes real-time pressure monitoring and early warning, and provides personalized relaxation or intervention strategies, which is of great significance for promoting the user's mental health and improving the quality of life. BRIEF DESCRIPTION OF DRAWINGS

[0045] The above and / or additional aspects and advantages of the application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0046] Figure 1 is a flowchart of a multi-modal mental stress evaluation method based on machine learning provided by an embodiment of the application;

[0047] Figure 2 is a flowchart of EEG and HRV data processing and feature extraction of a multi-modal mental stress evaluation method based on machine learning of an embodiment of the application;

[0048] Figure 3 is a schematic diagram of the overall framework of the combination of CNN-LSTM and SVM model of a multi-modal mental stress evaluation method based on machine learning of an embodiment of the application;

[0049] Figure 4 is a unit module diagram of a multi-modal mental stress evaluation system based on machine learning provided by an embodiment of the application. DETAILED DESCRIPTION

[0050] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0051] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right and the like, is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0052] In the description of the present application, one or more is meant to be one or more, and more than two is meant to be more than two, greater than, less than, more than and the like are understood to not include the number, above, below, within and the like are understood to include the number. If the first, second is described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of indicated technical features.

[0053] The present application provides a kind of multi-modal mental stress evaluation method and system based on machine learning, it is a kind of user stress state detection method and system combining convolutional neural network (CNN), long short-term memory network (LSTM) and support vector machine (SVM), using CNN-LSTM carries out multi-modal deep feature extraction to EEG and HRV signal, then using SVM discriminant classification to fusion feature, to realize the accurate prediction of user real-time stress state, solve the technical problems that the existing stress detection method cannot fully reflect the complex space-time characteristics and nonlinear relationship of EEG and HRV signal, lead to detection precision is not high.

[0054] Figure 1 It is the flowchart of a kind of multi-modal mental stress evaluation method based on machine learning provided in the embodiments of the present application. Please refer to Figure 1 The present application provides a kind of multi-modal mental stress evaluation method based on machine learning, comprising:

[0055] S110: collect the brain wave signal and heart rate variability signal of user;

[0056] In this step S110, EEG signals are collected by an electroencephalography device, and heart rate signals are collected by an electrocardiogram or PPG device.

[0057] Preferably, in this step, the collected electroencephalogram signals and heart rate variability signals are also pre-processed and preliminary feature extraction is performed, specifically including:

[0058] (1) Electroencephalogram (EEG) data: After band-pass filtering, artifact removal and correction, short-time Fourier transform (STFT) or wavelet transform is used to obtain a time-frequency matrix of (n, m). In electroencephalogram (EEG) signal processing, short-time Fourier transform (STFT) or wavelet transform is used to convert the EEG signal from time domain to frequency domain, resulting in a time-frequency matrix. Where (n, m) represents the dimension of a two-dimensional matrix, each row of the matrix represents the frequency distribution at a time point, and each column represents the time variation at a frequency.

[0059] (1) Band-pass filtering;

[0060] A. Purpose: To remove noise and unnecessary frequency components in the signal to ensure the accuracy of subsequent feature extraction and analysis.

[0061] B. Band-pass filter:

[0062] A band-pass filter allows signals within a specific frequency range to pass through while suppressing frequency components below and above that range.

[0063] A Butterworth filter is used, and its transfer function is:

[0064] ;

[0065] Where: represents the amplitude response of the filter at frequency f; represents the frequency variable; represents the cutoff frequency; represents the order of the filter, which determines the steepness of the filter.

[0066] C. Filter parameter settings:

[0067] EEG filter parameters: Band-pass range: 0.5Hz to 50Hz; Filter order: 4.

[0068] HRV filter parameters: Band-pass range: 0.5Hz to 40Hz; Filter order: 4.

[0069] D. Normalized cutoff frequency:

[0070] ;

[0071] where, represents the normalized cutoff frequency; represents the actual cutoff frequency represents the sampling rate, is the Nyquist frequency.

[0072] The denominator coefficient and the numerator coefficient of the filter are calculated using Python, and the filtered signal is obtained by inputting the band-pass filter.

[0073] (2) EEG artifact removal and correction;

[0074] EEG signals are often disturbed by various physiological and non-physiological artifacts, such as eye movement, blinking, electromyography, etc. These artifacts will affect the subsequent feature extraction and classification accuracy, and need to be effectively identified and removed.

[0075] Independent Component Analysis (ICA): used to decompose multivariate signals into independent components, which can separate artifacts from EEG signals.

[0076] ;

[0077] where, represents the observation signal matrix, with a shape of (p × q), where p is the number of channels and q is the number of time steps. represents the mixing matrix, which describes how independent components are mixed into observation signals, with a shape of (p × p). represents the independent component matrix, with a shape of (p × q), i.e. the source signal matrix.

[0078] Apply ICA decomposition: decompose the preprocessed EEG signal into independent components and mixing matrix , by observing the time waveform and spectrum of independent components, identify and remove components representing artifacts.

[0079] Transient artifact detection: transient artifacts such as blinking, instantaneous electromyographic interference, etc. are detected and corrected by statistical analysis method.

[0080] Artifact detection formula:

[0081] If , consider as an artifact.

[0082] where: : the signal value of the th channel at time ; : threshold factor, set to 3 in this invention; : standard deviation of the signal.

[0083] After finding the artifacts, an interpolation method is used to replace the abnormal values with the interpolated values of the valid signals before and after.

[0084] ;

[0085] represents the signal value of the i-th channel at time t. In other words, it is the amplitude or measurement value of the electroencephalogram (EEG) signal at a certain specific time point t and a specific channel i; represents the signal value of the i-th channel at time t-1, represents the signal value of the i-th channel at time t+1; in continuous signal recording, t-1 is the previous time point of t, and t+1 is the next time point of t.

[0086] Specifically, if is determined to be an artifact (i.e. its amplitude exceeds the threshold value determined by the threshold factor k and the signal standard deviation , the average value of the valid signals before and after is used to replace the abnormal value. This interpolation method helps to reduce the impact of artifacts on signal analysis, thereby improving the accuracy of subsequent feature extraction and classification.

[0087] (3) Time-frequency conversion;

[0088] In order to further extract the time-frequency features in the EEG signal, the present application adopts a time-frequency conversion module to convert the purified and corrected EEG signal from the time domain to the time-frequency domain, and the specific implementation includes:

[0089] Short-time Fourier transform (STFT):

[0090] Window function selection: Hanning window is selected as the window function to reduce spectral leakage.

[0091] Window length and overlap: the window length is set to 256 samples, and the window overlap is set to 128 samples, achieving an overlap rate of 50%.

[0092] STFT calculation:

[0093] ;

[0094] wherein, is the original EEG signal, is the window function, which can be selected as Hanning window in the present application, and the window is moving with time, T usually represents the time step of window movement; is the time parameter; ω is the frequency parameter; is a complex exponential function used to convert signals into the frequency domain.

[0095] Continuous Wavelet Transform (CWT):

[0096] Mother wavelet function selection: Morlet wavelet is selected as the mother wavelet function because of its good localization characteristics in time-frequency domain.

[0097] Scale parameter setting: The scale parameter range is set to 1 to 128 to cover the required frequency range.

[0098] CWT calculation:

[0099] ;

[0100] where x(t) is the original EEG signal; ψ(t) is the mother wavelet function; a is the scale parameter, which determines the width (or frequency component) of the wavelet function, and the scale parameter a is set to 1 to 128 to cover the required frequency range; b is the translation parameter, which determines the position of the wavelet function on the time axis.

[0101] The technical advantages of the time-frequency conversion of EEG signals in the present application:

[0102] Multi-resolution analysis: Wavelet transform provides multi-resolution analysis capability, which can capture the frequency components of the signal at different time scales, adapting to the non-stationary nature of EEG signals.

[0103] Dynamic feature capture: STFT and wavelet transform can capture the dynamic features of the frequency components in the EEG signal over time, providing more rich feature information.

[0104] Improve feature extraction effect: The signal after time-frequency conversion has higher information content and better feature separation capability, which helps to improve the feature extraction and classification performance of the subsequent deep learning model (CNN-LSTM).

[0105] (2) Heart Rate Variability (HRV) data: Obtain the R-R interval sequence and form a one-dimensional time series after appropriate normalization and filtering processing. The R-R interval is the time interval between two adjacent R waves in an electrocardiogram (ECG), which is the basis for heart rate variability (HRV) analysis. Heart rate variability (HRV) is an important indicator for evaluating the activity of the autonomic nervous system. By measuring the changes in R-R intervals, the rhythm changes of the heart can be analyzed, and the individual's physiological and psychological state can be understood. The R-R interval sequence is a one-dimensional time series, where each element represents the time length of an R-R interval.

[0106] (1) Artifact removal and correction of HRV signal:

[0107] The HRV signal is mainly based on the analysis of R-R interval sequence, but there may be outliers (such as jump errors, missed detection, etc.) in the detection process of R-R interval. Therefore, it is necessary to remove and correct the artifacts of the R-R interval sequence.

[0108] Identify abnormal R-R intervals using statistical methods:

[0109] ;

[0110] : the ith R-R interval; : the mean of R-R intervals; : the standard deviation of R-R intervals; : threshold factor, set to 3 in the present application;

[0111] Abnormal R-R interval correction: use a spline function written in python to perform smooth interpolation on the outliers, maintaining the continuity and smoothness of the signal.

[0112] S120: using a CNN-LSTM model to perform multi-modal deep feature extraction on the frequency spectrum of the brain wave signal and the time sequence of the heart rate variability signal, respectively, to obtain brain wave signal deep feature vectors and heart rate variability signal deep feature vectors;

[0113] This step S120 specifically includes the following steps:

[0114] S121: multi-modal deep feature extraction of the brain wave (EEG) signal (CNN-LSTM module):

[0115] In this step S121, a CNN-LSTM model is used to perform multi-modal deep feature extraction on the frequency spectrum of the brain wave signal, to obtain brain wave signal deep feature vectors, including:

[0116] Using a convolutional neural network (CNN) to extract convolutional features from the frequency spectrum of the brain wave signal, to obtain local time-frequency features of the brain wave signal;

[0117] Using a long short-term memory network (LSTM) to model the extracted local time-frequency features, to capture the dynamic patterns of the brain wave signal over time, and to output brain wave signal deep feature vectors. Specifically as follows:

[0118] Structure of CNN module (convolution + pooling + BN):

[0119] Input: EEG time-frequency matrix (n, m);

[0120] Different size 2D convolution kernels are used to extract time-frequency features of EEG, and the feature dimension is reduced through the max pooling layer.

[0121] Input shape: (T, G, 1) or (T, G) (T, G) (T, G) or (T, G) (T, G) (T, G),

[0122] Where T is the time step (time period slice or STFT frame number), and G is the frequency channel number (obtained after wavelet / Fourier transform).

[0123] 2D convolution is used:

[0124] ;

[0125] To extract local spatial features in the time-frequency plane.

[0126] Where: X represents the input feature map, which is a two-dimensional matrix after processing by the previous layer, usually representing a time-frequency graph or other spatial features. K represents the convolution kernel (Kernel), which is a filter matrix with a size of k' x k', used to extract specific local features. i and j are output position indexes, representing the current convolution operation in the row and column positions of the output feature map. and are local indexes within the convolution kernel, used to traverse the rows and columns of the convolution kernel.

[0127] Output: EEG convolution feature map (n, k). After processing by the convolution layer of the convolutional neural network (CNN), the original time-frequency matrix is converted into a new feature matrix, and the dimension may change. (n, k) represents the dimension of the EEG feature matrix after CNN processing, where n still represents the number of time points (e.g. after down-sampling or up-sampling processing), and k represents the new feature dimension.

[0128] LSTM module (EEG):

[0129] Input: EEG convolution feature sequence (n, k);

[0130] Use LSTM to capture the dynamic evolution of EEG features over time, and map the sequence to the final EEG feature vector.

[0131] Output: EEG deep feature vector E (dimension e_dim);

[0132] S122: Multimodal deep feature extraction of heart rate variability signal (CNN-LSTM module):

[0133] In step S122, a CNN-LSTM model is used to perform multi-modal deep feature extraction on the time series of the heart rate variability signal to obtain a heart rate variability signal deep feature vector, including:

[0134] The time series of the heart rate variability signal is input into a one-dimensional convolutional neural network (1D-CNN) to extract local time sequence pattern features.

[0135] The extracted local time sequence pattern features are modeled using a long short-term memory network (LSTM) to output a heart rate variability signal deep feature vector.

[0136] Specifically as follows:

[0137] CNN module (HRV):

[0138] Input: HRV time series (t, 1)

[0139] Where (t, 1) represents a two-dimensional matrix (or can be regarded as a vector), in heart rate variability (HRV) signal processing, after normalization and filtering processing, the R-R interval sequence forms a one-dimensional time series, and its dimension can be represented as (t, 1); t represents the length of the time series, i.e. the number of R-R intervals.

[0140] A one-dimensional convolution kernel is used to extract HRV local time sequence features, and a max-pooling down-sampling is performed to obtain more representative feature sequences.

[0141] Optionally, the input shape is (t, 1);

[0142] A one-dimensional convolution is adopted:

[0143] ;

[0144] In the above formula, X represents the input HRV one-dimensional sequence; represents a convolution kernel (filter); i represents the current position (time step) in the sequence; u represents the element index within the convolution kernel.

[0145] The one-dimensional convolution performs weighted summation of the convolution kernel and the corresponding part of the input sequence through a sliding window, which can capture the local time sequence pattern of HRV (such as the heart rate change in a specific interval).

[0146] Batch normalization (Batch Normalization): normalization after convolution to accelerate convergence and stabilize training.

[0147] Output: HRV convolutional feature sequence (t, h). Where (t, h) also represents the dimensions of a two-dimensional matrix, where t represents the number of rows of the matrix, and h represents the number of columns of the matrix. After processing by a one-dimensional convolutional neural network (1D-CNN), the original R-R interval sequence is converted into a new feature matrix, and the dimensions may change. (t, h) represents the dimensions of the HRV feature matrix after 1D-CNN processing, where t still represents the number of time points (e.g. after downsampling or upsampling processing), and h represents the new feature dimension.

[0148] LSTM module (HRV):

[0149] Input: HRV convolutional feature sequence (t, h);

[0150] Use LSTM to capture the dynamic evolution of the HRV signal on the time axis, and map the sequence into the final HRV feature vector.

[0151] Output: HRV deep feature vector H (dimension h_dim);

[0152] Preferably, in some embodiments, the output of the CNN in step S120 is interfaced with the LSTM module as follows:

[0153] The EEG matrix after 2D-CNN has an output dimension of , which is first flattened / reshaped into , and then input into the LSTM;

[0154] The HRV sequence after 1D-CNN can be directly input into the LSTM, which has a form of

[0155] Network hyperparameters:

[0156] Number of hidden units: 128; number of layers: 2; Dropout: 0.4 for preventing overfitting; activation: Sigmoid + Tanh (default).

[0157] S130: Concatenate the electroencephalogram signal deep feature vector and the heart rate variability signal deep feature vector to form a high-dimensional multi-modal feature vector representation;

[0158] This step S130 is a multi-modal feature fusion step, which is used to concatenate the EEG feature and the HRV feature vector extracted by CNN-LSTM into a multi-modal feature vector.

[0159] Preferably, this step can specifically include:

[0160] (1) Feature vector standardization:

[0161] Before concatenation, the EEG feature vector E and the HRV feature vector H are first standardized to ensure consistency in numerical range. Standardization can adopt Z-score standardization method, that is, subtracting the mean and dividing by the standard deviation, so that each feature has zero mean and unit variance. This helps to prevent the features of a certain modality from dominating in the fusion process and affecting the performance of the classifier.

[0162] (2) Feature dimension alignment:

[0163] Check if the dimensions of the EEG feature vector E and the HRV feature vector H are consistent. If the dimensions are inconsistent, the dimensions of both need to be adjusted to the same through feature selection or dimension reduction techniques (such as principal component analysis PCA), or a dimension mapping strategy is designed to map the feature vectors of different dimensions into a common feature space.

[0164] (3) Feature vector concatenation:

[0165] The standardized and dimension-aligned EEG feature vector E and HRV feature vector H are concatenated in a predetermined order to form a multi-modal feature vector F. The concatenation method can be simple concatenation, that is, arranging all elements of E and H in a new vector in order.

[0166] In this feature vector concatenation step, the following example is given for concatenating the standardized electroencephalogram (EEG) feature vector E and heart rate variability (HRV) feature vector H into a multi-modal feature vector F. Assuming the dimension of E is e_dim and the dimension of H is h_dim, the dimension of the concatenated multi-modal feature vector F will be e_dim+h_dim. Optionally, the following steps are included in this embodiment:

[0167] 3.1 Standardization:

[0168] Standardize the EEG feature vector E:

[0169] ;

[0170] where and are the mean and standard deviation of E, respectively.

[0171] Standardize the HRV feature vector H:

[0172] ;

[0173] where and are the mean and standard deviation of H, respectively.

[0174] After the above standardization, we can get , ,

[0175] 3.2 Feature vector splicing:

[0176] The normalized EEG feature vector and the HRV feature vector are spliced into a multi-modal feature vector F:

[0177] ;

[0178] In the above formula, [;] represents the vector splicing operation, that is, all elements of and are arranged in a new vector in order.

[0179] 3.3 Spliced feature vector:

[0180] Then the spliced feature vector F is:

[0181] ;

[0182] (4) Feature fusion strategy optimization:

[0183] This embodiment can consider using more advanced feature fusion strategies, such as weighted average, feature cross or deep learning-based fusion network, to further improve the expressiveness of the fused features. For example, a fusion layer can be designed to automatically adjust the weights of EEG and HRV features in the fusion process through learning, thereby optimizing the combination effect of multi-modal features.

[0184] (5) Fusion feature verification:

[0185] Before using the multi-modal feature vector F for stress state classification, the effectiveness of the fused features is verified through cross-validation or leave-out method. The performance improvement of the fused features compared to single modal features in the classification task is evaluated to ensure the effectiveness of the fusion strategy.

[0186] The following specific embodiments are given for the training and verification of the model:

[0187] 5.1 Data division:

[0188] Divide the labeled electroencephalogram signal and heart rate variability signal data set into training set and test set. For example, use 80% of the data as the training set and 20% of the data as the test set.

[0189] 5.2 Model training:

[0190] (1) CNN-LSTM training

[0191] Step 1: Train CNN-LSTM first, then extract bottleneck layer feature vectors for SVM.

[0192] Optimizer: Adam(lr=1e -3 ~1e -4 ) or RMSProp with mini-batch training; use Earlystopping to prevent overfitting.

[0193] Batch size and number of rounds will be determined based on data volume, aiming to balance convergence stability.

[0194] (2) The training process of the SVM model includes:

[0195] Use the training set to train the SVM model, with labels as stress state (stress / non-stress);

[0196] Use multi-modal feature vector F as input to the SVM model, and stress state as target output;

[0197] Optimize the hyperparameters of the SVM model, such as kernel function type (linear kernel, radial basis function kernel, etc.), penalty parameter, etc., through grid search;

[0198] Train the SVM classifier to classify new multi-modal feature vectors.

[0199] (3) Coupling of SVM and deep learning:

[0200] After training CNN-LSTM, freeze its weights and use it to output features F for all training samples. Then use traditional machine learning pipelines (such as scikit-learn) to perform grid search and five-fold cross-validation on features + labels to select the best SVM model.

[0201] (4) Real-time or near-real-time deployment:

[0202] Offline stage: Obtain and fix the parameters of CNN-LSTM + SVM;

[0203] Online stage: Only need to input new user data after preprocessing and time-frequency conversion into the network to obtain deep features, and then quickly call the trained SVM for judgment;

[0204] Latency optimization: Reduce the number of CNN layers or kernel size, increase stride to reduce computational complexity, also use GPU acceleration, model quantization / distillation, etc.

[0205] 5.3 Model testing:

[0206] Using the multi-modal feature vector F on the test set as input, the stress state prediction is performed through the trained SVM model.

[0207] The accuracy, precision, recall, and other indicators of the prediction results are calculated to evaluate the effectiveness of the fused features.

[0208] 5.4 Performance Comparison:

[0209] The classification performance of the multi-modal feature vector F is compared with that of single-modal features (e.g., using only EEG features or only HRV features).

[0210] If the classification performance of the multi-modal feature vector F is significantly better than that of single-modal features, it indicates that the fusion strategy is effective.

[0211] 5.5 Result Analysis:

[0212] The classification performance of the fused features under different stress states and their applicability in different populations (e.g., age, gender, occupation, etc.) are analyzed.

[0213] Based on the analysis results, the fusion strategy and SVM model are further optimized to improve the accuracy and reliability of stress state prediction.

[0214] S140: Classify the multi-modal feature representation using a support vector machine (SVM) to output a stress state discrimination result.

[0215] This step S140 specifically includes:

[0216] Classify the high-dimensional multi-modal feature vector representation using a pre-trained SVM model to output a stress state prediction result, including: the SVM model finds the optimal classification hyperplane in the high-dimensional feature space to distinguish between stress states and normal states and outputs a stress state label, including "stress" or "non-stress". Of course, "stress" can also be subdivided, such as high, medium, and low stress.

[0217] Preferably, the method further includes:

[0218] Transmit the stress state discrimination result to a health management system to display the user's stress state in real time and perform long-term stress data monitoring and analysis, including:

[0219] When the real-time monitoring stress state label is "stress", the health management system automatically triggers an alarm and reminds the user to rest or relax;

[0220] The health management system monitors and analyzes the user's long-term stress data to analyze the user's stress patterns and assist in developing personalized relaxation or intervention strategies.

[0221] According to the method and system for evaluating multi-modal mental stress based on machine learning, the CNN-LSTM is used for multi-modal deep feature extraction of the EEG and HRV signals, the CNN is used for extracting local time-frequency features, the LSTM is used for modeling dynamic evolution of time series, and the SVM is used for classifying the fused multi-modal features, and the complementarity and correlation between the multi-modal signals are fully utilized, so that the accurate prediction of the real-time stress state of the user is realized.

[0222] The above is the introduction of the method embodiment, and the scheme of the present application is further described through the system embodiment.

[0223] Figure 3 A module schematic diagram of the system for evaluating multi-modal mental stress based on machine learning is shown. Figure 3 As shown in the figure, the system for evaluating multi-modal mental stress based on machine learning 200 provided by the present application comprises:

[0224] The data acquisition module 201 acquires the brain wave signal and the heart rate variability signal of the user.

[0225] The feature extraction module 202 is used for using the CNN-LSTM model to perform multi-modal deep feature extraction on the spectrogram of the brain wave signal and the time series of the heart rate variability signal respectively, to obtain a brain wave signal deep feature vector and a heart rate variability signal deep feature vector.

[0226] The feature fusion module 203 is used for splicing the brain wave signal deep feature vector and the heart rate variability signal deep feature vector to form a high-dimensional multi-modal feature vector representation.

[0227] The stress prediction module 204 is used for using the support vector machine (SVM) to classify the multi-modal feature representation, and outputting a stress state discrimination result.

[0228] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described modules can refer to the corresponding process in the foregoing method embodiments, which will not be described herein.

[0229] In summary, the present application improves the accuracy and reliability of stress detection by combining machine learning and multi-modal signal processing technology, realizes real-time stress monitoring and early warning, and provides personalized relaxation or intervention strategies, which has important significance for promoting the mental health of users and improving the quality of life.

[0230] The above has been described in detail one embodiment of the present application, but the content is only the preferred embodiment of the present application, cannot be considered for limiting the scope of the present application. Any equivalent changes and improvements made in the scope of the present application, should still belong to the scope of the claims of the present application.

Claims

1. A machine learning based multi-modal stress assessment method, characterized in that, The method comprises the following steps: Collecting the brain wave signals and heart rate variability signals of a user; Using a CNN-LSTM model to perform multi-modal deep feature extraction on the frequency spectrum of the brain wave signals and the time sequence of the heart rate variability signals, respectively, to obtain a brain wave signal deep feature vector and a heart rate variability signal deep feature vector; Wherein, using the CNN-LSTM model to perform multi-modal deep feature extraction on the frequency spectrum of the brain wave signals to obtain the brain wave signal deep feature vector comprises: Using a convolutional neural network (CNN) to perform convolution feature extraction on the frequency spectrum of the brain wave signals to obtain the local time-frequency features of the brain wave signals, comprising: Using different sizes of two-dimensional convolution kernels to extract the time-frequency features of the frequency spectrum of the brain wave signals, and reducing the feature dimension through a max-pooling layer to output a brain wave signal convolution feature map, i.e. local time-frequency features; Wherein, using a two-dimensional convolution to extract local spatial features in the time-frequency plane comprises: ； wherein: X represents an input feature map; K represents a convolution kernel, which is a filter matrix with a size of k'xk', used to extract specific local features; i and j are output position indexes, representing the row and column positions of the current convolution operation in the output feature map; and are local indexes within the convolution kernel, used to traverse the rows and columns of the convolution kernel; Using a long short-term memory network (LSTM) to model the extracted local time-frequency features to capture the dynamic patterns of the brain wave signals over time, and output a brain wave signal deep feature vector; wherein, using the CNN-LSTM model to perform multi-modal deep feature extraction on the time sequence of the heart rate variability signals to obtain a heart rate variability signal deep feature vector comprises: Inputting the time sequence of the heart rate variability signals into a one-dimensional convolutional neural network (1D-CNN) to extract local time sequence pattern features, comprising: Inputting the time sequence of the heart rate variability signals into a one-dimensional convolutional neural network (1D-CNN), and reducing the sampling through a max-pooling layer to obtain a more representative feature sequence, to obtain a heart rate variability signal convolution feature sequence, i.e. local time sequence pattern features of the heart rate variability signals; Wherein, using a one-dimensional convolution to capture the local time sequence pattern of the HRV by weighting and summing the corresponding parts of the input sequence through a sliding window: ； Wherein, X represents the input HRV one-dimensional sequence; Convolution kernel; i represents the current position in the sequence; u represents the element index within the convolution kernel; the extracted local timing mode features are sequentially modeled using a long short-term memory network (LSTM), and a heart rate variability signal deep feature vector is output. Concatenating the brain wave signal deep feature vector and the heart rate variability signal deep feature vector to form a high-dimensional multi-modal feature vector representation; Using a support vector machine (SVM) to classify the multi-modal feature representation and output a stress state discrimination result. 2.The machine learning based multi-modal mental stress assessment method according to claim 1, characterized in that, Wherein, The method further comprises: Pretreating and preliminarily extracting features from the collected brain wave signals and heart rate variability signals, comprising: After band-pass filtering and artifact removal of the collected brain wave signals, using short-time Fourier transform (STFT) or wavelet transform to obtain a time-frequency matrix of the brain wave signals, i.e. a frequency spectrum of the brain wave signals; Wherein, the short-time Fourier transform (STFT) is: ； wherein, is the original EEG signal, is a window function, and the window is moving in time, T typically denotes the time step of the window movement; is a time parameter; ω is a frequency parameter; is a complex exponential function, used to transform the signal into the frequency domain; The wavelet transform (CWT) is: ； Wherein, ψ(t) is the mother wavelet function; a is the scale parameter, which is set to range from 1 to 128 to cover the required frequency range; b is the translation parameter, which determines the position of the wavelet function on the time axis; Obtaining the R-R interval sequence of the collected heart rate variability signals, and after normalization and filtering processing of the R-R interval sequence, forming a one-dimensional time sequence, comprising: artifact removal and correction of the R-R interval sequence, specifically comprising: Identify abnormal R-R intervals using statistical methods: ； : ith R-R interval; : mean of R-R intervals; : standard deviation of R-R intervals; : threshold factor, set to 3 specifically; Smooth the outliers using a spline function written in Python, maintaining the continuity and smoothness of the signal. 3.The machine learning based multi-modal mental stress assessment method of claim 1, wherein, Among them, The EEG signal deep feature vector and the heart rate variability signal deep feature vector are spliced to form a high-dimensional multi-modal feature vector representation, which includes: The EEG signal deep feature vector and the heart rate variability signal deep feature vector are respectively standardized and feature dimension aligned; The standardized and dimension-aligned EEG signal deep feature vector and heart rate variability signal deep feature vector are spliced in a predetermined order to form a multi-modal feature vector. 4.The machine learning based multi-modal mental stress assessment method according to claim 1, wherein, The use of support vector machine (SVM) for multi-modal feature representation classification outputs stress state discrimination results, including: Using a pre-trained SVM model to classify the high-dimensional multi-modal feature vector representation outputs the stress state prediction result, including: the SVM model finds the optimal classification hyperplane in the high-dimensional feature space to distinguish the stress state from the normal state and output the stress state label, the stress state label includes "stress" or "non-stress".

5. The machine learning based multi-modal stress assessment method according to claim 4, characterized in that, The method further includes: The stress state discrimination result is transmitted to the health management system, which displays the user's stress state in real time and monitors and analyzes long-term stress data, including: When the real-time monitoring stress state label is "stress", the health management system automatically triggers an alarm and reminds the user to rest or relax; The health management system monitors and analyzes the user's long-term stress data to analyze the user's stress pattern and assist in developing personalized relaxation or intervention strategies.

6. A machine learning based multi-modal stress assessment system, characterized in that, Including: Data acquisition module, collecting user's EEG signal and heart rate variability signal; Feature extraction module, for using CNN-LSTM model to extract multi-modal deep features from the frequency spectrum of the EEG signal and the time series of the heart rate variability signal, respectively, to obtain the EEG signal deep feature vector and the heart rate variability signal deep feature vector; Among them, using CNN-LSTM model to extract multi-modal deep features from the frequency spectrum of the EEG signal to obtain the EEG signal deep feature vector, including: Using a convolutional neural network (CNN) to extract convolutional features from the frequency spectrum of the EEG signal to obtain the local time-frequency features of the EEG signal, including: Using different sizes of two-dimensional convolution kernels to extract the time-frequency features of the frequency spectrum of the EEG signal, and reducing the feature dimension through the max-pooling layer to output the EEG signal convolutional feature map, i.e. local time-frequency features; Among them, using two-dimensional convolution to extract local spatial features in the time-frequency plane includes: ； wherein: X represents an input feature map; K represents a convolution kernel, which is a filter matrix with a size of k'xk', used to extract specific local features; i and j are output position indexes, representing the row and column positions of the current convolution operation in the output feature map; and are local indexes within the convolution kernel, used to traverse the rows and columns of the convolution kernel; Using a long short-term memory network (LSTM) to model the extracted local time-frequency features to capture the dynamic patterns of the EEG signal over time, outputting the EEG signal deep feature vector; Among them, using CNN-LSTM model to extract multi-modal deep features from the time series of the heart rate variability signal to obtain the heart rate variability signal deep feature vector, including: The time sequence of the heart rate variability signal is input into a one-dimensional convolutional neural network (1D-CNN) to extract local time sequence pattern features, including: The time sequence of the heart rate variability signal is input into a one-dimensional convolutional neural network (1D-CNN), and a more representative feature sequence is obtained through maximum pooling downsampling, to obtain a convolutional feature sequence of the heart rate variability signal, i.e., local time sequence pattern features of the heart rate variability signal; Wherein, a one-dimensional convolution is used to capture the local time sequence pattern of the HRV by weighting and summing the corresponding part of the convolution kernel and the input sequence through a sliding window: ； wherein X represents the input HRV one-dimensional sequence; denotes the convolution kernel; i denotes the current position in the sequence; u denotes the element index within the convolution kernel; A long short-term memory network (LSTM) is used to model the extracted local time sequence pattern features in time sequence, and a deep feature vector of the heart rate variability signal is output; The feature fusion module is used to splice the deep feature vector of the brain wave signal and the deep feature vector of the heart rate variability signal to form a high-dimensional multi-modal feature vector representation; The stress prediction module is used to classify the multi-modal feature representation using a support vector machine (SVM) to output a stress state discrimination result.

Images