Multi-modal mental stress assessment method and system based on machine learning

Abstract

The invention relates to the field of biological signal processing and emotion calculation, and discloses a multi-mode mental stress assessment method and system based on machine learning, and the method comprises the steps: collecting a brain wave signal and a heart rate variability signal of a user; using a CNN-LSTM model to perform multi-modal depth feature extraction on the spectrogram of the brain wave signal and the time sequence of the heart rate variability signal to obtain a brain wave signal depth feature vector and a heart rate variability signal depth feature vector; splicing the brain wave signal depth feature vector and the heart rate variability signal depth feature vector to form a high-dimensional multi-modal feature vector representation; and classifying the multi-modal feature representation by using a support vector machine, and outputting a pressure state judgment result. The real-time pressure state of the user can be accurately predicted, and the technical problem that an existing pressure detection method is not high in detection precision is solved.

Description

Technical Field

[0001] The present invention relates to the fields of biological signal processing and emotional computing, and in particular to a multimodal mental stress assessment method and system based on machine learning. Background Art

[0002] Electroencephalogram (EEG) signals and heart rate variability (HRV) signals are two physiological indicators widely used to assess an individual's psychological state and stress level. EEG signals reflect the electrical activity of brain neurons. By measuring the changes in electrical potential on the scalp, the activity patterns of the brain under different psychological states can be captured. HRV signals reflect the activity of the autonomic nervous system by measuring the changes in heartbeat intervals, which is closely related to an individual's stress level, emotional state, etc.

[0003] Traditional stress detection methods mainly rely on manually extracted features of EEG and HRV signals, such as the power of a specific frequency band of EEG, the time domain and frequency domain features of HRV, etc. However, these methods have obvious limitations. First, a single feature often cannot fully reflect the complex spatiotemporal characteristics of EEG and HRV signals, because both signals contain rich time-frequency information and dynamic change patterns. Secondly, simply fusing these features cannot effectively capture the nonlinear relationship between EEG and HRV signals, thereby limiting the accuracy and reliability of stress detection. Most of the existing machine learning-based stress detection methods only focus on signals of a single modality (such as using only EEG or HRV), or simply fuse multimodal features, failing to fully utilize the complementarity and correlation between multimodal signals. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention aims to provide a multimodal mental stress assessment method and system based on machine learning, specifically to a user stress state detection method and system combining a convolutional neural network (CNN), a long short-term memory network (LSTM) and a support vector machine (SVM). CNN-LSTM is used to perform multimodal deep feature extraction on EEG and HRV signals, and then SVM is used to discriminate and classify the fused features, thereby achieving accurate prediction of the user's real-time stress state, solving the technical problem that the existing stress detection method cannot fully reflect the complex spatiotemporal characteristics and nonlinear relationship of EEG and HRV signals, resulting in low detection accuracy.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] In a first aspect, the present invention provides a multimodal mental stress assessment method based on machine learning, comprising:

[0007] Collect the user's brain wave signals and heart rate variability signals;

[0008] Using a CNN-LSTM model, multimodal deep feature extraction is performed on the spectrum of the brain wave signal and the time series of the heart rate variability signal to obtain a deep feature vector of the brain wave signal and a deep feature vector of the heart rate variability signal;

[0009] Concatenating the deep feature vector of the brain wave signal and the deep feature vector of the heart rate variability signal to form a high-dimensional multimodal feature vector representation;

[0010] Support vector machine (SVM) is used to classify the multimodal feature representation and output the pressure state discrimination result.

[0011] Preferably, the method further comprises:

[0012] Preprocessing and preliminary feature extraction of collected EEG signals and heart rate variability signals, including:

[0013] After the collected EEG signals are bandpass filtered and artifacts removed, short-time Fourier transform (STFT) or wavelet transform is used to obtain the time-frequency matrix of the EEG signals, i.e., the spectrum of the EEG signals;

[0014] The RR interval sequence of the collected heart rate variability signal is obtained, and the RR interval sequence is normalized and filtered to form a one-dimensional time series.

[0015] Preferably, the CNN-LSTM model is used to perform multimodal deep feature extraction on the spectrum of the brain wave signal to obtain a deep feature vector of the brain wave signal, including:

[0016] Using a convolutional neural network (CNN) to perform convolution feature extraction on the spectrum of the brain wave signal to obtain local time-frequency features of the brain wave signal;

[0017] The extracted local time-frequency features are modeled using a long short-term memory network (LSTM) to capture the dynamic pattern of brain wave signals changing over time and output a deep feature vector of the brain wave signal.

[0018] Preferably, the CNN-LSTM model is used to perform multimodal deep feature extraction on the time series of the heart rate variability signal to obtain a deep feature vector of the heart rate variability signal, including:

[0019] The time series of heart rate variability signals is input into a one-dimensional convolutional neural network (1D-CNN) to extract local temporal pattern features;

[0020] A long short-term memory network (LSTM) is used to perform time series modeling on the extracted local time series pattern features, and a deep feature vector of the heart rate variability signal is output.

[0021] Preferably, the step of using a convolutional neural network (CNN) to perform convolution feature extraction on the spectrum of the brain wave signal to obtain the local time-frequency features of the brain wave signal includes:

[0022] The time-frequency features of the spectrum graph of the EEG signal are extracted using two-dimensional convolution kernels of different sizes, and the feature dimension is reduced through a maximum pooling layer to output a convolution feature graph of the EEG signal, i.e., a local time-frequency feature.

[0023] Preferably, the step of inputting the time series of the heart rate variability signal into a one-dimensional convolutional neural network (1D-CNN) to extract local temporal pattern features comprises:

[0024] The time series of the heart rate variability signal is input into a one-dimensional convolutional neural network (1D-CNN), and a more representative feature sequence is obtained through maximum pooling downsampling, thereby obtaining the convolution feature sequence of the heart rate variability signal, that is, the local temporal pattern feature of the heart rate variability signal.

[0025] Preferably, the step of concatenating the deep feature vector of the brain wave signal and the deep feature vector of the heart rate variability signal to form a high-dimensional multimodal feature vector representation comprises:

[0026] Respectively performing standardization processing and feature dimension alignment on the deep feature vector of the brain wave signal and the deep feature vector of the heart rate variability signal;

[0027] The standardized and dimensionally aligned deep feature vectors of the EEG signal and the deep feature vectors of the heart rate variability signal are concatenated in a predetermined order to form a multimodal feature vector.

[0028] Preferably, the method of using a support vector machine (SVM) to classify the multimodal feature representation and outputting a pressure state discrimination result includes:

[0029] The high-dimensional multimodal feature vector representation is classified using a pre-trained SVM model to output a stress state prediction result, including: the SVM model distinguishes the stress state from the normal state and outputs a stress state label by finding an optimal classification hyperplane in the high-dimensional feature space, and the stress state label includes "stress" or "non-stress".

[0030] Preferably, the method further comprises:

[0031] The stress state identification result is transmitted to the health management system to display the user's stress state in real time and conduct long-term stress data monitoring and analysis, including:

[0032] When the stress status label is detected as "stress" in real time, the health management system automatically triggers an alarm and reminds the user to rest or relax;

[0033] The health management system monitors and analyzes the user's long-term stress data, analyzes the user's stress pattern, and assists in formulating personalized relaxation or intervention strategies.

[0034] In a second aspect, the present invention further provides a multimodal mental stress assessment system based on machine learning, comprising:

[0035] Data collection module, collecting the user's brain wave signals and heart rate variability signals;

[0036] A feature extraction module, used to perform multimodal deep feature extraction on the spectrum of the brain wave signal and the time series of the heart rate variability signal using a CNN-LSTM model, to obtain a deep feature vector of the brain wave signal and a deep feature vector of the heart rate variability signal;

[0037] A feature fusion module, used for concatenating the deep feature vector of the EEG signal and the deep feature vector of the heart rate variability signal to form a high-dimensional multimodal feature vector representation;

[0038] The pressure prediction module is used to classify the multimodal feature representation using a support vector machine (SVM) and output the pressure state discrimination result.

[0039] Compared with the prior art, the present invention achieves the following beneficial effects:

[0040] (1) By combining convolutional neural networks (CNN), long short-term memory networks (LSTM) and support vector machines (SVM), the present invention can more comprehensively capture the complex spatiotemporal characteristics and nonlinear relationships of EEG and HRV signals. CNN is used to extract local time-frequency features, LSTM is used to model the dynamic evolution of time series, and SVM is used to classify the fused multimodal features, thereby achieving accurate prediction of the user's real-time stress state and improving the accuracy and reliability of stress detection.

[0041] (2) Compared with traditional stress detection methods, which often only focus on single-modal signals or simply fuse multimodal features, the present invention uses a CNN-LSTM model to perform deep feature extraction on EEG and HRV signals, and concatenates the extracted feature vectors to form a high-dimensional multimodal feature vector representation, thereby more effectively utilizing the complementarity and correlation of multimodal signals.

[0042] (3) The present invention transmits the stress state identification result to the health management system, which can display the user's stress state in real time and conduct long-term stress data monitoring and analysis. When the stress state is monitored in real time, the health management system can automatically trigger an alarm and remind the user to rest or relax, thereby helping the user to manage his or her stress state in a timely manner.

[0043] (4) By monitoring and analyzing users’ stress data over a long period of time, the health management system can analyze users’ stress patterns and assist in developing personalized relaxation or intervention strategies to better meet users’ personalized needs.

[0044] In summary, the present invention improves the accuracy and reliability of stress detection by combining machine learning and multimodal signal processing technology, realizes real-time stress monitoring and early warning, and provides personalized relaxation or intervention strategies, which is of great significance for promoting users' mental health and improving their quality of life. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0046] Figure 1 is a flowchart of a multimodal mental stress assessment method based on machine learning provided by an embodiment of the present invention;

[0047] Figure 2 It is a schematic diagram of the EEG and HRV data processing and feature extraction process of a multimodal mental stress assessment method based on machine learning in an embodiment of the present invention;

[0048] Figure 3 It is a schematic diagram of the overall framework of a CNN-LSTM and SVM model combined in a multimodal mental stress assessment method based on machine learning in an embodiment of the present invention;

[0049] Figure 4 This is a unit module diagram of a multimodal mental stress assessment system based on machine learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0052] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0053] The present invention provides a multimodal mental stress assessment method and system based on machine learning, which is a user stress state detection method and system combining a convolutional neural network (CNN), a long short-term memory network (LSTM) and a support vector machine (SVM). CNN-LSTM is used to perform multimodal deep feature extraction on EEG and HRV signals, and then SVM is used to discriminate and classify the fused features, thereby achieving accurate prediction of the user's real-time stress state, and solving the technical problem that the existing stress detection method cannot fully reflect the complex spatiotemporal characteristics and nonlinear relationship of EEG and HRV signals, resulting in low detection accuracy.

[0054] Figure 1 This is a schematic diagram of the process of a multimodal mental stress assessment method based on machine learning provided by an embodiment of the present invention. Figure 1 The present invention provides a multimodal mental stress assessment method based on machine learning, comprising:

[0055] S110: Collecting brain wave signals and heart rate variability signals of the user;

[0056] In this step S110, for example, EEG signals are collected by an electroencephalogram device, and heart rate signals are collected by an electrocardiogram or PPG device.

[0057] Preferably, this step also includes: preprocessing and preliminary feature extraction of the collected brain wave signals and heart rate variability signals, specifically including:

[0058] (I) EEG data: After bandpass filtering, artifact removal and correction of the signal, the short-time Fourier transform (STFT) or wavelet transform is used to obtain a (n, m) time-frequency matrix. In EEG signal processing, the EEG signal is converted from the time domain to the frequency domain using the short-time Fourier transform (STFT) or wavelet transform, resulting in a time-frequency matrix. (n, m) represents the dimension of a two-dimensional matrix, where each row of the matrix represents the frequency distribution at a time point, and each column represents the time change of a frequency.

[0059] (1) Bandpass filtering;

[0060] A. Purpose: To remove noise and unnecessary frequency components from the signal to ensure the accuracy of subsequent feature extraction and analysis.

[0061] B. Bandpass filter:

[0062] A bandpass filter allows signals within a specific frequency range to pass while rejecting frequency components below and above that range.

[0063] Using Butterworth filter, its transfer function is:

[0064] ;

[0065] in: represents the amplitude response of the filter at frequency f; represents frequency variable; represents the cut-off frequency; Represents the order of the filter, which determines the steepness of the filter.

[0066] C. Filter parameter settings:

[0067] EEG filter parameters: Bandpass range: 0.5Hz to 50Hz; Filter order: 4.

[0068] HRV filter parameters: Bandpass range: 0.5Hz to 40Hz; Filter order: 4.

[0069] D. Normalized cutoff frequency:

[0070] ;

[0071] in, represents the normalized cutoff frequency; Indicates the actual cutoff frequency represents the sampling rate, is the Nyquist frequency.

[0072] Use Python to calculate the denominator coefficient and numerator coefficient of the filter, and substitute them into the bandpass filter to get the filtered signal.

[0073] (2) EEG artifact removal and correction;

[0074] EEG signals are often interfered by a variety of physiological and non-physiological artifacts, such as eye movements, blinking, electromyography, etc. These artifacts will affect the accuracy of subsequent feature extraction and classification, and need to be effectively identified and removed.

[0075] Independent component analysis (ICA): It is used to decompose multivariate signals into independent components, which can separate artifacts in EEG signals from EEG signals.

[0076] ;

[0077] in, Represents the observation signal matrix, with a shape of (p×q), where p is the number of channels and q is the number of time steps. Represents the mixing matrix, which describes how independent components are mixed into the observed signal and has a shape of (p ×p). Represents the independent component matrix, with a shape of (p×q), that is, the source signal matrix.

[0078] Apply ICA decomposition: the preprocessed EEG signal Decompose into independent components and the mixing matrix , by observing the temporal waveforms and spectra of independent components, components representing artifacts are identified and removed.

[0079] Transient artifact detection: Transient artifacts, such as eye blinks and momentary electromyographic interference, are detected and corrected through statistical analysis.

[0080] Artifact detection formula:

[0081] if , then it is believed that It is a false trace.

[0082] in: : No. Channels at time The signal value at ; : Threshold factor, which is set to 3 in the present invention; : standard deviation of the signal.

[0083] After finding the artifacts, use the interpolation method to replace the outliers with interpolated values ​​of the previous and subsequent valid signals.

[0084] ;

[0085] Represents the signal value of the i-th channel at time t. In other words, it is the amplitude or measurement value of the EEG signal at a specific time point t and on a specific channel i; represents the signal value of the i-th channel at time t-1, Represents the signal value of the i-th channel at time t+1; in continuous signal recording, t-1 is the time point before t, and t+1 is the time point after t.

[0086] Specifically, if It is judged as an artifact (i.e. its amplitude exceeds the value determined by the threshold factor k and the signal standard deviation The threshold value determined by the The interpolation method helps to reduce the impact of artifacts on signal analysis, thereby improving the accuracy of subsequent feature extraction and classification.

[0087] (3) Time-frequency conversion;

[0088] In order to further extract the time-frequency features in the EEG signal, the present invention adopts a time-frequency conversion module to convert the purified and corrected EEG signal from the time domain to the time-frequency domain. The specific implementation includes:

[0089] Short Time Fourier Transform (STFT):

[0090] Window function selection: Select Hanning window as the window function to reduce spectrum leakage.

[0091] Window length and overlap: Set the window length to 256 samples and the window overlap to 128 samples to achieve a 50% overlap.

[0092] STFT calculation:

[0093] ;

[0094] in, is the original EEG signal, is a window function, and in the present invention, a Hanning window can be selected, and the window moves over time, and T usually represents the time step of the window movement; is the time parameter; ω is the frequency parameter; is a complex exponential function used to transform the signal into the frequency domain.

[0095] Wavelet Transform (CWT):

[0096] Mother wavelet function selection: Morlet wavelet is selected as the mother wavelet function because it has good localization characteristics in the time-frequency domain.

[0097] Scale parameter setting: Set the scale parameter range from 1 to 128 to cover the desired frequency range.

[0098] CWT calculation:

[0099] ;

[0100] Among them, x(t) is the original EEG signal; ψ(t) is the mother wavelet function; a is the scale parameter, which determines the width (or frequency component) of the wavelet function. The range of scale parameter a is set from 1 to 128 to cover the required frequency range; b is the translation parameter, which determines the position of the wavelet function on the time axis.

[0101] Technical advantages of time-frequency conversion for EEG signals in the present invention:

[0102] Multi-resolution analysis: Wavelet transform provides multi-resolution analysis capabilities, which can capture the frequency components of the signal at different time scales and adapt to the non-stationary nature of EEG signals.

[0103] Dynamic feature capture: STFT and wavelet transform can capture the dynamic characteristics of the frequency components in the EEG signal changing over time, providing richer feature information.

[0104] Improve feature extraction effect: The signal after time-frequency conversion has higher information content and better feature separation ability, which helps to improve the feature extraction and classification performance of subsequent deep learning models (CNN-LSTM).

[0105] (ii) Heart rate variability (HRV) data: Obtain the RR interval sequence, and form a one-dimensional time series after proper normalization and filtering of the sequence. Among them, the RR interval refers to the time interval between two adjacent R waves in the electrocardiogram (ECG), which is the basis of heart rate variability (HRV) analysis. Heart rate variability (HRV) is an important indicator for evaluating the activity of the autonomic nervous system. By measuring the changes in the RR interval, the rhythm changes of the heart can be analyzed, and then the physiological and psychological state of the individual can be understood. The RR interval sequence is a one-dimensional time series, in which each element represents the time length of an RR interval.

[0106] (1) Artifact removal and correction of HRV signals:

[0107] HRV signals are mainly analyzed based on RR interval sequences, but abnormal values ​​(such as jump errors, missed detections, etc.) may appear during the detection of RR intervals. Therefore, it is necessary to remove artifacts and correct the RR interval sequence.

[0108] Identify abnormal RR intervals using statistical methods:

[0109] ;

[0110] : the i-th RR interval; : the mean of the RR interval; : standard deviation of RR interval; : Threshold factor, which is set to 3 in the present invention;

[0111] Abnormal RR interval correction: Use the spline function written in Python to smoothly interpolate the outliers and maintain the continuity and smoothness of the signal.

[0112] S120: using a CNN-LSTM model to perform multimodal deep feature extraction on the spectrum of the brain wave signal and the time series of the heart rate variability signal, respectively, to obtain a deep feature vector of the brain wave signal and a deep feature vector of the heart rate variability signal;

[0113] The step S120 specifically includes the following steps:

[0114] S121: Multimodal deep feature extraction of EEG signals (CNN-LSTM module):

[0115] In step S121, a CNN-LSTM model is used to perform multimodal deep feature extraction on the spectrum of the brain wave signal to obtain a deep feature vector of the brain wave signal, including:

[0116] Using a convolutional neural network (CNN) to perform convolution feature extraction on the spectrum of the brain wave signal to obtain local time-frequency features of the brain wave signal;

[0117] The extracted local time-frequency features are modeled using a long short-term memory network (LSTM), capturing the dynamic pattern of brain wave signals changing over time and outputting a deep feature vector of the brain wave signal. The details are as follows:

[0118] The structure of the CNN module (convolution + pooling + BN):

[0119] Input: EEG time-frequency matrix (n, m);

[0120] The time-frequency features of EEG are extracted using two-dimensional convolution kernels of different sizes, and the feature dimensions are reduced through a maximum pooling layer.

[0121] Input shape: (T,G,1)(T,G,1)(T,G,1) or (T,G)(T,G)(T,G),

[0122] Where T is the time step (time slice or STFT frame number) and G is the number of frequency channels (obtained after wavelet / Fourier transform).

[0123] Using 2D convolution:

[0124] ;

[0125] Used to extract local spatial features in the time-frequency plane.

[0126] Where: X represents the input feature map, which is a two-dimensional matrix after processing by the previous layer, usually representing a time-frequency map or other spatial features. K represents the convolution kernel, which is a filter matrix of size k'×k', used to extract specific local features. i and j are output position indices, indicating the row and column positions of the current convolution operation in the output feature map. and It is a local index within the convolution kernel, used to traverse the rows and columns of the convolution kernel.

[0127] Output: EEG convolution feature map (n, k). After being processed by the convolutional layer of the convolutional neural network (CNN), the original time-frequency matrix will be converted into a new feature matrix, and its dimension may change. (n, k) represents the dimension of the EEG feature matrix after CNN processing, where n still represents the number of time points (for example, after downsampling or upsampling), and k represents the new feature dimension.

[0128] LSTM module (EEG):

[0129] Input: EEG convolution feature sequence (n, k);

[0130] LSTM is used to capture the dynamic evolution of EEG features over time and map the sequence into the final EEG feature vector.

[0131] Output: EEG deep feature vector E (dimension is e_dim);

[0132] S122: Multimodal deep feature extraction of heart rate variability signals (CNN-LSTM module):

[0133] In step S122, a CNN-LSTM model is used to perform multimodal deep feature extraction on the time series of the heart rate variability signal to obtain a deep feature vector of the heart rate variability signal, including:

[0134] The time series of heart rate variability signals is input into a one-dimensional convolutional neural network (1D-CNN) to extract local temporal pattern features;

[0135] A long short-term memory network (LSTM) is used to perform time series modeling on the extracted local time series pattern features, and a deep feature vector of the heart rate variability signal is output.

[0136] The details are as follows:

[0137] CNN module (HRV):

[0138] Input: HRV time series (t,1)

[0139] Among them, (t,1) represents a two-dimensional matrix (or can be regarded as a vector). In the heart rate variability (HRV) signal processing, the RR interval sequence is normalized and filtered to form a one-dimensional time series, whose dimension can be expressed as (t, 1); t represents the length of the time series, that is, the number of RR intervals.

[0140] A one-dimensional convolution kernel is used to extract the local temporal features of HRV, and a more representative feature sequence is obtained through maximum pooling and downsampling.

[0141] Optional, input shape: (t, 1);

[0142] Using one-dimensional convolution:

[0143] ;

[0144] In the above formula, X represents the input HRV one-dimensional sequence; represents the convolution kernel (filter); i represents the current position (time step) in the sequence; u represents the element index within the convolution kernel.

[0145] One-dimensional convolution uses a sliding window to perform weighted summation of the convolution kernel and the corresponding part of the input sequence, which can capture the local timing patterns of HRV (such as changes in heart rate in a specific interval).

[0146] Batch Normalization: Normalization is performed after convolution to accelerate convergence and stabilize training.

[0147] Output: HRV convolution feature sequence (t, h). (t, h) also represents the dimension of a two-dimensional matrix, where t represents the number of rows and h represents the number of columns. After being processed by a one-dimensional convolutional neural network (1D-CNN), the original RR interval sequence will be converted into a new feature matrix, and its dimension may change. (t, h) represents the dimension of the HRV feature matrix after 1D-CNN processing, where t still represents the number of time points (for example, after downsampling or upsampling), and h represents the new feature dimension.

[0148] LSTM module (HRV):

[0149] Input: HRV convolution feature sequence (t, h);

[0150] LSTM is used to capture the dynamic evolution of HRV signals on the time axis and map the sequence into the final HRV feature vector.

[0151] Output: HRV deep feature vector H (dimension is h_dim);

[0152] Preferably, in some embodiments, in step S120, the interface between the output of the CNN and the LSTM module is:

[0153] After the EEG matrix of 2D-CNN, the output dimension is as follows , first Flatten / Reshape , then input LSTM;

[0154] The HRV sequence after 1D-CNN can be directly used as LSTM input, as follows

[0155] Network hyperparameters:

[0156] Number of hidden units: 128; Number of layers: 2; Dropout: 0.4 to prevent overfitting; Activation: Sigmoid + Tanh (default).

[0157] S130: concatenating the deep feature vector of the brain wave signal and the deep feature vector of the heart rate variability signal to form a high-dimensional multimodal feature vector representation;

[0158] This step S130 is a multimodal feature fusion step, which is used to splice the EEG features extracted by CNN-LSTM and the HRV feature vector into a multimodal feature vector.

[0159] Preferably, this step may specifically include:

[0160] (1) Feature vector normalization:

[0161] Before concatenation, the EEG feature vector E and the HRV feature vector H are first standardized to ensure consistency in their numerical ranges. Standardization can be done using the Z-score standardization method, which is to subtract the mean and divide by the standard deviation so that each feature has zero mean and unit variance. This helps prevent the features of a certain modality from dominating the fusion process and affecting the performance of the classifier.

[0162] (2) Feature dimension alignment:

[0163] Check whether the dimensions of the EEG feature vector E and the HRV feature vector H are consistent. If the dimensions are inconsistent, it is necessary to adjust the dimensions of the two to the same through feature selection or dimensionality reduction techniques (such as principal component analysis PCA), or design a dimensional mapping strategy to map feature vectors of different dimensions into a common feature space.

[0164] (3) Feature vector concatenation:

[0165] The standardized and dimensionally aligned EEG feature vector E and HRV feature vector H are concatenated in a predetermined order to form a multimodal feature vector F. The concatenation method can be a simple concatenation, that is, all elements of E and H are arranged in sequence in a new vector.

[0166] In the feature vector concatenation step, the following example is specifically given for concatenating the normalized electroencephalogram (EEG) feature vector E and the heart rate variability (HRV) feature vector H into a multimodal feature vector F. Assuming that the dimension of E is e_dim and the dimension of H is h_dim, the dimension of the concatenated multimodal feature vector F will be e_dim+h_dim. Optionally, the following steps are included in this embodiment:

[0167] 3.1 Standardization processing:

[0168] Normalize the EEG feature vector E:

[0169] ;

[0170] in, and are the mean and standard deviation of E respectively.

[0171] Normalize the HRV feature vector H:

[0172] ;

[0173] in, and are the mean and standard deviation of H respectively.

[0174] After the above standardization, we can get: , ,

[0175] 3.2 Feature vector concatenation:

[0176] The standardized EEG feature vector and HRV feature vector Concatenate into a multimodal feature vector F:

[0177] ;

[0178] In the above formula, [;] represents the vector concatenation operation, that is, and All elements of are arranged sequentially in a new vector.

[0179] 3.3 Feature vector after concatenation:

[0180] Then the concatenated feature vector F is:

[0181] ;

[0182] (4) Feature fusion strategy optimization:

[0183] This embodiment may consider adopting more advanced feature fusion strategies, such as weighted averaging, feature crossover, or a fusion network based on deep learning, to further enhance the expressiveness of the fusion features. For example, a fusion layer may be designed to automatically adjust the weights of EEG and HRV features in the fusion process through learning, thereby optimizing the combination effect of multimodal features.

[0184] (5) Fusion feature verification:

[0185] Before using the multimodal feature vector F for stress state classification, verify the effectiveness of the fused features by cross-validation or holdout method, etc. Evaluate the performance improvement of the fused features over the single modality features in the classification task to ensure the effectiveness of the fusion strategy.

[0186] The following is a specific example of model training and verification:

[0187] 5.1 Data Division:

[0188] The labeled EEG signal and heart rate variability signal datasets are divided into a training set and a test set. For example, 80% of the data is used as a training set and 20% of the data is used as a test set.

[0189] 5.2 Model training:

[0190] (1) CNN-LSTM training

[0191] Step by step: first train only CNN-LSTM, then extract the bottleneck layer feature vector to SVM.

[0192] Optimizer: Adam(lr=1e -3 ~1e -4 ) or RMSProp, combined with mini-batch training; use Earlystopping to prevent overfitting.

[0193] The batch size and number of rounds will depend on the amount of data, with the goal of balancing the basis of convergence stability.

[0194] (2) The training process of the SVM model includes:

[0195] Use the training set with labels as stress states (stress / non-stress) to train the SVM model;

[0196] The multimodal feature vector F is used as the input of the SVM model, and the pressure state is used as the target output;

[0197] Optimize the hyperparameters of the SVM model through grid search, such as kernel function type (linear kernel, radial basis function kernel, etc.), penalty parameters, etc.

[0198] The trained SVM classifier is used to classify new multimodal feature vectors.

[0199] (3) Coupling of SVM and deep learning:

[0200] After CNN-LSTM is trained, its weights are frozen and used to output feature F for all training samples. Then, a traditional machine learning pipeline (such as scikit-learn) is used to perform grid search and five-fold cross validation on the features + labels to select the best SVM model.

[0201] (4) Real-time or near real-time deployment:

[0202] Offline stage: obtain and fix the parameters of CNN-LSTM + SVM;

[0203] Online stage: New user data only needs to be pre-processed and time-frequency converted before being input into the network to obtain deep features, and then the trained SVM is quickly called for judgment;

[0204] Latency optimization: Reduce the number of CNN layers or kernel size, increase stride to reduce the amount of computation, or use GPU acceleration, model quantization / distillation and other methods.

[0205] 5.3 Model testing:

[0206] The multimodal feature vector F on the test set is used as input to predict the pressure state through the trained SVM model.

[0207] Calculate the accuracy, precision, recall and other indicators of the prediction results to evaluate the effectiveness of the fusion features.

[0208] 5.4 Performance comparison:

[0209] The classification performance of the multimodal feature vector F is compared with that of a single modality feature (e.g., using only EEG features or only HRV features).

[0210] If the classification performance of the multimodal feature vector F is significantly better than that of the single modality feature, it means that the fusion strategy is effective.

[0211] 5.5 Result analysis:

[0212] Analyze the classification performance of fusion features under different stress states and their applicability in different populations (such as age, gender, occupation, etc.).

[0213] According to the analysis results, the fusion strategy and SVM model are further optimized to improve the accuracy and reliability of stress state prediction.

[0214] S140: Use a support vector machine (SVM) to classify the multimodal feature representation and output a pressure state discrimination result.

[0215] This step S140 specifically includes:

[0216] The high-dimensional multimodal feature vector representation is classified using a pre-trained SVM model to output a stress state prediction result, including: the SVM model distinguishes the stress state from the normal state and outputs a stress state label by finding an optimal classification hyperplane in the high-dimensional feature space, and the stress state label includes "stress" or "non-stress". Of course, "stress" can also be subdivided, for example, stress is divided into high, medium, and low.

[0217] Preferably, the method further comprises:

[0218] The stress state identification result is transmitted to the health management system to display the user's stress state in real time and conduct long-term stress data monitoring and analysis, including:

[0219] When the stress status label is detected as "stress" in real time, the health management system automatically triggers an alarm and reminds the user to rest or relax;

[0220] The health management system monitors and analyzes the user's long-term stress data, analyzes the user's stress pattern, and assists in formulating personalized relaxation or intervention strategies.

[0221] According to an embodiment of the present invention, a multimodal mental stress assessment method and system based on machine learning utilizes CNN-LSTM to perform multimodal deep feature extraction on EEG and HRV signals, CNN is used to extract local time-frequency features, LSTM is used to model the dynamic evolution of time series, and SVM is used to classify the fused multimodal features. The complementarity and correlation between multimodal signals are fully utilized, thereby achieving accurate prediction of the user's real-time stress state.

[0222] The above is an introduction to the method embodiment. The following is a further explanation of the solution of the present invention through a system embodiment.

[0223] Figure 3 FIG. 2 shows a module schematic diagram of a multimodal mental stress assessment system based on machine learning according to an embodiment of the present invention. Figure 3 As shown, the present invention provides a multimodal mental stress assessment system 200 based on machine learning, comprising:

[0224] The data collection module 201 collects the user's brain wave signals and heart rate variability signals;

[0225] A feature extraction module 202 is used to perform multimodal deep feature extraction on the spectrum of the brain wave signal and the time series of the heart rate variability signal using a CNN-LSTM model to obtain a deep feature vector of the brain wave signal and a deep feature vector of the heart rate variability signal;

[0226] A feature fusion module 203 is used to concatenate the deep feature vector of the brain wave signal and the deep feature vector of the heart rate variability signal to form a high-dimensional multimodal feature vector representation;

[0227] The pressure prediction module 204 is used to classify the multimodal feature representation using a support vector machine (SVM) and output a pressure state discrimination result.

[0228] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0229] In summary, the present invention improves the accuracy and reliability of stress detection by combining machine learning and multimodal signal processing technology, realizes real-time stress monitoring and early warning, and provides personalized relaxation or intervention strategies, which is of great significance for promoting users' mental health and improving their quality of life.

[0230] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made according to the scope of application of the present invention should still fall within the scope of the claims of the present invention.

Claims

1. A multimodal mental stress assessment method based on machine learning, characterized in that: include: Collect the user's brain wave signals and heart rate variability signals; Using a CNN-LSTM model, multimodal deep feature extraction is performed on the spectrum of the brain wave signal and the time series of the heart rate variability signal to obtain a deep feature vector of the brain wave signal and a deep feature vector of the heart rate variability signal; Concatenating the deep feature vector of the brain wave signal and the deep feature vector of the heart rate variability signal to form a high-dimensional multimodal feature vector representation; Support vector machine (SVM) is used to classify the multimodal feature representation and output the pressure state discrimination result.

2. The multimodal mental stress assessment method based on machine learning according to claim 1 is characterized in that: in, The method further comprises: Preprocessing and preliminary feature extraction of collected EEG signals and heart rate variability signals, including: After the collected EEG signals are bandpass filtered and artifacts removed, short-time Fourier transform (STFT) or wavelet transform is used to obtain the time-frequency matrix of the EEG signals, i.e., the spectrum of the EEG signals; The RR interval sequence of the collected heart rate variability signal is obtained, and the RR interval sequence is normalized and filtered to form a one-dimensional time series.

3. The multimodal mental stress assessment method based on machine learning according to claim 1 or 2, characterized in that: in, The CNN-LSTM model is used to perform multimodal deep feature extraction on the spectrum of the brain wave signal to obtain a deep feature vector of the brain wave signal, including: Using a convolutional neural network (CNN) to perform convolution feature extraction on the spectrum of the brain wave signal to obtain local time-frequency features of the brain wave signal; The extracted local time-frequency features are modeled using a long short-term memory network (LSTM) to capture the dynamic pattern of brain wave signals changing over time and output a deep feature vector of the brain wave signal.

4. The multimodal mental stress assessment method based on machine learning according to claim 1 or 2, characterized in that: in, The CNN-LSTM model is used to perform multimodal deep feature extraction on the time series of the heart rate variability signal to obtain a deep feature vector of the heart rate variability signal, including: The time series of heart rate variability signals is input into a one-dimensional convolutional neural network (1D-CNN) to extract local temporal pattern features; A long short-term memory network (LSTM) is used to perform time series modeling on the extracted local time series pattern features, and a deep feature vector of the heart rate variability signal is output.

5. The multimodal mental stress assessment method based on machine learning according to claim 3 is characterized in that: in, The method of using a convolutional neural network (CNN) to extract convolution features from the spectrum of the brain wave signal to obtain local time-frequency features of the brain wave signal includes: The time-frequency features of the spectrum graph of the EEG signal are extracted using two-dimensional convolution kernels of different sizes, and the feature dimension is reduced through a maximum pooling layer to output a convolution feature graph of the EEG signal, i.e., a local time-frequency feature.

6. The multimodal mental stress assessment method based on machine learning according to claim 4 is characterized in that: The step of inputting the time series of the heart rate variability signal into a one-dimensional convolutional neural network (1D-CNN) to extract local time series pattern features includes: The time series of the heart rate variability signal is input into a one-dimensional convolutional neural network (1D-CNN), and a more representative feature sequence is obtained through maximum pooling downsampling, thereby obtaining the convolution feature sequence of the heart rate variability signal, that is, the local temporal pattern feature of the heart rate variability signal.

7. The multimodal mental stress assessment method based on machine learning according to claim 1, characterized in that: in, The step of concatenating the deep feature vector of the brain wave signal and the deep feature vector of the heart rate variability signal to form a high-dimensional multimodal feature vector representation includes: Respectively performing standardization processing and feature dimension alignment on the deep feature vector of the brain wave signal and the deep feature vector of the heart rate variability signal; The standardized and dimensionally aligned deep feature vectors of the EEG signal and the deep feature vectors of the heart rate variability signal are concatenated in a predetermined order to form a multimodal feature vector.

8. The multimodal mental stress assessment method based on machine learning according to claim 1, characterized in that: The method uses a support vector machine (SVM) to classify the multimodal feature representation and output a pressure state discrimination result, including: The high-dimensional multimodal feature vector representation is classified using a pre-trained SVM model to output a stress state prediction result, including: the SVM model distinguishes the stress state from the normal state and outputs a stress state label by finding an optimal classification hyperplane in the high-dimensional feature space, and the stress state label includes "stress" or "non-stress".

9. The multimodal mental stress assessment method based on machine learning according to claim 8, characterized in that: The method further comprises: The stress state determination result is transmitted to the health management system to display the user's stress state in real time and conduct long-term stress data monitoring and analysis, including: When the stress status label is detected as "stress" in real time, the health management system automatically triggers an alarm and reminds the user to rest or relax; The health management system monitors and analyzes the user's long-term stress data, analyzes the user's stress pattern, and assists in formulating personalized relaxation or intervention strategies.

10. A multimodal mental stress assessment system based on machine learning, characterized in that: include: Data collection module, collecting the user's brain wave signals and heart rate variability signals; A feature extraction module, used to perform multimodal deep feature extraction on the spectrum of the brain wave signal and the time series of the heart rate variability signal using a CNN-LSTM model, to obtain a deep feature vector of the brain wave signal and a deep feature vector of the heart rate variability signal; A feature fusion module, used for concatenating the deep feature vector of the EEG signal and the deep feature vector of the heart rate variability signal to form a high-dimensional multimodal feature vector representation; The pressure prediction module is used to classify the multimodal feature representation using a support vector machine (SVM) and output the pressure state discrimination result.

Images