Fatigue evaluation method and system based on multi-modal feature fusion

Abstract

The invention provides a fatigue assessment method and system based on multi-modal feature fusion, and the method comprises the steps: carrying out the preprocessing of collected multi-dimensional physiological signals, including denoising and baseline drift removal; a forehead electroencephalogram signal and an electro-oculogram signal are adaptively separated from the forehead mixed electroencephalogram signal by adopting a variational mode decomposition algorithm, so that the problems of mode aliasing and endpoint effect are solved; and extracting a multi-modal feature vector, and inputting the multi-modal feature vector into the fatigue state evaluation model for training to obtain a trained fatigue state evaluation model. According to the method, the anti-interference performance of mixed electroencephalogram signal separation can be improved, so that the accuracy and stability of fatigue monitoring are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electroencephalogram signal processing, and more specifically, to a fatigue assessment method and system based on multimodal feature fusion. Background Art

[0002] In the related research of fatigue monitoring, commonly used physiological signals include electroencephalogram (EEG) signal, electrocardiogram (ECG) signal, photoplethysmogram (PPG) signal, electromyogram (EMG) signal and electrooculogram (EOG) signal. Among them, the fatigue monitoring method based on EEG signal can intuitively and reliably reflect the degree of fatigue of the human body, but because EEG signal is susceptible to interference and has large noise, it is easy to reduce the accuracy of monitoring results and the reliability is poor. Fatigue monitoring using single-modal signal is easily affected by external conditions and lacks objectivity, while the fatigue monitoring system using multi-modal feature fusion has more stable performance and more accurate results. In addition, direct collection of EOG signals requires electrodes to be attached around the eyes of the subjects, which will cause discomfort to the subjects if worn for a long time and is not suitable for long-term monitoring. Fatigue monitoring based on ECG signals requires electrodes to be attached to the chest of the subjects, which is also not suitable for long-term monitoring.

[0003] Empirical Mode Decomposition (EMD) algorithm, Blind Source Separation (BSS) algorithm, Independent Component Analysis (ICA) algorithm are widely used in the fields of noise reduction, feature extraction, signal separation, etc. However, the EMD algorithm has problems such as mode aliasing and endpoint effect, and is sensitive to the choice of decomposition parameters. The BSS algorithm's assumption that the source signals are independent or non-Gaussian does not always hold in practical applications, and the separation results depend on the choice of initial conditions. The ICA algorithm's assumption that the source signals are statistically independent does not always hold in practical applications, and the separation results are sensitive to noise. Summary of the invention

[0004] The present invention provides a fatigue assessment method and system based on multi-modal feature fusion to improve the accuracy and stability of a fatigue monitoring system.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] The present invention proposes a fatigue assessment method based on multimodal feature fusion, comprising:

[0007] S1: Collecting multi-dimensional original physiological state data of several test persons, wherein the multi-dimensional original physiological state data of each test person includes original four-lead EEG signals, original photoplethysmographic signals and fatigue state perception level;

[0008] S2: performing noise reduction preprocessing on the original four-lead EEG signal and the original photoplethysmogram signal to obtain a noise-reduced four-lead EEG signal and a noise-reduced photoplethysmogram signal, wherein the noise-reduced four-lead EEG signal includes a left forehead noise-reduced mixed EEG signal, a right forehead noise-reduced mixed EEG signal, a first noise-reduced EEG signal, and a second noise-reduced EEG signal;

[0009] S3: Using the variational mode decomposition algorithm, the left forehead denoised mixed EEG signal and the right forehead denoised mixed EEG signal are separated to obtain the left forehead denoised EEG signal, the left forehead denoised electrooculogram signal, the right forehead denoised EEG signal and the right forehead denoised electrooculogram signal;

[0010] S4: extracting multimodal features of the left frontal denoised EEG segmented signal, the left frontal denoised EEG segmented signal, the right frontal denoised EEG segmented signal, the right frontal denoised EEG segmented signal, the first denoised EEG segmented signal, the second denoised EEG segmented signal and the denoised photoplethysmogram segmented signal to form a multimodal feature vector set;

[0011] S5: constructing a fatigue state assessment model, inputting the multimodal feature vector set and the fatigue state perception level into the fatigue state assessment model for training, and obtaining a trained fatigue state assessment model;

[0012] S6: The user's four-lead EEG signals and photoplethysmographic signals are collected at intervals and input into the trained fatigue state assessment model to obtain fatigue state assessment results.

[0013] Preferably, the S1 includes:

[0014] Collect multi-dimensional original physiological state data of several test persons, wherein the multi-dimensional original physiological state data of each test person includes original four-lead EEG signal Fp1_raw , EEG Fp2_raw , EEG F7_raw , EEG F8_raw , original photoplethysmogram signal PPG raw and fatigue state perception level, where EEG Fp1_raw , EEG Fp2_raw , EEG F7_raw , EEG F8_raw Respectively represent the left forehead original mixed EEG signal, the right forehead original mixed EEG signal, the first original EEG signal and the second original EEG signal;

[0015] The fatigue state perception levels include severe fatigue, moderate fatigue, slight fatigue, and no fatigue.

[0016] Preferably, the S2 includes:

[0017] EEG Fp1_raw , EEG Fp2_raw , EEG F7_raw , EEG F8_raw The signals were processed by first bandpass filtering, power frequency interference and baseline drift in turn to obtain the left forehead noise reduction mixed EEG signal Fp1_mix , right forehead noise reduction mixed EEG signal Fp2_mix , the first noise-reduced EEG signal F7 and the second noise-reduced EEG signal F8 ;

[0018] About PPG raw The signal is subjected to second bandpass filtering, power frequency interference and baseline drift processing to obtain a noise-reduced photoplethysmogram signal PPG;

[0019] The first bandpass filtering process is to use a bandpass filter that allows a frequency range of 0.5 to 75 Hz to pass through, the second bandpass filtering process is to use a bandpass filter that allows a frequency range of 0.5 to 5 Hz to pass through, and the power frequency interference and baseline drift processing is to use a 50 Hz notch filter to filter out the influence of 50 Hz power frequency noise interference and baseline drift.

[0020] Preferably, the S4 includes:

[0021] Divide the left forehead noise-reduced EEG signal, the left forehead noise-reduced electrooculogram signal, the right forehead noise-reduced EEG signal, the right forehead noise-reduced electrooculogram signal, the first noise-reduced EEG signal, the second noise-reduced EEG signal and the noise-reduced photoplethysmogram signal into a number of segmented signals respectively, and input each segment of the left forehead noise-reduced EEG segmented signal, the right forehead noise-reduced EEG segmented signal, the first noise-reduced EEG segmented signal and the second noise-reduced EEG segmented signal into a δ wave bandpass filter, a θ wave bandpass filter, an α wave bandpass filter and a β wave bandpass filter respectively to extract the δ wave signal, the θ wave signal, the α wave signal and the β wave signal of the corresponding segmented signal, use the Welch algorithm to estimate the power spectral density of the signal, and obtain the power spectral density of each segmented signal δ wave signal, the θ wave signal, the α wave signal and the β wave signal;

[0022] Calculate the heart rate and energy ratio of each segment of the noise-reduced photoplethysmogram segmented signal;

[0023] The final blink frequency is calculated according to each segment signal of the left forehead noise-reduced electrooculogram segment signal and the right forehead noise-reduced electrooculogram segment signal;

[0024] The EEG Fp1 , EEG Fp2 , EEG F7 and EEG F8 The power spectrum density of each segmented signal of δ wave signal, θ wave signal, α wave signal, and β wave signal, the heart rate and energy ratio of each segment signal of the noise-reduced photoelectric volume pulse wave segmented signal, EOG Fp1 ,EOG Fp2 The final blink frequencies calculated for each signal segment in compose a set of multimodal feature vectors.

[0025] Preferably, the delta wave bandpass filter allows frequencies to pass in a range of 0.5 to 4 Hz, the theta wave bandpass filter allows frequencies to pass in a range of 4 to 8 Hz, the alpha wave bandpass filter allows frequencies to pass in a range of 8 to 13 Hz, and the beta wave bandpass filter allows frequencies to pass in a range of 13 to 32 Hz.

[0026] Preferably, the step of calculating the heart rate and energy ratio of each segment of the noise-reduced photoplethysmogram segmented signal comprises:

[0027] Perform fast Fourier transform on each segment of the noise-reduced photoplethysmogram segmented signal to obtain the frequency domain signal, extract the frequency corresponding to the peak value of the frequency domain signal amplitude, and calculate the heart rate and energy ratio. The calculation formula is:

[0028] HR=60×f max

[0029]

[0030] Where HR represents heart rate, f max Indicates the frequency corresponding to the peak value of the frequency domain signal amplitude, E LF / HF Represents energy ratio, E LF and E HF They respectively represent the low-frequency energy of 0.04 Hz to 0.15 Hz and the high-frequency energy of 0.15 Hz to 0.4 Hz in each segment of the noise-reduced photoplethysmogram segmented signal.

[0031] Preferably, the calculating the final blink frequency according to each segment signal of the left forehead noise-reduced electrooculogram segment signal and the right forehead noise-reduced electrooculogram segment signal comprises:

[0032]

[0033] Among them, BR represents the final blink frequency, BR Fp1 =60×f Fp1，max represents the blink frequency of the left forehead noise-reduced electrooculogram segmented signal, BR Fp2 =60×f Fp2，maxRepresents the blink frequency of the right forehead noise-reduced electrooculogram segmented signal.

[0034] Preferably, the fatigue assessment method further comprises:

[0035] Calculate the corresponding sleep index and deep sleep index for the left frontal EEG segmented signal, the right frontal EEG segmented signal, the first denoised EEG segmented signal, and the second denoised EEG segmented signal, respectively, and calculate the average sleep index and the average deep sleep index as the final sleep index SI and deep sleep index DI;

[0036] A fatigue relief instruction is generated according to the fatigue status assessment result, the final sleep index SI and the deep sleep index DI, and the fatigue relief device is controlled to relieve the fatigue of the user.

[0037] Preferably, the initial fatigue state assessment model is any one of a random forest model, a support vector machine, a decision tree, and a neural network model.

[0038] The present invention also proposes a fatigue assessment system based on multimodal feature fusion, which is used to implement the above method. The system comprises:

[0039] A data acquisition module, used to collect multi-dimensional original physiological state data of a number of test persons, wherein the multi-dimensional original physiological state data of each test person includes original four-lead EEG signals, original photoplethysmographic pulse wave signals and fatigue state perception level;

[0040] a preprocessing module, used for performing noise reduction preprocessing on the original four-lead EEG signal and the original photoplethysmogram signal to obtain a noise-reduced four-lead EEG signal and a noise-reduced photoplethysmogram signal, wherein the noise-reduced four-lead EEG signal includes a left frontal noise-reduced mixed EEG signal, a right frontal noise-reduced mixed EEG signal, a first noise-reduced EEG signal and a second noise-reduced EEG signal;

[0041] A signal separation module is used to separate the left forehead noise reduction mixed EEG signal and the right forehead noise reduction mixed EEG signal by using a variational mode decomposition algorithm to obtain a left forehead noise reduction EEG signal, a left forehead noise reduction electrooculogram signal, a right forehead noise reduction EEG signal, and a right forehead noise reduction electrooculogram signal;

[0042] A multimodal feature extraction module, used to extract multimodal features of the left forehead noise-reduced EEG segmented signal, the left forehead noise-reduced EEG segmented signal, the right forehead noise-reduced EEG segmented signal, the right forehead noise-reduced EEG segmented signal, the first noise-reduced EEG segmented signal, the second noise-reduced EEG segmented signal and the noise-reduced photoplethysmogram segmented signal to form a multimodal feature vector set;

[0043] A model training module is used to construct a fatigue state assessment model, input a multimodal feature vector set and a fatigue state perception level into the fatigue state assessment model for training, and obtain a trained fatigue state assessment model;

[0044] The model verification module is used to collect the user's four-lead EEG signals and photoplethysmographic signals at intervals, input them into the trained fatigue state assessment model, and obtain the fatigue state assessment results.

[0045] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0046] The present invention proposes a fatigue assessment method based on multimodal feature fusion, preprocesses the collected multidimensional physiological signals, including denoising and baseline drift removal, and uses a variational modal decomposition algorithm to adaptively separate the frontal EEG signal and the electrooculogram signal from the frontal mixed EEG signal, overcomes the problems of modal aliasing and endpoint effect, extracts multimodal feature vectors, and inputs them into the fatigue state assessment model for training to obtain a trained fatigue state assessment model. The present invention can improve the anti-interference of the mixed EEG signal separation, thereby improving the accuracy and stability of fatigue monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of the fatigue assessment method described in Example 1;

[0048] Figure 2 This is a schematic diagram of the structure of the fatigue assessment system described in Example 3;

[0049] Figure 3 This is a schematic diagram of the structure of the fatigue relief device described in Example 4;

[0050] Figure 4 This is a schematic diagram of the working process of the fatigue relief device described in Example 4. DETAILED DESCRIPTION

[0051] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0052] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0053] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0054] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0055] Example 1

[0056] This embodiment provides a fatigue assessment method based on multimodal feature fusion. Figure 1 As shown, including:

[0057] S1: Collecting multi-dimensional original physiological state data of several test persons, wherein the multi-dimensional original physiological state data of each test person includes original four-lead EEG signals, original photoplethysmographic signals and fatigue state perception level;

[0058] S2: performing noise reduction preprocessing on the original four-lead EEG signal and the original photoplethysmogram signal to obtain a noise-reduced four-lead EEG signal and a noise-reduced photoplethysmogram signal, wherein the noise-reduced four-lead EEG signal includes a left forehead noise-reduced mixed EEG signal, a right forehead noise-reduced mixed EEG signal, a first noise-reduced EEG signal, and a second noise-reduced EEG signal;

[0059] S3: Using the variational mode decomposition algorithm, the left forehead denoised mixed EEG signal and the right forehead denoised mixed EEG signal are separated to obtain the left forehead denoised EEG signal, the left forehead denoised electrooculogram signal, the right forehead denoised EEG signal and the right forehead denoised electrooculogram signal;

[0060] S4: extracting multimodal features of the left frontal denoised EEG segmented signal, the left frontal denoised EEG segmented signal, the right frontal denoised EEG segmented signal, the right frontal denoised EEG segmented signal, the first denoised EEG segmented signal, the second denoised EEG segmented signal and the denoised photoplethysmogram segmented signal to form a multimodal feature vector set;

[0061] S5: constructing a fatigue state assessment model, inputting the multimodal feature vector set and the fatigue state perception level into the fatigue state assessment model for training, and obtaining a trained fatigue state assessment model;

[0062] S6: The user's four-lead EEG signals and photoplethysmographic signals are collected at intervals and input into the trained fatigue state assessment model to obtain fatigue state assessment results.

[0063] In the specific implementation process, firstly, the multi-dimensional original physiological state data of several test persons are collected, and then the multi-dimensional original physiological state data are processed to obtain the left forehead noise reduction EEG signal, the left forehead noise reduction electrooculogram signal, the right forehead noise reduction EEG signal, and the right forehead noise reduction electrooculogram signal; the multi-modal features of the left forehead noise reduction EEG segmented signal, the left forehead noise reduction electrooculogram segmented signal, the right forehead noise reduction EEG segmented signal, the right forehead noise reduction electrooculogram segmented signal, the first noise reduction EEG segmented signal, the second noise reduction EEG segmented signal and the noise reduction photoelectric volume pulse wave segmented signal are extracted to form a multi-modal feature vector set; a fatigue state assessment model is constructed, and the multi-modal feature vector set and the fatigue state perception level are input into the fatigue state assessment model for training to obtain a trained fatigue state assessment model; the user's four-lead EEG signal and photoelectric volume pulse wave signal are collected at intervals, and input into the trained fatigue state assessment model to obtain a fatigue state assessment result.

[0064] Example 2

[0065] The present invention proposes a fatigue assessment method based on multimodal feature fusion, comprising:

[0066] S1: Collecting multi-dimensional original physiological state data of several test persons, wherein the multi-dimensional original physiological state data of each test person includes original four-lead EEG signals, original photoplethysmographic signals and fatigue state perception level;

[0067] EEG signals can be divided into five categories: delta wave, theta wave, alpha wave, beta wave, and gamma wave. Delta wave is highly correlated with unconscious state, with a frequency between 0.5 and 4 Hz. In the awake state, the value of delta wave is not active, and it is usually active in deep sleep, unconsciousness and other states. Theta wave is highly correlated with subconscious state, with a frequency between 4 and 8 Hz, and it is usually active in the state of interrupted consciousness and deep relaxation of the body. Alpha wave is highly correlated with relaxation state, with a frequency between 8 and 13 Hz, and it is usually active in a state of relaxation but not feeling tired. When stimulated externally, the value of alpha wave will decrease. Beta wave is highly correlated with concentration state, with a frequency between 13 and 32 Hz, and it is usually active in the state of wakefulness, thinking and high alertness. Gamma wave is highly correlated with high-level psychological activities, with a frequency above 32 Hz, and it is usually active in the state of concentration, memory, and meditation.

[0068] Collect multi-dimensional original physiological state data of several test persons, wherein the multi-dimensional original physiological state data of each test person includes original four-lead EEG signal Fp1_raw , EEG Fp2_raw , EEG F7_raw , EEG F8_raw , original photoplethysmogram signal PPG raw , where EEG Fp1_raw , EEG Fp2_raw, EEG F7_raw , EEG F8_raw They represent the left forehead original mixed EEG signal, the right forehead original mixed EEG signal, the first original EEG signal and the second original EEG signal respectively. The original photoplethysmogram signal is collected at the finger position. After each sample is collected, the measurement personnel are asked to fill in the Karolinska Sleepiness Scale (KSS) to obtain the fatigue state perception level corresponding to the current sample.

[0069] The fatigue state perception levels include severe fatigue, moderate fatigue, slight fatigue, and no fatigue.

[0070] The specific collection process is as follows:

[0071] (1) All subjects participated in the experiment voluntarily and signed the informed consent form before the experiment began. All subjects were between 14 and 35 years old. All subjects had no history of mental illness and maintained a good work and rest schedule before the experiment began. None of them took sedatives or stimulants.

[0072] (2) Samples with Karolinska Sleepiness Scale scores of 1-2, 3-4, 5-6, and 7-9 were divided into no fatigue, mild fatigue, moderate fatigue, and severe fatigue states respectively.

[0073] (3) The 10 / 20 electrode lead positioning standard calibrated by the International Electroencephalography Association was used, and the subjects wore the signal collection cap of the EEG device. Before each subject performed the memory task, the EEG signal and PPG signal were collected for 3 minutes in the resting state. After completing the memory task, the EEG signal and PPG signal were collected for another 3 minutes in the resting state. Then, the Karolinska Sleepiness Scale was filled out.

[0074] (4) Collect a large number of samples corresponding to non-fatigue, mild fatigue, moderate fatigue, and severe fatigue states to form a training set.

[0075] S2: performing noise reduction preprocessing on the original four-lead EEG signal and the original photoplethysmogram signal to obtain a noise-reduced four-lead EEG signal and a noise-reduced photoplethysmogram signal, wherein the noise-reduced four-lead EEG signal includes a left forehead noise-reduced mixed EEG signal, a right forehead noise-reduced mixed EEG signal, a first noise-reduced EEG signal, and a second noise-reduced EEG signal;

[0076] Since EEG signals are easily affected by environmental noise level power frequency noise and baseline drift, and the amplitude is extremely low, it is necessary to denoise these EEG signals first. Fp1_raw , EEG Fp2_raw , EEG F7_raw , EEG F8_rawThe signals were processed by first bandpass filtering, power frequency interference and baseline drift in turn to obtain the left forehead noise reduction mixed EEG signal Fp1_mix , right forehead noise reduction mixed EEG signal Fp2_mix , the first noise-reduced EEG signal F7 and the second noise-reduced EEG signal F8 ;

[0077] About PPG raw The signal is subjected to second bandpass filtering, power frequency interference and baseline drift processing to obtain a noise-reduced photoplethysmogram signal PPG;

[0078] The first bandpass filtering process is to use a bandpass filter that allows a frequency range of 0.5 to 75 Hz to pass through, the second bandpass filtering process is to use a bandpass filter that allows a frequency range of 0.5 to 5 Hz to pass through, and the power frequency interference and baseline drift processing is to use a 50 Hz notch filter to filter out the influence of 50 Hz power frequency noise interference and baseline drift.

[0079] S3: Using the variational mode decomposition (VMD) algorithm, the left forehead denoised mixed EEG signal and the right forehead denoised mixed EEG signal are separated to obtain the left forehead denoised EEG signal, the left forehead denoised electrooculogram signal, the right forehead denoised EEG signal and the right forehead denoised electrooculogram signal;

[0080] S4: extracting multimodal features of the left frontal denoised EEG segmented signal, the left frontal denoised EEG segmented signal, the right frontal denoised EEG segmented signal, the right frontal denoised EEG segmented signal, the first denoised EEG segmented signal, the second denoised EEG segmented signal and the denoised photoplethysmogram segmented signal to form a multimodal feature vector set;

[0081] After VMD algorithm processing, EEG Fp1_mix Signals and EEG Fp2_mix The signal is adaptively decomposed into M1 and M2 intrinsic mode functions (IMFs), u k (n) and v k (n) represents EEG Fp1_mix , EEG Fp2_mix The nth signal value of the kth IMF of the signal, each intrinsic mode function corresponds to a center frequency, and the sum of the individual intrinsic mode functions constitutes the original signal, as follows:

[0082]

[0083]

[0084] Perform Fast Fourier Transform (FFT) on each intrinsic mode function to obtain the corresponding center frequency (the center frequency is the frequency corresponding to the amplitude peak in the frequency domain) f u_k and f v_k ,Right now:

[0085] f u_k =FFT(u k (n)), k = 1, ..., M1

[0086] f v_k =FFT(v k (n)), k = 1, ..., M2

[0087] After obtaining the center frequencies corresponding to each IMF, u_k and f v_k Select the corresponding u in the range of 0.5Hz to 10Hz k (n) and v k (n) as EOG after separation Fp1 and EOG Fp2 signal, respectively, from the center frequency f u_k and f v_k Select the corresponding u in the range of 10Hz to 32Hz k (n) and v k (n) As the separated left frontal noise-reduced EEG signal Fp1 and right forehead noise reduction EEG Fp2 Signal.

[0088] Then, the left forehead noise reduction EEG signal, the left forehead noise reduction electrooculogram signal, the right forehead noise reduction EEG signal, the right forehead noise reduction electrooculogram signal, the first noise reduction EEG signal, the second noise reduction EEG signal and the noise reduction photoelectric volume pulse wave signal are divided into several segmented signals, that is, the EEG signal is respectively Fp1 , EEG Fp2 , EEG F7 , EEG F8 ,EOG Fp1 ,EOG Fp2The PPG signal is cut, and the cutting time length is 2s per segment. Then each segment of the left frontal denoised EEG segmented signal, the right frontal denoised EEG segmented signal, the first denoised EEG signal and the second denoised EEG signal is input into the δ wave bandpass filter, the θ wave bandpass filter, the α wave bandpass filter and the β wave bandpass filter to extract the corresponding segmented signal's δ wave signal, the θ wave signal, the α wave signal and the β wave signal, and the Welch algorithm is used to estimate the power spectrum density of the signal to obtain the power spectrum density of each segmented signal δ wave signal, the θ wave signal, the α wave signal and the β wave signal, specifically: for an EEG signal x(n) of length N, n = 0, 1, ..., N-1, the signal is divided into L overlapping segments, and there is usually a 50% overlap between adjacent segments, then the lth segment signal x l The power spectral density of (n) is defined as:

[0089]

[0090] Where M is the length of each signal segment, w[n] is the windowing function (such as Hamming window or Hanning window), and |w[n]|2 is the energy correction factor of the window function. The power spectral density of the signal is:

[0091]

[0092] The delta wave bandpass filter allows a passing frequency range of 0.5 to 4 Hz, the theta wave bandpass filter allows a passing frequency range of 4 to 8 Hz, the alpha wave bandpass filter allows a passing frequency range of 8 to 13 Hz, and the beta wave bandpass filter allows a passing frequency range of 13 to 32 Hz.

[0093] Perform fast Fourier transform on each segment of the noise-reduced photoelectric volume pulse wave segment signal, assuming that the PPG signal is x PPG =[x0, x1, …, x N-1 ], where N is the number of sample points, the sampling rate of the PPG acquisition device is Fs, and the specific calculation formula is:

[0094]

[0095] Where X[k] represents x PPG The frequency domain signal, x PPG [n] represents x PPG Then, take the frequency f corresponding to the peak value of the amplitude in the frequency domain signal X[k] max , and calculate the heart rate and energy ratio, the calculation formula is:

[0096] HR=60×f max

[0097]

[0098] Where HR represents heart rate, f max Indicates the frequency corresponding to the peak value of the frequency domain signal amplitude, E LF / HF It represents the energy ratio, which reflects the stability of the sympathetic and vagus nerve activities. LF and E HF They represent the low-frequency energy of 0.04Hz to 0.15Hz and the high-frequency energy of 0.15Hz to 0.4Hz in each segment of the noise-reduced photoplethysmogram segmented signal. The energy is defined as the sum of the squares of the amplitudes of all sample points, and the specific calculation formula is:

[0099]

[0100] Where E represents x PPG energy.

[0101] The final blink frequency is calculated based on each segment of the left forehead noise-reduced electrooculogram (EOG) signal and the right forehead noise-reduced EOG signal. The specific calculation process is as follows:

[0102]

[0103] Among them, BR represents the final blink frequency, BR Fp1 =60×f Fp1，max represents the blink frequency of the left forehead noise-reduced electrooculogram segmented signal, BR Fp2 =60×f Fp2，max Represents the blink frequency of the right forehead noise-reduced electrooculogram segmented signal.

[0104] The EEG Fp1 , EEG Fp2 , EEG F7 and EEG F8 The power spectrum density of each segmented signal of δ wave signal, θ wave signal, α wave signal, and β wave signal, the heart rate and energy ratio of each segment signal of the noise-reduced photoelectric volume pulse wave segmented signal, EOG Fp1 ,EOG Fp2 The final blink frequencies calculated for each signal segment in compose a set of multimodal feature vectors.

[0105] S5: constructing a fatigue state assessment model, inputting the multimodal feature vector set and the fatigue state perception level into the fatigue state assessment model for training, and obtaining a trained fatigue state assessment model;

[0106] The initial fatigue state assessment model is any one of a random forest model, a support vector machine, a decision tree, and a neural network model.

[0107] S6: The user's four-lead EEG signals and photoplethysmographic signals are collected at intervals and input into the trained fatigue state assessment model to obtain fatigue state assessment results.

[0108] The fatigue assessment method further comprises:

[0109] The corresponding sleep index and deep sleep index are calculated for the left frontal EEG segmented signal, the right frontal EEG segmented signal, the first denoised EEG segmented signal, and the second denoised EEG segmented signal, respectively, and the average sleep index and the average deep sleep index are calculated as the final sleep index SI and deep sleep index DI, specifically:

[0110] Calculate the sleep index SI. Taking the left frontal EEG segmented signal as an example, its signal representation is x Fp1 (n), the sleep index calculation formula is:

[0111]

[0112] Among them, SI Fp1 Represents the sleep index of the left frontal EEG segment signal, The value range of K1 is 0.3~0.5, the value range of K2 is 0.7~1, and E δ，Fp1 、E θ，Fp1 、E α，Fp1 and E β，Fp1 From x Fp1 (n) The energy of the extracted delta wave, theta wave, alpha wave, and beta wave. The calculation formula for the deep sleep index is:

[0113]

[0114] By analogy, the sleep index and deep sleep index of the right frontal EEG segmented signal, the first denoised EEG segmented signal and the second denoised EEG segmented signal can be obtained respectively, and the final sleep index SI and deep sleep index DI are obtained by averaging them. The specific calculation formula is:

[0115]

[0116] Among them, SI Fp2 、SI F7 and SI F8 Respectively represent the sleep index of the right frontal EEG segmented signal, the first noise-reduced EEG segmented signal, and the second noise-reduced EEG segmented signal, DI Fp2 ,DI F7 and DI F8 They respectively represent the deep sleep index of the right frontal EEG segmented signal, the first denoised EEG segmented signal and the second denoised EEG segmented signal.

[0117] The sleep state is obtained based on the sleep index SI and the deep sleep index DI: if SI < Th1, it is determined as the wake period; if SI ≥ Th1 and DI ≥ Th2, it is determined as the light sleep period; if SI ≥ Th1 and DI < Th2, it is determined as the deep sleep period. Usually, Th1 is taken as 1.5 - 4 and Th2 is taken as 0.8 - 2.

[0118] Based on the final sleep index SI and the deep sleep index DI, the sleep state of the user can be obtained, and then a fatigue relief instruction is generated according to the fatigue state assessment result and the sleep state, controlling the fatigue relief device to relieve the fatigue of the user.

[0119] Embodiment 3

[0120] This embodiment provides a fatigue assessment system based on multi-modal feature fusion for implementing the method described in Embodiment 1 or 2, as Figure 2 shown, the system includes:

[0121] A data acquisition module for acquiring multi-dimensional raw physiological state data of a number of testers, and the multi-dimensional raw physiological state data of each tester includes raw four-lead electroencephalogram signals, raw photoplethysmogram signals, and fatigue state perception levels;

[0122] A preprocessing module for performing noise reduction preprocessing on the raw four-lead electroencephalogram signals and raw photoplethysmogram signals to obtain noise-reduced four-lead electroencephalogram signals and noise-reduced photoplethysmogram signals, and the noise-reduced four-lead electroencephalogram signals include left frontal noise-reduced mixed electroencephalogram signals, right frontal noise-reduced mixed electroencephalogram signals, first noise-reduced electroencephalogram signals, and second noise-reduced electroencephalogram signals;

[0123] A signal separation module for separately separating the left frontal noise-reduced mixed electroencephalogram signal and the right frontal noise-reduced mixed electroencephalogram signal by using the variational mode decomposition algorithm to obtain a left frontal noise-reduced electroencephalogram signal, a left frontal noise-reduced electrooculogram signal, a right frontal noise-reduced electroencephalogram signal, and a right frontal noise-reduced electrooculogram signal;

[0124] A multi-modal feature extraction module for extracting multi-modal features of the left frontal noise-reduced electroencephalogram segmented signals, left frontal noise-reduced electrooculogram segmented signals, right frontal noise-reduced electroencephalogram segmented signals, right frontal noise-reduced electrooculogram segmented signals, first noise-reduced electroencephalogram segmented signals, second noise-reduced electroencephalogram segmented signals, and noise-reduced photoplethysmogram segmented signals to form a multi-modal feature vector set;

[0125] A model training module for constructing a fatigue state assessment model, inputting the multi-modal feature vector set and the fatigue state perception level into the fatigue state assessment model for training to obtain a trained fatigue state assessment model;

[0126] The model verification module is used to collect the user's four-lead EEG signals and photoplethysmographic signals at intervals, input them into the trained fatigue state assessment model, and obtain the fatigue state assessment results.

[0127] Example 4

[0128] This embodiment provides a fatigue relief device, such as Figure 3 As shown, it includes a massage device, an EEG signal acquisition terminal, a PPG signal acquisition terminal, a host computer and a fatigue assessment system. The fatigue assessment system includes a fatigue state assessment module and a sleep state assessment module, which are used to implement the fatigue assessment method described in Example 3. The massage device includes a main control module, a music module, a massage module and a temperature control module. The main control module can receive control instructions from the host computer and control the music module, the massage module and the temperature control module. The music module can play music. The massage module has multiple working massage modes. The massage module can perform different degrees of massage on the user in different modes. The temperature control module has a heating function.

[0129] like Figure 4 As shown, it is a schematic diagram of the working process of the fatigue relief device. Before the massage starts, the EEG signal acquisition terminal and the PPG signal acquisition terminal respectively collect the user's four-lead EEG signal and PPG signal for a period of time, and then transmit the data back to the fatigue assessment system through the host computer. The fatigue assessment system processes the data to obtain the user's fatigue status assessment result; and automatically selects the massage mode according to the user's fatigue status assessment result.

[0130] After the massage starts, the EEG signal acquisition terminal and the PPG signal acquisition terminal collect the user's four-lead EEG signals at intervals, and calculate the sleep index SI and deep sleep index DI, and automatically issue control instructions to the massage equipment mode according to the sleep state, helping the user to gradually relax from a fatigue state to a deep sleep state.

[0131] The user's sleep state can be obtained based on the final sleep index SI and deep sleep index DI. The sleep state is divided into wakefulness, light sleep expectation and deep sleep. Then, fatigue relief instructions are generated based on the fatigue state assessment results and sleep state, and the fatigue relief device is controlled to relieve the user's fatigue. Generally, severe fatigue or moderate fatigue will coexist with light sleep. The massage mode is selected according to the state assessment results output by the fatigue state assessment model; after the massage starts, the massage device is automatically adjusted according to the sleep state, specifically:

[0132] a. If the user is in a state of severe fatigue and the sleep state is in the light sleep stage, the goal is to relax quickly and let the user enter a deep sleep state: use a gentle full-body massage to avoid excessive stimulation, the massage time is 15 to 20 minutes, focusing on the shoulders, back and legs, and start the heating function, synchronized with the massage time; play soothing white noise or light music, the volume is low, and the duration is about 30 minutes. When the user enters a deep sleep stage (the sleep state is deep sleep), turn down the music volume and turn off the massage function.

[0133] b. If the user is in a state of moderate fatigue and the sleep state is in the light sleep stage, the goal is to let the user enter a deep sleep state through moderate massage and relaxation: use a medium-intensity full-body massage for 20 to 25 minutes, focusing on the waist and legs, and start the heating function, synchronized with the massage time; play soothing white noise or light music at a low volume for about 30 minutes. When the user enters a deep sleep stage (the sleep state is in the deep sleep state), turn down the music volume and turn off the massage function.

[0134] c. If the user is slightly tired and the sleep state is awake, the goal is to let the user enter a light sleep state and gradually enter a deep sleep state through easy massage and relaxation: use gentle local massage to avoid excessive stimulation, the massage time is 25 to 30 minutes, focus on the shoulders, back and neck, and start the heating function, synchronized with the massage time; play soothing white noise or light music, the volume is low, and the duration is 30-40 minutes. When the user enters the light sleep stage (the sleep state is light sleep), the massage intensity is automatically reduced; after entering the deep sleep stage (the sleep state is deep sleep), turn down the music volume and turn off the massage function.

[0135] d. If the user is not tired and the sleep state is awake, the goal is to gradually relax the body through continuous relaxation and gradual stimulation, and finally enter light sleep and deep sleep: use a gentle full body massage, gradually transition to medium intensity, the massage time is 30 to 35 minutes, and start the heating function, synchronized with the massage time; play soothing white noise or light music, the volume is low, and the duration is about 40 to 50 minutes. When the user enters the light sleep period, the massage intensity is automatically reduced; after entering the deep sleep period, turn down the music volume and turn off the massage function.

[0136] The same or similar reference numerals correspond to the same or similar components;

[0137] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0138] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A fatigue assessment method based on multimodal feature fusion, characterized in that: include: S1: Collecting multi-dimensional original physiological state data of several test persons, wherein the multi-dimensional original physiological state data of each test person includes original four-lead EEG signals, original photoplethysmographic signals and fatigue state perception level; S2: performing noise reduction preprocessing on the original four-lead EEG signal and the original photoplethysmogram signal to obtain a noise-reduced four-lead EEG signal and a noise-reduced photoplethysmogram signal, wherein the noise-reduced four-lead EEG signal includes a left forehead noise-reduced mixed EEG signal, a right forehead noise-reduced mixed EEG signal, a first noise-reduced EEG signal, and a second noise-reduced EEG signal; S3: Using the variational mode decomposition algorithm, the left forehead denoised mixed EEG signal and the right forehead denoised mixed EEG signal are separated to obtain the left forehead denoised EEG signal, the left forehead denoised electrooculogram signal, the right forehead denoised EEG signal and the right forehead denoised electrooculogram signal; S4: extracting multimodal features of the left frontal denoised EEG segmented signal, the left frontal denoised EEG segmented signal, the right frontal denoised EEG segmented signal, the right frontal denoised EEG segmented signal, the first denoised EEG segmented signal, the second denoised EEG segmented signal and the denoised photoplethysmogram segmented signal to form a multimodal feature vector set; S5: constructing a fatigue state assessment model, inputting the multimodal feature vector set and the fatigue state perception level into the fatigue state assessment model for training, and obtaining a trained fatigue state assessment model; S6: The user's four-lead EEG signals and photoplethysmographic signals are collected at intervals and input into the trained fatigue state assessment model to obtain fatigue state assessment results.

2. The fatigue assessment method based on multimodal feature fusion according to claim 1 is characterized in that: The S1 includes: Collect multi-dimensional original physiological state data of several test persons, wherein the multi-dimensional original physiological state data of each test person includes original four-lead EEG signal Fp1_raw , EEG Fp2_raw , EEG F7_raw , EEG F8_raw , original photoplethysmogram signal PPG raw and fatigue state perception level, where EEG Fp1_raw , EEG Fp2_raw , EEG F7_raw , EEG F8_raw Respectively represent the left forehead original mixed EEG signal, the right forehead original mixed EEG signal, the first original EEG signal and the second original EEG signal; The fatigue state perception levels include severe fatigue, moderate fatigue, slight fatigue, and no fatigue.

3. The fatigue assessment method based on multimodal feature fusion according to claim 2 is characterized in that: The S2 includes: EEG Fp1_raw , EEG Fp2_raw , EEG F7_raw , EEG F8_raw The signals were processed by first bandpass filtering, power frequency interference and baseline drift in turn to obtain the left forehead noise reduction mixed EEG signal Fp1_mix , right forehead noise reduction mixed EEG signal Fp2_mix , the first noise-reduced EEG signal F7 and the second noise-reduced EEG signal F8 ; About PPG raw The signal is subjected to second bandpass filtering, power frequency interference and baseline drift processing to obtain a noise-reduced photoplethysmogram signal PPG; The first bandpass filtering process is to use a bandpass filter that allows a frequency range of 0.5 to 75 Hz to pass through, the second bandpass filtering process is to use a bandpass filter that allows a frequency range of 0.5 to 5 Hz to pass through, and the power frequency interference and baseline drift processing is to use a 50 Hz notch filter to filter out the influence of 50 Hz power frequency noise interference and baseline drift.

4. The fatigue assessment method based on multimodal feature fusion according to claim 2 is characterized in that: The S4 includes: Divide the left forehead noise-reduced EEG signal, the left forehead noise-reduced electrooculogram signal, the right forehead noise-reduced EEG signal, the right forehead noise-reduced electrooculogram signal, the first noise-reduced EEG signal, the second noise-reduced EEG signal and the noise-reduced photoplethysmogram signal into a number of segmented signals respectively, and input each segment of the left forehead noise-reduced EEG segmented signal, the right forehead noise-reduced EEG segmented signal, the first noise-reduced EEG segmented signal and the second noise-reduced EEG segmented signal into a δ wave bandpass filter, a θ wave bandpass filter, an α wave bandpass filter and a β wave bandpass filter respectively to extract the δ wave signal, the θ wave signal, the α wave signal and the β wave signal of the corresponding segmented signal, use the Welch algorithm to estimate the power spectral density of the signal, and obtain the power spectral density of each segmented signal δ wave signal, the θ wave signal, the α wave signal and the β wave signal; Calculate the heart rate and energy ratio of each segment of the noise-reduced photoplethysmogram segmented signal; The final blink frequency is calculated according to each segment signal of the left forehead noise-reduced electrooculogram segment signal and the right forehead noise-reduced electrooculogram segment signal; The EEG Fp1 , EEG Fp2 , EEG F7 and EEG F8 The power spectrum density of each segmented signal of δ wave signal, θ wave signal, α wave signal, and β wave signal, the heart rate and energy ratio of each segment signal of the noise-reduced photoelectric volume pulse wave segmented signal, EOG Fp1 ,EOG Fp2 The final blink frequencies calculated for each signal segment in compose a set of multimodal feature vectors.

5. The fatigue assessment method based on multimodal feature fusion according to claim 4 is characterized in that: The delta wave bandpass filter allows a passing frequency range of 0.5 to 4 Hz, the theta wave bandpass filter allows a passing frequency range of 4 to 8 Hz, the alpha wave bandpass filter allows a passing frequency range of 8 to 13 Hz, and the beta wave bandpass filter allows a passing frequency range of 13 to 32 Hz.

6. The fatigue assessment method based on multimodal feature fusion according to claim 4 is characterized in that: The step of calculating the heart rate and energy ratio of each segment of the noise-reduced photoplethysmogram segmented signal comprises: Perform fast Fourier transform on each segment of the noise-reduced photoplethysmogram segmented signal to obtain the frequency domain signal, extract the frequency corresponding to the peak value of the frequency domain signal amplitude, and calculate the heart rate and energy ratio. The calculation formula is: HR=60×f max Where HR represents heart rate, f max Indicates the frequency corresponding to the peak value of the frequency domain signal amplitude, E LF / HF Represents energy ratio, E LF and E HF They respectively represent the low-frequency energy of 0.04 Hz to 0.15 Hz and the high-frequency energy of 0.15 Hz to 0.4 Hz in each segment of the noise-reduced photoplethysmogram segmented signal.

7. The fatigue assessment method based on multimodal feature fusion according to claim 4 is characterized in that: The method of calculating the final blink frequency according to each segment signal of the left forehead noise reduction electrooculogram segment signal and the right forehead noise reduction electrooculogram segment signal comprises: Among them, BR represents the final blink frequency, BR Fp1 =60×f Fp1,max represents the blink frequency of the left forehead noise-reduced electrooculogram segmented signal, BR Fp2 =60×f Fp2,max Represents the blink frequency of the right forehead noise-reduced electrooculogram segmented signal.

8. The fatigue assessment method based on multimodal feature fusion according to claim 1 is characterized in that: The fatigue assessment method further comprises: Calculate the corresponding sleep index and deep sleep index for the left frontal EEG segmented signal, the right frontal EEG segmented signal, the first denoised EEG segmented signal, and the second denoised EEG segmented signal, respectively, and calculate the average sleep index and the average deep sleep index as the final sleep index SI and deep sleep index DI; A fatigue relief instruction is generated according to the fatigue status assessment result, the final sleep index SI and the deep sleep index DI, and the fatigue relief device is controlled to relieve the fatigue of the user.

9. The fatigue assessment method based on multimodal feature fusion according to claim 1 is characterized in that: The initial fatigue state assessment model is any one of a random forest model, a support vector machine, a decision tree, and a neural network model.

10. A fatigue assessment system based on multimodal feature fusion, characterized in that: For implementing the method according to any one of claims 1 to 9, the system comprises: A data acquisition module, used to collect multi-dimensional original physiological state data of a number of test persons, wherein the multi-dimensional original physiological state data of each test person includes original four-lead EEG signals, original photoplethysmographic pulse wave signals and fatigue state perception level; a preprocessing module, used for performing noise reduction preprocessing on the original four-lead EEG signal and the original photoplethysmogram signal to obtain a noise-reduced four-lead EEG signal and a noise-reduced photoplethysmogram signal, wherein the noise-reduced four-lead EEG signal includes a left frontal noise-reduced mixed EEG signal, a right frontal noise-reduced mixed EEG signal, a first noise-reduced EEG signal and a second noise-reduced EEG signal; A signal separation module is used to separate the left forehead noise reduction mixed EEG signal and the right forehead noise reduction mixed EEG signal by using a variational mode decomposition algorithm to obtain a left forehead noise reduction EEG signal, a left forehead noise reduction electrooculogram signal, a right forehead noise reduction EEG signal, and a right forehead noise reduction electrooculogram signal; A multimodal feature extraction module, used to extract multimodal features of the left forehead noise-reduced EEG segmented signal, the left forehead noise-reduced EEG segmented signal, the right forehead noise-reduced EEG segmented signal, the right forehead noise-reduced EEG segmented signal, the first noise-reduced EEG segmented signal, the second noise-reduced EEG segmented signal and the noise-reduced photoplethysmogram segmented signal to form a multimodal feature vector set; A model training module is used to construct a fatigue state assessment model, input a multimodal feature vector set and a fatigue state perception level into the fatigue state assessment model for training, and obtain a trained fatigue state assessment model; The model verification module is used to collect the user's four-lead EEG signals and photoplethysmographic signals at intervals, input them into the trained fatigue state assessment model, and obtain the fatigue state assessment results.

Images