Wearable fatigue monitoring and feedback method based on deep learning

Abstract

The invention discloses a wearable fatigue monitoring and feedback method based on deep learning. The method comprises the steps that a heart rate variability signal, a gamma wave band electroencephalogram signal and body movement posture information of a user are collected in real time; denoising, normalizing and synchronously fusing the acquired multi-mode signals; extracting a fatigue state representation vector in real time by adopting a Mamba linear state space sequence model; estimating a user fatigue index in real time based on a lightweight full-connection neural network decoder and constructing an individual fatigue threshold dynamic model; calculating a phase synchronization index of the electroencephalogram signal in real time; and generating and outputting an individualized 40Hz gamma wave band sensory nerve stimulation feedback signal in real time based on the fatigue index and the phase synchronization index. According to the invention, high-robustness fatigue identification and low-delay feedback adjustment in a complex motion noise environment are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of physiological signal monitoring, and in particular to a wearable fatigue monitoring and feedback method based on deep learning. BACKGROUND

[0002] With the continuous acceleration of the rhythm of modern social production and life, the monitoring and effective relief of human fatigue state has become one of the important technical needs in the current health management and safe operation field. In recent years, wearable physiological signal monitoring devices have gradually become an important technical route in the field of real-time monitoring of fatigue state. By real-time acquisition of physiological indicators such as human heart rate variability signals, electroencephalogram signals, and body movement postures, fatigue state recognition and feedback intervention are carried out. The existing technologies widely used at present mainly include fatigue recognition technologies based on traditional machine learning and conventional deep learning methods, which realize the basic recognition of fatigue state through feature extraction and classification analysis of single modal or simply fused physiological signals.

[0003] However, in real application environment, especially in conditions with complex motion interference and noise influence, the existing technology lacks stability and robustness in the fusion analysis of multi-modal physiological signals, and it is difficult to accurately capture the dynamic change trend of the user's fatigue state in real time, resulting in high feedback intervention delay and significantly reduced feedback stimulation effect. In addition, the existing fatigue feedback stimulation technology generally uses fixed parameter visual, auditory or tactile stimulation signals, and does not fully consider the individual differences and real-time changes of fatigue state, making it difficult to dynamically and real-timely optimize the parameter combination of the stimulation signal, which significantly limits the fatigue intervention effect. The above shortcomings directly restrict the application effect of the existing wearable fatigue monitoring and feedback technology in long-term actual scenes, and further optimization is urgently needed.

[0004] Therefore, how to provide a wearable fatigue monitoring and feedback method based on deep learning is a problem that those skilled in the art need to solve. SUMMARY

[0005] One object of the present application is to provide a wearable fatigue monitoring and feedback method based on deep learning. In view of the problem of insufficient robustness of the existing technology in real-time and accurate determination of fatigue state and low-delay individualized feedback in complex motion noise environment, a technical scheme is proposed, which comprises real-time acquisition and fusion of multi-modal physiological signals, deep extraction of fatigue state representation vectors by using Mamba linear state space sequence model, construction of individual fatigue threshold dynamic model and dynamic optimization of stimulation feedback parameters. The present application has the advantages of high fatigue recognition accuracy and timely and effective feedback.

[0006] According to the wearable fatigue monitoring and feedback method based on deep learning of the embodiment of the present application, the method comprises the following steps:

[0007] Collect the heart rate variability signal, gamma band electroencephalogram signal and body movement posture information of the user to form an initial multi-modal sequence signal matrix;

[0008] Remove noise in the initial multi-modal sequence signal matrix and perform normalization processing, synchronously fuse the time stamp and sampling rate to form standardized input sequence data;

[0009] Input the standardized input sequence data into a feature dynamic extraction module based on a Mamba linear state space sequence model, capture long-term time series dependence in a complex motion noise environment in real time by using an adaptive selective state space kernel, and obtain a fatigue state representation vector;

[0010] Input the fatigue state representation vector into a lightweight fully connected neural network decoder to estimate the user fatigue index in real time, and construct an individual fatigue threshold dynamic model based on the user historical sequence data to determine the user fatigue state in real time;

[0011] Based on the gamma band electroencephalogram signal of the user collected in real time by the ear clip type electroencephalogram electrode, the phase synchronization index of the gamma band electroencephalogram signal of the user is calculated to represent the real-time synchronization level of the sensory nerve gamma band of the user;

[0012] Based on the real-time determined fatigue index and phase synchronization index, the phase parameters, frequency parameters and intensity parameters of the visual stimulation signal, auditory stimulation signal or tactile stimulation signal are dynamically adjusted through real-time closed-loop optimization to generate a gamma band sensory nerve stimulation feedback scheme;

[0013] The wearable feedback device outputs the stimulation signal corresponding to the individualized gamma band sensory nerve stimulation feedback scheme to the user in real time to form a real-time closed-loop feedback.

[0014] Optionally, the collecting the heart rate variability signal, gamma band electroencephalogram signal and body movement posture information of the user to form an initial multi-modal sequence signal matrix is specifically:

[0015] A wrist-type photoelectric plethysmogram sensor is worn at the radial artery of the wrist of the user to collect the pulse wave original signal of the user in real time and calculate the heart rate variability signal;

[0016] Conductive gel electrodes are worn on the bilateral earlobes of the user in a differential electrode manner to collect the electroencephalogram potential signal in the gamma band range of the cerebral cortex of the user in real time;

[0017] The electroencephalogram signal acquisition front-end amplification circuit filters and digitizes the electroencephalogram potential signal to generate real-time gamma band electroencephalogram signal data;

[0018] A three-axis inertial measurement unit is fixedly installed at the C7 vertebra position at the back of the user's neck, and three-axis acceleration data and three-axis angular velocity data of the user's head, neck and torso are collected in real time, which are converted into body movement posture information through an Euler angle coordinate transformation algorithm;

[0019] The real-time calculated heart rate variability signal, the real-time generated gamma band electroencephalogram signal data, and the real-time acquired body movement posture information are marked and aligned with a unified timestamp, and stored as an initial multi-modal sequence signal matrix at a fixed sampling rate.

[0020] Optionally, the noise in the initial multi-modal sequence signal matrix is removed and normalized, the timestamp and the sampling rate are synchronized and fused to form standardized input sequence data, specifically:

[0021] A third-order Butterworth low-pass filtering algorithm is used to filter the heart rate variability signal calculated in real time by the wrist-type photoelectric plethysmogram sensor, to generate a filtered heart rate variability signal.

[0022] A fourth-order Chebyshev band-pass filtering algorithm is used to filter the gamma band electroencephalogram signal data generated in real time by the electroencephalogram signal acquisition front-end amplification circuit, to generate filtered gamma band electroencephalogram signal data.

[0023] An adaptive Kalman filtering algorithm is used to dynamically suppress noise of the body movement posture information acquired in real time by the three-axis inertial measurement unit, to generate filtered body movement posture information.

[0024] The filtered heart rate variability signal, the filtered gamma band electroencephalogram signal data, and the filtered body movement posture information are respectively subjected to minimum-maximum normalization processing, to generate normalized multi-modal signal data.

[0025] The internal clocks of the wrist-type photoelectric plethysmogram sensor, the ear clip-type electroencephalogram electrode, and the three-axis inertial measurement unit are periodically synchronized and calibrated based on the network time protocol, and the sampling time instants of each modal signal data are synchronized at a millisecond-level clock. The normalized multi-modal signal data is aligned and fused point by point and channel by channel according to a unified sampling rate and a unified timestamp, to generate standardized input sequence data.

[0026] Optionally, the feature dynamic extraction module based on the Mamba linear state space sequence model includes a local state space kernel adaptive perception unit, a cross-modal nonlinear synchronous fusion unit, and a kernel parameter online optimization unit.

[0027] The local state space kernel adaptive perception unit is configured to calculate a short-time energy entropy value of each channel in the standardized input sequence data in real time, adjust a local segment length and a kernel function local bandwidth of the state space kernel on line according to the short-time energy entropy value, generate a non-uniform segmented local state space kernel, and output a local enhanced fatigue state representation vector.

[0028] The cross-modal nonlinear synchronous fusion unit is configured to calculate a dynamic mutual information metric value between different modal channels of the standardized input sequence data in real time, generate a cross-modal attention weight coefficient according to the dynamic mutual information metric value, and fuse the cross-modal attention weight coefficient with the local enhanced fatigue state representation vector to output a cross-modal synergistically enhanced fatigue state representation vector.

[0029] The kernel parameter on-line optimization unit is configured to estimate a mean vector, a covariance matrix and a skewness coefficient of the standardized input sequence data in real time, dynamically adjust a state transition parameter matrix and a gain matrix of the local state space kernel according to the mean vector, the covariance matrix and the skewness coefficient, and output a dynamically optimized fatigue state representation vector.

[0030] Optionally, the standardized input sequence data is input into the feature dynamic extraction module based on the Mamba linear state space sequence model, and a long-term time sequence dependence relationship in a complex motion noise environment is captured in real time by using the adaptive selective state space kernel to obtain the fatigue state representation vector. Specifically,

[0031] The standardized input sequence data is segmented by a preset sliding time window, and each sliding time window is sent into the local state space kernel adaptive perception unit to calculate a short-time energy entropy value of each channel in real time.

[0032] The short-time energy entropy value is used to determine a local segment position index of each channel in real time, and the local segment length and the kernel function local bandwidth of the state space kernel are dynamically adjusted to generate a non-uniform segmented local state space kernel.

[0033] The non-uniform segmented local state space kernel is used to process the standardized input sequence data in the sliding time window channel by channel to capture a local long-term time sequence dependence relationship and obtain a local enhanced fatigue state representation vector.

[0034] The dynamic mutual information metric value is calculated between different channels of the standardized input sequence data in the sliding time window point by point to generate a cross-modal attention weight coefficient in real time.

[0035] The cross-modal attention weight coefficient is fused with the local enhanced fatigue state representation vector to generate a cross-modal synergistically enhanced fatigue state representation vector.

[0036] The mean vector, covariance matrix and skewness coefficient of the normalized input sequence data in the sliding time window are calculated, the state transition parameter matrix and gain matrix of the state space kernel are dynamically adjusted, the state space kernel parameters are optimized online, and the fatigue state representation vector after dynamic optimization is output.

[0037] Optionally, the fatigue state representation vector is input into the lightweight fully connected neural network decoder to estimate the user fatigue index in real time, and an individual fatigue threshold dynamic model is constructed based on the user historical sequence data to determine the user fatigue state in real time, specifically:

[0038] The fatigue state representation vector is input into the lightweight fully connected neural network decoder and is mapped to multiple hidden layer nodes in real time, and each hidden layer node is connected to the input feature with a dynamic weight parameter;

[0039] The fatigue index historical sequence data of the user in the past 30 minutes is counted, the fatigue index historical sequence data is segmented by time window, and the median and interquartile range of the fatigue index sequence of each time window are calculated.

[0040] According to the median and interquartile range of the fatigue index sequence in each time window, a historical fatigue index distribution state space is constructed, and the dynamic evolution trajectory of the historical fatigue index distribution state space is recorded in real time;

[0041] The dynamic evolution trajectory of the historical fatigue index distribution state space is converted into an individual fatigue threshold dynamic model, and the upper and lower thresholds in the individual fatigue threshold dynamic model are updated in real time;

[0042] After the lightweight fully connected neural network decoder output layer nonlinearly combines the hidden layer node feature vectors, the current user fatigue index is output in real time;

[0043] The current user fatigue index is compared with the upper and lower thresholds of the individual fatigue threshold dynamic model, and when the fatigue index is higher than the real-time upper threshold, it is determined that the user is in a fatigue state, and the user fatigue state determination result is obtained.

[0044] Optionally, the individual fatigue threshold dynamic model specifically includes a fatigue index historical data segmentation module, a dynamic probability density real-time estimation module, a probability density manifold construction module, a dynamic manifold feature extraction module, a fatigue threshold space construction module, and a real-time threshold adaptive updating module:

[0045] The fatigue index historical data segmentation module is used to segment the fatigue index historical sequence data in the past 30 minutes into subsegments with a preset time window, and generate historical sequence data subsegments.

[0046] The dynamic probability density real-time estimation module is configured to perform Gaussian kernel density estimation on historical sequence data sub-segments in real time to generate a fatigue index probability density distribution of a current sub-segment.

[0047] The probability density manifold construction module is configured to take a peak point of the probability density distribution as a manifold center anchor point, define a local manifold structure according to a gradient direction of the probability density distribution, connect local manifold structures of sub-segments, and form a continuously evolving probability density manifold structure.

[0048] The dynamic manifold feature extraction module is configured to calculate a local curvature value of the continuously evolving probability density manifold structure and a distance change rate between adjacent peak points on the manifold structure, and obtain a dynamic evolution feature of the manifold structure.

[0049] The fatigue threshold space construction module is configured to define an upper bound of an individual fatigue threshold in a region with a higher local curvature value of the continuously evolving probability density manifold structure, define a lower bound of the individual fatigue threshold in a region with a lower local curvature value, and construct an individual fatigue threshold space.

[0050] The real-time threshold adaptive updating module is configured to call the dynamic probability density real-time estimation module, the probability density manifold construction module, the dynamic manifold feature extraction module and the fatigue threshold space construction module based on a newly added historical sequence data sub-segment to update the individual fatigue threshold space in real time, adjust the upper and lower bounds of the individual fatigue threshold, and obtain a dynamic threshold of an individual fatigue index.

[0051] Optionally, the user's gamma band electroencephalogram signal collected in real time based on the ear clip electroencephalogram electrode is used to calculate a phase synchronization index of the user's gamma band electroencephalogram signal, which represents the real-time synchronization level of the user's sensory nerve gamma band, and specifically comprises the following steps:

[0052] The gamma band electroencephalogram signal data collected in real time by the ear clip electroencephalogram electrode is segmented by sliding window processing with a preset time window to generate a gamma band electroencephalogram signal segment.

[0053] The Hilbert transform is performed on the gamma band electroencephalogram signal segment in each channel in the sliding time window to obtain an instantaneous analytic signal and an instantaneous phase value of each channel electroencephalogram signal segment.

[0054] The difference between the instantaneous phase values of each two electroencephalogram channels is determined, and the complex exponential mapping is performed on the instantaneous phase value difference, and then the vector average operation is performed to obtain a phase synchronization measure value between each pair of electroencephalogram channels.

[0055] The phase synchronization measure value between each pair of electroencephalogram channels is normalized and averaged according to the number of channel pairs to obtain an electroencephalogram phase synchronization index.

[0056] Based on the electroencephalogram phase synchronization index, the arithmetic mean and the standard deviation of the electroencephalogram phase synchronization index of each sliding time window in the last 1 minute are calculated to generate the dynamic statistical characteristics of the electroencephalogram phase synchronization of the user;

[0057] The electroencephalogram phase synchronization index is compared with the dynamic statistical characteristics of the electroencephalogram phase synchronization to determine the real-time synchronization level of the sensory nerve gamma wave band of the user.

[0058] Optionally, the fatigue index and the phase synchronization index determined in real time are used to dynamically adjust the phase parameter, the frequency parameter and the intensity parameter of the visual stimulation signal, the auditory stimulation signal or the tactile stimulation signal through real-time closed-loop optimization to generate a gamma wave band sensory nerve stimulation feedback scheme, specifically:

[0059] The fatigue index and the electroencephalogram phase synchronization index of the current user are collected in real time, and a sensory nerve stimulation feedback adjustment factor is calculated according to a preset ratio;

[0060] Based on the sensory nerve stimulation feedback adjustment factor, the initial phase parameter, the frequency parameter and the intensity parameter of the visual stimulation signal are generated in real time, wherein the initial phase parameter is inversely linearly related to the electroencephalogram phase synchronization index of the user, and the intensity parameter is a reference intensity of the brightness of the visual stimulation light source;

[0061] Based on the sensory nerve stimulation feedback adjustment factor, the initial phase parameter, the frequency parameter and the intensity parameter of the auditory stimulation signal are generated in real time, wherein the initial phase parameter is inversely linearly related to the electroencephalogram phase synchronization index of the user, and the intensity parameter is a reference intensity of the volume of the auditory stimulation sound wave;

[0062] Based on the sensory nerve stimulation feedback adjustment factor, the initial phase parameter, the frequency parameter and the intensity parameter of the tactile stimulation signal are generated in real time, wherein the initial phase parameter is inversely linearly related to the electroencephalogram phase synchronization index of the user, and the intensity parameter is a reference intensity of the driving voltage of the vibration motor of the tactile stimulation;

[0063] The change rate of the fatigue index and the change trend of the electroencephalogram phase synchronization index of the user after the stimulation signal is applied are monitored in real time, and the phase parameter, the frequency parameter and the intensity parameter of the visual stimulation signal, the auditory stimulation signal or the tactile stimulation signal are dynamically adjusted;

[0064] According to the phase parameter, the frequency parameter and the intensity parameter of the visual stimulation signal, the auditory stimulation signal or the tactile stimulation signal that are dynamically adjusted, a gamma wave band sensory nerve stimulation feedback scheme is generated.

[0065] Optionally, the stimulation signal corresponding to the individualized gamma wave band sensory nerve stimulation feedback scheme is output to the user in real time through the wearable feedback device to form a real-time closed-loop feedback, specifically:

[0066] The visual stimulation signal parameters in the gamma band sensory nerve stimulation feedback scheme adapted to the user are sent to the wearable visual stimulation device worn by the user in real time, and the corresponding light signals are generated by the light emitting diode array in real time;

[0067] The auditory stimulation signal parameters in the gamma band sensory nerve stimulation feedback scheme adapted to the user are sent to the wearable auditory stimulation device worn by the user in real time, and the corresponding sound wave signals are generated by the micro speaker in real time;

[0068] The tactile stimulation signal parameters in the gamma band sensory nerve stimulation feedback scheme adapted to the user are sent to the wearable tactile stimulation device worn by the user in real time, and the corresponding mechanical vibration is generated by the built-in vibration motor in real time;

[0069] The fatigue index change condition and the electroencephalogram phase synchrony index change trend of the user after the stimulation signal is applied are monitored and recorded in real time, and the stimulation signal output time is adjusted in real time according to the electroencephalogram phase synchrony change trend;

[0070] The motion posture state of the user during the application of the stimulation signal is monitored in real time, the head, neck and trunk posture parameters of the user are acquired in real time through the inertial measurement unit, and if the posture parameters exceed the preset posture range, the stimulation signal output is paused;

[0071] The sensory nerve stimulation feedback adjustment factor is updated and fed back in real time according to the fatigue index and the electroencephalogram phase synchrony index change trend of the user, and a complete real-time closed-loop feedback is formed.

[0072] The beneficial effects of the present application are:

[0073] (1) The present application realizes deep feature extraction of multi-modal signals by fusing heart rate variability signals, gamma band electroencephalogram signals and motion posture information in real time, and adopts a Mamba linear state space sequence model to effectively improve the robustness and accuracy of fatigue state recognition in a complex motion noise environment, and enhance the reliability and precision of fatigue state monitoring.

[0074] (2) The present application realizes dynamic adjustment and real-time output of sensory nerve gamma band stimulation feedback parameters by constructing an individual fatigue threshold dynamic model and a real-time closed-loop feedback optimization method, significantly improves the timeliness and effectiveness of individualized feedback regulation, and shows better adaptability and effect in long-time fatigue monitoring and feedback intervention scenarios.

[0075] (3) The application solves the problems of high fatigue feedback delay and insufficient individual adaptability in the prior art through a lightweight fully connected neural network decoder and a dynamic probability density manifold method in real-time monitoring and feedback of fatigue state, breaks through the feedback intervention bottleneck caused by fixed stimulation parameters, realizes low-delay and individualized fatigue feedback regulation, and effectively improves the practical application ability of wearable devices in health management and safe operation fields. BRIEF DESCRIPTION OF DRAWINGS

[0076] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application, and should not be taken as limiting the application. In the drawings:

[0077] Fig. 1 A whole flowchart of a wearable fatigue monitoring and feedback method based on deep learning is provided for the application;

[0078] Fig. 2 A multi-modal signal fusion and fatigue state representation vector extraction flowchart of a wearable fatigue monitoring and feedback method based on deep learning is provided for the application;

[0079] Fig. 3 An individual fatigue threshold dynamic modeling and closed-loop feedback optimization flowchart of a wearable fatigue monitoring and feedback method based on deep learning is provided for the application. DETAILED DESCRIPTION

[0080] The application will now be described in further detail with reference to the drawings. These drawings are simplified schematic diagrams, which only schematically show the basic structure of the application, and thus only show the components relevant to the application.

[0081] REFERENCE Figs. 1-3 A wearable fatigue monitoring and feedback method based on deep learning, comprising:

[0082] A wrist-type photoelectric plethysmogram sensor is used to collect the heart rate variability signal of a user in real time, an ear clip-type electroencephalogram electrode is used to collect the gamma band electroencephalogram signal of the user in real time, and an inertial measurement unit is used to collect the body movement posture information of the user in real time, forming an initial multi-modal sequence signal matrix;

[0083] Noise interference in the initial multi-modal sequence signal matrix is removed, the filtered multi-modal sequence signal matrix is normalized, and the timestamps and sampling rates of the multi-modal sequence signal matrix are synchronously fused to form standardized input sequence data;

[0084] The standardized input sequence data is input into a feature dynamic extraction module based on a Mamba linear state space sequence model, long-term time sequence dependence of the standardized input sequence data in a complex motion noise environment is captured in real time by adaptive selective state space kernel verification, and a high-robustness fatigue state representation vector is obtained;

[0085] The fatigue state representation vector is input into a lightweight fully connected neural network decoder to estimate a fatigue index of the current user in real time, and an individual fatigue threshold dynamic model is constructed based on historical sequence data of the user, so that real-time determination of the fatigue state of the current user is realized;

[0086] Based on the user's gamma band electroencephalogram signal collected in real time by the ear clip electroencephalogram electrode, a phase synchronization index of the user's current gamma band electroencephalogram signal is calculated to represent the real-time synchronization level of the user's sensory nerve gamma band;

[0087] Based on the real-time determined fatigue index and the calculated phase synchronization index, the phase parameters, frequency parameters and intensity parameters of the visual stimulation signal, the auditory stimulation signal or the tactile stimulation signal are dynamically adjusted in real time through real-time closed-loop optimization, and a 40Hz gamma band sensory nerve stimulation feedback scheme suitable for the current user is generated in real time;

[0088] The individualized 40Hz gamma band sensory nerve stimulation feedback scheme corresponding to the stimulation signal is output to the current user in real time through a wearable feedback device to form a real-time closed-loop feedback.

[0089] In the embodiment, the heart rate variability signal of the user is collected in real time by the wrist photoplethysmogram sensor, the gamma band electroencephalogram signal of the user is collected in real time by the ear clip electroencephalogram electrode, and the body motion posture information of the user is collected in real time by the inertial measurement unit, forming an initial multi-modal sequence signal matrix, specifically:

[0090] A wrist photoplethysmogram sensor is worn on the radial artery of the user's wrist to continuously collect the pulse wave raw signal of the user by photoelectric reflection method, and the heart rate variability signal is calculated in real time based on the time interval between pulse wave peaks;

[0091] Conductive gel electrodes are worn on the user's bilateral earlobes, and the electroencephalogram potential signal in the gamma band range of the user's cerebral cortex is collected by a differential electrode method, and is transmitted in real time to a brain electro signal acquisition front-end amplification circuit through an electrode lead;

[0092] The brain electro signal acquisition front-end amplification circuit filters the received electroencephalogram potential signal in a band-pass filtering manner, digitizes the filtered signal and generates real-time gamma band electroencephalogram signal data;

[0093] A three-axis inertial measurement unit is fixedly installed at the position of C7 vertebra on the back of the user's neck, and real-time collection of three-axis acceleration data and three-axis angular velocity data of the user's head, neck and torso in the forward and backward inclination, left and right inclination and horizontal rotation directions is performed, and the body movement posture information is converted into Euler angle coordinate transformation algorithm;

[0094] The real-time calculated heart rate variability signal, the real-time generated gamma band electroencephalogram signal data and the real-time acquired body movement posture information are marked and aligned with a unified time stamp, and stored as an initial multi-modal sequence signal matrix at a fixed sampling rate.

[0095] In the embodiment, the noise interference in the initial multi-modal sequence signal matrix is removed, the filtered multi-modal sequence signal matrix is normalized, and the time stamp and the sampling rate of the multi-modal sequence signal matrix are synchronously fused to form standardized input sequence data, specifically:

[0096] A third-order Butterworth low-pass filter algorithm is used to filter the heart rate variability signal calculated in real time by the wrist-type photoelectric plethysmogram sensor, the filter cutoff frequency is set to 0.5 Hz, the high-frequency interference components caused by the user's body micro-movement in the heart rate variability signal are removed, and a filtered heart rate variability signal containing only physiological rhythms is generated;

[0097] A fourth-order Chebyshev band-pass filter algorithm is used to filter the gamma band electroencephalogram signal data generated in real time by the electroencephalogram signal acquisition front-end amplification circuit, the filter passband ripple parameter is set to 0.5 dB, the passband range is strictly limited to 30 Hz to 50 Hz, the user's electromyographic artifacts, environmental electromagnetic interference and low-frequency physiological signal components are filtered out, and pure gamma band electroencephalogram signal data is generated;

[0098] An adaptive Kalman filter algorithm is used to perform dynamic noise suppression processing on the body movement posture information acquired in real time by the three-axis inertial measurement unit, the process noise covariance matrix and the observation noise covariance matrix are adaptively estimated based on the real-time updated motion state vector, the random drift and sensor inherent high-frequency vibration interference in the user's posture measurement process are filtered out, and smooth and stable body movement posture information is generated;

[0099] The filtered heart rate variability signal, the filtered gamma band electroencephalogram signal data and the filtered body movement posture information are respectively subjected to minimum-maximum normalization processing, specifically, the current sampling value of each signal data is subtracted from the minimum value in the past 60 seconds time window of the signal data, and then divided by the difference between the maximum value and the minimum value in the time window, the signal data is standardized to the interval of 0 to 1, and normalized multi-modal signal data is generated;

[0100] The internal clocks of the wrist photoplethysmography sensor, the ear clip electroencephalogram electrode and the three-axis inertial measurement unit are periodically synchronized and calibrated in each data acquisition cycle based on the network time protocol, and the sampling time instants of the multi-modal signal data are synchronized at the millisecond level clock, the normalized multi-modal signal data is aligned and fused point by point and channel by channel according to the unified sampling rate and the unified time stamp, and finally the standardized input sequence data.

[0101] In the embodiment, the feature dynamic extraction module based on the Mamba linear state space sequence model comprises a local state space kernel adaptive perception unit, a cross-modal nonlinear synchronous fusion unit and a kernel parameter online optimization unit.

[0102] The local state space kernel adaptive perception unit is configured to calculate the short-time energy entropy value of each channel in the standardized input sequence data in real time, adjust the local segment length and the local bandwidth of the state space kernel according to the short-time energy entropy value, generate a non-uniform segmented local state space kernel, capture the local long-term time sequence dependence of the standardized input sequence data in a complex motion noise environment, and output a locally enhanced fatigue state representation vector.

[0103] The cross-modal nonlinear synchronous fusion unit is configured to calculate the dynamic mutual information metric value between the modal channels of the standardized input sequence data in a local time window in real time, generate a cross-modal attention weight coefficient according to the dynamic mutual information metric value, fuse the cross-modal attention weight coefficient with the locally enhanced fatigue state representation vector, extract the asynchronous nonlinear cooperative relationship between different modalities of the standardized input sequence data, and output a cross-modal cooperative enhanced fatigue state representation vector.

[0104] The kernel parameter online optimization unit is configured to estimate the mean vector, the covariance matrix and the skewness coefficient in a sliding time window of the standardized input sequence data in real time, dynamically adjust the state transition parameter matrix and the gain matrix of the local state space kernel according to the mean vector, the covariance matrix and the skewness coefficient, online optimize the state space kernel parameters of the local state space kernel, and output a dynamically optimized fatigue state representation vector.

[0105] In the embodiment, the feature dynamic extraction module based on the Mamba linear state space sequence model is used to capture the long-term time sequence dependence of the standardized input sequence data in a complex motion noise environment in real time by using an adaptive selective state space kernel, obtain a high-robustness fatigue state representation vector, and specifically:

[0106] The standardized input sequence data is segmented into sliding time windows with a length of 10 seconds and a step of 1 second, and the sliding time window is sent into the local state space kernel adaptive perception unit one by one, and the short-time energy entropy value of each sampling point in the sliding time window is calculated in real time;

[0107] The local segment position index of each channel is determined in real time based on the short-time energy entropy value calculated in the sliding time window, and the local segment length and the kernel function local bandwidth of the state space kernel are dynamically adjusted according to the position index, and the non-uniform segmented local state space kernel is generated for each channel;

[0108] The non-uniform segmented local state space kernel is used to process the standardized input sequence data in the sliding time window for each channel, and the local long-term time sequence dependence of the standardized input sequence data in a complex motion noise environment is captured in real time, and a local enhanced fatigue state representation vector is obtained;

[0109] The dynamic mutual information metric value is calculated point by point between different channels in the sliding time window, and the cross-modal attention weight coefficient is generated in real time according to the dynamic mutual information metric value;

[0110] The cross-modal attention weight coefficient and the local enhanced fatigue state representation vector are fused to extract the asynchronous nonlinear cooperative relationship between different modalities in the standardized input sequence data, and a cross-modal cooperative enhanced fatigue state representation vector is generated;

[0111] The mean vector, covariance matrix and skewness coefficient of the standardized input sequence data in the sliding time window are calculated, and the state transition parameter matrix and the gain matrix of the state space kernel are dynamically adjusted according to the mean vector, the covariance matrix and the skewness coefficient, and the state space kernel parameters are optimized online, and finally the dynamically optimized fatigue state representation vector is output.

[0112] In the embodiment, the fatigue state representation vector is input into the lightweight fully connected neural network decoder to estimate the fatigue index of the current user in real time, and an individual fatigue threshold dynamic model is constructed based on the historical sequence data of the user, so that the real-time determination of the fatigue state of the current user is realized, specifically:

[0113] The fatigue state representation vector is input into the input layer of the lightweight fully connected neural network decoder, and is mapped to a plurality of hidden layer nodes in real time, and each hidden layer node is connected to the input feature with a dynamic weight parameter;

[0114] The fatigue index historical sequence data output by the lightweight fully connected neural network decoder in the past 30 minutes of the user is statistically calculated in real time, and the fatigue index historical sequence data is segmented into time windows, and the median and quartile difference of the fatigue index sequence in each time window are calculated;

[0115] According to the median and the quartile difference of the fatigue index sequence in each time window, a historical fatigue index distribution state space is constructed with the median as the core, and a dynamic evolution trajectory of the historical fatigue index distribution state space over time is recorded in real time;

[0116] The dynamic evolution trajectory of the historical fatigue index distribution state space is converted into a real-time updated individual fatigue threshold dynamic model, and the upper and lower thresholds in the individual fatigue threshold dynamic model are updated in real time;

[0117] After the output layer of the lightweight fully connected neural network decoder performs nonlinear combination on the hidden layer node feature vectors, the current user fatigue index is output in real time;

[0118] The current user fatigue index is compared with the upper and lower thresholds in the real-time updated individual fatigue threshold dynamic model. When the current user fatigue index is higher than the real-time upper threshold of the individual fatigue threshold dynamic model, it is determined that the user is in a fatigue state, and a real-time determination result of the current user fatigue state is obtained.

[0119] In the embodiment, the individual fatigue threshold dynamic model specifically includes a fatigue index historical data segmentation module, a dynamic probability density real-time estimation module, a probability density manifold construction module, a dynamic manifold feature extraction module, a fatigue threshold space construction module, and a real-time threshold adaptive updating module:

[0120] The fatigue index historical data segmentation module is configured to segment the fatigue index historical sequence data output by the lightweight fully connected neural network decoder within 30 minutes in a sliding window manner with a time window length of 1 minute and a step length of 10 seconds, and generate a plurality of historical sequence data subsegments in real time;

[0121] The dynamic probability density real-time estimation module is configured to perform Gaussian kernel density estimation on each historical sequence data subsegment in real time, and generate a fatigue index probability density distribution of the current subsegment by performing Gaussian kernel function smoothing processing on each data point in real time;

[0122] The probability density manifold construction module is configured to define a local manifold structure in real time according to the gradient direction of the probability density distribution with the peak point of the probability density distribution as a manifold center anchor point, and gradually connect the local manifold structures of the subsegments to form a continuously evolving probability density manifold structure in real time;

[0123] The dynamic manifold feature extraction module is configured to calculate the local curvature value of the continuously evolving probability density manifold structure and the distance change rate between adjacent peak points on the manifold structure in real time, and obtain curvature feature values and distance change rates that represent the dynamic evolution characteristics of the manifold structure in real time;

[0124] The fatigue threshold space construction module is configured to define a higher region of the local curvature value of the continuously evolving probability density manifold structure as an upper bound of the individual fatigue threshold in real time, and define a lower region of the local curvature value as a lower bound of the individual fatigue threshold in real time, to construct the individual fatigue threshold space evolving in real time.

[0125] The real-time threshold adaptive updating module is configured to repeatedly call the dynamic probability density real-time estimation module, the probability density manifold construction module, the dynamic manifold feature extraction module and the fatigue threshold space construction module based on the newly added historical sequence data sub-fragments, to update the individual fatigue threshold space in real time and adaptively adjust the upper and lower bounds of the individual fatigue threshold, and to obtain the individual fatigue index real-time dynamic threshold.

[0126] In the embodiment, the user's current gamma band electroencephalogram signal is calculated based on the user's gamma band electroencephalogram signal collected in real time by the ear clip electroencephalogram electrode, to obtain a phase synchronization index of the user's current gamma band electroencephalogram signal, which is used to represent the real-time synchronization level of the user's sensory nerve gamma band.

[0127] The gamma band electroencephalogram signal data collected in real time by the ear clip electroencephalogram electrode is segmented by sliding window processing with a time window length of 2 seconds and a step length of 0.5 seconds, and the gamma band electroencephalogram signal fragments to be analyzed are generated by sliding the time window one by one.

[0128] The Hilbert transform is performed on the gamma band electroencephalogram signal fragments in each sliding time window channel by channel, and the instantaneous analytic signal and the corresponding instantaneous phase value of each channel electroencephalogram signal fragment are calculated and obtained.

[0129] The difference between the instantaneous phase values of each two electroencephalogram channels in each sliding time window is determined in real time, and the complex exponential mapping is performed on the instantaneous phase value difference, and then the vector average operation is performed, to obtain the phase synchronization measure value between each pair of electroencephalogram channels in real time.

[0130] The phase synchronization measure values obtained between all pairs of electroencephalogram channels are normalized and averaged in terms of the number of channel pairs in real time, to obtain the electroencephalogram phase synchronization index of the current sliding time window.

[0131] The arithmetic mean and the standard deviation of the electroencephalogram phase synchronization indices of each sliding time window in the last continuous 1 minute are calculated in real time based on the electroencephalogram phase synchronization index of the current sliding time window, to generate the dynamic statistical characteristics of the current user's electroencephalogram phase synchronization.

[0132] The electroencephalogram phase synchronization index of the current sliding time window is compared with the dynamic statistical characteristics of the current user's electroencephalogram phase synchronization in real time, to determine the real-time synchronization level of the user's sensory nerve gamma band.

[0133] In this embodiment, the fatigue index based on the real-time determination and the phase synchronization index calculated are used to dynamically adjust the phase parameter, frequency parameter and intensity parameter of the visual stimulation signal, auditory stimulation signal or tactile stimulation signal in real time through real-time closed-loop optimization, to generate a 40Hz gamma band sensory neural stimulation feedback scheme adapted to the current user in real time, specifically:

[0134] The fatigue index and the brain electrical phase synchronization index of the current user are collected in real time, and the sensory neural stimulation feedback adjustment factor is calculated using the fatigue index and the brain electrical phase synchronization index. The sensory neural stimulation feedback adjustment factor is calculated by real-time weighting of the fatigue index and the brain electrical phase synchronization index according to a preset ratio;

[0135] Based on the sensory neural stimulation feedback adjustment factor, the initial phase parameter, frequency parameter and intensity parameter of the visual stimulation signal are generated in real time, wherein the initial phase parameter of the visual stimulation signal is set to be inversely linearly related to the current brain electrical signal phase synchronization index of the user, the frequency parameter is initially set to 40Hz, and the initial intensity parameter is set to the reference intensity of the visual stimulation light source brightness;

[0136] Based on the sensory neural stimulation feedback adjustment factor, the initial phase parameter, frequency parameter and intensity parameter of the auditory stimulation signal are generated in real time, wherein the initial phase parameter of the auditory stimulation signal is set to be inversely linearly related to the current brain electrical signal phase synchronization index of the user, the frequency parameter is initially set to 40Hz, and the initial intensity parameter is set to the reference intensity of the auditory stimulation sound wave volume;

[0137] Based on the sensory neural stimulation feedback adjustment factor, the initial phase parameter, frequency parameter and intensity parameter of the tactile stimulation signal are generated in real time, wherein the initial phase parameter of the tactile stimulation signal is set to be inversely linearly related to the current brain electrical signal phase synchronization index of the user, the frequency parameter is initially set to 40Hz, and the initial intensity parameter is set to the reference intensity of the tactile stimulation vibration motor driving voltage;

[0138] The change rate of the fatigue index and the change trend of the brain electrical phase synchronization index of the user after the stimulation signal is applied are monitored in real time, and the phase parameter, frequency parameter and intensity parameter of the visual stimulation signal, auditory stimulation signal or tactile stimulation signal are dynamically adjusted according to the change rate of the fatigue index and the change trend of the brain electrical phase synchronization index of the user;

[0139] According to the phase parameter, frequency parameter and intensity parameter of the dynamically adjusted visual stimulation signal, auditory stimulation signal or tactile stimulation signal, a 40Hz gamma band sensory neural stimulation feedback scheme adapted to the current user is generated in real time.

[0140] In this embodiment, the individualized 40Hz gamma band sensory neural stimulation feedback scheme corresponding to the stimulation signal is output in real time to the current user through the wearable feedback device, forming a real-time closed-loop feedback, specifically:

[0141] The visual stimulation signal parameters in the 40Hz gamma band sensory neural stimulation feedback scheme adapted to the current user are sent in real time to the wearable visual stimulation device worn by the user, and the light signal corresponding to the parameters is generated in real time by the light-emitting diode array of the visual stimulation device to apply real-time stimulation to the user's visual system.

[0142] The auditory stimulation signal parameters in the 40Hz gamma band sensory neural stimulation feedback scheme adapted to the current user are sent in real time to the wearable auditory stimulation device worn by the user, and the sound wave signal corresponding to the parameters is generated in real time by the micro speaker of the auditory stimulation device to apply real-time stimulation to the user's auditory system.

[0143] The tactile stimulation signal parameters in the 40Hz gamma band sensory neural stimulation feedback scheme adapted to the current user are sent in real time to the wearable tactile stimulation device worn by the user, and the mechanical vibration corresponding to the parameters is generated in real time by the vibration motor built-in in the tactile stimulation device to apply real-time stimulation to the user's body surface tactile system.

[0144] The changes in fatigue index and the trend of phase synchronization index of the user's sensory neural gamma band EEG after the application of visual, auditory and tactile stimulation signals are monitored and recorded in real time, and the output duration of the stimulation signal is adjusted in real time according to the trend of phase synchronization of the user's sensory neural gamma band EEG signal.

[0145] The user's motion posture state during the application of the stimulation signal is monitored in real time, and the head, neck and trunk posture parameters of the user are obtained in real time by the inertial measurement unit. If the posture parameters exceed the preset posture range, the output of the stimulation signal is suspended in real time.

[0146] According to the changes in the user's fatigue index and the trend of phase synchronization of the EEG, the sensory neural stimulation feedback adjustment factor is updated and fed back in real time, forming a complete real-time closed-loop feedback with fatigue monitoring and feedback as the core.

[0147] Example 1:

[0148] In order to verify the feasibility of the application in implementation, the application is applied to a certain intelligent wearable device research and development project, mainly aiming at real-time fatigue state monitoring and feedback intervention for users who work continuously for a long time, in order to reduce the decline in work efficiency and safety risks caused by fatigue state. In the actual application scene, the user wears a wearable monitoring device based on a wrist-type photoplethysmography sensor, an ear clip-type electroencephalogram electrode and an inertial measurement unit, and real-time collection of the user's heart rate variability signal, gamma band electroencephalogram signal and body movement posture information forms an initial multi-modal sequence signal matrix. Traditional fatigue monitoring methods usually use a single physiological signal or simple multi-modal signal fusion, which is difficult to accurately distinguish fatigue state from external motion noise interference, resulting in low fatigue recognition accuracy and large feedback delay. Moreover, due to the lack of effective individualized feedback optimization strategies, the feedback stimulation effect is not good.

[0149] In the specific implementation process, first, the third-order Butterworth low-pass filtering algorithm is used to process the real-time collected heart rate variability signal, and the cutoff frequency is set to 0.5 Hz to remove motion artifact interference; at the same time, the fourth-order Chebyshev band-pass filtering algorithm is used to process the real-time collected gamma band electroencephalogram signal, and the passband range is 30 Hz to 50 Hz to effectively filter out electromyographic interference and environmental noise; and the adaptive Kalman filtering algorithm is used to process the posture information obtained by the three-axis inertial measurement unit in real time, effectively reducing the measurement drift and sensor inherent vibration interference. Subsequently, after the filtered multi-modal signals are normalized by minimum-maximum, a unified sampling rate and a unified timestamp are aligned and fused through the network time protocol to generate standardized input sequence data.

[0150] Next, the standardized input sequence data is segmented by using a sliding time window length of 10 seconds and a step length of 1 second, the local segment length and the kernel function local bandwidth of the state space kernel are dynamically determined by real-time calculation of short-time energy entropy values, and a non-uniform segmented local state space kernel is generated, so as to real-time capture the local long-term time sequence dependence in the complex motion environment. The dynamic mutual information metric value between different modalities is calculated in real time to generate cross-modality attention weight coefficients, which are fused with the local enhanced representation vector to finally obtain a high-robustness cross-modality collaborative enhanced fatigue state representation vector.

[0151] Subsequently, the fatigue state representation vector is input into a lightweight fully connected neural network decoder in real time to dynamically map and output the real-time fatigue index of the current user; and the historical fatigue index sequence data of the user in the past 30 minutes is statistically calculated, the median and the interquartile range of each segment of data are calculated, the historical fatigue index distribution state space with the median as the core is constructed, and the individual fatigue threshold dynamic model is updated in real time. By comparing the current fatigue index with the individual fatigue threshold dynamic model in real time, the real-time fatigue state of the user is accurately determined.

[0152] Meanwhile, the phase synchrony index of the gamma band electroencephalogram signal of the user is calculated in real time through Hilbert transform to evaluate the real-time synchrony level of the sensory nerve gamma band of the user; the sensory nerve stimulation feedback adjustment factor is calculated in combination with the real-time fatigue index and the brain electrical phase synchrony index, a 40Hz gamma band visual, auditory and tactile stimulation feedback scheme adapted to the current state of the user is dynamically generated through a real-time closed-loop optimization algorithm, and individualized stimulation signals are applied in real time through an array of light-emitting diodes, a micro speaker and a vibration motor to realize closed-loop feedback regulation of the fatigue state of the user.

[0153] In the actual effect evaluation link, 5 healthy volunteers are selected to test in a continuous 2-hour high-intensity cognitive task scenario, the fatigue index prediction value obtained by the method is compared and analyzed with the actual cognitive performance and subjective fatigue score, and the results are shown in Table 1.

[0154] Table 1 Comparison data table of predicted value and actual evaluation index of fatigue monitoring method

[0155]

[0156] From the data given in Table 1, it can be clearly seen that the fatigue index predicted by the method has a significant correlation and consistency with the actual task reaction time and subjective fatigue score. Specifically, taking user U01 as an example, the fatigue index prediction value reaches 0.78, the actual task reaction time measurement value is 485ms, the prediction value is 492ms, the difference between the two is only 7ms, the error is about 1.44%, the actual subjective fatigue score is 7.9, and the predicted score is 7.8, the difference is very small, indicating that the method has high accuracy in fatigue state evaluation. For user U02, the fatigue index prediction value is lower, which is 0.65, compared with the actual task reaction time 435ms, the predicted reaction time 428ms has an error of about 1.61%, and the actual subjective fatigue score 6.5 and the predicted score 6.7 have an error of only 0.2, which further verifies that the method also has stable and reliable prediction performance in the condition of moderate fatigue state.

[0157] The fatigue index prediction value of user U03 is 0.82, which belongs to a higher fatigue state level, the actual value of the task reaction time is 512ms, the prediction value is 519ms, the error is 1.37%, the actual value of the subjective score is 8.1, and the prediction value is 8.0, the deviation is only 0.1, which shows that the method can accurately reflect the fatigue state of the user in the condition of high fatigue state. User U04 is in a lower fatigue state level, the fatigue index prediction value is 0.56, the actual reaction time is 406ms, the prediction value is 412ms, the error is controlled within 1.48%, and the error of the actual value and the predicted value of the subjective score is only 0.2, which again reflects that the method still has excellent monitoring accuracy in the situation of lower fatigue state.

[0158] The fatigue index prediction value of the user U05 is 0.71, the task reaction time measured value is 453ms, the prediction value is 447ms, the error is about 1.32%, the actual subjective fatigue score is 7.2 points, and the prediction score is 7.3 points, the error is negligible, which again confirms that the method has high precision and good stability under different fatigue degrees. The above data analysis can clearly show that the present application can extract fatigue state features in depth by real-time fusion of multi-modal physiological signals and implement dynamic closed-loop feedback control, which has obvious advantages over traditional technology in fatigue state prediction accuracy and stability, and can effectively improve the reliability and effect of fatigue recognition and feedback intervention in practical application.

[0159] The present application can extract fatigue state features in depth by real-time fusion of multi-modal physiological signals based on the Mamba linear state space sequence model, construct an individual fatigue threshold dynamic model, and dynamically generate an individualized sensory stimulation feedback scheme combined with a real-time closed-loop optimization mechanism, which significantly improves the fatigue state monitoring accuracy and feedback intervention effectiveness compared with traditional methods, and has clear practical application value and promotion potential.

[0160] The above is only the preferred embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical solution and inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A deep learning based wearable fatigue monitoring and feedback method, characterized in that, The method comprises the following steps: Collecting heart rate variability signals, gamma band electroencephalogram signals and body movement posture information of a user to form an initial multi-modal sequence signal matrix; Removing noise in the initial multi-modal sequence signal matrix and performing normalization processing, synchronously fusing time stamps and sampling rates to form standardized input sequence data; Inputting the standardized input sequence data into a feature dynamic extraction module based on a Mamba linear state space sequence model, using an adaptive selective state space kernel to capture long-term time series dependence in a complex motion noise environment in real time to obtain a fatigue state representation vector; Inputting the fatigue state representation vector into a lightweight fully connected neural network decoder to estimate a user fatigue index in real time, and constructing an individual fatigue threshold dynamic model based on historical sequence data of the user to determine the fatigue state of the user in real time; Based on the gamma band electroencephalogram signals of the user collected in real time by ear clip electrodes, calculating a phase synchrony index of the gamma band electroencephalogram signals of the user to represent the real-time synchronization level of the sensory nerve gamma band of the user; Based on the real-time determined fatigue index and phase synchrony index, dynamically adjusting the phase parameters, frequency parameters and intensity parameters of the visual stimulation signal, the auditory stimulation signal or the tactile stimulation signal through real-time closed-loop optimization to generate a gamma band sensory nerve stimulation feedback scheme; Outputting the stimulation signal corresponding to the individualized gamma band sensory nerve stimulation feedback scheme to the user through a wearable feedback device to form a real-time closed-loop feedback.

2. The wearable fatigue monitoring and feedback method based on deep learning according to claim 1, characterized in that, The method for collecting heart rate variability signals, gamma band electroencephalogram signals and body movement posture information of a user to form an initial multi-modal sequence signal matrix comprises the following steps: Wearing a wrist-type photoelectric plethysmogram sensor on the radial artery of the wrist of the user to collect raw pulse wave signals of the user in real time and calculate heart rate variability signals; Wearing conductive gel electrodes on the bilateral earlobes of the user to collect electroencephalogram signals of the cerebral cortex of the user in the gamma band range in real time in a differential electrode mode; Filtering and digitizing the electroencephalogram signals by an electroencephalogram signal acquisition front-end amplification circuit to generate real-time gamma band electroencephalogram signal data; Fixing a three-axis inertial measurement unit on the C7 vertebra of the back of the neck of the user to collect three-axis acceleration data and three-axis angular velocity data of the head, neck and torso of the user in real time, and converting the data into body movement posture information through an Euler angle coordinate transformation algorithm; Aligning the heart rate variability signals calculated in real time, the gamma band electroencephalogram signal data generated in real time and the body movement posture information acquired in real time with a unified time stamp marker, and storing them as an initial multi-modal sequence signal matrix at a fixed sampling rate.

3. The wearable fatigue monitoring and feedback method based on deep learning according to claim 1, characterized in that, The method for removing noise in the initial multi-modal sequence signal matrix and performing normalization processing, synchronously fusing time stamps and sampling rates to form standardized input sequence data comprises the following steps: Filtering the heart rate variability signals calculated in real time by the wrist-type photoelectric plethysmogram sensor using a third-order Butterworth low-pass filter algorithm to generate filtered heart rate variability signals; Filtering the gamma band electroencephalogram signal data generated in real time by the electroencephalogram signal acquisition front-end amplification circuit using a fourth-order Chebyshev band-pass filter algorithm to generate filtered gamma band electroencephalogram signal data; Adaptive Kalman filtering algorithm is used to process the body motion posture information obtained by the triaxial inertial measurement unit in real time to generate filtered body motion posture information; The filtered heart rate variability signal, the filtered gamma band electroencephalogram signal data and the filtered body motion posture information are subjected to minimum-maximum normalization processing respectively to generate normalized multi-modal signal data; The internal clocks of the wrist photoplethysmography sensor, the ear clip electroencephalogram electrode and the triaxial inertial measurement unit are periodically synchronized based on the network time protocol, and the sampling time instants of the multi-modal signal data are synchronized at the millisecond level, so that the normalized multi-modal signal data are aligned and fused point by point and channel by channel according to a unified sampling rate and a unified timestamp to generate standardized input sequence data.

4. The wearable fatigue monitoring and feedback method based on deep learning according to claim 1, characterized in that, The feature dynamic extraction module based on the Mamba linear state space sequence model comprises a local state space kernel adaptive perception unit, a cross-modal nonlinear synchronous fusion unit and a kernel parameter online optimization unit: The local state space kernel adaptive perception unit is configured to calculate the short-time energy entropy value of each channel in the standardized input sequence data in real time, adjust the local segment length and the local bandwidth of the kernel function of the state space kernel online according to the short-time energy entropy value, generate a non-uniform segmented local state space kernel, and output a locally enhanced fatigue state representation vector; The cross-modal nonlinear synchronous fusion unit is configured to calculate the dynamic mutual information metric value between the channels of the standardized input sequence data in real time, generate a cross-modal attention weight coefficient according to the dynamic mutual information metric value, and fuse the cross-modal attention weight coefficient with the locally enhanced fatigue state representation vector to output a cross-modal synergistically enhanced fatigue state representation vector; The kernel parameter online optimization unit is configured to estimate the mean vector, the covariance matrix and the skewness coefficient of the standardized input sequence data in real time, dynamically adjust the state transition parameter matrix and the gain matrix of the local state space kernel according to the mean vector, the covariance matrix and the skewness coefficient, and output a dynamically optimized fatigue state representation vector.

5. The wearable fatigue monitoring and feedback method based on deep learning according to claim 1, characterized in that, The standardized input sequence data is input into the feature dynamic extraction module based on the Mamba linear state space sequence model, and the long-term temporal dependence relationship in a complex motion noise environment is captured in real time by using an adaptive selective state space kernel to obtain a fatigue state representation vector, specifically as follows: The standardized input sequence data is segmented by a preset sliding time window, and each sliding time window is sent into the local state space kernel adaptive perception unit to calculate the short-time energy entropy value of each channel in real time; The local segment position index of each channel is determined in real time based on the short-time energy entropy value, and the local segment length and the local bandwidth of the kernel function of the state space kernel are dynamically adjusted to generate a non-uniform segmented local state space kernel; The non-uniform segmented local state space kernel is used to process the standardized input sequence data in the sliding time window channel by channel to capture the local long-term temporal dependence relationship and obtain a locally enhanced fatigue state representation vector; The dynamic mutual information metric value is calculated point by point between different channels of the standardized input sequence data in the sliding time window, and the cross-modal attention weight coefficient is generated in real time; The cross-modal attention weight coefficient is fused with the locally enhanced fatigue state representation vector to generate a cross-modal synergistically enhanced fatigue state representation vector; The mean vector, covariance matrix and skewness coefficient of the standardized input sequence data in the sliding time window are calculated, the state transition parameter matrix and gain matrix of the state space kernel are dynamically adjusted, the state space kernel parameters are optimized online, and the dynamically optimized fatigue state representation vector is output.

6. The wearable fatigue monitoring and feedback method based on deep learning according to claim 1, characterized in that, The fatigue state representation vector is input into the lightweight fully connected neural network decoder to estimate the user fatigue index in real time, and an individual fatigue threshold dynamic model is constructed based on the user historical sequence data to determine the user fatigue state in real time, specifically: The fatigue state representation vector is input into the lightweight fully connected neural network decoder, and is mapped to multiple hidden layer nodes in real time, each hidden layer node being connected to the input feature with a dynamic weight parameter; The fatigue index historical sequence data of the user in the past 30 minutes is counted, the fatigue index historical sequence data is segmented by time window, and the median and interquartile range of the fatigue index sequence of each time window are calculated; According to the median and interquartile range of the fatigue index sequence in each time window, a historical fatigue index distribution state space is constructed, and the dynamic evolution trajectory of the historical fatigue index distribution state space is recorded in real time; The dynamic evolution trajectory of the historical fatigue index distribution state space is converted into an individual fatigue threshold dynamic model, and the upper and lower thresholds of the individual fatigue threshold dynamic model are adaptively updated in real time; The current user fatigue index is output after the hidden layer node feature vector is nonlinearly combined by the output layer of the lightweight fully connected neural network decoder; The current user fatigue index is compared with the upper and lower thresholds of the individual fatigue threshold dynamic model, and when the fatigue index is higher than the real-time upper threshold, it is determined that the user is in a fatigue state, and the user fatigue state determination result is obtained.

7. The wearable fatigue monitoring and feedback method based on deep learning according to claim 1, characterized in that, The individual fatigue threshold dynamic model specifically includes a fatigue index historical data segmentation module, a dynamic probability density real-time estimation module, a probability density manifold construction module, a dynamic manifold feature extraction module, a fatigue threshold space construction module, and a real-time threshold adaptive updating module: The fatigue index historical data segmentation module is used to segment the fatigue index historical sequence data in the past 30 minutes into sub-pieces by sliding window segmentation with a preset time window, and generate historical sequence data sub-pieces; The dynamic probability density real-time estimation module is used to perform Gaussian kernel density estimation on the historical sequence data sub-pieces in real time to generate the fatigue index probability density distribution of the current sub-piece; The probability density manifold construction module is used to take the peak point of the probability density distribution as the manifold center anchor point, define the local manifold structure according to the gradient direction of the probability density distribution, connect the local manifold structures of the sub-pieces, and form a continuously evolving probability density manifold structure; The dynamic manifold feature extraction module is used to calculate the local curvature value of the continuously evolving probability density manifold structure and the distance change rate between adjacent peak points on the manifold structure, and obtain the dynamic evolution features of the manifold structure; The fatigue index historical sequence data of the user in the past 30 minutes is counted, the fatigue index historical sequence data is segmented by time window, and the median and interquartile range of the fatigue index sequence of each time window are calculated; According to the median and interquartile range of the fatigue index sequence in each time window, a historical fatigue index distribution state space is constructed, and the dynamic evolution trajectory of the historical fatigue index distribution state space is recorded in real time; The dynamic evolution trajectory of the historical fatigue index distribution state space is converted into an individual fatigue threshold dynamic model, and the upper and lower thresholds of the individual fatigue threshold dynamic model are adaptively updated in real time; The current user fatigue index is output after the hidden layer node feature vector is nonlinearly combined by the output layer of the lightweight fully connected neural network decoder; The current user fatigue index is compared with the upper and lower thresholds of the individual fatigue threshold dynamic model, and when the fatigue index is higher than the real-time upper threshold, it is determined that the user is in a fatigue state, and the user fatigue state determination result is obtained. The fatigue threshold space construction module is configured to define an upper limit of the individual fatigue threshold in a region with a higher local curvature value of the continuously evolving probability density manifold structure and define a lower limit of the individual fatigue threshold in a region with a lower local curvature value, and construct the individual fatigue threshold space. The real-time threshold adaptive updating module is configured to call the dynamic probability density real-time estimation module, the probability density manifold construction module, the dynamic manifold feature extraction module, and the fatigue threshold space construction module based on the newly added historical sequence data sub-fragments, update the individual fatigue threshold space in real time, adjust the upper and lower limits of the individual fatigue threshold, and obtain the dynamic threshold of the individual fatigue index. 8.The deep learning based wearable fatigue monitoring and feedback method of claim 1, wherein, The user's gamma band electroencephalogram signal is collected in real time based on the ear clip type electroencephalogram electrode, the phase synchronization index of the user's gamma band electroencephalogram signal is calculated, and the real-time synchronization level of the user's sensory nerve gamma band is represented. Specifically, the gamma band electroencephalogram signal data collected in real time by the ear clip type electroencephalogram electrode is segmented by sliding window processing with a preset time window to generate a gamma band electroencephalogram signal fragment. The Hilbert transform is performed on the gamma band electroencephalogram signal fragment in the sliding time window channel by channel to obtain the instantaneous analytic signal and the instantaneous phase value of each channel electroencephalogram signal fragment. The difference between the instantaneous phase values of each two electroencephalogram channels is determined, and the complex exponential mapping is performed on the instantaneous phase value difference, and then the vector average operation is performed to obtain the phase synchronization measurement value between the electroencephalogram channel pairs. The phase synchronization measurement value between the electroencephalogram channel pairs is normalized and averaged according to the number of channel pairs to obtain the electroencephalogram phase synchronization index. Based on the electroencephalogram phase synchronization index, the arithmetic mean and the standard deviation of the electroencephalogram phase synchronization index of each sliding time window in the last continuous 1 minute are calculated to generate the dynamic statistical characteristics of the user's electroencephalogram phase synchronization. The electroencephalogram phase synchronization index and the dynamic statistical characteristics of the electroencephalogram phase synchronization are compared numerically to determine the real-time synchronization level of the user's sensory nerve gamma band. Based on the real-time determination of the fatigue index and the phase synchronization index, the phase parameters, the frequency parameters and the intensity parameters of the visual stimulation signal, the auditory stimulation signal or the tactile stimulation signal are dynamically adjusted in real time through real-time closed-loop optimization to generate a gamma band sensory nerve stimulation feedback scheme. Specifically, 9.The fatigue monitoring and feedback method based on deep learning according to claim 1, wherein, The fatigue index and the electroencephalogram phase synchronization index of the current user are collected in real time, and the sensory nerve stimulation feedback adjustment factor is calculated according to a preset ratio. Based on the sensory nerve stimulation feedback adjustment factor, the initial phase parameters, the frequency parameters and the intensity parameters of the visual stimulation signal are generated in real time, wherein the initial phase parameters are inversely linearly related to the electroencephalogram phase synchronization index of the user, and the intensity parameter is the reference intensity of the brightness of the visual stimulation light source. Based on the sensory nerve stimulation feedback adjustment factor, the initial phase parameters, the frequency parameters and the intensity parameters of the auditory stimulation signal are generated in real time, wherein the initial phase parameters are inversely linearly related to the electroencephalogram phase synchronization index of the user, and the intensity parameter is the reference intensity of the volume of the auditory stimulation sound wave. ​ The initial phase parameter, frequency parameter and intensity parameter of the haptic stimulation signal are generated in real time based on the sensory nerve stimulation feedback adjustment factor, wherein the initial phase parameter is inversely linearly related to the phase synchronization index of the user's brain electrical signal, and the intensity parameter is the reference intensity of the driving voltage of the haptic stimulation vibration motor; The phase parameter, frequency parameter and intensity parameter of the visual stimulation signal, auditory stimulation signal or haptic stimulation signal are dynamically adjusted in real time according to the change rate of the user's fatigue index and the change trend of the brain electrical phase synchronization index after the stimulation signal is applied; The γ-band sensory nerve stimulation feedback scheme is generated according to the dynamically adjusted phase parameter, frequency parameter and intensity parameter of the visual stimulation signal, auditory stimulation signal or haptic stimulation signal.

10. The wearable fatigue monitoring and feedback method based on deep learning according to claim 1, characterized in that, The stimulation signal corresponding to the individualized γ-band sensory nerve stimulation feedback scheme is output to the user in real time through the wearable feedback device to form a real-time closed-loop feedback, specifically: The visual stimulation signal parameters in the γ-band sensory nerve stimulation feedback scheme adapted to the user are sent to the wearable visual stimulation device worn by the user in real time, and the corresponding light signal is generated in real time through the light-emitting diode array; The auditory stimulation signal parameters in the γ-band sensory nerve stimulation feedback scheme adapted to the user are sent to the wearable auditory stimulation device worn by the user in real time, and the corresponding sound wave signal is generated in real time through the micro speaker; The haptic stimulation signal parameters in the γ-band sensory nerve stimulation feedback scheme adapted to the user are sent to the wearable haptic stimulation device worn by the user in real time, and the corresponding mechanical vibration is generated in real time through the built-in vibration motor; The change of the user's fatigue index and the change trend of the brain electrical phase synchronization index after the stimulation signal is applied are monitored and recorded in real time, and the output time of the stimulation signal is adjusted in real time according to the change trend of the brain electrical phase synchronization; The motion posture state of the user during the application of the stimulation signal is monitored in real time, and the head, neck and trunk posture parameters of the user are obtained in real time through the inertial measurement unit. If the posture parameters exceed the preset posture range, the output of the stimulation signal is paused; The sensory nerve stimulation feedback adjustment factor is updated and fed back in real time according to the change trend of the user's fatigue index and brain electrical phase synchronization index, and a complete real-time closed-loop feedback is formed.

Images