Fatigue detection method, apparatus and system

Abstract

The application discloses a fatigue detection method, device and system, wherein the method comprises the following steps: acquiring a physiological signal of a detected person; wherein the physiological signal comprises an electroencephalogram signal and a peripheral physiological signal; performing signal processing on the physiological signal to obtain a processed physiological signal; wherein the processed physiological signal comprises a processed electroencephalogram signal and a processed peripheral physiological signal; performing feature extraction on the processed electroencephalogram signal and the processed peripheral physiological signal to obtain a multi-scale fusion feature; and inputting the multi-scale fusion feature into a fatigue detection model to obtain a fatigue state of the detected person.

Description

Technical Field

[0001] This invention relates to the field of fatigue technology, and more specifically, to a fatigue detection method, apparatus, and system. Background Technology

[0002] Mental fatigue refers to a decline in brain function caused by prolonged mental activity, stress, emotional fluctuations, or cognitive load. It can be caused by a variety of factors, including prolonged cognitive work, emotional stress, insufficient sleep, excessive stress, and decreased motivation. Mental fatigue may lead to reduced work efficiency, memory loss, difficulty concentrating, mood swings, impaired judgment, and deteriorating work performance. Chronic mental fatigue may also increase the risk of mental health problems such as chronic fatigue syndrome, anxiety disorders, and depression. Therefore, it is necessary to detect mental fatigue.

[0003] Existing methods for detecting mental fatigue include subjective questionnaires, behavioral measurements, and psychological measurements. However, these methods have low accuracy and are difficult to objectively reflect the degree of mental fatigue in an individual. Summary of the Invention

[0004] One objective of this invention is to provide a new technical solution for fatigue detection.

[0005] According to a first aspect of the present invention, a fatigue detection method is provided, comprising:

[0006] Acquire physiological signals from the subject; wherein, the physiological signals include electroencephalogram (EEG) signals and peripheral physiological signals;

[0007] The physiological signals are processed to obtain processed physiological signals; wherein, the processed physiological signals include processed electroencephalogram (EEG) signals and processed peripheral physiological signals.

[0008] Feature extraction is performed on the processed EEG signals and the processed peripheral physiological signals to obtain multi-scale fusion features;

[0009] The multi-scale fusion features are input into the fatigue detection model to obtain the fatigue state of the subject.

[0010] Optionally, the step of extracting features from the processed EEG signal and the processed peripheral physiological signal to obtain multi-scale fusion features includes:

[0011] Feature extraction is performed on the processed EEG signals and the processed peripheral physiological signals to obtain multi-scale features;

[0012] The multi-scale features are fused to obtain multi-scale fused features.

[0013] Optionally, the step of extracting features from the processed EEG signal and the processed peripheral physiological signal to obtain multi-scale features includes:

[0014] The processed EEG signal and the processed peripheral physiological signal are input into a convolutional neural network to obtain local spatiotemporal features;

[0015] The processed EEG signal and the processed peripheral physiological signal are subjected to frequency domain transformation to obtain frequency domain data of the EEG signal and frequency domain data of the peripheral physiological signal; wherein, the frequency domain data is time-frequency data;

[0016] The frequency domain data of the EEG signal is input into a recurrent neural network to obtain EEG time series features;

[0017] The frequency domain data of the peripheral physiological signals are input into the recurrent neural network to obtain peripheral time series features.

[0018] Optionally, after obtaining the frequency domain data of the electroencephalogram (EEG) signal and the frequency domain data of the peripheral physiological signal, the method further includes:

[0019] The frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal are subjected to spectral simplification processing in order to extract time series features from the frequency domain data after spectral simplification.

[0020] Optionally, the sampling duration and sampling frequency of the EEG signal and the peripheral physiological signal are the same, and the signal processing of the physiological signal to obtain the processed physiological signal includes:

[0021] The EEG signal is subjected to a bandpass filter at a set frequency to obtain a filtered EEG signal.

[0022] The filtered EEG signal and the peripheral physiological signal are segmented according to a set segment length to obtain the processed physiological signal.

[0023] Optionally, the process of training the fatigue detection model includes:

[0024] Obtain a sample set; wherein the sample set includes a training sample set and a test sample set, and the training sample set includes training EEG signals and training peripheral physiological signals corresponding to different fatigue states;

[0025] Feature extraction is performed on the training EEG signal and training peripheral physiological signal of each sample in the training sample set to obtain a multi-scale fusion feature set;

[0026] The initial fatigue detection model is trained using the multi-scale fusion feature set to obtain the trained fatigue detection model.

[0027] The trained fatigue detection model is evaluated using the test sample set to obtain model evaluation index values;

[0028] If the model evaluation index value is greater than or equal to the preset index value, the trained fatigue detection model shall be used as the fatigue detection model.

[0029] Optionally, obtaining the sample set includes:

[0030] The fatigue self-assessment values ​​and physiological signals of multiple subjects were obtained; wherein, the physiological signals included electroencephalogram (EEG) signals and peripheral physiological signals.

[0031] The physiological signal is segmented according to a set segment length to obtain multiple signal segments;

[0032] Based on the fatigue self-assessment values, the multiple signal segments are labeled to obtain multiple samples;

[0033] The multiple samples are segmented to obtain the training sample set and the test sample set.

[0034] Optionally, the evaluation index values ​​include at least one of accuracy, sensitivity, and specificity.

[0035] According to a second aspect of this disclosure, a fatigue detection device is provided, including a memory and a processor, the memory being used to store executable instructions; the processor being used to operate under the control of the instructions to perform the method as described in the first aspect.

[0036] According to a third aspect of this disclosure, a fatigue detection system is provided, comprising: an electroencephalogram (EEG) sensor, a peripheral sensor, and a fatigue detection device as described in the second aspect;

[0037] The electroencephalogram (EEG) sensor is used to collect the EEG signals of the subject and transmit them to the fatigue detection device.

[0038] The peripheral sensor is used to collect peripheral physiological signals of the subject and send them to the fatigue detection device.

[0039] One of the advantages of this invention is that by comprehensively analyzing electroencephalogram (EEG) signals and peripheral physiological signals, a more comprehensive understanding of the mental state of the subject can be obtained. Furthermore, this method of fatigue detection based on multimodal physiological signals can provide more dimensional information, which helps to more accurately assess the subject's level of fatigue and potential risks. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0041] Figure 1 This is a schematic diagram of the hardware structure of a fatigue detection system according to an embodiment of the present invention;

[0042] Figure 2 This is a schematic flowchart of a fatigue detection method according to an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of a network structure for extracting multi-scale fusion features according to an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of a fatigue detection device according to an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of a fatigue detection system according to an embodiment of the present invention. Detailed Implementation

[0046] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0047] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0048] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0049] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0050] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0051] The complexity and unpredictability of flight missions require pilots to maintain a high level of concentration for extended periods. Prolonged periods of concentration and vigilance deplete cognitive resources, leading to mental fatigue. This fatigue may manifest as inattention, decreased reaction time, weakened cognitive function, and impaired decision-making ability. Accumulated mental fatigue not only increases the risk of operational errors but may also cause pilots to make judgment mistakes at critical moments, thereby jeopardizing the success and safety of the flight mission. Therefore, it is necessary to assess the level of pilot mental fatigue.

[0052] Existing fatigue assessment methods mainly rely on single physiological signals, such as facial image signals or eye movement feature signals. Although these methods can provide some information about fatigue status, they often cannot comprehensively and accurately reflect the pilot's mental state, resulting in an inaccurate assessment of whether the pilot is experiencing mental fatigue.

[0053] This application embodiment, by comprehensively analyzing the pilot's electroencephalogram (EEG) signals and peripheral physiological signals, can provide a more comprehensive understanding of the pilot's mental state. Furthermore, this fatigue detection method based on multimodal physiological signals can provide more dimensional information, helping to more accurately assess the pilot's fatigue level and potential risks.

[0054] The following description uses the detection of pilot mental fatigue as an example to illustrate the principle of this embodiment. However, it is understood that the specific application of mental fatigue detection is not limited to this. In fact, the mental fatigue detection method of this embodiment can be used to detect mental fatigue of any object in any scenario. Without departing from the principle described in this embodiment, the detection method of this embodiment can be applied to the detection of mental fatigue of any object in any scenario.

[0055] <Hardware Configuration>

[0056] Figure 1 This is a block diagram of the hardware configuration of a fatigue detection system 100 according to an embodiment of the present invention.

[0057] like Figure 1 As shown, the fatigue detection system 100 includes an electroencephalogram (EEG) sensor 1000, a peripheral sensor 2000, and a fatigue detection device 3000.

[0058] The brainwave sensor can be an EEG (electroencephalography) sensor, an ECoG (electrocortical electroencephalography) sensor, a SEEG (stereoelectroencephalography) sensor, etc., and there is no limitation here.

[0059] In some embodiments, the brainwave sensor is an EEG sensor.

[0060] In this embodiment, EEG measures the electroencephalogram (EEG) signals generated by the brain by placing electrodes on the scalp. The electrodes can be arranged according to the international 10-20 system.

[0061] The EEG sensor can collect the pilot's brain signals during flight missions at a first sampling frequency of 200Hz and send them to the fatigue detection device 3000.

[0062] In this embodiment, refer to Figure 1 As shown, the EEG sensor 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, etc.

[0063] Processor 1100 may be a mobile processor. Memory 1200 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), and non-volatile memory such as a hard disk. Interface device 1300 includes, for example, a USB interface and a headphone jack. Communication device 1400 is capable of wired or wireless communication. Communication device 1400 may include short-range communication devices, such as any device that performs short-range wireless communication based on short-range wireless communication protocols such as Hilink, WiFi (IEEE 802.11), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, and LiFi. Communication device 1400 may also include long-range communication devices, such as any device that performs WLAN, GPRS, or 2G / 3G / 4G / 5G long-range communication. Display device 1500 is, for example, an LCD screen or a touch screen, used to display the acquired EEG signals. Input device 1600 may include, for example, a touch screen or a keyboard.

[0064] In this embodiment, the memory 1200 of the EEG sensor 1000 is used to store instructions for controlling the processor 1100 to operate in order to at least execute the fatigue detection method implemented by the EEG sensor according to any embodiment of the present invention. Those skilled in the art can design the instructions according to the disclosed scheme of the present invention. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

[0065] Despite Figure 1 The invention illustrates multiple devices of the EEG sensor 1000, but the invention may refer to only some of these devices; for example, the EEG sensor 1000 may refer only to the memory 1200 and the processor 1100.

[0066] Peripheral sensors, also known as peripheral physiological monitoring sensors (PPMs), are a class of devices used to monitor and record human peripheral physiological activities.

[0067] In this application, the peripheral sensor may be at least one of an electrocardiogram (ECG) sensor, an electrical activity of the skin (EDA) sensor, an electrooculogram (EOG) sensor, an electromyography (sEMG) sensor, and a heart rate detector.

[0068] In some embodiments, the peripheral sensors may be electrocardiogram (ECG) sensors and electrical skin activity (EDA) sensors. The ECG sensor acquires signals of cardiac electrical activity, while the EDA sensor acquires signals of changes in skin conductance.

[0069] The peripheral sensor collects the pilot's peripheral physiological signals during the flight mission at a second sampling frequency and sends them to the fatigue detection device 3000.

[0070] In this embodiment, refer to Figure 1 As shown, the peripheral sensor 2000 may include a processor 2100, a memory 2200, an interface device 2300, a communication device 2400, a display device 2500, an input device 2600, etc.

[0071] Processor 2100 may be a mobile processor. Memory 2200 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), and non-volatile memory such as a hard disk. Interface device 2300 includes, for example, a USB interface and a headphone jack. Communication device 2400 may be capable of wired or wireless communication. Communication device 2400 may include short-range communication devices, such as any device that performs short-range wireless communication based on short-range wireless communication protocols such as Hilink, WiFi (IEEE 802.11), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, and LiFi. Communication device 2400 may also include long-range communication devices, such as any device that performs WLAN, GPRS, or 2G / 3G / 4G / 5G long-range communication. Display device 2500 is, for example, an LCD screen or a touch screen, used to display the collected peripheral physiological signals. Input device 2600 may include, for example, a touch screen or a keyboard.

[0072] Despite Figure 1 The invention illustrates multiple devices of the peripheral sensor 2000; however, the invention may refer to only some of these devices, for example, the peripheral sensor 2000 may refer only to the memory 2200 and the processor 2100.

[0073] The fatigue detection device 3000 is used to receive brain signals collected by the EEG sensor 1000 at a first sampling frequency during the flight mission, and to receive peripheral physiological signals collected by the peripheral sensor 2000 at a second sampling frequency during the flight mission, and to perform fatigue detection based on the brain signals and peripheral physiological signals.

[0074] The fatigue detection device 3000 may include a processor 3100, a memory 3200, an interface device 3300, a communication device 3400, a display device 3500, an input device 3600, etc.

[0075] Processor 3100 may be a mobile processor. Memory 3200 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), and non-volatile memory such as a hard disk. Interface device 3300 includes, for example, a USB interface and a headphone jack. Communication device 3400 may be capable of wired or wireless communication. Communication device 3400 may include short-range communication devices, such as any device that performs short-range wireless communication based on short-range wireless communication protocols such as Hilink, WiFi (IEEE 802.11), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, and LiFi. Communication device 3400 may also include long-range communication devices, such as any device that performs WLAN, GPRS, or 2G / 3G / 4G / 5G long-range communication. Display device 3500 is, for example, an LCD screen or a touch screen. Input device 3600 may include, for example, a touch screen or a keyboard.

[0076] In this embodiment, the memory 3200 of the fatigue detection device 3000 is used to store instructions for controlling the processor 3100 to operate to at least execute the fatigue detection method performed by the fatigue detection device 3000 according to any embodiment of the present invention. Those skilled in the art can design the instructions according to the disclosed scheme of the present invention. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

[0077] Despite Figure 1 The present invention illustrates multiple devices of the fatigue detection device 3000, but may refer to only some of these devices; for example, the fatigue detection device 3000 may refer only to the memory 3200 and the processor 3100.

[0078] It should be understood that, despite Figure 1Only one EEG sensor 1000 and one peripheral sensor 2000, and one fatigue detection device 3000 are shown, but this does not mean that the number of each is limited. The fatigue detection system 100 may contain multiple EEG sensors 1000 and / or peripheral sensors 2000 and / or fatigue detection devices 3000.

[0079] <Method Implementation>

[0080] Figure 2 This is a flowchart illustrating a fatigue detection method according to an embodiment of the present invention, which can be implemented by a fatigue detection device 3000.

[0081] according to Figure 2 As shown, the fatigue detection method of this embodiment may include the following steps S2100 to S2400:

[0082] Step S2100: Obtain the physiological signals of the subject being tested.

[0083] In this embodiment, corresponding to a flight scenario, the subject of the test is the pilot. The pilot's physiological signals are acquired, including electroencephalogram (EEG) signals and peripheral physiological signals. That is, in a flight scenario, step S2100 can also involve acquiring the EEG signals collected by the EEG sensor at a first sampling frequency and the peripheral physiological signals collected by the peripheral sensor at a second sampling frequency during the pilot's flight mission.

[0084] In this embodiment, the brainwave sensor can be, for example, an EEG (electroencephalography) sensor, with a first sampling frequency of 200 Hz. The electrode arrangement of the EEG sensor can be based on a 10-20 system layout. The peripheral sensor can be, for example, an electrocardiogram (ECG) sensor and an electrical activity of the skin (EDA) sensor, with a second sampling frequency of 200 Hz or other sampling frequencies, such as 260 Hz, which is not limited here.

[0085] Those skilled in the art should understand that the types of EEG sensors and peripheral sensors described herein are merely examples and should not be construed as limitations on EEG sensors and peripheral sensors. EEG sensors can also be other types of EEG sensors, such as ECoG (electrocortical electroencephalography), and peripheral sensors can also be other types of sensors, such as electromyography (EMG) sensors.

[0086] Step S2200: Perform signal processing on the physiological signal to obtain the processed physiological signal.

[0087] In this embodiment, signal processing can be, for example, amplification, segmentation, noise reduction, etc., and is not limited here. After signal processing of the EEG signal and the peripheral physiological signal, processed EEG signal and processed peripheral physiological signal are obtained, that is, the processed physiological signal includes processed EEG signal and processed peripheral physiological signal.

[0088] In some embodiments, the sampling duration and sampling frequency of the EEG sensor and the peripheral sensor are the same, for example, both are 1 hour, and the sampling frequency can be, for example, 200 Hz. In this case, the EEG signals collected by the EEG sensor and the peripheral physiological signals collected by the peripheral sensor are synchronized in time, which facilitates fatigue detection and analysis based on the corresponding performance of the two signals at the same time.

[0089] Based on this, in step S2200, the physiological signal is processed to obtain the processed physiological signal, including: steps S2200.1 to S2200.2.

[0090] Step S2200.1: Perform bandpass filtering on the EEG signal at a set frequency to obtain the filtered EEG signal.

[0091] In this embodiment, the set frequency can be 0.5-40Hz.

[0092] Those skilled in the art should understand that the bandpass filtering process in the 0.5-40Hz frequency range is common knowledge in the field. It is used to remove signal components that are not within this frequency range in order to reduce artifacts and highlight useful signals of brain activity. Therefore, the specific processing procedure of bandpass filtering will not be described here.

[0093] Step S2200.2: With a set segment length, perform signal segmentation processing on the filtered EEG signal and the peripheral physiological signal respectively to obtain the processed physiological signal.

[0094] In this embodiment, since the sampling frequency and sampling duration of the EEG signal and the peripheral physiological signal are the same, if the two signals are segmented with a set segment length, frames of two channels with the same duration can be provided during the sliding window process.

[0095] In some examples, the segment length can be set to, for example, 30 seconds, one minute, etc., and the overlap can be 0 seconds, etc.

[0096] According to the embodiments of this application, by performing signal segmentation processing on the filtered physiological signal with a set segment length, multiple signal segments can be obtained. Then, feature extraction is performed on these multiple signal segments. Compared with feature extraction on the signal without signal segmentation processing, feature extraction on these multiple signal segments can facilitate the extraction of features related to mental fatigue and reduce the omission of features.

[0097] Step S2300: Feature extraction is performed on the processed EEG signal and the processed peripheral physiological signal to obtain multi-scale fusion features.

[0098] In this embodiment, multiple single-scale features, i.e., multi-scale features, can be extracted from the processed EEG signal and the processed peripheral physiological signal. Then, the multi-scale features of the processed EEG signal and the processed peripheral physiological signal are fused to obtain multi-scale fused features. Among them, the single-scale features can be time-domain features (such as mean, variance, standard deviation, maximum and minimum values, skewness, kurtosis, etc.), frequency-domain features (power spectral density, spectral center frequency, etc.), nonlinear features (signal approximate entropy, sample entropy, Lyapunov exponent, etc., nonlinear dynamic features), etc., which are not limited here.

[0099] In some embodiments, step S2300 involves extracting features from the processed EEG signal and the processed peripheral physiological signal to obtain multi-scale fusion features, including steps S2300.1 to S2300.2.

[0100] Step S2300.1: Extract features from the processed EEG signals and processed peripheral physiological signals to obtain multi-scale features.

[0101] In this embodiment, multi-scale features may include time-domain features, frequency-domain features, etc.

[0102] In some embodiments, the network architecture for extracting multi-scale features can be as follows: Figure 3 As shown, it includes three parallel networks: a convolutional neural network (CNN) and two long short-term memory (LSTM) networks. Based on... Figure 3 The feature extraction network architecture shown in step S2300.1 extracts features from the processed EEG features and the processed peripheral physiological signals to obtain multi-scale features, including steps SA1 to SA4.

[0103] Step SA1 involves inputting the processed EEG signals and processed peripheral physiological signals into a convolutional neural network to obtain local spatiotemporal features.

[0104] In this embodiment, similar to existing convolutional neural networks, the CNN used in this step consists of six consecutive layers. Each layer involves one convolution, one max pooling, and one batch normalization layer. Dropout and average pooling layers are also applied. Unlike existing convolutional neural networks, an additional layer with 10 neurons is introduced into the CNN—a fully connected (dense) layer after the convolutional and pooling layers and before the output layer. The purpose of this additional layer is to integrate the features extracted by the preceding convolutional and pooling layers, learn the complex relationships between features, and select features related to mental fatigue.

[0105] Local spatiotemporal features can be local time series patterns (such as oscillations at a specific frequency or periodic patterns of heartbeats), frequency features (such as alpha waves, beta waves, theta waves, and delta waves in EEG, and heart rate variability (HRV) in ECG signals), waveform features (such as P waves, QRS waves, and T waves in ECG signals, and spike waves or other abnormal waveforms in EEG signals), etc.

[0106] The following example illustrates the process of a convolutional neural network extracting local spatiotemporal features:

[0107] Assuming the sampling duration of the EEG signal and the peripheral physiological signal is t, and the sampling frequency is the same, the processed EEG signal can be represented by vector x1 and the processed peripheral physiological signal can be represented by vector x2. Therefore, the sliding window process of signal segmentation can provide frames with two channels of the same size, that is, the i-th frame fi is composed of the processed EEG signal. and processed peripheral physiological signal vectors By cascading Each frame It extracts features independently of other frames; that is, it uses a two-channel CNN to analyze features simultaneously in each frame. and Furthermore, convolution is performed on each channel with a one-dimensional filter to simultaneously extract features from the processed EEG signal and the processed peripheral physiological signal. An additional layer of 10 neurons then performs selection processing on the features extracted by convolution to select features related to mental fatigue, thereby improving the targeting of feature extraction.

[0108] Step SA2 involves performing frequency domain transformation on the processed EEG signal and the processed peripheral physiological signal to obtain the frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal.

[0109] In this embodiment, the processed EEG signal and the processed peripheral physiological signal can be converted to the frequency domain by short-time Fourier transform to obtain the frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal.

[0110] Continuing with the example above, the i-th frame fi is composed of processed EEG signals. and processed peripheral physiological signal vectors By cascading To facilitate short-time Fourier transform, each frame can be... Divide into n m-dimensional blocks 。 Here, "block" refers to a finer-grained segmentation performed within each frame, with each block containing m sample points. For each of the n m-dimensional blocks into which the signal is segmented, a short-time Fourier transform (STFT) is applied to convert the time domain to the frequency domain, yielding frequency domain data of the EEG signal and the peripheral physiological signal.

[0111] Among them, frequency domain data is time-frequency data, that is, frequency domain data can provide information on the frequency distribution of the signal at different points in time.

[0112] Those skilled in the art should understand that the short-time Fourier transform technique is not a conventional technique in the field, therefore the specific process of the short-time Fourier transform will not be described in detail here.

[0113] Step SA3: Input the frequency domain data of the EEG signal into a recurrent neural network to obtain EEG time series features.

[0114] In this embodiment, a Recurrent Neural Network (RNN) is a neural network specifically designed to process sequential data. It captures the temporal dependencies, i.e., time-series features, by recursively applying the same weights to each element in the sequence. RNNs can characterize the temporal evolution of different frames. Long Short-Term Memory (LSTM) networks are a special type of RNN that can capture the memory effect between consecutive frames. They are designed to address the gradient vanishing and gradient exploding problems encountered by traditional RNNs when processing long sequences of data; that is, LSTM networks belong to the RNN family.

[0115] The architecture of an LSTM network consists of two layers, each containing n LSTM units. The complexity of this architecture depends on two parameters: the output y of the LSTM network and the input m of the first-layer units. It is worth noting that the network complexity decreases as n increases.

[0116] Step SA4: Input the frequency domain data of the peripheral physiological signal into the recurrent neural network to obtain peripheral time series features.

[0117] In this embodiment, the recurrent neural network is... Figure 3 The Long Short-Term Memory (LSTM) network in the middle.

[0118] In some embodiments, after obtaining the frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal in step SA2, before performing steps SA3 and SA4 to input the signal into the recurrent neural network, the method further includes: performing spectral simplification processing on the frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal to extract time series features from the spectrally simplified frequency domain data.

[0119] In this embodiment, since the result of STFT frequency domain transformation typically includes positive and negative frequency components, and the negative frequency component is the complex conjugate of the positive frequency component, retaining only the positive spectrum component for time series feature extraction does not affect the integrity of the features. Therefore, after obtaining the frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal, spectral simplification processing is performed on these two frequency domain data respectively to remove the negative frequency component.

[0120] According to the embodiments of this application, before extracting time series features from the frequency domain data, spectral simplification processing is performed on the frequency domain data. This can reduce the amount of data in the frequency domain data while retaining most of the useful information, thereby improving the efficiency of time series feature extraction without affecting the integrity of the time series features.

[0121] Step S2300.2: The multi-scale features are fused to obtain multi-scale fused features.

[0122] Continuing with the example above, such as Figure 3 As shown, the outputs of the three networks are local spatiotemporal features, time-series features of EEG signals, and peripheral time-series features. The outputs of these three networks are passed to a dense layer, which is used to fuse the multi-scale features to obtain multi-scale fused features.

[0123] According to the embodiments of this application, by fusing multi-scale features into a single vector, not only can the performance of the subsequent fatigue detection model be improved, but the structure of the fatigue detection model can also be simplified, the computational burden reduced, and the robustness of the fatigue detection model improved.

[0124] Step S2400: Input the multi-scale fusion features into the fatigue detection model to obtain the fatigue state of the subject.

[0125] In this embodiment, the fatigue detection model can be a binary classification model. That is, after inputting multi-scale fusion features into the fatigue detection model, the fatigue state of the detected subject is obtained as either "drowsy state" or "awake state". In a flight scenario, the fatigue state of the pilot is obtained as either "drowsy state" or "awake state".

[0126] In some embodiments, the process of training the fatigue detection model includes steps S3100 to S3500.

[0127] Step S3100: Obtain the sample set.

[0128] In this embodiment, the sample set includes a training sample set and a test sample set. The training sample set includes training EEG signals and training peripheral physiological signals corresponding to different fatigue states; specifically, the training sample set includes training EEG signals and training peripheral physiological signals corresponding to the "awake state" and the "drowsy state." Similarly, the test sample set also includes test EEG signals and test peripheral physiological signals corresponding to different fatigue states; the test sample set includes test EEG signals and test peripheral physiological signals corresponding to the "awake state" and the "drowsy state."

[0129] The “drowsy state” and “awake state” in the sample set are the sample labels, that is, the true values ​​of the samples, which reflect the pilots’ true fatigue state.

[0130] In some embodiments, obtaining the sample set in step S3100 includes steps S3100.1 to S3100.4.

[0131] Step S3100.1: Obtain fatigue self-assessment values ​​and physiological signals from multiple subjects.

[0132] In this embodiment, the subjects had experience operating flight simulation games to simulate real flight. Furthermore, an experimental paradigm was designed to induce mental fatigue in the subjects, allowing for the collection of their physiological signals and self-assessment values ​​of fatigue. These physiological signals included electroencephalogram (EEG) signals and peripheral physiological signals.

[0133] For example, the experimental paradigm for the flight scenario is specifically designed as follows:

[0134] I. Experimental Equipment

[0135] Flight simulator: A flight simulator that provides a flight control environment that simulates real flight, including a flight joystick, electric control loads, and other control options.

[0136] EEG sensor: 32-channel EEG system with a sampling frequency of 2000Hz. The 32 electrodes are arranged according to the international 10-20 system.

[0137] PPMs (peripheral sensors): collect peripheral physiological signals at a sampling frequency of 2000Hz.

[0138] Wireless keyboard: Attached to the flight joystick, used to record the subject's sleepiness rating input.

[0139] II. Experimental Procedure

[0140] Twenty healthy university students (4 women and 16 men, mean age 25.7 ± 2.6) participated in the experiment. Each participant had experience operating flight simulation games and was asked to avoid alcohol and caffeine the day before the experiment and to get 6-8 hours of sleep.

[0141] Before the experiment began, subjects wore EEG and PPM sensors to collect their electroencephalogram (EEG) and peripheral physiological signals during the experiment. The equipment was calibrated and signal quality was ensured. Then, the experimental procedure was initiated as follows:

[0142] 1) Subjects underwent simulated night flight, with the flight environment set as a monotonous, low-stimulation night flight scenario to maximize the induction of fatigue.

[0143] 2) A beeping sound is played every 5 minutes, which is played through the audio system of the flight simulator.

[0144] 3) After hearing the buzzer, the subjects entered their subjective sleepiness score according to the Karolinska Sleepiness Scale (KSS), that is, the fatigue self-assessment value, with a score range of 1 to 9, where 1 indicates complete wakefulness and 9 indicates extreme drowsiness. The KSS scale is shown in Table 1 below.

[0145] 4) The experiment lasted for 1 hour, and the physiological signals of the subjects were obtained, namely, electroencephalogram (EEG) signals and peripheral physiological signals. In addition, a fatigue self-assessment value was recorded every 5 minutes, for a total of 12 times.

[0146] Table 1. Karolinska Sleepiness Scale Scoring

[0147]

[0148] Step S3100.2: The physiological signal is segmented by setting a segment length to obtain multiple signal segments.

[0149] In this embodiment, the segment length can be set to 30 seconds or 1 minute, etc., and the specific value of the segment length is not limited here.

[0150] It should be noted that the segment length setting must correspond to the fatigue self-assessment value. If the segment length is set to 30 seconds and the fatigue self-assessment value is set every 5 minutes, then every 10 signal segments correspond to one fatigue self-assessment value. If the segment length is set to 1 minute and the fatigue self-assessment value is set every 5 minutes, then every 5 signal segments correspond to one fatigue self-assessment value.

[0151] Step S3100.3: Based on the fatigue self-assessment value, the multiple signal segments are labeled with data to obtain multiple samples.

[0152] In this embodiment, after data annotation of multiple signal segments, one signal segment corresponds to one sample. The number of signal segments is the same as the number of samples.

[0153] For example, if a certain signal segment corresponds to a fatigue self-assessment value of Level 9 (extreme drowsiness), then that signal segment is marked as a sleepy state. If a certain signal segment corresponds to a fatigue self-assessment value of Level 3 (alertness), then that signal segment is marked as a conscious state.

[0154] Step S3100.4: Perform sample segmentation on the multiple samples to obtain the training sample set and the test sample set.

[0155] For example, 70% of these multiple samples can be used as training samples to obtain the training sample set, and 30% can be used as test samples to obtain the test sample set.

[0156] Step S3200: Extract features from the training EEG signals and training peripheral physiological signals of each sample in the training sample set to obtain a multi-scale fusion feature set.

[0157] In this embodiment, the method of extracting features from the training EEG signal and training peripheral physiological signal of any sample in the training sample set to obtain multi-scale fusion features is the same as the method described above, for example, the same as the method described in step S2300, and will not be elaborated here.

[0158] Step S3300: Train the initial fatigue detection model by fusing a multi-scale feature set to obtain the trained fatigue detection model.

[0159] In this embodiment, a suitable machine learning or deep learning algorithm is selected as the base model of the fatigue detection model, such as support vector machine, random forest, neural network, etc., and the model parameters of the base model are initialized to obtain the initial fatigue detection model.

[0160] The initial fatigue detection model is trained using a multi-scale fusion feature set with labels, and the model parameters are adjusted using optimization algorithms (such as gradient descent) to minimize the error between the labels and the model's predictions.

[0161] For example, the model parameters are adjusted using the "Adam" optimizer, and after each iteration, the iterated model is trained using a multi-scale fused feature set to obtain the trained model. Experiments show that after 300 iterations, the error between the label and the model's predictions stabilizes, meeting practical detection requirements.

[0162] Step S3400: Evaluate the trained fatigue detection model using the test sample set to obtain model evaluation index values.

[0163] Continuing with the example above, after each iteration, the trained model is evaluated using a test sample set to obtain the model evaluation index value.

[0164] In one example, the performance of a fatigue detection model can be evaluated using a confusion matrix. This matrix shows the relationship between the model's predictions and the actual labels.

[0165] In some embodiments, the evaluation metric values ​​include at least one of accuracy, sensitivity, and specificity.

[0166] In this embodiment, the fatigue detection model is a binary classification model. For binary classification problems, the confusion matrix usually contains four elements: true positive, false positive, true negative, and false negative.

[0167] In this context, a True Positive (TP) refers to the number of samples correctly predicted as positive (e.g., drowsy state). A False Positive (FP) refers to the number of samples incorrectly predicted as positive (e.g., drowsy state) but actually negative (e.g., awake state). A True Negative (TN) refers to the number of samples correctly predicted as negative (e.g., awake state). A False Negative (FN) refers to the number of samples incorrectly predicted as negative (e.g., awake state) but actually positive (e.g., drowsy state).

[0168] Sensitivity is a metric that measures a model's ability to identify the positive class (drowsy state). It is calculated as the ratio of the number of true positives (TP) to the total number of elements in that class (drowsy state) (TP+FN).

[0169] Specificity is a metric that measures a model’s ability to identify the negative class (awake state). It is calculated as the ratio of the number of true negatives (TN) to the total number of elements in that class (awake state) (TN+FP).

[0170] Sensitivity indicates how well the model identifies drowsy states. Specificity indicates how well the model excludes drowsy states (i.e., correctly identifies waking states).

[0171] In binary classification problems, accuracy intuitively tells the model the percentage of samples that it correctly predicts out of the total number of samples. For example, if a model has an accuracy of 90% on the test set, it means that the model correctly predicts the class of 90% of the samples.

[0172] Those skilled in the art should understand that when evaluating a model, it can be evaluated by at least one of sensitivity, specificity, and accuracy, without limitation here.

[0173] Step S3500: If the model evaluation index value is greater than or equal to the preset index value, the trained fatigue detection model is used as the fatigue detection model.

[0174] In this embodiment, if sensitivity is used as the evaluation index, the preset index value can be, for example, sensitivity of 97.3%. If specificity is used as the evaluation index, the preset index value can be, for example, specificity of 96.6%. If accuracy is used as the evaluation index, the preset index value can be, for example, accuracy of 87%.

[0175] According to the embodiments of this application, by comprehensively analyzing electroencephalogram (EEG) signals and peripheral physiological signals, a more comprehensive understanding of the pilot's mental state can be obtained. Furthermore, this method of fatigue detection based on multimodal physiological signals can provide more dimensional information, which helps to more accurately assess the pilot's fatigue level and potential risks.

[0176] <Device Embodiment>

[0177] Figure 4 This is a schematic block diagram of a fatigue detection device according to an embodiment of the present invention.

[0178] In this embodiment, as Figure 4 As shown, the fatigue detection device 400 includes a memory 410 and a processor 420. The memory 410 is used to store executable instructions; the processor 420 is used to operate according to the control of the instructions to execute the method described in any of the above method embodiments.

[0179] Figure 5 This is a schematic diagram of a fatigue detection system according to an embodiment of the present invention.

[0180] In this embodiment, as Figure 5 As shown, the fatigue detection system 500 includes: an electroencephalogram (EEG) sensor 510, a peripheral sensor 520, and a fatigue detection device 530.

[0181] The electroencephalogram (EEG) sensor 510 is used to collect the EEG signals of the subject and send them to the fatigue detection device 530.

[0182] The peripheral sensor 520 is used to collect peripheral physiological signals of the subject and send them to the fatigue detection device 530.

[0183] In some embodiments, the fatigue detection device 530 is as follows: Figure 4 The fatigue detection device or such Figure 1 The fatigue detection device shown.

[0184] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0185] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0186] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0187] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0188] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0189] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0190] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0191] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0192] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A fatigue detection method, characterized in that, include: Acquire physiological signals from the subject; wherein, the physiological signals include electroencephalogram (EEG) signals and peripheral physiological signals; The physiological signals are processed to obtain processed physiological signals; wherein, the processed physiological signals include processed electroencephalogram (EEG) signals and processed peripheral physiological signals. Feature extraction is performed on the processed EEG signals and the processed peripheral physiological signals to obtain multi-scale fusion features; The multi-scale fusion features are input into the fatigue detection model to obtain the fatigue state of the subject. The process involves extracting features from the processed EEG signals and the processed peripheral physiological signals to obtain multi-scale fusion features, including: Feature extraction is performed on the processed EEG signals and the processed peripheral physiological signals to obtain multi-scale features; The multi-scale features are fused to obtain multi-scale fused features; The process involves extracting features from the processed EEG signals and the processed peripheral physiological signals to obtain multi-scale features, including: The processed EEG signal and the processed peripheral physiological signal are input into a convolutional neural network to obtain local spatiotemporal features; The processed EEG signal and the processed peripheral physiological signal are subjected to frequency domain transformation to obtain frequency domain data of the EEG signal and frequency domain data of the peripheral physiological signal; wherein, the frequency domain data is time-frequency data; The frequency domain data of the EEG signal is input into a recurrent neural network to obtain EEG time series features; The frequency domain data of the peripheral physiological signals are input into the recurrent neural network to obtain peripheral time series features.

2. The method according to claim 1, characterized in that, After obtaining the frequency domain data of the electroencephalogram (EEG) signal and the frequency domain data of the peripheral physiological signal, the method further includes: The frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal are subjected to spectral simplification processing in order to extract time series features from the frequency domain data after spectral simplification.

3. The method according to claim 1, characterized in that, The sampling duration and sampling frequency of the electroencephalogram (EEG) signal and the peripheral physiological signal are the same. The step of signal processing the physiological signal to obtain the processed physiological signal includes: The EEG signal is subjected to a bandpass filter at a set frequency to obtain a filtered EEG signal. The filtered EEG signal and the peripheral physiological signal are segmented according to a set segment length to obtain the processed physiological signal.

4. The method according to claim 1, characterized in that, The process of training the fatigue detection model includes: Obtain a sample set; wherein the sample set includes a training sample set and a test sample set, and the training sample set includes training EEG signals and training peripheral physiological signals corresponding to different fatigue states; Feature extraction is performed on the training EEG signal and training peripheral physiological signal of each sample in the training sample set to obtain a multi-scale fusion feature set; The initial fatigue detection model is trained using the multi-scale fusion feature set to obtain the trained fatigue detection model. The trained fatigue detection model is evaluated using the test sample set to obtain model evaluation index values; If the model evaluation index value is greater than or equal to the preset index value, the trained fatigue detection model shall be used as the fatigue detection model.

5. The method according to claim 4, characterized in that, The acquisition of the sample set includes: The fatigue self-assessment values ​​and physiological signals of multiple subjects were obtained; wherein, the physiological signals included electroencephalogram (EEG) signals and peripheral physiological signals. The physiological signal is segmented according to a set segment length to obtain multiple signal segments; Based on the fatigue self-assessment values, the multiple signal segments are labeled to obtain multiple samples; The multiple samples are segmented to obtain the training sample set and the test sample set.

6. The method according to claim 4, characterized in that, The evaluation index values ​​include at least one of accuracy, sensitivity, and specificity.

7. A fatigue detection device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the fatigue detection method as described in any one of claims 1-6.

8. A fatigue detection system, characterized in that, include: EEG sensor, peripheral sensor and fatigue detection device as described in claim 7; The electroencephalogram (EEG) sensor is used to collect the EEG signals of the subject and transmit them to the fatigue detection device. The peripheral sensor is used to collect the peripheral physiological signals of the subject and send them to the fatigue detection device.

Images