Fatigue detection method, device and system

Abstract

The invention discloses a fatigue detection method, device and system, and the method comprises the steps: obtaining a physiological signal of a detected person; wherein the physiological signals comprise electroencephalogram signals and peripheral physiological signals; performing signal processing on the physiological signal to obtain a processed physiological signal; wherein the processed physiological signal comprises a processed electroencephalogram signal and a processed peripheral physiological signal; performing feature extraction on the processed electroencephalogram signal and the processed peripheral physiological signal to obtain a multi-scale fusion feature; and inputting the multi-scale fusion features into a fatigue detection model to obtain the fatigue state of the detected person.

Description

Technical Field

[0001] The present invention relates to the field of fatigue technology, and more specifically, to a fatigue detection method, device and system. Background Art

[0002] Mental fatigue refers to a state of decreased brain function caused by prolonged mental activity, stress, mood swings, or cognitive load. It may be caused by a variety of factors, such as prolonged cognitive work, emotional stress, lack of sleep, excessive stress, decreased motivation, etc. Mental fatigue may lead to reduced work efficiency, memory loss, distraction, mood swings, decreased judgment, and worsened work performance. Long-term mental fatigue may also increase the risk of mental health problems, such as chronic fatigue syndrome, anxiety, and depression. Therefore, it is necessary to detect mental fatigue.

[0003] Existing methods for detecting mental fatigue include subjective questionnaires, behavioral measurements, psychological measurements, etc. However, these methods have low accuracy and are difficult to objectively reflect the degree of mental fatigue of an individual. Summary of the invention

[0004] An object of an embodiment of the present invention is to provide a new technical solution for fatigue detection.

[0005] According to a first aspect of the present invention, there is provided a fatigue detection method, comprising:

[0006] Acquiring physiological signals of the subject; wherein the physiological signals include electroencephalogram signals and peripheral physiological signals;

[0007] Performing signal processing on the physiological signal to obtain a processed physiological signal; wherein the processed physiological signal includes a processed EEG signal and a processed peripheral physiological signal;

[0008] Extracting features from the processed EEG signal and the processed peripheral physiological signal to obtain multi-scale fusion features;

[0009] The multi-scale fusion features are input into a fatigue detection model to obtain the fatigue state of the subject.

[0010] Optionally, the extracting features of the processed EEG signal and the processed peripheral physiological signal to obtain multi-scale fusion features includes:

[0011] Extracting features from the processed EEG signals and the processed peripheral physiological signals to obtain multi-scale features;

[0012] The multi-scale features are fused to obtain multi-scale fused features.

[0013] Optionally, the extracting features of the processed EEG signal and the processed peripheral physiological signal to obtain multi-scale features includes:

[0014] Inputting the processed EEG signal and the processed peripheral physiological signal into a convolutional neural network to obtain local spatiotemporal features;

[0015] Performing frequency domain conversion processing on the processed EEG signal and the processed peripheral physiological signal to obtain frequency domain data of the EEG signal and frequency domain data of the peripheral physiological signal; wherein the frequency domain data is time-frequency data;

[0016] Inputting the frequency domain data of the EEG signal into a recurrent neural network to obtain EEG time series features;

[0017] The frequency domain data of the peripheral physiological signal is input into the recurrent neural network to obtain peripheral time series features.

[0018] Optionally, after obtaining the frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal, the method further includes:

[0019] The frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal are subjected to spectrum simplification processing, so as to extract time series features through the frequency domain data after the spectrum simplification processing.

[0020] Optionally, the sampling duration and sampling frequency of the EEG signal and the peripheral physiological signal are the same, and the signal processing of the physiological signal to obtain the processed physiological signal includes:

[0021] Performing bandpass filtering of a set frequency on the EEG signal to obtain a filtered EEG signal;

[0022] The filtered EEG signal and the peripheral physiological signal are respectively processed in segments according to the set segment length to obtain the processed physiological signal.

[0023] Optionally, the process of training the fatigue detection model includes:

[0024] Acquire a sample set; wherein the sample set includes a training sample set and a test sample set, and the training sample set includes training EEG signals and training peripheral physiological signals corresponding to different fatigue states;

[0025] Extracting features from the training EEG signal and the training peripheral physiological signal of each sample in the training sample set to obtain a multi-scale fusion feature set;

[0026] The initial fatigue detection model is trained by using the multi-scale fusion feature set to obtain a trained fatigue detection model;

[0027] Evaluate the trained fatigue detection model using the test sample set to obtain a model evaluation index value;

[0028] When the model evaluation index value is greater than or equal to the preset index value, the trained fatigue detection model is used as the fatigue detection model.

[0029] Optionally, obtaining a sample set includes:

[0030] Obtaining fatigue self-assessment values ​​and physiological signals of multiple subjects; wherein the physiological signals include electroencephalogram signals and peripheral physiological signals;

[0031] Performing signal segmentation processing on the physiological signal with a set segment length to obtain a plurality of signal segments;

[0032] Performing data labeling on the multiple signal segments according to the fatigue self-assessment values ​​to obtain multiple samples;

[0033] The multiple samples are segmented to obtain the training sample set and the test sample set.

[0034] Optionally, the evaluation index value includes at least one of accuracy, sensitivity, and specificity.

[0035] According to a second aspect of the present disclosure, a fatigue detection device is provided, comprising a memory and a processor, wherein the memory is used to store executable instructions; the processor is used to operate according to the control of the instructions to execute the method as described in the first aspect.

[0036] According to a third aspect of the present disclosure, there is provided a fatigue detection system, comprising: an electroencephalogram sensor, a peripheral sensor, and the fatigue detection device as described in the second aspect;

[0037] The electroencephalogram sensor is used to collect the electroencephalogram signal of the subject and send it to the fatigue detection device;

[0038] The peripheral sensor is used to collect peripheral physiological signals of the subject and send them to the fatigue detection device.

[0039] One beneficial effect of the present invention is that by comprehensively analyzing EEG signals and peripheral physiological signals, a more comprehensive understanding of the mental state of the subject can be obtained. In addition, this method of fatigue detection based on multimodal physiological signals can provide information in more dimensions, which helps to more accurately assess the fatigue level and potential risks of the subject. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0041] Figure 1 is a schematic diagram of the hardware structure of a fatigue detection system according to an embodiment of the present invention;

[0042] Figure 2 is a schematic flow chart of a fatigue detection method according to an embodiment of the present invention;

[0043] Figure 3 is a schematic diagram of a network structure for extracting multi-scale fusion features according to an embodiment of the present invention;

[0044] Figure 4 is a schematic structural diagram of a fatigue detection device according to an embodiment of the present invention;

[0045] Figure 5 is a schematic structural diagram of a fatigue detection system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0047] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0048] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0049] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0050] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0051] The complexity and unpredictability of flight missions require pilots to maintain a high level of mental concentration for a long time. Long-term mental concentration and vigilance maintenance will lead to the consumption of cognitive resources, resulting in mental fatigue. This fatigue may manifest as distraction, decreased reaction speed, weakened cognitive function, and weakened decision-making ability. The accumulation of mental fatigue may not only increase the risk of operational errors, but also cause pilots to make errors in judgment at critical moments, thereby endangering the success and safety of the flight mission. Therefore, it is necessary to assess the degree of mental fatigue of pilots.

[0052] Existing fatigue assessment methods mainly rely on single physiological signals, such as facial image signals or eye movement feature signals. Although these methods can provide certain fatigue status information, they often cannot fully and accurately reflect the mental state of the pilot, resulting in an inaccurate assessment of whether the pilot is mentally fatigued.

[0053] The embodiment of the present application can provide a more comprehensive understanding of the pilot's mental state by comprehensively analyzing the pilot's EEG signals and peripheral physiological signals. In addition, this method of fatigue detection based on multimodal physiological signals can provide information in more dimensions, which helps to more accurately assess the pilot's fatigue level and potential risks.

[0054] The principle of this embodiment is described below by taking the detection of mental fatigue of pilots as an example, but it can be understood that the specific application of mental fatigue detection is not limited to this. In fact, the mental fatigue detection method of this embodiment can be used to detect mental fatigue of any object in any scene. Without departing from the principle described in this embodiment, the detection method of this embodiment can be applied to the detection of mental fatigue of any object in any scene.

[0055] <Hardware Configuration>

[0056] Figure 1 is a block diagram of a hardware configuration of a fatigue detection system 100 according to an embodiment of the present invention.

[0057] like Figure 1 As shown, the fatigue detection system 100 includes an electroencephalogram sensor 1000 and a peripheral sensor 2000 , and a fatigue detection device 3000 .

[0058] The EEG sensor may be an EEG (electroencephalogram) sensor, an ECoG (electrocorticogram) sensor, a SEEG (stereoencephalogram) sensor, etc., which is not limited here.

[0059] In some embodiments, the electroencephalographic sensor is an EEG sensor.

[0060] In this embodiment, EEG measures the electrical signals generated by the brain by placing electrodes on the scalp, wherein the electrodes may be arranged according to the international 10-20 system.

[0061] The EEG sensor can collect EEG signals of the pilot during the flight mission at a first sampling frequency and send them to the fatigue detection device 3000. The first sampling frequency is 200 Hz.

[0062] In this embodiment, refer to Figure 1 As shown, the EEG sensor 1000 may include a processor 1100 , a memory 1200 , an interface device 1300 , a communication device 1400 , a display device 1500 , an input device 1600 , and the like.

[0063] The processor 1100 may be a mobile processor. The memory 1200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, etc. The interface device 1300 may include, for example, a USB interface, a headphone interface, etc. The communication device 1400 may, for example, be capable of wired or wireless communication. The communication device 1400 may include a short-range communication device, for example, any device for short-range wireless communication based on short-range wireless communication protocols such as Hilink protocol, WiFi (IEEE 802.11 protocol), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, LiFi, etc. The communication device 1400 may also include a remote communication device, for example, any device for WLAN, GPRS, 2G / 3G / 4G / 5G remote communication. The display device 1500 may be, for example, a liquid crystal display, a touch display, etc., and the display device 1500 is used to display the collected EEG signals. The input device 1600 may include, for example, a touch screen, a keyboard, etc.

[0064] In this embodiment, the memory 1200 of the EEG sensor 1000 is used to store instructions, which are used to control the processor 1100 to operate to at least perform the fatigue detection method implemented by the EEG sensor according to any embodiment of the present invention. Technicians can design instructions according to the scheme disclosed in the present invention. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0065] Despite Figure 1 , multiple devices of the EEG sensor 1000 are shown; however, the present invention may only involve some of the devices, for example, the EEG sensor 1000 only involves the memory 1200 and the processor 1100 .

[0066] Peripheral sensors, namely, peripheral physiological signal sensors (Peripheral Physiological Monitoring Sensors, referred to as PPMs) are a type of equipment used to monitor and record peripheral physiological activities of the human body.

[0067] In the present application, the peripheral sensor may be at least one of an electrocardiogram (ECG) sensor, an electrodermal activity (EDA) sensor, an electrooculogram (EOG) sensor, a sEMG sensor, and a heart rate detector.

[0068] In some embodiments, the peripheral sensor may be an electrocardiogram (ECG) sensor and an electrodermal activity (EDA) sensor. The ECG sensor collects signals of cardiac electrical activity, while the EDA sensor collects signals of changes in skin conductance.

[0069] The peripheral sensor collects peripheral physiological signals of the pilot during the execution of the flight mission at the second sampling frequency and sends them to the fatigue detection device 3000 .

[0070] In this embodiment, refer to Figure 1 As shown, the peripheral sensor 2000 may include a processor 2100, a memory 2200, an interface device 2300, a communication device 2400, a display device 2500, an input device 2600, and the like.

[0071] The processor 2100 may be a mobile version processor. The memory 2200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, etc. The interface device 2300 may include, for example, a USB interface, a headphone interface, etc. The communication device 2400 may, for example, be capable of wired or wireless communication. The communication device 2400 may include a short-range communication device, for example, any device for short-range wireless communication based on short-range wireless communication protocols such as Hilink protocol, WiFi (IEEE 802.11 protocol), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, LiFi, etc. The communication device 2400 may also include a remote communication device, for example, any device for WLAN, GPRS, 2G / 3G / 4G / 5G remote communication. The display device 2500 may be, for example, a liquid crystal display, a touch display, etc., and the display device 2500 is used to display the collected peripheral physiological signals. The input device 2600 may include, for example, a touch screen, a keyboard, etc.

[0072] Despite Figure 1 2 shows multiple devices of the peripheral sensor 2000 , but the present invention may only involve some of the devices, for example, the peripheral sensor 2000 only involves the memory 2200 and the processor 2100 .

[0073] The fatigue detection device 3000 is used to receive the EEG signals of the pilot collected by the EEG sensor 1000 at a first sampling frequency during the performance of the flight mission, and to receive the peripheral physiological signals of the pilot collected by the peripheral sensor 2000 at a second sampling frequency during the performance of the flight mission, and to perform fatigue detection based on the EEG signals and the peripheral physiological signals.

[0074] The fatigue detection device 3000 may include a processor 3100 , a memory 3200 , an interface device 3300 , a communication device 3400 , a display device 3500 , an input device 3600 , and the like.

[0075] The processor 3100 may be a mobile version processor. The memory 3200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, etc. The interface device 3300 may include, for example, a USB interface, a headphone interface, etc. The communication device 3400 may, for example, be capable of wired or wireless communication. The communication device 3400 may include a short-range communication device, for example, any device for short-range wireless communication based on short-range wireless communication protocols such as Hilink protocol, WiFi (IEEE 802.11 protocol), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, LiFi, etc. The communication device 3400 may also include a remote communication device, for example, any device for WLAN, GPRS, 2G / 3G / 4G / 5G remote communication. The display device 3500 may be, for example, a liquid crystal display, a touch display, etc. The input device 3600 may include, for example, a touch screen, a keyboard, etc.

[0076] In this embodiment, the memory 3200 of the fatigue detection device 3000 is used to store instructions, and the instructions are used to control the processor 3100 to operate to at least perform the fatigue detection method performed by the fatigue detection device 3000 according to any embodiment of the present invention. A technician can design instructions according to the scheme disclosed in the present invention. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0077] Despite Figure 1 , multiple devices of the fatigue detection device 3000 are shown; however, the present invention may only involve some of the devices, for example, the fatigue detection device 3000 only involves the memory 3200 and the processor 3100.

[0078] It should be understood that although Figure 1Only one EEG sensor 1000, one peripheral sensor 2000, and one fatigue detection device 3000 are shown, but this does not mean to limit their respective numbers. The fatigue detection system 100 may include multiple EEG sensors 1000 and / or peripheral sensors 2000 and / or fatigue detection devices 3000.

[0079] <Method Example>

[0080] Figure 2 3 is a flow chart of a fatigue detection method according to an embodiment of the present invention, which can be implemented by a fatigue detection device 3000.

[0081] according to Figure 2 As shown, the fatigue detection method of this embodiment may include the following steps S2100 to S2400:

[0082] Step S2100, obtaining physiological signals of the subject.

[0083] In this embodiment, corresponding to the flight scenario, the detected person is a pilot. The pilot's physiological signals are obtained, wherein the physiological signals include EEG signals and peripheral physiological signals. That is, in the flight scenario, step S2100 may also be to obtain the EEG signals collected by the EEG sensor at the first sampling frequency and the peripheral physiological signals collected by the peripheral sensor at the second sampling frequency during the pilot's flight mission.

[0084] In this embodiment, the EEG sensor may be, for example, an EEG (electroencephalogram) sensor, the first sampling frequency may be 200 Hz, and the electrode arrangement of the EEG sensor may be arranged based on a 10-20 system. The peripheral sensor may be, for example, an electrocardiogram (ECG) sensor and an electrodermal activity (EDA) sensor, etc., the second sampling frequency may be 200 Hz, or other sampling frequencies, such as 260 Hz, which are not limited here.

[0085] Those skilled in the art should understand that the types of EEG sensors and peripheral sensors described here are merely examples and should not be taken as limitations on EEG sensors and peripheral sensors. The EEG sensors may also be other types of EEG sensors, such as ECoG (electrocorticogram), and the peripheral sensors may also be other types of sensors, such as electromyographic signal (EMG) sensors.

[0086] Step S2200: performing signal processing on the physiological signal to obtain a processed physiological signal.

[0087] In this embodiment, the signal processing may be, for example, amplification, segmentation, noise reduction, etc., which are not limited here. After the EEG signal and the peripheral physiological signal are processed, the processed EEG signal and the processed peripheral physiological signal are obtained, that is, the processed physiological signal includes the processed EEG signal and the processed peripheral physiological signal.

[0088] In some embodiments, the sampling duration and sampling frequency of the EEG sensor and the peripheral sensor are the same, for example, both are 1 hour, and the sampling frequency can be, for example, both are 200 Hz. At this time, the EEG signal collected by the EEG sensor and the peripheral physiological signal collected by the peripheral sensor are synchronized in time, which can facilitate fatigue detection and analysis based on the corresponding performance of the two signals at the same time.

[0089] Based on this, in step S2200, signal processing is performed on the physiological signal to obtain a processed physiological signal, including: steps S2200.1 to S2200.2.

[0090] Step S2200.1, performing bandpass filtering of a set frequency on the EEG signal to obtain a filtered EEG signal.

[0091] In this embodiment, the set frequency may be 0.5-40 Hz.

[0092] Those skilled in the art should understand that, in electroencephalogram (EEG) signal processing, the processing step of 0.5-40 Hz bandpass filtering is common knowledge in the art, and is intended to remove signal components that are not within the frequency band to reduce artifacts and highlight useful signals of brain activity. Therefore, the specific processing process of bandpass filtering will not be described here.

[0093] Step S2200.2, by setting the segment length, respectively performing signal segmentation processing on the filtered EEG signal and the peripheral physiological signal to obtain the processed physiological signal.

[0094] In this embodiment, since the sampling frequency and sampling duration of the EEG signal and the peripheral physiological signal are the same, if the two signals are segmented with a set segment length, frames of two channels with the same duration can be provided in the sliding window process.

[0095] In some examples, the set segment length can be, for example, 30 seconds, one minute, etc., with an overlap of 0 seconds, etc.

[0096] According to an embodiment of the present application, by performing signal segmentation processing on the filtered physiological signal with a set segment length, multiple signal segments can be obtained, and then subsequent feature extraction is performed on the multiple signal segments. Compared with extracting features from signals that have not been segmented, extracting features from the multiple signal segments can facilitate the extraction of features related to mental fatigue and reduce feature omissions.

[0097] Step S2300, performing feature extraction on the processed EEG signal and the processed peripheral physiological signal to obtain multi-scale fusion features.

[0098] In this embodiment, multiple single-scale features of the processed EEG signal and the processed peripheral physiological signal, that is, multi-scale features, can be extracted, and then the multi-scale features of the processed EEG signal and the processed peripheral physiological signal are fused to obtain multi-scale fusion features. Among them, the single-scale features can be time domain features (such as mean, variance, standard deviation, maximum and minimum values, skewness, kurtosis, etc.), frequency domain features (power spectral density, spectrum center frequency and other features), nonlinear features (approximate entropy of the signal, sample entropy, Lyapunov exponent and other nonlinear dynamic features), etc., which are not limited here.

[0099] In some embodiments, step S2300 performs feature extraction on the processed EEG signal and the processed peripheral physiological signal to obtain multi-scale fusion features, including: step S2300.1 to step S2300.2.

[0100] Step S2300.1, extracting features from the processed EEG signal and the processed peripheral physiological signal to obtain multi-scale features.

[0101] In this embodiment, the multi-scale features may include time domain features, frequency domain features, and the like.

[0102] In some embodiments, the network architecture for extracting multi-scale features can be as follows: Figure 3 As shown in Figure 1, it includes three parallel networks: a convolutional neural network (CNN) and two long short-term memory (LSTM) networks. Figure 3 The network architecture of feature extraction shown in the figure, in step S2300.1, feature extraction is performed on the processed EEG features and the processed peripheral physiological signals to obtain multi-scale features, including: steps SA1 to SA4.

[0103] Step SA1, inputting the processed EEG signal and the processed peripheral physiological signal into a convolutional neural network to obtain local spatiotemporal features.

[0104] In this embodiment, like the existing convolutional neural network, the CNN (convolutional neural network) used in this step consists of 6 consecutive layers, each layer involves 1 convolution, 1 maximum pooling and 1 batch normalization layer, and Dropout and average pooling layers are also applied. Unlike the existing convolutional neural network, an extra layer with 10 neurons is introduced into the convolutional neural network (CNN), that is, a fully connected (dense) layer after the convolutional layer and pooling layer of the CNN and before the output layer. The role of this extra layer is to integrate the features extracted by the previous convolutional layer and pooling layer, learn the complex relationship between features and select features related to mental fatigue.

[0105] Local spatiotemporal features can be local time series patterns (such as oscillations of a specific frequency or periodic patterns of heartbeats), frequency features (such as alpha waves, beta waves, theta waves, and delta waves in EEG, and heart rate variability HRV in ECG signals), waveform features (such as P waves, QRS waves, and T waves in ECG signals, and spike waves or other abnormal waveforms in EEG signals), etc.

[0106] The following example describes the process of extracting local spatiotemporal features using a convolutional neural network:

[0107] Assuming that the sampling duration of the EEG signal and the peripheral physiological signal is t and the sampling frequency is the same, the processed EEG signal can be represented by the vector x1 and the processed peripheral physiological signal can be represented by the vector x2, so the sliding window process of signal segmentation can provide frames with two channels of the same size, that is, the i-th frame fi consists of the processed EEG signal and the processed peripheral physiological signal vector Through cascading Every frame It is feature extraction independent of other frames, that is, two channels of CNN are used to analyze simultaneously in each frame and In addition, convolution is performed with a one-dimensional filter on each channel to simultaneously extract features from the processed EEG signal and the processed peripheral physiological signal. The features extracted by convolution are then processed selectively through an additional layer of 10 neurons to select features related to mental fatigue and improve the pertinence of feature extraction.

[0108] Step SA2, performing frequency domain conversion processing on the processed EEG signal and the processed peripheral physiological signal to obtain frequency domain data of the EEG signal and frequency domain data of the peripheral physiological signal.

[0109] In this embodiment, the processed EEG signal and the processed peripheral physiological signal can be converted into the frequency domain by short-time Fourier transform to obtain the frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal.

[0110] Continuing the above example, the i-th frame fi consists of the processed EEG signal and the processed peripheral physiological signal vector Through cascading In order to facilitate the short-time Fourier transform, each frame can be The signal is divided into n m-dimensional blocks. The fast here is obtained by a finer-grained segmentation within each frame, and each block contains m sample points. For each of the n m-dimensional blocks into which the signal is divided, a short-time Fourier transform (STFT) is applied to convert the time domain to the frequency domain to obtain the frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal.

[0111] The frequency domain data is time-frequency data, that is, the frequency domain data can provide frequency distribution information of the signal at different time points.

[0112] Those skilled in the art should understand that short-time Fourier transform technology is a conventional technical means in this field, so the specific process of short-time Fourier transform is not described in detail here.

[0113] Step SA3: input the frequency domain data of the EEG signal into a recurrent neural network to obtain EEG time series features.

[0114] In this embodiment, a recursive neural network (RNN) is a neural network specifically used to process sequence data, which processes each element in the sequence by recursively applying the same weights, thereby being able to capture the time dependency in the sequence, i.e., time series features. A recursive neural network (RNN) can be used to characterize the time evolution of different frames. A long short-term memory (LSTM) network is a special type of recursive neural network (RNN), which can capture the memory effect between consecutive frames and can be designed to solve the gradient vanishing and gradient exploding problems encountered by traditional RNN networks when processing long sequence data, i.e., an LSTM network belongs to an RNN network.

[0115] The architecture of the LSTM network consists of two layers of n LSTM units each. The complexity of this architecture depends on two parameters: the output y of the LSTM network and the input m of the first layer unit. It is worth noting that as n increases, the complexity of the network decreases.

[0116] Step SA4, inputting the frequency domain data of the peripheral physiological signal into the recurrent neural network to obtain peripheral time series features.

[0117] In this embodiment, the recursive neural network is Figure 3 Long short-term memory (LSTM) networks in .

[0118] In some embodiments, after obtaining the frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal in step SA2, before executing steps SA3 and SA4 to input the signals into the recursive neural network, the method further includes: performing spectral simplification processing on the frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal to extract time series features through the frequency domain data after the spectral simplification processing.

[0119] In this embodiment, since the result of the STFT frequency domain conversion processing usually contains positive frequency and negative frequency parts, and the negative frequency part is the complex conjugate of the positive frequency part, only the positive frequency spectrum part can be retained for time series feature extraction without affecting the integrity of the feature. Therefore, after obtaining the frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal, the two frequency domain data are respectively subjected to spectrum simplification processing to remove the negative frequency part.

[0120] According to an embodiment of the present application, before time series feature extraction is performed on the frequency domain data, spectrum simplification processing is performed on the frequency domain data, which can reduce the amount of frequency domain data while maintaining most of the useful information, thereby improving the efficiency of time series feature extraction without affecting the integrity of the time series features.

[0121] Step S2300.2, fusing the multi-scale features to obtain multi-scale fused features.

[0122] Continuing with the above example, Figure 3 As shown in the figure, the outputs of the three networks are local spatiotemporal features, time series features of EEG signals, and peripheral time series features. The outputs of the three networks are passed to a dense layer, which is used to fuse multi-scale features to obtain multi-scale fusion features.

[0123] According to the embodiment of the present application, by fusing multi-scale features into a vector, not only can the performance of subsequent fatigue detection models for detection thereof be improved, but also the structure of the fatigue detection model can be simplified, the computational burden can be reduced, and the robustness of the fatigue detection model can be improved.

[0124] Step S2400: input the multi-scale fusion features into a fatigue detection model to obtain the fatigue state of the subject.

[0125] In this embodiment, the fatigue detection model can be a binary classification model, that is, after the multi-scale fusion features are input into the fatigue detection model, the fatigue state of the detected person is obtained as "drowsy state" or "awake state". In the flight scene, the fatigue state of the pilot is obtained as "drowsy state" or "awake state".

[0126] In some embodiments, the process of training the fatigue detection model includes: steps S3100 to S3500.

[0127] Step S3100, obtaining a sample set.

[0128] In this embodiment, the sample set includes a training sample set and a test sample set. Among them, the training sample set includes training EEG signals and training peripheral physiological signals corresponding to different fatigue states, that is, the training sample set includes training EEG signals and training peripheral physiological signals corresponding to the "awake state", and training EEG signals and training peripheral physiological signals corresponding to the "drowsy state". Similarly, the test sample set also includes test EEG signals and test peripheral physiological signals corresponding to different fatigue states, and the test sample set includes test EEG signals and test peripheral physiological signals corresponding to the "awake state", and test EEG signals and test peripheral physiological signals corresponding to the "drowsy state".

[0129] The “drowsy state” and “awake state” in the sample set are sample labels, that is, the true values ​​of the samples, which reflect the actual fatigue state of the pilot.

[0130] In some embodiments, obtaining the sample set in step S3100 includes: steps S3100.1 to S3100.4.

[0131] Step S3100.1, obtaining fatigue self-assessment values ​​and physiological signals of multiple subjects.

[0132] In this embodiment, the multiple subjects have experience in operating flight driving games, so as to simulate the real flight process through flight driving games, and the experimental paradigm is designed to induce mental fatigue of the subjects, so as to collect the physiological signals and fatigue self-assessment values ​​of the subjects. The physiological signals include EEG signals and peripheral physiological signals.

[0133] Exemplarily, the experimental paradigm of the flight scenario is specifically designed as follows:

[0134] 1. Experimental equipment

[0135] Flight Simulator: A flight simulator that provides a flight control environment that simulates real flight, including flight joysticks, electric control loads, and other control options.

[0136] EEG sensor: 32-channel EEG system with a sampling frequency of 2000 Hz. The 32 electrodes are arranged according to the international 10-20 system.

[0137] PPMs sensor (peripheral sensor): collects peripheral physiological signals with a sampling frequency of 2000Hz.

[0138] Wireless keyboard: Attached to the flight joystick, used to record the subject's sleepiness score input.

[0139] 2. Experimental process

[0140] Twenty healthy college students (4 females and 16 males, average age 25.7±2.6) participated in the experiment. Each subject had experience in operating flight driving games and was asked to avoid drinking alcohol and coffee and get 6-8 hours of sleep the day before the experiment.

[0141] Before the experiment begins, the subjects wear EEG and PPMs sensors to collect their EEG signals and peripheral physiological signals during the experiment. Calibrate the equipment and ensure good signal quality. Then, start the following experimental process:

[0142] 1) The subjects underwent a simulated night flight, and the flight environment was set to a monotonous, low-stimulation night flight scene to maximize the induction of fatigue.

[0143] 2) A beep will sound every 5 minutes through the flight simulator's audio system.

[0144] 3) After hearing the beep, the subject enters a subjective sleepiness score based on the Karolinska Sleepiness Scale (KSS), i.e., a self-assessment of fatigue, with a score range of 1 to 9, with 1 indicating full wakefulness and 9 indicating extreme sleepiness. The KSS scale is shown in Table 1 below.

[0145] 4) The experiment lasted for 1 hour, and the subject's physiological signals, namely, EEG signals and peripheral physiological signals, were obtained. Moreover, a fatigue self-assessment value corresponded to every 5 minutes, and was recorded 12 times in total.

[0146] Table 1 Karolinska Sleepiness Scale Scores

[0147]

[0148] Step S3100.2, performing signal segmentation processing on the physiological signal by setting the segment length to obtain multiple signal segments.

[0149] In this embodiment, the set segment length can be 30 seconds or 1 minute, etc., and the specific value of the set segment length is not limited here.

[0150] It should be noted that the segment length needs to correspond to the fatigue self-assessment value. If the segment length is set to 30 seconds and the fatigue self-assessment value is one value every 5 minutes, then every 10 signal segments correspond to one fatigue self-assessment value. If the segment length is set to 1 minute and the fatigue self-assessment value is one value every 5 minutes, then every 5 signal segments correspond to one fatigue self-assessment value.

[0151] Step S3100.3, annotating the multiple signal segments according to the fatigue self-assessment values ​​to obtain multiple samples.

[0152] In this embodiment, after data annotation is performed on multiple signal segments, one signal segment corresponds to one sample, and the number of signal segments is consistent with the number of samples.

[0153] For example, if the fatigue self-assessment value corresponding to a certain signal segment is Level 9 (extremely sleepy), the signal segment is marked as a sleepy state. If the fatigue self-assessment value corresponding to a certain signal segment is Level 3 (alert), the signal segment is marked as a wakeful state.

[0154] Step S3100.4, performing sample segmentation on the multiple samples to obtain the training sample set and the test sample set.

[0155] For example, 70% of the multiple samples may be used as training samples to obtain a training sample set, and 30% may be used as testing samples to obtain a testing sample set.

[0156] Step S3200 , extracting features from the training EEG signal and the training peripheral physiological signal of each sample in the training sample set to obtain a multi-scale fusion feature set.

[0157] In this embodiment, feature extraction is performed on the training EEG signal and the training peripheral physiological signal of any sample in the training sample set to obtain multi-scale fusion features in the same manner as described above, for example, the same manner as described in step S2300, which will not be elaborated here.

[0158] Step S3300: training the initial fatigue detection model by using a multi-scale fusion feature set to obtain a trained fatigue detection model.

[0159] In this embodiment, a suitable machine learning or deep learning algorithm is selected as the basic model of the fatigue detection model, such as a support vector machine, a random forest, a neural network, etc., and the model parameters of the basic model are initialized to obtain an initial fatigue detection model.

[0160] An initial fatigue detection model is trained using a multi-scale fused feature set with labels, and the model parameters are adjusted through an optimization algorithm (such as gradient descent) to minimize the error between the labels and the model's predicted values.

[0161] For example, the model parameters are adjusted by the "Adam" optimizer, and after each iteration, the iterated model is trained by the multi-scale fusion feature set to obtain a trained model. According to experimental measurements, after 300 iterations, the error between the label and the model's predicted value is basically stable, meeting actual detection requirements.

[0162] Step S3400: Evaluate the trained fatigue detection model using the test sample set to obtain a model evaluation index value.

[0163] Continuing with the above example, after each iteration, the trained model is evaluated using the test sample set to obtain the model evaluation index value.

[0164] In one example, the performance of the fatigue detection model can be evaluated through a confusion matrix, which can show the relationship between the prediction results of the fatigue detection model and the actual labels.

[0165] In some embodiments, the evaluation index value includes at least one of accuracy, sensitivity, and specificity.

[0166] In this embodiment, the fatigue detection model is a binary classification model. For a binary classification problem, a confusion matrix usually includes four elements: true positive, false positive, true negative, and false negative.

[0167] Among them, True Positive (TP) refers to the number of samples correctly predicted by the model as positive (e.g., sleepy state). False Positive (FP) refers to the number of samples incorrectly predicted by the model as positive (e.g., sleepy state) but actually negative (e.g., awake state). True Negative (TN) refers to the number of samples correctly predicted by the model as negative (e.g., awake state). False Negative (FN) refers to the number of samples incorrectly predicted by the model as negative (e.g., awake state) but actually positive (e.g., sleepy state).

[0168] Sensitivity is an indicator that measures the ability of the model to identify the positive class (sleepy state). It is calculated by the ratio of the number of true positives (TP) to the total number of elements in this class (sleepy state) (TP+FN).

[0169] Specificity is a measure of the model's ability to identify the negative class (awake state). It is calculated by the ratio of the number of true negatives (TN) to the total number of elements in that class (awake state) (TN+FP).

[0170] Sensitivity tells us how effective the model is at identifying sleepy states, while specificity tells us how effective the model is at excluding sleepy states (i.e., correctly identifying awake states).

[0171] In a binary classification problem, accuracy can intuitively tell the model the percentage of samples that are correctly predicted out of the total number of samples. For example, if a model has an accuracy of 90% on the test sample set, it means that the model correctly predicts the category of 90% of the samples.

[0172] Those skilled in the art should understand that when evaluating a model, the model can be evaluated by at least one of sensitivity, specificity, and accuracy, which is not limited here.

[0173] Step S3500, when the model evaluation index value is greater than or equal to the preset index value, the trained fatigue detection model is used as the fatigue detection model.

[0174] In this embodiment, if sensitivity is used as the evaluation index value, the preset index value may be, for example, 97.3% sensitivity, if specificity is used as the evaluation index value, the preset index value may be, for example, 96.6% specificity, and if accuracy is used as the evaluation index value, the preset index value may be, for example, 87% accuracy.

[0175] According to the embodiments of the present application, by comprehensively analyzing EEG signals and peripheral physiological signals, a more comprehensive understanding of the pilot's mental state can be obtained. In addition, this method of fatigue detection based on multimodal physiological signals can provide information in more dimensions, which helps to more accurately assess the pilot's fatigue level and potential risks.

[0176] <Device Example>

[0177] Figure 4 4 is a principle block diagram of a fatigue detection device according to an embodiment of the present invention.

[0178] In this embodiment, if Figure 4 As shown, the fatigue detection device 400 includes a memory 410 and a processor 420, wherein the memory 410 is used to store executable instructions; the processor 420 is used to operate according to the control of the instructions to execute the method described in any of the above method embodiments.

[0179] Figure 5 4 is a principle block diagram of a fatigue detection system according to an embodiment of the present invention.

[0180] In this embodiment, if Figure 5 As shown, the fatigue detection system 500 includes: an electroencephalogram sensor 510 , a peripheral sensor 520 and a fatigue detection device 530 .

[0181] The EEG sensor 510 is used to collect EEG signals of the subject and send them to the fatigue detection device 530 .

[0182] The peripheral sensor 520 is used to collect peripheral physiological signals of the subject and send them to the fatigue detection device 530 .

[0183] In some embodiments, the fatigue detection device 530 is as follows Figure 4 The fatigue detection device or Figure 1 The fatigue detection device shown.

[0184] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0185] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0186] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0187] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0188] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0189] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0190] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0191] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0192] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A fatigue detection method, characterized in that: include: Acquiring physiological signals of the subject; wherein the physiological signals include electroencephalogram signals and peripheral physiological signals; Performing signal processing on the physiological signal to obtain a processed physiological signal; wherein the processed physiological signal includes a processed EEG signal and a processed peripheral physiological signal; Extracting features from the processed EEG signals and the processed peripheral physiological signals to obtain multi-scale fusion features; The multi-scale fusion features are input into a fatigue detection model to obtain the fatigue state of the subject.

2. The method according to claim 1, characterized in that The step of extracting features from the processed EEG signal and the processed peripheral physiological signal to obtain multi-scale fusion features includes: Extracting features from the processed EEG signal and the processed peripheral physiological signal to obtain multi-scale features; The multi-scale features are fused to obtain multi-scale fused features.

3. The method according to claim 2, characterized in that The step of extracting features from the processed EEG signal and the processed peripheral physiological signal to obtain multi-scale features includes: Inputting the processed EEG signal and the processed peripheral physiological signal into a convolutional neural network to obtain local spatiotemporal features; Performing frequency domain conversion processing on the processed EEG signal and the processed peripheral physiological signal to obtain frequency domain data of the EEG signal and frequency domain data of the peripheral physiological signal; wherein the frequency domain data is time-frequency data; Inputting the frequency domain data of the EEG signal into a recurrent neural network to obtain EEG time series features; The frequency domain data of the peripheral physiological signal is input into the recurrent neural network to obtain peripheral time series features.

4. The method according to claim 3, characterized in that After obtaining the frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal, the method further includes: The frequency domain data of the EEG signal and the frequency domain data of the peripheral physiological signal are subjected to spectrum simplification processing, so as to extract time series features through the frequency domain data after the spectrum simplification processing.

5. The method according to claim 1, characterized in that The sampling duration and sampling frequency of the EEG signal and the peripheral physiological signal are the same, and the signal processing of the physiological signal to obtain the processed physiological signal includes: Performing bandpass filtering of a set frequency on the EEG signal to obtain a filtered EEG signal; The filtered EEG signal and the peripheral physiological signal are respectively processed in segments according to the set segment length to obtain the processed physiological signal.

6. The method according to claim 1, characterized in that The process of training the fatigue detection model includes: Acquire a sample set; wherein the sample set includes a training sample set and a test sample set, and the training sample set includes training EEG signals and training peripheral physiological signals corresponding to different fatigue states; Extracting features from the training EEG signal and the training peripheral physiological signal of each sample in the training sample set to obtain a multi-scale fusion feature set; The initial fatigue detection model is trained by using the multi-scale fusion feature set to obtain a trained fatigue detection model; Evaluate the trained fatigue detection model using the test sample set to obtain a model evaluation index value; When the model evaluation index value is greater than or equal to the preset index value, the trained fatigue detection model is used as the fatigue detection model.

7. The method according to claim 6, characterized in that The obtaining of the sample set comprises: Obtaining fatigue self-assessment values ​​and physiological signals of multiple subjects; wherein the physiological signals include electroencephalogram signals and peripheral physiological signals; Performing signal segmentation processing on the physiological signal with a set segment length to obtain a plurality of signal segments; Performing data labeling on the multiple signal segments according to the fatigue self-assessment values ​​to obtain multiple samples; The multiple samples are segmented to obtain the training sample set and the test sample set.

8. The method according to claim 6, characterized in that The evaluation index value includes at least one of accuracy, sensitivity, and specificity.

9. A fatigue detection device, characterized in that: It comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the fatigue detection method according to any one of claims 1 to 8 are implemented.

10. A fatigue detection system, characterized in that: include: An electroencephalogram sensor, a peripheral sensor, and a fatigue detection device as claimed in claim 9; The electroencephalogram sensor is used to collect the electroencephalogram signal of the subject and send it to the fatigue detection device; The peripheral sensor is used to collect peripheral physiological signals of the subject and send them to the fatigue detection device.

Images