Fatigue monitoring system and method

Abstract

The invention relates to a fatigue monitoring system and method. The fatigue monitoring system comprises a head-mounted device and a terminal device. The head-mounted device comprises a physiological signal acquisition module, the physiological signal acquisition module is used for acquiring physiological signals of a user, and the physiological signals comprise electroencephalogram signals, pulse rate and HRV signals and body movement signals; the terminal equipment is in communication connection with the physiological signal acquisition module and is used for respectively inputting the electroencephalogram signal, the pulse rate and HRV signal and the body movement signal into a first fatigue evaluation model, a second fatigue evaluation model and a third fatigue evaluation model which are completely trained; obtaining a first fatigue grade, a second fatigue grade and a third fatigue grade which are respectively output by the three models; and the terminal equipment is also used for performing correlation analysis on the first fatigue grade, the second fatigue grade and the third fatigue grade, and obtaining a final fatigue grade according to an output result of the correlation analysis. Through the method and the device, the problem of relatively low fatigue monitoring accuracy of the pilot in related technologies is solved.

Description

Technical Field

[0001] The present application belongs to the technical field of wearable devices, and in particular, relates to a fatigue monitoring system and method. Background Art

[0002] With the advancement of science and technology and the continuous improvement of the level of aircraft automation, the proportion of flight accidents caused by equipment failure has dropped from 85% in the middle of the last century to 12% to 21% today, while the proportion of flight accidents caused by pilots has now reached 70%. Among the flight accidents caused by pilots, accidents related to pilot fatigue account for 21%. Therefore, reducing flight accidents caused by pilot fatigue is of great significance to aviation safety.

[0003] Cross-time zone flights and irregular work schedules have made pilot fatigue an increasingly prominent problem. Therefore, conducting research on pilot aviation fatigue is of great significance for improving aviation safety and protecting the lives and property of passengers. At present, pilot fatigue monitoring mainly relies on subjective reports and behavioral performance tests, which have certain limitations. For example, subjective reports may be affected by individual differences and self-assessment biases, while behavioral performance tests may not be accurate enough due to environmental factors and test conditions.

[0004] Currently, no effective solution has been proposed to address the problem of low accuracy in pilot fatigue monitoring in related technologies. Summary of the invention

[0005] The embodiments of the present application provide a fatigue monitoring system and method to at least solve the problem of low accuracy in pilot fatigue monitoring in the related art.

[0006] In a first aspect, an embodiment of the present application provides a fatigue monitoring system, comprising a head-mounted device and a terminal device; wherein the head-mounted device comprises a physiological signal acquisition module, wherein the physiological signal acquisition module is used to collect physiological signals of the user, and the physiological signals include electroencephalogram signals, pulse rate and HRV signals, and body movement signals; the terminal device is communicatively connected to the physiological signal acquisition module, and is used to input the electroencephalogram signals, pulse rate and HRV signals, and body movement signals into a first fatigue assessment model, a second fatigue assessment model, and a third fatigue assessment model that have been trained, respectively, and obtain the first fatigue level, the second fatigue level, and the third fatigue level output by the first fatigue assessment model, the second fatigue assessment model, and the third fatigue assessment model, respectively; the terminal device is also used to perform correlation analysis on the first fatigue level, the second fatigue level, and the third fatigue level, and obtain the final fatigue level based on the output result of the correlation analysis.

[0007] In some embodiments, the fatigue monitoring system further includes an alarm module; wherein the alarm module is communicatively connected to the terminal device, and the terminal device is further configured to control the alarm module to execute an alarm action when the final fatigue level exceeds a preset threshold.

[0008] In some embodiments, the fatigue monitoring system further includes a feedback receiving module; wherein the feedback receiving module is communicatively connected to the terminal device, the feedback receiving module is used to receive user feedback information on the warning action, and the terminal device is also used to update the final fatigue level according to the feedback information.

[0009] In some embodiments, the feedback information includes at least one of head tapping feedback information and eye movement feedback information; wherein, when the feedback information includes head tapping feedback information, the terminal device is also used to extract the frequency and strength of the user's head tapping from the head tapping feedback information, and update the final fatigue level according to the frequency and strength of the user's head tapping; when the feedback information includes eye movement feedback information, the terminal device is also used to extract the user's blinking frequency and gaze point duration from the eye movement feedback information, and update the final fatigue level according to the user's blinking frequency and gaze point duration.

[0010] In some embodiments, the terminal device is further configured to increase the final fatigue level by a preset fixed value when the feedback receiving module does not receive feedback information from the user regarding the warning action within a preset time period.

[0011] In some embodiments, a correlation analysis is performed on the first fatigue level, the second fatigue level, and the third fatigue level, and a final fatigue level is obtained based on the output results of the correlation analysis, including: performing a correlation analysis on the first fatigue level, the second fatigue level, and the third fatigue level to obtain a correlation coefficient between the first fatigue level, the second fatigue level, and the third fatigue level; and performing weighted integration of the first fatigue level, the second fatigue level, and the third fatigue level based on the correlation coefficient between the first fatigue level, the second fatigue level, and the third fatigue level to obtain a final fatigue level.

[0012] In some embodiments, the terminal device is also used to preprocess the EEG signals, pulse rate, HRV signals and body movement signals; and extract EEG features from the preprocessed EEG signals, extract HRV features from the preprocessed pulse rate and HRV signals, and extract body movement features from the preprocessed body movement signals; the terminal device is also used to input the EEG features, HRV features and body movement features into the first fatigue assessment model, the second fatigue assessment model and the third fatigue assessment model, respectively, and obtain the first fatigue level, the second fatigue level and the third fatigue level output by the first fatigue assessment model, the second fatigue assessment model and the third fatigue assessment model, respectively.

[0013] In some embodiments, the physiological signal acquisition module includes EEG electrodes, a PPG sensor and an accelerometer; wherein the EEG electrodes are used to collect EEG signals; the PPG sensor is used to collect pulse rate and HRV signals; and the accelerometer is used to collect body movement signals.

[0014] In some embodiments, the EEG electrodes include an Fp1 electrode, an Fp2 electrode, an O1 electrode and an O2 electrode; wherein the Fp1 electrode is arranged on the inner surface of the head-mounted device near the left forehead of the user's head, the Fp2 electrode is arranged on the inner surface of the head-mounted device near the right forehead of the user's head, the O1 electrode is arranged on the inner surface of the head-mounted device near the left back of the user's head, and the O2 electrode is arranged on the inner surface of the head-mounted device near the right back of the user's head.

[0015] In the second aspect, an embodiment of the present application provides a fatigue monitoring method, including: obtaining physiological signals of a user, the physiological signals including electroencephalogram signals, pulse rate, HRV signals and body movement signals; inputting the electroencephalogram signals, pulse rate, HRV signals and body movement signals into a first fatigue assessment model, a second fatigue assessment model and a third fatigue assessment model that have been trained completely, respectively; obtaining a first fatigue level, a second fatigue level and a third fatigue level output by the first fatigue assessment model, the second fatigue assessment model and the third fatigue assessment model, respectively; performing correlation analysis on the first fatigue level, the second fatigue level and the third fatigue level, and obtaining the final fatigue level based on the output results of the correlation analysis.

[0016] Compared with the related art, the embodiment of the present application provides a fatigue monitoring system capable of real-time fatigue monitoring of pilots. The fatigue monitoring system includes a head-mounted device and a terminal device. The head-mounted device collects physiological signals such as EEG signals, pulse rate, HRV signals, and body movement signals of the pilot, and the terminal device can use multiple fatigue assessment models trained for different physiological signals of the pilot to output multiple fatigue levels, and perform correlation analysis on the output multiple fatigue levels to obtain the user's final fatigue level. By inputting different physiological signals into different fatigue assessment models respectively, and performing correlation analysis on the different fatigue levels output by the multiple fatigue assessment models, the pilot's final fatigue level can be obtained. A fatigue level that better matches the pilot can be obtained, thereby improving the accuracy of fatigue monitoring. Through this application, the problem of low accuracy of fatigue monitoring of pilots in the related art is solved, and the technical effect of improving the accuracy of fatigue monitoring of pilots is achieved.

[0017] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 is a structural schematic diagram of a fatigue monitoring system according to an embodiment of the present application;

[0020] Figure 2 is a schematic structural diagram of a head mounted device according to an embodiment of the present application;

[0021] Figure 3 is a flow chart of a fatigue monitoring method according to an embodiment of the present application;

[0022] Figure 4 It is a structural diagram of a terminal device according to an embodiment of the present application.

[0023] Among them, the reference numerals in the figure are: 100-fatigue monitoring system; 110-head-mounted device; 111-physiological signal acquisition module; 120-terminal device; 121-mainboard; 122-analog signal input port; 123-digital signal input and power supply port; 200-EEG electrode; 201-Fp1 electrode; 202-Fp2 electrode; 203-O1 electrode; 204-O2 electrode; 210-PPG sensor; 220-elastic band; 400-processor; 410-memory; 420-computer program. DETAILED DESCRIPTION

[0024] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0025] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0026] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0027] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0028] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0029] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0030] With the advancement of science and technology and the continuous improvement of the level of aircraft automation, the proportion of flight accidents caused by equipment failure has dropped from 85% in the middle of the last century to 12% to 21% today, while the proportion of flight accidents caused by pilots has now reached 70%. Among the flight accidents caused by pilots, accidents related to pilot fatigue account for 21%. Therefore, reducing flight accidents caused by pilot fatigue is of great significance to aviation safety.

[0031] Cross-time zone flights and irregular work schedules have made pilot fatigue an increasingly prominent problem. Therefore, conducting research on pilot aviation fatigue is of great significance for improving aviation safety and protecting the lives and property of passengers. At present, pilot fatigue monitoring mainly relies on subjective reports and behavioral performance tests, which have certain limitations. For example, subjective reports may be affected by individual differences and self-assessment biases, while behavioral performance tests may not be accurate enough due to environmental factors and test conditions.

[0032] Currently, no effective solution has been proposed to address the problem of low accuracy in pilot fatigue monitoring in related technologies.

[0033] In view of this, an embodiment of the present application provides a fatigue monitoring system capable of real-time fatigue monitoring of pilots, the fatigue monitoring system includes a head-mounted device and a terminal device, the head-mounted device collects the pilot's electroencephalogram signal, pulse rate and HRV (Heart Rate Variability) signal and body movement signal and other physiological signals, and the terminal device can use multiple fatigue assessment models trained for different physiological signals of the pilot to output multiple fatigue levels, and perform correlation analysis on the output multiple fatigue levels to obtain the user's final fatigue level, by inputting different physiological signals into different fatigue assessment models respectively, and performing correlation analysis on the different fatigue levels output by the multiple fatigue assessment models, so as to obtain the pilot's final fatigue level, a fatigue level that better matches the pilot can be obtained, and the accuracy of fatigue monitoring is improved. Through this application, the problem of low accuracy of fatigue monitoring of pilots in the related art is solved, and the technical effect of improving the accuracy of fatigue monitoring of pilots is achieved.

[0034] The following will be combined Figure 1 For an explanation of a fatigue monitoring system 100 provided in an embodiment of the present application, please refer to Figure 1 , Figure 1 is a schematic diagram of the structure of a fatigue monitoring system 100 according to an embodiment of the present application. Figure 1 As shown, the fatigue monitoring system 100 may include a head-mounted device 110 and a terminal device 120; wherein the head-mounted device 110 includes a physiological signal acquisition module 111, wherein the physiological signal acquisition module 111 is used to collect the user's physiological signals, and the physiological signals include electroencephalogram signals, pulse rate and HRV signals, and body movement signals; the terminal device 120 is communicatively connected to the physiological signal acquisition module 111, and is used to input the electroencephalogram signals, pulse rate and HRV signals, and body movement signals into the first fatigue assessment model, the second fatigue assessment model, and the third fatigue assessment model that have been trained, respectively, and obtain the first fatigue level, the second fatigue level, and the third fatigue level output by the first fatigue assessment model, the second fatigue assessment model, and the third fatigue assessment model, respectively; the terminal device 120 is also used to perform correlation analysis on the first fatigue level, the second fatigue level, and the third fatigue level, and obtain the final fatigue level according to the output result of the correlation analysis.

[0035] In one embodiment, the terminal device 120 can control the head mounted device 110 and provide power to the head mounted device 110. The terminal device 120 can be an independent host computing device, an onboard computing device integrated with the head mounted device 110, a mobile device, or any other hardware platform capable of controlling the head mounted device 110 and providing power to the head mounted device 110.

[0036] In this embodiment, if Figure 1 As shown, the terminal device 120 may be a main control computing device that can be fixed on the collar of the user.

[0037] In other embodiments, the terminal device 120 may be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiments of the present application do not impose any restrictions on the specific type of the terminal device 120.

[0038] In one embodiment, the head mounted device 110 may be a portable device that is worn directly on the user's head, or integrated into the user's clothing or accessories (eg, a hat).

[0039] In this embodiment, if Figure 1 As shown, the head mounted device 110 may be a headband that can be fixed on the user's head.

[0040] In other embodiments, the style of the head mounted device 110 may include a helmet, a hat, etc. The embodiment of the present application does not impose any limitation on the specific style of the head mounted device 110.

[0041] In one embodiment, the first fatigue assessment model may be specially trained for EEG signals, the second fatigue assessment model may be specially trained for pulse rate and HRV signals, and the third fatigue assessment model may be specially trained for body motion signals. The first fatigue assessment model, the second fatigue assessment model, and the third fatigue assessment model may be trained based on at least one of the following algorithms: Support Vector Machine (SVM), Random Forest (RF), deep learning model (e.g., Convolutional Neural Networks (CNN) or Recurrent Neural Network (RNN)), etc.

[0042] In this embodiment, the preset segmented training data set can be used to complete the training of the first fatigue assessment model, the second fatigue assessment model and the third fatigue assessment model, and the preset verification data set can be used to adjust the model parameters (such as learning rate and number of iterations, etc.) to optimize the generalization ability of the above three models.

[0043] Specifically, the first training data set can be used to complete the training of the first fatigue assessment model, and the first verification data set can be used to adjust the parameters of the first fatigue assessment model, wherein the first training data set and the first verification data set only include EEG signal-related data; the second training data set can be used to complete the training of the second fatigue assessment model, and the second verification data set can be used to adjust the parameters of the second fatigue assessment model, wherein the second training data set and the second verification data set only include pulse rate and HRV signal-related data; the third training data set can be used to complete the training of the third fatigue assessment model, and the third verification data set can be used to adjust the parameters of the third fatigue assessment model, wherein the third training data set and the third verification data set only include body motion signal-related data.

[0044] By constructing different fatigue assessment models for different physiological signals and performing correlation analysis on the fatigue levels output by different fatigue assessment models to obtain the user's final fatigue level, the accuracy and stability of user fatigue monitoring can be improved.

[0045] In one embodiment, the fatigue monitoring system 100 may further include an alarm module; wherein the alarm module is in communication connection with the terminal device 120, and the terminal device 120 is further configured to control the alarm module to perform an alarm action when the final fatigue level exceeds a preset threshold.

[0046] Specifically, the warning module can be integrated into the terminal device 120 and / or the head-mounted device 110. For example, the warning module can be integrated into the terminal device 120, and it can notify the user through the screen of the terminal device 120 (if any) when the final fatigue level exceeds a preset threshold; or, the warning module can notify the user by emitting voice, vibration, light effect, etc. The embodiments of the present application do not impose any restrictions on the specific type of the warning module.

[0047] In this embodiment, the fatigue level can be subdivided into 0 to 100, and the preset threshold can be set based on the actual application scenario of the fatigue monitoring system 100. For example, when the actual application scenario of the fatigue monitoring system 100 provided in the embodiment of the present application requires a higher monitoring sensitivity, the preset threshold can be set lower to 50; and when the actual application scenario of the fatigue monitoring system 100 does not require a higher monitoring sensitivity, the preset threshold can be set higher to 80.

[0048] In this embodiment, a user using the fatigue monitoring system 100 provided in the embodiment of the present application can provide feedback on the warning action issued by the warning module.

[0049] In one embodiment, the fatigue monitoring system 100 may further include a feedback receiving module; wherein the feedback receiving module is communicatively connected to the terminal device 120, the feedback receiving module is used to receive user feedback information on the warning action, and the terminal device 120 is also used to update the final fatigue level according to the feedback information.

[0050] In this embodiment, the feedback receiving module may be integrated into the terminal device 120 and / or the head mounted device 110, and the feedback receiving module may correspond to the user's feedback method. For example, when the user's feedback method is tapping the head mounted device 110, the feedback receiving module may be a pressure sensitive sensor integrated into the head mounted device 110; when the user's feedback method is voice reminder, the feedback receiving module may be a voice receiving module integrated into the terminal device 120 and / or the head mounted device 110; when the user's feedback method is gazing, the feedback receiving module may be a camera device connected to the terminal device 120. The embodiment of the present application does not impose any restrictions on the specific type of the feedback receiving module.

[0051] Specifically, if the user realizes that he is in a fatigue driving state after discovering the warning action executed by the warning module, he can tap the head mounted device 110 lightly, so that the terminal device 120 can determine whether the user's feedback information has been received according to the pressure data changes detected by the pressure sensitive sensor; in addition, the above warning action can remind the user through voice, and can also ask the user to feedback whether the warning action is correct through voice. For example, if the user believes that he is not in a fatigue driving state, he can reply that the warning is inaccurate through voice after receiving the warning action; in addition, the above warning action can be communicated with the terminal device 120 through voice. 0, the display device connected to the terminal device 120 reminds the user (for example, by emitting a flashing light effect, etc.), and then receives feedback information from the user through the touch screen through the display device. For example, when a correct or incorrect control is displayed in the display device, the user can touch the corresponding control to send feedback information; finally, the user's feedback information can be received through a camera device connected to the terminal device 120 for communication. For example, after the warning module performs the warning action, the camera device can collect the user's blinking frequency and gaze point duration. If the user's blinking frequency and gaze point duration reach a preset threshold, it is determined that the user is in a fatigue driving state or a non-fatigue driving state. The embodiment of the present application does not impose any restrictions on the method of obtaining the above feedback information.

[0052] In this embodiment, the feedback information may include at least one of head tapping feedback information and eye movement feedback information; wherein, when the feedback information includes head tapping feedback information, the terminal device 120 is also used to extract the frequency and strength of the user's head tapping from the head tapping feedback information, and update the final fatigue level according to the frequency and strength of the user's head tapping; when the feedback information includes eye movement feedback information, the terminal device 120 is also used to extract the user's blinking frequency and gaze point duration from the eye movement feedback information, and update the final fatigue level according to the user's blinking frequency and gaze point duration.

[0053] Specifically, after receiving the user's feedback information, the terminal device 120 can analyze from the user's feedback information whether the user's current fatigue state meets the output final fatigue level. For example, when the frequency and strength of the user's head tapping are greater than a preset threshold, the terminal device 120 can determine that the output final fatigue level is higher than the user's current fatigue state. At this time, the output final fatigue level can be subtracted from a preset fixed value. The terminal device 120 can also determine that the output final fatigue level is lower than the user's current fatigue state when the user's blinking frequency and gaze point duration are less than a preset threshold. At this time, the output final fatigue level can be increased by a preset fixed value.

[0054] In addition, after receiving the user's feedback information, the terminal device 120 can also update the first fatigue assessment model, the second fatigue assessment model, and the third fatigue assessment model according to the feedback information, so that the first fatigue assessment model, the second fatigue assessment model, and the third fatigue assessment model can obtain more accurate fatigue determination results. Specifically, the terminal device 120 can update the above fatigue assessment model each time after receiving feedback information, and can also update the fatigue assessment model when the feedback information shows an alarm error within a continuous period of time. The embodiment of the present application does not impose any restrictions on the update method.

[0055] Finally, the terminal device 120 can also increase the final fatigue level by a preset fixed value when the feedback receiving module does not receive the user's feedback information on the warning action within a preset time period. The terminal device 120 in the fatigue monitoring system 100 provided in the embodiment of the present application can update the final fatigue level and the parameters in the above-mentioned fatigue assessment model according to the feedback message, so that the fatigue monitoring system 100 provided in the embodiment of the present application can more accurately monitor the user's fatigue driving.

[0056] In one embodiment, a correlation analysis is performed on the first fatigue level, the second fatigue level, and the third fatigue level, and a final fatigue level is obtained based on the output result of the correlation analysis, including: performing a correlation analysis on the first fatigue level, the second fatigue level, and the third fatigue level to obtain a correlation coefficient between the first fatigue level, the second fatigue level, and the third fatigue level; and performing weighted integration on the first fatigue level, the second fatigue level, and the third fatigue level based on the correlation coefficient between the first fatigue level, the second fatigue level, and the third fatigue level to obtain a final fatigue level.

[0057] In this embodiment, statistical methods (such as Pearson correlation coefficient) or other correlation analysis methods can be used to analyze the correlation between the first fatigue level, the second fatigue level and the third fatigue level, which can help determine the contribution and consistency of the physiological signals corresponding to each fatigue level to the fatigue level assessment.

[0058] According to the result of the correlation analysis, the first fatigue level, the second fatigue level and the third fatigue level may be weightedly integrated.

[0059] Specifically, a weighted average method may be used, wherein the weight in the weighted average may be determined by the correlation coefficients among the first fatigue level, the second fatigue level, and the third fatigue level. For example, if the physiological signal corresponding to a fatigue level is highly correlated with the other two physiological signals, the weight of the fatigue level in the weighted average will be greater.

[0060] By taking the weighted integrated result as the final fatigue level, signals from multiple angles including EEG signals, pulse rate and HRV signals, and body movement signals can be integrated to more accurately reflect the user's fatigue state.

[0061] In one embodiment, the terminal device 120 is also used to preprocess the EEG signals, pulse rate, HRV signals and body movement signals; and extract EEG features from the preprocessed EEG signals, extract HRV features from the preprocessed pulse rate and HRV signals, and extract body movement features from the preprocessed body movement signals; the terminal device 120 is also used to input the EEG (Electroencephalogram, brain wave) features, HRV features and body movement features into the first fatigue assessment model, the second fatigue assessment model and the third fatigue assessment model, respectively, and obtain the first fatigue level, the second fatigue level and the third fatigue level output by the first fatigue assessment model, the second fatigue assessment model and the third fatigue assessment model, respectively.

[0062] In this embodiment, the preprocessing may include filtering, denoising, normalization, segment cutting, etc.

[0063] Specifically, the terminal device 120 can perform bandpass filtering on the EEG signal to remove power frequency interference, high-frequency noise and low-frequency drift, and retain the frequency range related to fatigue (for example, 0.3 Hz to 45 Hz); in addition, the terminal device 120 can also use signal processing algorithms (such as independent component analysis (ICA) or wavelet transform, etc.) to separate and remove non-EEG noise such as eye movement and electromyography; in addition, the terminal device 120 can also normalize the signal to eliminate the differences in physiological characteristics between different individuals, so that the data are comparable between different users; in addition, the terminal device 120 can also divide the continuous physiological signal into short time series (for example, a 30-second data frame window) to facilitate subsequent feature extraction and model training; in addition, before performing feature extraction, the terminal device 120 can also perform necessary conversion on the pre-processed physiological signal, for example, through Fast Fourier Transform (Fast Fourier Transform, FFT) to obtain frequency domain features, or extract more detailed information through time-frequency analysis methods; finally, the terminal device 120 can also perform data cleaning on the physiological signal to remove any obviously erroneous data points or damaged data records.

[0064] In one embodiment, the terminal device 120 can extract HRV features from the preprocessed pulse rate and HRV signals and extract HRV features from the pulse wave signal, which reflect the activity of the autonomic nervous system and are closely related to the user's fatigue state.

[0065] In one embodiment, the terminal device 120 may use the triaxial accelerometer data to calculate the body motion level, which reflects the user's activity level and movement pattern and can be used as auxiliary information for fatigue assessment.

[0066] In this embodiment, the EEG features may include frequency domain features extracted from the EEG signal, including power spectral density (PSD), synchronization index and brain wave frequency band ratio, entropy (sample entropy, fuzzy entropy, approximate entropy and power spectral entropy) and other features. These features help reveal fatigue-related changes in brain activity. Time-frequency analysis tools such as wavelet transform can be used to extract time-frequency domain features of the signal, such as short-time energy and short-time coherence, which help capture the non-stationary characteristics of the signal.

[0067] In this embodiment, HRV features may include time domain and frequency domain features extracted from pulse rate and HRV signals, such as SDNN (standard deviation of NN intervals), RMSSD (root mean square of successive differences), and spectral power distribution (PSD). These indicators reflect the fluctuation of heart rate and are closely related to the state of the autonomic nervous system.

[0068] In this embodiment, the body motion features may be body motion index and activity intensity calculated based on body motion signals, and these features may be used to assess physical fatigue and activity level.

[0069] Figure 2 is a schematic diagram of the structure of a head mounted device 110 according to an embodiment of the present application, such as Figure 2 As shown, in one embodiment, the physiological signal acquisition module 111 includes an EEG electrode 200, a PPG (Photoplethysmography) sensor, and an accelerometer ( Figure 2 wherein the EEG electrode 200 is used to collect EEG signals; the PPG sensor 210 is used to collect pulse rate and HRV signals; and the accelerometer is used to collect body motion signals.

[0070] In this embodiment, the accelerometer can be a three-axis accelerometer, and the EEG electrode 200 can include an Fp1 electrode 201, an Fp2 electrode 202, an O1 electrode 203 and an O2 electrode 204; wherein the Fp1 electrode 201 is arranged on the inner surface of the head-mounted device 110 close to the left forehead of the user's head, the Fp2 electrode 202 is arranged on the inner surface of the head-mounted device 110 close to the right forehead of the user's head, the O1 electrode 203 is arranged on the inner surface of the head-mounted device 110 close to the left back of the user's head, and the O2 electrode 204 is arranged on the inner surface of the head-mounted device 110 close to the right back of the user's head.

[0071] like Figure 2 As shown, in one embodiment, the head mounted device 110 may include an elastic band 220 to accommodate users with different head sizes.

[0072] like Figure 2As shown, in one embodiment, the terminal device 120 may include a mainboard 121, which includes an analog signal input port 122 and a digital signal input and power supply port 123, wherein the EEG electrode 200 and the accelerometer can be connected to the analog signal input port 122 via wire or wirelessly, and the PPG sensor 210 can be connected to the digital signal input and power supply port 123 via wire or wirelessly. Through this result, the analog signal routing line can be physically isolated from the digital signal routing line and the mainboard 121 power supply line to avoid routing overlap and EMC (Electromagnetic Compatibility) interference.

[0073] The following is a description of a fatigue monitoring method provided by an embodiment of the present application in conjunction with the accompanying drawings. Figure 3 , Figure 3 is a flow chart of a fatigue monitoring method according to an embodiment of the present application. Figure 3 As shown, the method includes steps S301 to S304:

[0074] Step S301, obtaining the user's physiological signals, which include EEG signals, pulse rate, HRV signals, and body movement signals.

[0075] Step S302, inputting the EEG signal, pulse rate and HRV signal and body movement signal into the first fatigue assessment model, the second fatigue assessment model and the third fatigue assessment model which have been trained completely respectively.

[0076] Step S303: acquiring the first fatigue level, the second fatigue level and the third fatigue level respectively output by the first fatigue assessment model, the second fatigue assessment model and the third fatigue assessment model.

[0077] Step S304: performing correlation analysis on the first fatigue level, the second fatigue level and the third fatigue level, and obtaining a final fatigue level according to an output result of the correlation analysis.

[0078] In this embodiment, the fatigue monitoring method can be implemented by the fatigue monitoring system 100 provided in the above embodiment of the present application, or it can be implemented by an external terminal device coupled to the fatigue monitoring system 100 provided in the above embodiment of the present application, and the present application does not make any limitation here.

[0079] In this embodiment, the pilot's physiological signals such as brain wave signals, pulse rate, HRV (Heart Rate Variability) signals, and body movement signals can be collected, and multiple fatigue assessment models trained for different physiological signals of the pilot can be used to output multiple fatigue levels, and correlation analysis can be performed on the output multiple fatigue levels to obtain the user's final fatigue level. Different physiological signals are input into different fatigue assessment models, and correlation analysis is performed on different fatigue levels output by multiple fatigue assessment models to obtain the pilot's final fatigue level. A fatigue level that better matches the pilot can be obtained, thereby improving the accuracy of fatigue monitoring. Through this application, the problem of low accuracy of fatigue monitoring of pilots in related technologies is solved, and the technical effect of improving the accuracy of fatigue monitoring of pilots is achieved.

[0080] It should be noted that the detailed content of each of the above steps is based on the same concept as the system embodiment of the present application. Its specific functions and technical effects can be found in the system embodiment part and will not be repeated here.

[0081] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0082] Figure 4 is a schematic diagram of the structure of a terminal device according to an embodiment of the present application. Figure 4 As shown, the terminal device 120 includes: at least one processor 400 ( Figure 4 Only one is shown in the figure) a processor, a memory 410, and a computer program 420 stored in the memory 410 and executable on at least one processor 400, and when the processor 400 executes the computer program 420, the steps in any of the above fatigue monitoring method embodiments are implemented.

[0083] The terminal device 120 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device 120 may include but is not limited to a processor 400 and a memory 410. It is understood by those skilled in the art that Figure 4 It is only an example of the terminal device 120 and does not constitute a limitation on the terminal device 120. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0084] The processor 400 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0085] In some embodiments, the memory 410 may be an internal storage unit of the terminal device 120, such as a hard disk or memory of the terminal device 120. In other embodiments, the memory 410 may also be an external storage device of the terminal device 120, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device 120. In other embodiments, the memory 410 may also include both an internal storage unit of the terminal device 120 and an external storage device. The memory 410 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of a computer program 420, etc. The memory 410 may also be used to temporarily store data that has been output or is to be output.

[0086] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above fatigue monitoring method embodiment can be implemented.

[0087] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above fatigue monitoring method embodiment when executing the computer program product.

[0088] The present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0089] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0090] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0091] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0092] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0093] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A fatigue monitoring system, characterized in that: The fatigue monitoring system includes a head-mounted device and a terminal device; wherein, The head-mounted device includes a physiological signal acquisition module, wherein the physiological signal acquisition module is used to collect physiological signals of the user, and the physiological signals include EEG signals, pulse rate and HRV signals, and body movement signals; The terminal device is communicatively connected with the physiological signal acquisition module, and is used to input the EEG signal, the pulse rate and HRV signal, and the body movement signal into the first fatigue assessment model, the second fatigue assessment model, and the third fatigue assessment model that have been trained, respectively, and obtain the first fatigue level, the second fatigue level, and the third fatigue level output by the first fatigue assessment model, the second fatigue assessment model, and the third fatigue assessment model, respectively; The terminal device is further used to perform a correlation analysis on the first fatigue level, the second fatigue level and the third fatigue level, and obtain a final fatigue level according to an output result of the correlation analysis.

2. The fatigue monitoring system according to claim 1, characterized in that: The fatigue monitoring system also includes an alarm module; wherein the alarm module is in communication connection with the terminal device, and the terminal device is further configured to control the alarm module to perform an alarm action when the final fatigue level exceeds a preset threshold.

3. The fatigue monitoring system according to claim 2, characterized in that: The fatigue monitoring system also includes a feedback receiving module; wherein the feedback receiving module is communicatively connected to the terminal device, the feedback receiving module is used to receive user feedback information on the warning action, and the terminal device is also used to update the final fatigue level according to the feedback information.

4. The fatigue monitoring system according to claim 3, characterized in that: The feedback information includes at least one of head tapping feedback information and eye movement feedback information; wherein, When the feedback information includes the head tapping feedback information, the terminal device is further used to extract the frequency and strength of the user tapping the head from the head tapping feedback information, and update the final fatigue level according to the frequency and strength of the user tapping the head; When the feedback information includes the eye movement feedback information, the terminal device is further used to extract the user's blinking frequency and gaze point duration from the eye movement feedback information, and update the final fatigue level according to the user's blinking frequency and gaze point duration.

5. The fatigue monitoring system according to claim 3 or 4, characterized in that: The terminal device is further configured to increase the final fatigue level by a preset fixed value when the feedback receiving module does not receive any feedback information from the user regarding the warning action within a preset time period.

6. The fatigue monitoring system according to any one of claims 1 to 4, characterized in that: Performing a correlation analysis on the first fatigue level, the second fatigue level, and the third fatigue level, and obtaining a final fatigue level according to an output result of the correlation analysis includes: performing a correlation analysis on the first fatigue level, the second fatigue level, and the third fatigue level to obtain a correlation coefficient between the first fatigue level, the second fatigue level, and the third fatigue level; The first fatigue level, the second fatigue level and the third fatigue level are weightedly integrated according to the correlation coefficient among the first fatigue level, the second fatigue level and the third fatigue level to obtain the final fatigue level.

7. The fatigue monitoring system according to any one of claims 1 to 4, characterized in that: The terminal device is also used to preprocess the EEG signal, the pulse rate and HRV signal, and the body movement signal; and extract EEG features from the preprocessed EEG signal, extract HRV features from the preprocessed pulse rate and HRV signal, and extract body movement features from the preprocessed body movement signal; The terminal device is also used to input the EEG features, the HRV features and the body movement features into the first fatigue assessment model, the second fatigue assessment model and the third fatigue assessment model, respectively, and obtain the first fatigue level, the second fatigue level and the third fatigue level output by the first fatigue assessment model, the second fatigue assessment model and the third fatigue assessment model, respectively.

8. The fatigue monitoring system according to any one of claims 1 to 4, characterized in that: The physiological signal acquisition module includes EEG electrodes, PPG sensors and accelerometers; wherein, The EEG electrodes are used to collect the EEG signals; The PPG sensor is used to collect the pulse rate and HRV signals; and The accelerometer is used to collect the body motion signal.

9. The fatigue monitoring system according to claim 8, characterized in that: The EEG electrodes include an Fp1 electrode, an Fp2 electrode, an O1 electrode and an O2 electrode; Among them, the Fp1 electrode is set on the inner surface of the head-mounted device close to the left forehead of the user's head, the Fp2 electrode is set on the inner surface of the head-mounted device close to the right forehead of the user's head, the O1 electrode is set on the inner surface of the head-mounted device close to the left back of the user's head, and the O2 electrode is set on the inner surface of the head-mounted device close to the right back of the user's head.

10. A fatigue monitoring method, characterized in that: include: Acquire physiological signals of the user, wherein the physiological signals include electroencephalogram signals, pulse rate and HRV signals, and body movement signals; Inputting the EEG signal, the pulse rate and HRV signal, and the body movement signal into a first fatigue assessment model, a second fatigue assessment model, and a third fatigue assessment model that have been fully trained, respectively; acquiring a first fatigue level, a second fatigue level, and a third fatigue level respectively output by the first fatigue assessment model, the second fatigue assessment model, and the third fatigue assessment model; A correlation analysis is performed on the first fatigue level, the second fatigue level, and the third fatigue level, and a final fatigue level is obtained according to an output result of the correlation analysis.

Images