A method for predicting brain fatigue in advance

Abstract

The present invention discloses a method for predicting brain fatigue in advance, which can be used for vehicle drivers, pilots, air traffic controllers, instrument monitoring personnel, etc. It includes: step 1: synchronously obtain pupil diameter data and ambient light brightness data of the current person being measured; step 2: preprocess the pupil diameter data to obtain preprocessed pupil diameter data; step 3: define the adjustment ability index according to the pupil diameter data, and calculate the time series of the adjustment ability index; step 4: use an adaptive filter to remove ambient light interference to obtain the time series of the corrected adjustment ability index; step 5: construct a brain fatigue adjustment ability prediction model according to the time series of the corrected adjustment ability index; step 6: predict brain fatigue according to the prediction model and a preset threshold, and output the prediction result. The present invention advances the traditional "measurement after fatigue manifestation" to "measurement before fatigue manifestation", and can intelligently predict the time of brain fatigue manifestation and give an early alarm.

Description

Technical Field

[0001] The invention relates to the technical field of nonlinear signal processing, and in particular to a method for predicting brain fatigue in advance. Background Art

[0002] Long hours of work such as flying, driving, and air traffic control often cause mental fatigue in the personnel involved. Mental fatigue is sometimes called mental fatigue or cognitive fatigue, which represents a psychobiological state caused by long-term cognitive activities.

[0003] In order to stably carry out cognitive tasks, the human brain has evolved a real-time regulation system for brain fatigue. This system helps people overcome physiological fatigue. The overt brain fatigue is the result of the combined effect of two internal factors. One is the gradually accumulated physiological brain fatigue, which reflects the continuous consumption of cognitive resources. The other is the regulation system, which reflects the regulation and compensation of higher centers. It can mask physiological fatigue in the early stage. Usually, after 30-120 minutes of tasks, physiological fatigue can no longer be masked and brain fatigue will appear.

[0004] However, the "masking effect" is also caused by the role of this regulatory system. The masking effect has caused great difficulties in the detection of brain fatigue. Conventional indicators are reflected through facial expressions, eyelid closure rate, EEG, etc., which belong to "fatigue post-manifestation measurement" and have the defect of late detection time, which is a serious safety hazard. Therefore, the biggest difficulty in measuring brain fatigue is the "masking effect" of brain fatigue.

[0005] At present, brain fatigue measurement technology is mainly divided into the following schools:

[0006] The first is the oral fatigue warning device:

[0007] For example, the application number is 200920160060.4, which is a mouth-held fatigue warning device. Under normal circumstances, the teeth are clenched and no alarm is sounded. When fatigued or distracted, the alarm is released and the buzzer starts to sound. This method requires the person to bite the alarm at all times, which interferes with the person being tested. Moreover, when fatigue is detected, the fatigue has reached a very dangerous level, and the reminder effect on safe driving needs to be improved.

[0008] The second is the fatigue measurement of facial features:

[0009] For example, in the Chinese patent "Automobile Driver Fatigue State Prediction System Patent" with application number 201310229235.3, it uses the time the driver's eyes are closed as a feature to activate the alarm.

[0010] In the Chinese patent "Automobile Driver Fatigue State Prediction System" with application number 201410602818.0, it uses the comparison of fatigue and non-fatigue facial or eye features to activate the alarm.

[0011] In the Chinese patent application number 201510467585.2, entitled "A patent for a wearable glasses fatigue warning device and working method", the principle of fatigue detection is achieved by detecting the eyelid closure rate.

[0012] The US patent application number US5570698A, "System for monitoring eyes for detecting sleep behavior", determines whether a person is in a drowsy state by detecting the position of the eyes.

[0013] It can be seen that the above methods all belong to "post-fatigue measurement". When fatigue is measured, brain fatigue has already manifested itself. At this time, fatigue has reached a very dangerous level, posing serious safety hazards. The reminder effect on safe driving needs to be improved.

[0014] Third, the size of the pupil diameter is directly used as a measure of fatigue:

[0015] For example, Ma Jinfei's study "Analysis of the Empirical Validity of Pupil Diameter Size in Detecting Driver Fatigue" shows that in general road traffic scene video experiments, as the driver's self-reported sleepiness increases, the pupil diameter shrinks, which may be caused by passive fatigue. The article also points out that sports with time pressure, such as racing, often require continuous mobilization of attention resources and coordination of perceptual activities, and the psychological workload is large, which makes the driver's pupils dilate and prone to active fatigue. It can be seen that the change trends of pupil diameter for passive fatigue (ordinary boring tasks) and active fatigue (highly concentrated tasks) are just opposite. Complex driving situations often include both passive fatigue and active fatigue. Therefore, it is difficult to reliably detect driver fatigue by directly measuring the size of the pupil diameter.

[0016] In the "Fatigue Driving Detection Method" with application number 201610027827.0, the driver's fatigue level is judged by detecting the driver's pupil diameter and eye closure ratio. The pupil diameter is negatively correlated with the fatigue level, and the eye closure ratio is positively correlated with the fatigue level. The "negative correlation between pupil diameter and fatigue level" mentioned in the patent is also aimed at passive fatigue, while the change trend of pupil diameter for active fatigue (highly concentrated tasks) is just the opposite. Because complex driving situations often involve both passive and active fatigue, similar to the technology in Ma Jinfei's article, it is difficult to reliably detect driver fatigue by directly measuring the size of the pupil diameter.

[0017] The Chinese patent application number 201310579177.7, "Visual fatigue detection and relief method", detects the diameter of the user's pupil and the frequency of changes in the user's pupil diameter. By comparing the frequency of changes in the pupil diameter with a predetermined value of the frequency indicating visual fatigue, the visual fatigue of the user is understood. However, the invention detects the frequency of changes in the pupil diameter, which is only related to visual fatigue, not brain fatigue.

[0018] In summary, most of the current brain fatigue measurement methods reflect the external manifestations of brain fatigue from different aspects, reflecting the results of the regulation system's regulation of physiological brain fatigue (physiological fatigue is masked). Therefore, it can only be measured after physiological brain fatigue accumulates to a large enough level (the masking effect fails) and brain fatigue manifests itself. Summary of the invention

[0019] In view of the above-mentioned problems, the present invention aims to provide a method for advance prediction of driving fatigue, which adopts the adjustment ability of the measurement and adjustment system itself as the key technology to overcome the masking effect and achieve "pre-fatigue measurement": that is, physiological brain fatigue before brain fatigue is manifested can be measured, and physiological brain fatigue can be detected within less than 5 minutes after the start of the task, and brain fatigue detection is advanced from the traditional "post-fatigue measurement" to "pre-fatigue measurement", thereby further intelligently predicting brain fatigue.

[0020] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0021] A method for predicting brain fatigue in advance, characterized in that it comprises the following steps:

[0022] Step 1: synchronously obtain the pupil diameter data and ambient light brightness data of the current person being measured;

[0023] Step 2: Preprocess the pupil diameter data, remove outliers, and perform linear interpolation on the gaps after removing outliers to obtain preprocessed pupil diameter data;

[0024] Step 3: Define the accommodation ability index based on the preprocessed pupil diameter data, and obtain the time series of the accommodation ability index by calculating the accommodation ability index.

[0025] Step 4: Based on the collected ambient light brightness data, calculate the ambient light index to obtain a time series of the ambient light index;

[0026] Step 5: Based on the time series of the ambient light index, an adaptive filter is used to remove the ambient light interference to obtain the time series of the corrected adjustment capability index;

[0027] Step 6: Construct a prediction model of brain fatigue regulation ability based on the time series of the corrected regulation ability index;

[0028] Step 7: Perform brain fatigue prediction according to the established brain fatigue regulation ability prediction model and a preset threshold, and output the prediction result and give an alarm.

[0029] Further, the specific steps of Step 3 are as follows:

[0030] Step 31: Denote the pupil diameter time series as x 1 (i), 1 ≤ i ≤ N, where N is the length of the time series, and sequentially select the j-th point, the (j + 1)-th point, …, the (j + m - 1)-th point from this time series in the order from front to back to form a vector u m (j):

[0031] u m (j) = [x 1 (j + k): 0 ≤ k ≤ m - 1];

[0032] where j represents the serial number of the sequentially selected points; k represents the offset; m represents the number of elements in the vector;

[0033] Step 32: Define the distance between the vectors u m (i) and u m (j) as d:

[0034]

[0035] Step 33: Calculate the standard deviation of the pupil diameter time series x 1 (i) as r, and count the number of all pairs of vectors u m (i) and u m (j) that satisfy d < r, and denote the obtained number as A m ;

[0036] Step 34: Based on A m , calculate S 1 :

[0037]

[0038] Step 35: Average every two values of the pupil diameter time series x (1) (i), and use the obtained average value as a new value to form a new time series x (2) (i), and repeat Steps 21 - 24 to obtain S (2) ;

[0039] Step 36: Repeat Step 25, and sequentially obtain S (3) , S (4) , …, S (τ) , where fs is the sampling rate;

[0040] Step 37: Define the formula for the accommodation ability index as:

[0041]

[0042] Step 38: Calculate S using a sliding window, and the formula for setting the sliding window width is:

[0043]

[0044] where, F s is the sampling frequency of the pupil diameter;

[0045] Step 39: According to the calculated accommodation ability index S(n).

[0046] Furthermore, the specific steps of Step 4 are:

[0047] Step 41: Collect the ambient light time series y(i), 1 ≤ i ≤ N through an environmental sensor, where N is the length of the time series, and take m points from it to form a vector u m (j):

[0048] u m (j) = [y(j + k): 0 ≤ k ≤ m - 1];

[0049] Step 42: Define the distance between the vectors u m (i) and u m (j) as d:

[0050]

[0051] Step 43: Calculate the standard deviation of the time series y(i) as r, and count the number of all pairs of vectors u m (i) and u m (j) that satisfy d < r, and denote it as A m ;

[0052] Step 44: Obtain T according to formula (1);

[0053] Step 45: Calculate T using a sliding window to obtain the ambient light index time series T1, T2, T3, ….

[0054] Furthermore, the steps of the adaptive filter for removing ambient light interference in Step 5 are:

[0055] Step 51: Redefine the sliding window sequence of the accommodation ability index S(n) obtained in Step 39 as d(n), n = 1, 2, …;

[0056] Step 52: Express d(n) as:

[0057] d(n)=S(n)+T1(n),n=1,2,…

[0058] Among them, T1(n) is the pupil accommodation index component caused by ambient light stimulation; S(n) is the real accommodation ability index component;

[0059] Step 53: Based on d(n), the error e(n) between d(n) and the adaptive filter output z(n) is obtained:

[0060] e(n)=d(n)-z(n),n=1,2,…;

[0061] Step 54: Calculate the weight vector w of the adaptive filter by iterative method:

[0062] w(n+1)=w(n)+μ[p-Rw(n)]

[0063] Where n represents the number of iterations, μ represents a positive constant, p is the cross-correlation vector of the data, and R is the correlation matrix of the data;

[0064] Step 55: Obtain the time series of the corrected regulatory capability index based on e(n).

[0065] Furthermore, the prediction model constructed in step 6 includes an advanced prediction model for predicting when the time of occurrence of brain fatigue manifestation is greater than 5 minutes from the current time, and a near-threshold accurate prediction model for predicting when the time of occurrence of brain fatigue manifestation is less than or equal to 5 minutes from the current time.

[0066] Furthermore, the steps of establishing the brain fatigue advance prediction model include:

[0067] Step 61: Define the loss function for the time series of the moment-adjusted capability index:

[0068]

[0069] Step 62: For each input time point t n , the predicted estimate is:

[0070]

[0071] Step 63: Define the average loss function as:

[0072]

[0073] Step 64: Solve the parameters that minimize L(p,b) according to formula (7):

[0074]

[0075] and:

[0076]

[0077]

[0078] Furthermore, the steps for establishing the accurate prediction model of brain fatigue near the threshold are:

[0079] In the process of real-time recording, first record a period of time to obtain the time series of the moment-adjustment capability index (t1, S1), (t2, S2),… (t N2 ,S N2 ), an autoregressive model is established, the order of the autoregressive model is selected as 50, the data window is 100s, and it is updated every 1s to obtain a linear model for predicting the waveform of 25s.

[0080] Furthermore, the specific steps of step 7 are:

[0081] Step 71: obtaining a threshold straight line through a number of set thresholds;

[0082] Step 72: Draw a linear prediction line based on the predicted estimated value obtained by the linear model for predicting brain fatigue regulation ability. Based on the intersection of the prediction line and the threshold straight line, obtain the time from the manifestation of brain fatigue to the current moment. If the time reaches the preset alarm threshold, output the prediction result.

[0083] The beneficial effects of the present invention are:

[0084] First, compared with conventional brain fatigue detection methods, the present invention directly measures the "regulatory ability" of the regulatory system itself to quantify physiological brain fatigue, and determines the key brain area of ​​the regulatory system - the locus coeruleus-norepinephrine (LC-NE) system, which is an important link in the motivational regulation of the overall brain alertness, thereby overcoming the masking effect of the body's regulatory system on fatigue and achieving pre-fatigue prediction.

[0085] Secondly, the present invention adopts a non-contact measurement and regulation method. Since the discharge of the locus coeruleus is synchronized with the change of pupil diameter, they are jointly controlled by the central nervous system (through the paragigantic nucleus GPi). LC-NE plays an important role in sustained attention and alertness. The change of pupil diameter is activated by the locus coeruleus (LC) nerve, so this change reflects the motivational gain regulation activity of the locus coeruleus. The measurement of pupil diameter in the present invention is non-contact measurement by infrared optical equipment, which realizes non-contact measurement and regulation.

[0086] Thirdly, the present invention also determines how to quantify "regulatory ability". The present invention uses a prediction method to detect physiological brain fatigue by using the adjustment ability index of pupil changes to spy on the LC-NE system's alertness motivation gain adjustment "ability". In continuous tasks, the pupil diameter gain adjustment ability index shows a "double effect" of volatility and a slow downward trend, and through algorithms, theoretical analysis, research on concentration and non-concentration modes, and key technologies of time series prediction models, accurate advance prediction is achieved, thereby realizing intelligent intervention.

[0087] Finally, in order to further reduce the negative impact of pupil light reflex, the present invention also proposes to remove the ambient light reflection to obtain a corrected physiological brain fatigue index, which can further improve the accuracy of the linear model predicting the physiological brain fatigue regulation ability, thereby improving the fatigue prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 is a flow chart of the present invention;

[0089] Figure 2 A flow chart of removing the light reflection effect of the present invention;

[0090] Figure 3 It is the cognitive task moment-regulatory ability index curve for online courses;

[0091] Figure 4 (a)-(c) are the experimental instructions and flow chart for verifying the regulatory ability in both the focused mode and the non-focused mode;

[0092] Figure 5 Provides environment for pupil data collection;

[0093] Figure 6 (a)-(b) are the pupil diameter data before and after interpolation, respectively;

[0094] Figure 7 (a)-(b) is the pupil diameter and pupil accommodation index; where, Figure 7 (a) is the pupil diameter accommodation index of 12 subjects; Figure 7 (b) is the pupil diameter accommodation ability index of the subjects in both modes;

[0095] Figure 8 This is a performance test scenario for the method of the present invention;

[0096] Fig. 9 An interface for running PERCLOS to detect eyelid closure rate;

[0097] Fig.10 (a)-(b) are test result diagrams of subject 1 in the functional test;

[0098] Fig.11 (a)-(b) are test result diagrams of subject 2 in the functional test;

[0099] Fig.12 (a)-(b) are the test result diagrams of subject 3 in the functional test. DETAILED DESCRIPTION

[0100] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0101] Example 1

[0102] Since the brain fatigue regulation system is transmitted to the locus coeruleus-norepinephrine system through the prefrontal cortex, the secretion of norepinephrine is controlled to regulate the overall brain alertness level. The pupil diameter is highly consistent with the discharge of the locus coeruleus. Therefore, the present invention proposes a pupil diameter analysis algorithm that can reflect the regulation ability of the locus coeruleus. Based on this algorithm, Figure 1 As shown, the present invention proposes a method for predicting brain fatigue in advance, which includes the following steps:

[0103] Step 1: Data Acquisition

[0104] Obtaining pupil diameter data of the person to be measured;

[0105] Step 2: Data Preprocessing

[0106] Preprocessing the obtained pupil diameter data to remove outliers, and performing linear interpolation on the gaps after removing the outliers, thereby obtaining preprocessed pupil diameter data;

[0107] Step 3: Calculate the regulatory capability index

[0108] A time window is taken for the preprocessed pupil diameter data, and the time window is 50 seconds to obtain the corresponding pupil diameter time series data, define the accommodation ability index, and calculate the pupil diameter time series data within the time window to obtain the time series of the accommodation ability index;

[0109] Specifically include:

[0110] (1) Define the accommodation ability index based on the pupil diameter time series

[0111] First, let the pupil diameter time series be x 1 (i), 1<<i<<N, where N is the length of the time series. From the time series, we select the jth point, j+1th point, ..., j+m-1th point in order from the front to the back, a total of m points (m is generally 2) to form a vector u m (j):

[0112] um (j) = [x 1 (j + k): 0 << k << m - 1]

[0113] Among them, the j variable represents the serial number of the points sequentially selected in the forward order in this time series; the k variable represents the offset, and the range is from 0 to m - 1; m represents the dimension of the embedding vector u m (j), that is, the number of elements in the vector;

[0114] Secondly, define the distance between two vectors u m (i) and u m (j) as d:

[0115]

[0116] Again, calculate the standard deviation of the pupil diameter time series x 1 (i) as r, and count all pairs of vectors u m (u m (i) and u m (j)) where the distance d < r. Count a pair of vectors that meet the above conditions as 1, and record the obtained count as the scalar A m , and calculate S m based on A 1 :

[0117]

[0118] Among them, the superscript of S 1 indicates that it is calculated from the pupil diameter time series x 1 , and the superscript (1) represents the original signal.

[0119] Again, calculate the average value of every 2 values of the pupil diameter time series x (1) (i), and use the obtained average value as the new value, thereby forming a new pupil diameter time series x (2) (i). Repeat the above steps, and obtain S (2) according to formula (1), and so on, to obtain S (3) , S (4) , …, S (τ) , where τ is a number between 10 and 30, and this value depends on the sampling rate of the pupil diameter. The higher the sampling rate f s , the larger the τ value. The symbol represents rounding down. For example, when the sampling rate is 100 Hz, τ can be set to 10;

[0120] Finally, define the accommodation ability index as:

[0121]

[0122] (2) Calculate the Adjustability Index through Sliding Window

[0123] The present invention uses a sliding window to calculate S, and the sliding window width is determined by the following formula:

[0124]

[0125] Among them, f s is the sampling frequency of pupil diameter;

[0126] The step length of the sliding window movement is between 0.1-10s. The shorter the step length, the more conducive it is to the subsequent prediction modeling (improving prediction accuracy), but the amount of calculation is also greater. Therefore, after many experiments, the step length of the sliding window movement of the present invention is set to 1s.

[0127] Through sliding window calculation, we can get the time series of moment-adjustment capability index, (t1, S1), (t2, S2), … (t N2 ,S N2 );

[0128] The inventors have proved through experiments (see Example 3 for details) that the time series contains both components reflecting physiological brain fatigue (manifested as a downward trend) and the switching between the focus mode and the non-focus mode (manifested as periodic oscillations). It is also proved that the focus mode (good for driving, flying, etc.) is the peak, and the non-focus mode (bad for driving, flying, etc.) is the trough. The superposition of the trough and the downward trend means that only the low point of the time series needs to be predicted to predict the brain state that is unfavorable to the work task;

[0129] Step 4: Predictive Modeling

[0130] According to the obtained time series of the moment-regulatory ability index, a linear prediction model is built to construct a linear model for predicting physiological brain fatigue regulatory ability, thereby obtaining fatigue-related parameters (i.e., the slope and intercept p after linear fitting). * ,b * or autoregressive model coefficients), the predictive linear model includes the brain fatigue advance prediction model (i.e. the slope and intercept p after the straight line fitting * ,b * ) and an accurate prediction model of brain fatigue near the threshold (i.e., autoregressive model);

[0131] (1) The brain fatigue advance prediction model is used to predict the situation where the occurrence of brain fatigue manifestation is greater than 5 minutes from the current time. The construction steps are as follows:

[0132] First, define the loss function:

[0133]

[0134] Secondly, for each input time point t n , the predicted estimated value of physiological brain fatigue regulation ability is:

[0135]

[0136] Among them, p represents the slope and b represents the intercept;

[0137] Again, the loss function is used to calculate the difference between the predicted estimate and the true value:

[0138]

[0139] The average loss function is defined as:

[0140]

[0141] Finally, the parameters (p, b) that minimize L(p, b) are solved by formula (7):

[0142]

[0143] get:

[0144]

[0145]

[0146] Among them, p * ,b * They are the slope and intercept of the linear model predicting the physiological system's brain fatigue regulation ability;

[0147] In summary, Formula 5, Formula 8 and Formula 9 are the linear models for predicting physiological brain fatigue regulation ability constructed by the present invention. Physiological brain fatigue increases approximately linearly with the work task. At this time, the linear model for predicting brain fatigue regulation ability constructed by the present invention uses the negative slope and the starting value at the beginning of the task as linear prediction parameters.

[0148] (2) The accurate prediction model of brain fatigue near the threshold is used to predict the situation where the occurrence of brain fatigue manifestation is less than or equal to 5 minutes from the current time. The construction steps are as follows:

[0149] In the process of real-time recording, record a period of time and obtain the time series of the moment-adjustment capability index (t1, S1), (t2, S2), … (t N2 ,S N2 ), an autoregressive model is established. The order of the autoregressive model can be selected as 50, the data window is 100s, and it is updated every 1s to predict the waveform of 25s.

[0150] Step 5: Predictive Alarm

[0151] Brain fatigue prediction is performed based on the established physiological brain fatigue model and preset thresholds;

[0152] The linear prediction parameter p obtained according to the physiological brain fatigue model * ,b * , 3-5 (or more) thresholds are determined through experiments, for example, early alarm threshold (20 minutes in advance), mid-term alarm threshold (5 minutes), late alarm threshold (1 minute) and critical alarm threshold (20 seconds), and threshold curves are obtained through these thresholds. Then, a linear prediction line is drawn based on the predicted estimated value obtained by the linear model for predicting brain fatigue regulation ability. The system can adjust the system at any time based on the linear prediction line (i.e., p * ,b * ) and the threshold straight line to calculate the time from the threshold as a reminder.

[0153] The time from the appearance of brain fatigue to the present can be obtained through the intersection of the prediction line and the alarm threshold line. If the preset alarm time (threshold) is reached, the prediction result is output. For example, if the preset alarm time is 1 hour, 40 minutes, 20 minutes, 10 minutes in advance, etc., the prediction result is output when the threshold is reached.

[0154] Example 2

[0155] On the basis of Example 1, in order to eliminate the influence of ambient light, an adaptive filtering link is added. By synchronously collecting pupil diameter and ambient light, the influence of ambient light is eliminated, and the time series of the moment of eliminating the influence of ambient light-adjustment ability index (t1, S1), (t2, S2), ... (t N2 ,S N2 ), and then build a linear model for predicting physiological brain fatigue regulation ability based on the time series of the moment-regulation ability index to obtain fatigue-related parameters.

[0156] The pupil light reflection effect refers to the change in pupil diameter caused by changes in light. In order to reduce the negative impact of pupil light reflection, the present invention collects changes in ambient light through an environmental sensor, calculates the ambient light index, and eliminates the influence of ambient light through an adaptive filter, and then obtains the corrected adjustment ability index.

[0157] Specifically, the ambient light index method is calculated as follows:

[0158] Step 1: Collect the ambient light time series y(i) through the environmental sensor, 1<<i<<N, where N is the length of the time series, and take m points to form a vector u m (j):

[0159] um $(j)=[y(j + k):0\lt\lt k\lt\lt m - 1]$;

[0160] Step 2: Define the vector $u$ m $(i)$ and $u$ m The distance between $(j)$ is $d$:

[0161]

[0162] Step 3: Calculate the standard deviation of the time series $y(i)$ as $r$, and count all groups of vectors $u$ that satisfy $d\lt r$ m $(i)$ and $u$ m $(j)$ and denote the quantity as $A$ m ;

[0163] Step 4: Calculate $T$:

[0164]

[0165] Step 5: Calculate $T$ using a sliding window to obtain the ambient light index time series $T_1, T_2, T_3, \ldots T$ N .

[0166] Step 1: The obtained accommodation ability index $S(n)$ may be mixed with the true accommodation ability index caused by the pupil and the false component caused by ambient light stimulation. Therefore, re - define the sliding window sequence of $S(n)$ as $d(n), n = 1, 2, \ldots$;

[0167] Step 2: $d(n)$ is mixed with the true accommodation ability index component $S(n), n = 1, 2, \ldots$ caused by the pupil and the false component $T_1(n), n = 1, 2, \ldots$ caused by ambient light stimulation. Thus, $d(n)$ is expressed as:

[0168] $d(n)=S(n)+T_1(n), n = 1, 2, \ldots$

[0169] where $T_1(n)$ is the pupil accommodation index component caused by ambient light and is strongly correlated with the above - calculated ambient light index $T(n)$;

[0170] As Figure 2 shown, the steps to remove ambient light interference using an adaptive filter are:

[0171] $e(n)=d(n)-z(n), n = 1, 2, \ldots$

[0172] where $e(n)$ is the error between $d(n)$ and the output $z(n)$ of the adaptive filter.

[0173] Step 3: The weight vector $w$ of the adaptive filter can be obtained by the iterative method:

[0174] w(n+1)=w(n)+μ[p-Rw(n)]

[0175] Here n represents the number of iterations, μ represents a positive constant, p is the cross-correlation vector of the data, and R is the correlation matrix of the data.

[0176] Step 4: Based on the solved e(n), the time series of the corrected adjustment capability index is obtained to eliminate the influence of ambient light.

[0177] Example 3

[0178] In order to further verify the prediction effect of the method proposed in the present invention, the following experiments were carried out respectively.

[0179] 1. Double effect experiment in pupil diameter accommodation index

[0180] Through 16 college students completing the cognitive tasks of the online course, the pupil diameters of the 16 college students were collected and calculated, and the time-adjustment ability curve was calculated, such as Figure 3 As shown. As can be seen from the figure, the regulatory ability curve shows two obvious effects: overall decline and volatility. The overall downward trend can be explained by the decline in regulatory ability caused by fatigue; but how to explain the volatility? According to prior art records, when people do cognitive tasks, their brains usually switch between focus mode (exploitation) and non-focus mode (exploration). The volatility of the regulatory ability curve we observed is very likely caused by the switching between the two modes. The non-focus mode requires less cognitive effort, so how should the brain's work performance and regulatory ability change? Should they decline at the same time? In order to verify this, we further conducted experimental verification to explore whether the focus mode and the non-focus mode correspond to peaks or troughs, respectively.

[0181] 2. Verify whether the focus mode or the non-focus mode represents a high adjustment ability index

[0182] (1) Experimental Materials

[0183]

[0184] The Qixin aSee Pro eye tracker mainly used in this experiment to measure and record pupil data is a high-precision, multi-functional eye tracking instrument. It uses a unique multi-point correction algorithm to solve the error problems such as visual field distortion that exist in traditional eye trackers, and improves the accuracy and reliability of the test results. It also provides professional eye movement data analysis software that supports real-time recording, storage, playback and analysis of eye movement data, allowing users to have a more comprehensive understanding of the meaning and rules behind the eye movement data. In addition, the Qixin aSee Pro eye tracker also supports real-time data collection and storage, multi-channel data synchronous collection, and multiple eye movement task design and analysis functions, making the test results more accurate.

[0185] (2) Experimental reagents

[0186] Greentech GT5 conductive paste Wuhan Greentech Technology Co., Ltd., China

[0187] (3) Subjects

[0188] This experiment recruited 19 volunteers (5 males and 14 females, average age: 21.14 years, age range: 19-27 years). All participants had normal or corrected-to-normal vision and gave written informed consent, in accordance with the requirements of the Medical Ethics Committee of the Air Force Medical University.

[0189] (4) Experimental methods

[0190] i) Experimental paradigm

[0191] The process of this experimental task is as follows Figure 4 (c) There were 60 trials in total. In each trial, the subject first saw the instruction “Please empty your mind and stay relaxed” (e.g. Figure 4 (a)). After that, a "+" sign appeared in the center of the screen, and the subject was required to look at the '+' to stabilize the subject's pupil attention position and prevent eye movement from interfering with the collection of pupil diameter. The '+' sign continued to appear for 15 seconds, during which time the subject was required to remain in a non-focused state and not engage in any mental activity. After 15 seconds, the screen appeared with the instruction "When 'T' appears, press 'Space' immediately" (e.g. Figure 4 (b)), the instructions last for 3 seconds, and then the target object 'T' will appear at any time within 1 second to 6 seconds. At this time, the subject ends the previous non-focused state and enters the focused state, always alert to the appearance of the target 'T'. After 'T' appears, the subject presses the space bar and continues to the next trial.

[0192] The experimental program was written using E-prime software. The focus state instructions and target 'T' were in red font, and other instructions were in black font. The background was white to reduce the impact of light changes on pupil diameter. The light was kept constant throughout the experiment, and the experimental environment was kept quiet.

[0193] ii) Data collection

[0194] Conduct the experiment in an environment with appropriate brightness and stable lighting, such as Figure 5 As shown. Appropriate brightness is conducive to collecting pupil data in cognitive tasks. The subject's head was fixed with a bracket throughout the experiment to avoid the reduction of data efficiency caused by head movement. The data was recorded using Qixin Yiwei aSee Pro eye tracker, including information such as the position of the gaze point and pupil diameter. The data collection and experimental process were recorded in the supporting aSeeStudio software, and a three-point calibration of both eyes was performed before each experiment. At the same time, the Borycon EEG device was used to record the subject's EEG data during the experiment.

[0195] iii) Data processing

[0196] The eye tracker sampling frequency is set to 100Hz. The actual sampling rate is lower than the set value, but it does not affect the processing of pupil data. The screen recording video and original pupil data in the experiment can be obtained through the aSeeStudio software.

[0197] S1: Experimental time points

[0198] The time points where the events occurred in the experiment were marked to facilitate the subsequent data segmentation. There were 60 trials in the experiment, and there were two time points in each trial, namely the start time of the non-focused state and the start time of the focused state, so a total of 120 time points needed to be marked. The traditional method is to watch the screen video recorded in the experiment and manually mark the time points. However, this method is labor-intensive and takes a lot of time and effort. To this end, Matlab software is used to write an automated processing program, and the video frames that need to be marked are selected as key frames. The program will automatically find video frames similar to the key frames, thereby automatically marking all time points and saving them.

[0199] S2: Data Segmentation

[0200] In the above process, the time points of the start of the non-focus state and the focus state have been obtained, and the two are connected to form 120 time periods. The original pupil diameter data is segmented according to the corresponding time points to generate 60 non-focus mode time periods and 60 focus mode time periods.

[0201] S3: Data Cleansing

[0202] During blinking, invalid data detected by the eye tracker will be recorded as -1. When -1 appears too many times in a data segment, subsequent calculations will be invalid. Therefore, when the number of -1 in a data segment accounts for more than 50% of the total data, this segment of data will be discarded.

[0203] S4: Data interpolation

[0204] After data cleaning, there are still invalid data in the data segment (such as Figure 6 (a)), use the two data points before and after the -1 point to perform linear interpolation to replace the -1 point (such as Figure 6 (b)).

[0205] (5) Pupil data processing results

[0206] Among the 19 subjects who completed the above experiment, the data of 5 subjects had a large amount of abnormal data and low data efficiency. Therefore, the data of these 5 subjects were discarded, and the data of the remaining 14 subjects were processed and analyzed. Figure 7 (a) shows the results of pupil diameter accommodation index of 12 subjects. In each figure, the blue column represents the average value of pupil diameter accommodation index of the subjects in the non-focused state, the red column represents the average value of pupil diameter accommodation index of the subjects in the focused state, and the blue error bar above represents the corresponding standard error. Figure 7 (b) shows the pupil diameter adjustment ability index of the subjects in the two modes. It can be seen that the pupil diameter adjustment ability index of the subjects in the focused state (i.e., tense) is higher than the entropy in the non-focused state (i.e., relax) regardless of the individual or the whole.

[0207] In addition, a paired t-test was performed on the pupil diameter accommodation ability index of all subjects in the two modes, and the statistical test result was P=8.4452e-33 (P<0.05), which was significantly different.

[0208] (6) Experimental conclusion

[0209] This experiment mainly compares the pupil diameter adjustment ability index in the non-focused mode and the focused mode, records the pupil diameter of the subjects, and performs analysis and calculation. It can be seen from the experimental results that the pupil diameter adjustment ability index is higher in the focused mode. The subjects make less cognitive effort in the non-focused mode, and when the subjects switch to the focused state, their cognitive functions are activated, and people are more likely to be in a state of concentration. This explains that the volatility observed in the pupil diameter adjustment ability index is the switching of the two working modes. Since the trough in the pupil diameter adjustment ability represents the non-focused mode, it is not conducive to driving, and the trend of the pupil diameter adjustment ability decreasing is not conducive to driving. The two are consistent, and the lower the more unfavorable it is for driving. This proves the feasibility of the present invention to make advance predictions of brain fatigue manifestations through the pupil diameter adjustment ability index curve.

[0210] 3. Experiment of the advanced prediction method proposed by the present invention

[0211] (1) Performance test content

[0212] In order to further verify the performance of the fatigue advance prediction method proposed in the present invention, this experiment was tested through three indicators and compared with the perclos fatigue detection algorithm to verify the accuracy of the prediction method of the present invention.

[0213] (2) Test environment

[0214] Software environment: Visual Studio 2022, MATLAB R2022a, Anaconda 2022

[0215] (3) Subjects

[0216] Five volunteers (3 males and 2 females, average age: 20.6 years old, age range: 19-27 years old) were recruited. All five volunteers participated in the simulated driving experiment.

[0217] (4) Experimental process

[0218] S1: Align the human eyeball with the eye tracker at a certain distance, and calibrate according to the eye tracker calibration software process to ensure that the experimental data collected is true and reliable. Adjust the camera of another computer to ensure that facial information can be collected, such as Figure 8 As shown;

[0219] S2: Use software X-plane for simulated driving;

[0220] S3: When the person being tested starts the simulated driving, the fatigue advance prediction method code of the present invention is run (the test software designed based on the prediction method of the present invention), and the perclos fatigue detection algorithm code is run at the same time as the fatigue detection control to verify the accuracy of the prediction method. Fig. 9 As shown;

[0221] S4: Conduct test item 1 and test item 2 respectively;

[0222] i) Test Item 1: Whether the software can successfully draw the linear prediction and AR model images after running for five minutes, and whether it can display the predicted fatigue time in the text display area;

[0223] ii) Test Item 2: Compare the time error between the intelligent prediction time of brain fatigue of the present invention and the time error between the fatigue reported by the perclos program, and determine whether it is within the acceptable 3 minutes;

[0224] S5: Test and record the results of the above test items, as shown in Table 1.

[0225] Table 1 Functional test data record table

[0226]

[0227] In the brain fatigue intelligent prediction test experiment, it can be seen in the tests of multiple subjects that as the driving task progresses, the drawn curve forms a trend of volatility and a slow decline. After the software has been running for 5 minutes, a prediction curve appears and the predicted fatigue time is displayed. The error with the fatigue time measured by the perclos program is no more than 3 minutes.

[0228] (5) Experimental results

[0229] The gain regulation signal regulation ability index in the non-focus mode and the focus mode was compared, and the pupil data and EEG data of the subjects were recorded and analyzed and calculated. It can be seen from the experimental results that the pupil regulation ability index is higher in the focus mode. The subjects made less cognitive effort in the non-focus mode, and the functional state of the subjects as a complex system was at a lower level. When the subjects switched to the focus state, their cognitive functions were activated. This also shows that when the regulation ability index of the gain regulation signal is at a high level, people are more likely to be in a state of concentration. When it decreases, it may indicate that they are in a non-focused state such as distraction or fatigue.

[0230] Therefore, it can be concluded that this experiment proves that the fluctuation in the pupil diameter accommodation ability index is the switching of two working modes, the peak is the focus mode, and the trough is the non-focus mode.

[0231] Combined with the brain fatigue intelligent prediction test, it can be found that by simulating the fatigue condition of users caused by long-term driving, and using the PERCLOS fatigue detection algorithm as a comparative judgment in real time, the functions of the brain fatigue intelligent prediction system are correct according to three different test indicators. It successfully draws linear predictions and AR model images and displays the predicted remaining fatigue time. It also gives timely stimulation when the user has brain fatigue to improve concentration. That is, users can predict fatigue conditions in advance and avoid driving risks based on this system.

[0232] (6) Functional testing

[0233] The function of the prediction method proposed in the present invention was verified by conducting a teaching content learning task test on three subjects.

[0234] First, subject 1 watched the video teaching content and recorded the pupil diameter ( Fig.10 a first line). According to the method of the present invention, the physiological brain fatigue index ( Fig.10 a The second row) black line, establish a prediction model, obtain the parameters of the prediction model according to the data, and draw the prediction line ( Fig.10 Red dashed line in b);

[0235] The prediction line parameters are: (Unit: Knight). The slope is -0.0012506t n The intercept is 1.2805 nits per second. The threshold of apparent brain fatigue determined by experiment is 0.2 nits. So, according to calculation, fatigue can be reached after about 14 minutes.

[0236] Secondly, subject 2 performed the same experimental steps as above, and the results were as follows Fig.11 a- Fig.11 As shown in b, the prediction line parameters are: (Unit: nat) According to calculations, fatigue can be reached after about 101 minutes.

[0237] Finally, subject 3 performed the same experimental steps as above, and the results were as follows Fig.12 a- Fig.12 b, and its prediction line parameters are: (Unit: nat) According to calculations, fatigue can be reached after about 149 minutes.

[0238] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for predicting brain fatigue in advance, characterized in that: The following steps are involved: Step 1: synchronously obtain the pupil diameter data and ambient light brightness data of the current person being measured; Step 2: Preprocess the pupil diameter data, remove outliers, and perform linear interpolation on the gaps after removing outliers to obtain preprocessed pupil diameter data; Step 3: Based on the preprocessed pupil diameter data, define the accommodation ability index, and obtain the time series of the accommodation ability index by calculating the accommodation ability index; Step 4: Based on the collected ambient light brightness data, the ambient light index is calculated to obtain a time series of the ambient light index; Step 5: Based on the time series of the ambient light index, an adaptive filter is used to remove the ambient light interference to obtain the time series of the corrected adjustment capability index; Step 6: Construct a prediction model of brain fatigue regulation ability based on the time series of the corrected regulation ability index; Step 7: Predict brain fatigue based on the established brain fatigue regulation ability prediction model and the preset threshold, and output the prediction results and alarm; The specific steps of step 3 are: Step 31: Let the pupil diameter time series be x 1 (i), 1≤i≤N, where N is the length of the time series, and the jth point, j+1th point, ..., j+m-1th point are selected from the time series in order from the front to the back, a total of m points to form the vector u m (j): u m (j)=[x 1 (j+k):0≤k≤m-1]; Among them, j represents the serial number of the points selected in sequence; k represents the offset; m represents the number of elements in the vector; Step 32: Define vector u m (i) and u m The distance between (j) is d: d=max k＝0,1,…,m-1 |x 1 (i+k)-x 1 (j+k)|; Step 33: Calculate the pupil diameter time series x 1 (i) has a standard deviation of r, and count all pairs of vectors u m (i) and u m (j) that satisfy d < r, and denote the resulting count as A m ; Step 34: Based on A m , calculate S 1 : Step 35: Convert pupil diameter time series to x (1) (i) Calculate the average of every two values ​​and use the average as the new value to form a new time series x (2) (i) Repeat steps 31 to 34 to obtain S (2) ; Step 36: Repeat step 35 to obtain S (3) ,S (4) ,…,S (τ) ,in f s is the sampling rate; Step 37: Define the formula for the regulatory capability index as: Step 38: Use a sliding window to calculate S, and the setting formula of the sliding window width is: Among them, F s is the sampling frequency of pupil diameter; Step 39: According to the calculated regulation capability index S(n), a time series of the regulation capability index is obtained.

2. A method for predicting brain fatigue in advance as claimed in claim 1, characterized in that: The specific steps of step 4 are: Step 41: Collect the ambient light time series y(i) through the environmental sensor, 1≤i≤N, where N is the length of the time series, and take m points from it to form a vector u' m (j): u' m (j)=[y(j+k):0≤k≤m-1]; Step 42: Define vector u' m (i) and u' m The distance between (j) is d': of max k＝0,1,...,m-1 |y(i+k)-y(j+k)|; Step 43: Calculate the standard deviation of the time series y(i) as r', and count all the vectors u'(i) and u'(j) that satisfy d' < r', and denote the count as A'. m (i) and u' m (j), and denote the count as A'. m ; Step 44: Obtain T by the following formula: Step 45: Calculate T using a sliding window to obtain the ambient light index time series T1, T2, T3, ...

3. A method for predicting brain fatigue in advance as claimed in claim 2, characterized in that: The steps of removing ambient light interference by the adaptive filter in step 5 are as follows: Step 51: redefine the sliding window sequence of the adjustment capability index S(n) obtained in step 39 as d(n), n=1, 2, ...; Step 52: Express d(n) as: d(n)=S(n)+T1(n),n=1,2,… Among them, T1(n) is the pupil accommodation index component caused by ambient light stimulation; S(n) is the real accommodation ability index component; Step 53: Based on d(n), the error e(n) between d(n) and the adaptive filter output z(n) is obtained: e(n)=d(n)-z(n),n=1,2,…; Step 54: Calculate the weight vector w of the adaptive filter by iterative method: w(n+1)=w(n)+μ[p-Rw(n)] Where n represents the number of iterations, μ represents a positive constant, p is the cross-correlation vector of the data, and R is the correlation matrix of the data; Step 55: Obtain the time series of the corrected regulatory capability index based on e(n).

4. A method for predicting brain fatigue in advance as claimed in claim 1, characterized in that: The prediction model constructed in step 6 includes an advanced prediction model for predicting when the time of occurrence of brain fatigue is greater than 5 minutes from the current time, and a precise prediction model for brain fatigue near the threshold for predicting when the time of occurrence of brain fatigue is less than or equal to 5 minutes from the current time.

5. A method for predicting brain fatigue in advance as claimed in claim 4, characterized in that: The steps of establishing the advanced prediction model of brain fatigue include: Step 61: Define the loss function for the time series of the moment-adjusted capability index: Step 62: For each input time point t n , the predicted estimate is: Step 63: Define the average loss function as: Step 64: Solve the parameters that minimize L(p,b) according to formula (7): and:

6. A method for predicting brain fatigue in advance as claimed in claim 4, characterized in that: The steps for establishing the accurate prediction model of brain fatigue near the threshold are as follows: In the process of real-time recording, first record a period of time to obtain the time series of the moment-adjustment capability index (t1, S1), (t2, S2),… (t N2 ,S N2 ), an autoregressive model is established, the order of the autoregressive model is selected as 50, the data window is 100s, and it is updated every 1s to obtain a linear model for predicting the waveform of 25s.

7. A method for predicting brain fatigue in advance as claimed in claim 1, characterized in that: The specific steps of step 7 are: Step 71: obtaining a threshold straight line through a plurality of set thresholds; Step 72: Draw a linear prediction line based on the predicted estimated value obtained by the linear model for predicting brain fatigue regulation ability. Based on the intersection of the prediction line and the threshold straight line, obtain the time from the manifestation of brain fatigue to the current moment. If the time reaches the preset alarm threshold, output the prediction result.

Images