A fatigue driving monitoring method based on graph regularized extreme learning machine

Abstract

The present invention discloses a fatigue driving monitoring method based on graph regularized extreme learning machine, comprising: collecting the electrooculogram signal and facial image information of the driver during driving by means of an electrooculogram signal acquisition device and a camera; preprocessing the electrooculogram signal and facial image information respectively; extracting eye movement features of the preprocessed facial image information, classifying the preprocessed electrooculogram signal into three types of eye movement signals and extracting relevant features; determining that the driver is fatigue driving and giving an alarm when the eye closing duration in the eye movement feature exceeds a preset threshold; calculating the average value of the same features extracted from the electrooculogram signal and the facial image information when the eye closing duration in the eye movement feature does not exceed the preset threshold; inputting the average value and the extracted different features into the trained graph regularized extreme learning machine classification model, and discriminating the fatigue state by means of a GELM classifier. The monitoring method is more objective and accurate.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle driving fatigue monitoring and data processing technology, and in particular relates to a method for fatigue driving monitoring based on a graph regularized extreme learning machine. Background Art

[0002] According to the World Health Organization (WHO) 2015 Global Road Safety Status Report, 1.25 million people die in traffic accidents every year and millions more are seriously injured in traffic accidents. It is estimated that the main cause of death among young people is road traffic injuries, especially those aged 15-29. During driving, the driver's fatigue mental state seriously affects driving behavior and is a major hidden danger to traffic safety driving. In Europe, a study of 19 countries found that 17% of drivers felt drowsy while driving, and 7% of those who fell asleep had an accident.

[0003] Although researchers have made significant progress in monitoring fatigue status, there are still few related studies that can be put into practice due to the complexity of the driving environment and the imperfection of fatigue mechanisms in the transportation field. Other types of illegal driving such as overloading, speeding, overcrowding, drunk driving, etc. can reduce the occurrence of accidents through effective supervision measures. However, there are no quantitative and scientific measurement standards and detection methods for fatigue driving. Unlike other illegal driving behaviors, fatigue driving reflects a complex operating behavior that is difficult to judge and identify. How to correctly and scientifically define driving fatigue and implement more accurate real-time monitoring is the main research focus of researchers.

[0004] The term fatigue refers to a combination of symptoms such as impaired performance and subjective sleepiness. The European Transport Safety Council (ETSC) defines fatigue as "a preoccupation with the inability or unwillingness to continue an activity, usually because it has been going on for too long". Since distracting activities can take many forms, some literature divides distraction into 4 categories from the driver's functional perspective: visual distraction, cognitive distraction, auditory distraction, and biomechanical distraction. Even with intensive research, the term fatigue still does not have a universally accepted definition. Therefore, it is difficult to determine the level of accidents related to fatigue.

[0005] Studies have shown that 25-30% of driving accidents are related to fatigue. When a driver is tired and starts to fall asleep, the following symptoms can be observed: excessive yawning, confused thinking and seemingly unclear, feeling frustrated and irritable, slower reactions and responses, daydreaming, unable to open eyes, burning sensation in eyes, slow steering, difficulty concentrating, head or body swaying due to dozing off, vehicle deviating from the road or entering another lane, falling asleep while driving, shallow breathing, accelerated heartbeat, etc. Different individuals show different symptoms and degrees, so there is no specific method to measure the degree of fatigue. In the prior art, NTSC studies have shown that the degree of fatigue or sleepiness is a function of the amount of activity associated with the physiological wakefulness of the brain. Several factors can affect this physiological wakefulness, thereby lowering the fatigue threshold, such as sleep disorders, the lowest point of the circadian rhythm, hard work before driving, etc. These factors are not related to activity, but will cause the fatigue effect to appear faster. Therefore, fatigue cannot be simply regarded as a function of driving time.

[0006] The existing various driving fatigue monitoring methods generally have their own limitations. For example, the subjective evaluation method is widely used, but it has expectation bias and will interfere with normal driving, and is not suitable for continuous real-time monitoring of mental fatigue. In addition, it is unrealistic to always require drivers to report their status. In addition, the visual monitoring method is greatly affected by lighting factors. In the case of sunlight reflection and glasses reflection, the performance may drop by 30%. Among the current visual-based monitoring methods, the frontal face works well, but extreme head postures will lead to errors in the monitoring results. In addition, monitoring driver behavior based on vehicle operation information requires modification of the vehicle structure, which is unrealistic and unwise in reality. The development of vehicle automation also has a certain restrictive effect on the technology of obtaining fatigue status through vehicle movement. The recognition technology based on physiological signals belongs to one of the objective detection methods. It mainly judges whether the driver is in a state of driving fatigue by analyzing the change law of physiological signals in the driver's body. The extracted physiological signals mainly include EEG, ECG, EOG, EMG, heart rate and respiratory rate. Because physiological signals are sensitive to the fatigue state of the human body and have high monitoring accuracy, related research has developed rapidly in recent years. Among them, EOG signals have become a potential fatigue monitoring technology because of their relatively low cost, low power consumption, fast response speed, no obstruction of the driver's line of sight, and wide eye movement tracking capabilities.

[0007] However, the monitoring of driving fatigue only relies on EOG signals, which is relatively simple in form and cannot fully and accurately express the driving fatigue. Summary of the invention

[0008] The main purpose of the present invention is to provide a fatigue driving monitoring method based on a graph regularized extreme learning machine that at least partially solves the above-mentioned technical problems, processes and analyzes the driver's electrooculographic signals and facial image information, and realizes accurate monitoring of fatigue driving.

[0009] To achieve the above object, the technical solution adopted by the present invention is:

[0010] A fatigue driving monitoring method based on graph regularized extreme learning machine, comprising:

[0011] S10, collecting eye contact signals and facial image information of the driver during driving by using an eye contact signal collection device and a camera respectively;

[0012] S20, pre-processing the electrooculogram signal and the facial image information respectively;

[0013] S30, extracting eye movement features from the preprocessed facial image information, and classifying the preprocessed electrooculogram signals into three types and extracting relevant features;

[0014] S40, when the eye closing time in the eye movement feature exceeds a preset threshold, determining that the driver is driving fatigued and issuing an alarm;

[0015] S50. When the eye closure duration in the eye movement feature does not exceed a preset threshold, the average value of the same features extracted from the electrooculogram signal and the facial image information is calculated; the average value and the extracted different features are input into the trained graph regularized extreme learning machine classification model, and the fatigue state is determined by the GELM classifier.

[0016] Furthermore, in step S10, the driver's electrooculogram signal is collected once at a preset time interval during driving; and the parameters of the electrooculogram signal collection device are determined according to the amplitude and frequency of the eye movement signal.

[0017] Furthermore, in step S20, the electrooculogram signal is preprocessed, including:

[0018] S201, denoising and reconstructing the electrooculogram signal using a wavelet function;

[0019] S202, performing normalization processing on the reconstructed electrooculogram signal;

[0020] S203 , using a window function to perform secondary effective extraction on the normalized data to obtain a local minimum value, a local maximum value, and their corresponding positions and differences.

[0021] Furthermore, after step S201 and before step S202, the following steps are further included:

[0022] Unbiased risk estimation threshold is used for automatic threshold selection to achieve secondary denoising.

[0023] Furthermore, an unbiased risk estimation threshold is used for automatic threshold selection to achieve secondary denoising, including:

[0024] For signal s(i), take the absolute value of each data point, sort them from small to large, and take the square of each point as a replacement for the new signal data point. The formula is as follows:

[0025] f(k)=(sort(|s|)) 2 , (k=0, 1, ···, N-1); sort(·) is the sorting function;

[0026] The square root of the kth data point of f(k) is the threshold λ of the point k ,

[0027] The risk generated by the threshold at this point is Rish(k):

[0028]

[0029] Where k represents the kth data point, k = 0, 1, ..., N-1; N is the total number of data points; i represents the index value of the k data points; j represents a point from i-1 to the kth point;

[0030] The threshold corresponding to the minimum risk point is taken as the automatic threshold to achieve secondary denoising.

[0031] Furthermore, in step S20, the facial image information is preprocessed, including:

[0032] According to the initial face frame position and the eye area positioning based on the image gradient, the image processing position is compressed to the target area and used as the input of the machine algorithm to identify and extract the eye image.

[0033] Furthermore, in step S30, the pre-processed electrooculogram signal is classified into EOG signals, including:

[0034] The KNN algorithm is used to divide eye movement signals into three stages according to their different characteristics: blink stage, fixation stage, and saccade stage.

[0035] According to the difference, relevant characteristic data of three stages of blinking are extracted, including: blinking duration, eye closing duration, eye opening duration, eye closing speed, eye opening speed, maximum eye closing speed, minimum eye closing speed and total energy of the blinking process.

[0036] Furthermore, in step S30, extracting eye movement features from the pre-processed facial image information includes:

[0037] Using a median filter to perform denoising on the pre-processed facial image information;

[0038] The maximum inter-class variance method is used to extract the eye area image and perform binarization processing;

[0039] Based on the binary eye area image, the blinking frequency per unit time, the blinking action time, and the proportion of eye closure time per unit time are obtained.

[0040] Further, the extracted different features are input into a trained graph regularized extreme learning machine classification model to output a recognition result; including:

[0041] Performing a Concat operation on the electrooculogram signals and image information in the extracted different features to obtain a multi-feature fusion result;

[0042] The multi-feature fusion result is input into the trained graph regularized extreme learning machine classification model, and the recognition result is output.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] A fatigue driving monitoring method based on graph regularized extreme learning machine includes: collecting the driver's electrooculogram signal and facial image information during driving by an electrooculogram signal acquisition device and a camera respectively; preprocessing the electrooculogram signal and facial image information respectively; extracting eye movement features from the preprocessed facial image information, classifying the preprocessed electrooculogram signal into three types of eye movement signals and extracting relevant features; determining that the driver is fatigue driving and giving an alarm when the eye closing time in the eye movement feature exceeds a preset threshold; calculating the average value of the same features extracted from the electrooculogram signal and the facial image information when the eye closing time in the eye movement feature does not exceed the preset threshold; inputting the average value and the extracted different features into a trained graph regularized extreme learning machine classification model, and discriminating the fatigue state by a GELM classifier. The monitoring method is more objective and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a method for monitoring driving fatigue based on a graph regularized extreme learning machine provided in an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of fatigue driving monitoring based on a graph regularized extreme learning machine provided in an embodiment of the present invention;

[0047] Figure 3EOG signal processing flow chart provided by an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of peak and trough detection results and calculated differences provided in an embodiment of the present invention;

[0049] Figure 5a A schematic diagram of eye movement signal separation results in the saccade phase provided by an embodiment of the present invention;

[0050] Figure 5b A schematic diagram of the eye movement signal separation result during the blinking phase provided by an embodiment of the present invention;

[0051] Figure 5c A schematic diagram of eye movement signal separation results in the fixation phase provided by an embodiment of the present invention;

[0052] Figure 6 This is a flowchart of image information processing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0054] In the description of the present invention, it should be noted that the terms "upper", "lower", "inner", "outer", "front end", "rear end", "two ends", "one end", "the other end" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0055] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "provided with", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0056] The fatigue driving monitoring method based on graph regularized extreme learning machine provided by the present invention adopts the monitoring realized by fusing the electrooculogram signal with the visual image information. The electrooculogram signal involved contains rich eyelid movement feature information in the electrooculogram, which can usually be extracted from the video signal, and different types of eye movement features can also be extracted from the electrooculogram, including rapid eye movement, slow eye movement, and eyelid movement. These eye movements are all related to human sleep or fatigue and are easy to be obtained from the video signal. The present invention also uses them as fatigue detection indicators. The EOG signal is easier to monitor and has a higher signal-to-noise ratio, so it is more suitable for large-scale implementation than the EEG signal. Slow eye movement has been proven to be a reliable indicator of the onset of sleep and has been studied in many different types of sleep-related studies. Blinking, rapid scanning and gazing are the main ways of eye movement when humans are awake.

[0057] The present invention provides a fatigue driving monitoring method based on a graph regularized extreme learning machine. The graph regularized extreme learning machine (GELM) is used to monitor fatigue in mixed electrooculogram signals and image information, so as to achieve higher-precision fatigue driving monitoring. Compared with a single electrooculogram feature, the GELM model can achieve better performance in the feature classification of the fusion electrooculogram signal and image information.

[0058] Reference Figure 1 As shown, the method includes:

[0059] S10, collecting eye contact signals and facial image information of the driver during driving by using an eye contact signal collection device and a camera respectively;

[0060] S20, pre-processing the electrooculogram signal and the facial image information respectively;

[0061] S30, extracting eye movement features from the preprocessed facial image information, and classifying the preprocessed electrooculogram signals into three types and extracting relevant features;

[0062] S40, when the eye closing time in the eye movement feature exceeds a preset threshold, determining that the driver is driving fatigued and issuing an alarm;

[0063] S50. When the eye closure duration in the eye movement feature does not exceed a preset threshold, the average value of the same features extracted from the electrooculogram signal and the facial image information is calculated; the average value and the extracted different features are input into the trained graph regularized extreme learning machine classification model, and the fatigue state is determined by the GELM classifier.

[0064] The fatigue driving monitoring principle of this embodiment based on EOG and visual information is as follows: Figure 2 As shown. The experiment collects EOG signal and image information data in real time. The collected electrooculogram signal and image information are first preprocessed, and then the image information feature extraction and EOG signal classification are performed. Closing the eyes for more than two seconds is considered a dangerous driving behavior. The t in the image feature is used to represent the length of time the eyes are closed. Driving is a very important behavior. When dangerous driving behavior occurs, that is, t is greater than, for example, 2s, it will be immediately determined as fatigue driving and an alarm will be issued. Under normal circumstances, the length of time a person closes his eyes will not reach 2 seconds. After extracting the features of the two signals, the average value of the signals that can be extracted from the image and the electrooculogram signal, such as the blinking frequency, is used to obtain the value. Then, the different feature quantities of the two signals are input into the GELM classifier to realize the entire process of fatigue driving monitoring.

[0065] The following is a detailed description of each of the above steps:

[0066] In step S10, the electrooculogram signal and the driver's facial image information are collected, wherein the electrooculogram signal is collected by the collection device Emotive, and the electrodes are arranged on the human head to avoid the disadvantages of facial electrodes. The facial image information is captured by the high-definition industrial camera MER-132-30UC. Taking the driving time of 2 hours as an example, data is collected every 15 minutes, and each collection time is 3 minutes. Since the amplitude range of the eye movement signal is between 0.4 and 10mv, and the frequency is between 0 and 30Hz. According to the sampling law, selecting a sampling frequency of 128Hz can meet the requirements, the scanning speed is selected as 400ms / div, the sensitivity is 1mv, the time constant is 0.2s, and the filtering frequency is 100Hz.

[0067] In one embodiment, in step S20, preprocessing the electrooculogram signal includes:

[0068] S201, denoising and reconstructing the electrooculogram signal using a wavelet function;

[0069] S202, performing normalization processing on the reconstructed electrooculogram signal;

[0070] S203 , using a window function to perform secondary effective extraction on the normalized data to obtain a local minimum value, a local maximum value, and their corresponding positions and differences.

[0071] And in step S30, the pre-processed electrooculogram signal is classified into EOG signals, including:

[0072] The KNN algorithm is used to divide eye movement signals into three stages according to their different characteristics: blink stage, fixation stage, and saccade stage.

[0073] According to the difference, relevant characteristic data of three stages of blinking are extracted, including: blinking duration, eye closing duration, eye opening duration, eye closing speed, eye opening speed, maximum eye closing speed, minimum eye closing speed and total energy of the blinking process.

[0074] Specific reference Figure 3 As shown, the data processing flow of the collected EOG signal is shown. Since the problem studied by the present invention is used to realize the actual traffic safety assurance system, there is a lot of noise at its application site. The EOG signal reconstructed by wavelet denoising can ensure that the error of subsequent feature extraction is small. After that, the reconstructed signal after normalization processing can ignore the influence of different orders of magnitude on different feature input classifiers during subsequent feature fusion. Then, the signal characteristics can be studied in segments through signal windowing processing to avoid the excessive redundancy of the data volume of the signal acquisition sample points and affect the calculation speed. For the EOG signal after preprocessing, after peak and trough detection, it can be divided into three stages of eye movement signals according to the different characteristics of different eye movement signals: blinking stage, fixation stage, and scanning stage. By extracting different features of different eye movement signals, preparation for the next step of feature fusion is made.

[0075] Among them, wavelet analysis is a time-frequency analysis method for signals. It has the characteristics of multi-resolution analysis and can focus on any details of the signal for multi-resolution time-frequency analysis, which is superior to the Fourier analysis algorithm. The Emotive forehead signal acquisition device is equipped with 14 signal acquisition channels. The collected electrical signals include forehead electrooculogram signals and forehead electroencephalogram signals, as well as some other interference signals. First, the collected signals are preliminarily processed using a wavelet function with a scale of 3. The experimental acquisition frequency is 0-128Hz, while the frequency of the electrooculogram signal is within 0-60Hz. The high-frequency signal can be preliminarily filtered out through the decomposition process of the wavelet transform. Wavelet analysis decomposes the signal into two parts: approximation and details. The approximation part can be decomposed into the second layer of approximation and details, and this can be repeated. For an N-layer decomposition, there are N+1 ways to decompose the signal. The details of the wavelet packet analysis are the same as the approximation part, and can also be decomposed. For the N-layer decomposition, it produces 2 N There are many different ways. Baseline drift comes from many sources, such as interfering background signals or the influence of electrodes, which is a slow signal change superimposed on the electrooculogram signal. Baseline drift has little effect on rapid eye movements, but has a greater impact on other eye movements. The use of wavelet transform can effectively reduce the impact of baseline drift. Using the Daubechies wavelet series function to perform a three-layer decomposition of the electrooculogram signal by wavelet transform, signals of different frequency bands can be obtained. The A2 of the two-layer wavelet decomposition is the electrooculogram signal of 0 to 32Hz. The reconstructed decomposition coefficient gives an estimate of the baseline drift. Subtracting this estimate from each original signal can generate a correction signal to reduce the baseline drift.

[0076] Assume that the original signal is x(n) and the denoised signal is x′(n), then the signal-to-noise ratio is defined as:

[0077]

[0078] The root mean square error between the original signal and the denoised signal is defined as:

[0079]

[0080] In signal processing, the higher the signal-to-noise ratio of the research signal, the smaller the root mean square error between the denoised signal and the original signal. In this way, the denoised signal is more similar to the original signal. The EOG signal is non-repetitive and has a higher signal-to-noise ratio than the EEG signal. EOG has a baseline drift problem. Baseline drift is a slow signal change superimposed on the EOG signal, but it is mostly unrelated to eye movement. It has many sources, such as interfering background signals or electrode position errors. Baseline drift has little effect on rapid eye movements, but has a greater impact on other eye movements.

[0081] In this embodiment, an unbiased risk estimation threshold is used for automatic threshold selection. For signal s(i), the absolute value of each data point is taken, and then they are sorted from small to large. Then, the square of each point is taken as a replacement for the new signal data point. The specific method can be expressed as follows (3):

[0082] f(k)=(sort(|s|)) 2 , (k=0,1,···,N-1) (3)

[0083] sort(·) is the sorting function;

[0084] The square root of the kth data point of f(k) is the threshold λ of the point k ,

[0085] The risk generated by the threshold at this point is Rish(k):

[0086]

[0087] Where k represents the kth data point, k = 0, 1, ..., N-1; N is the total number of data points; i represents the index value of the k data points; j represents a point from i-1 to the kth point;

[0088] The threshold corresponding to the minimum risk point is taken as the automatic threshold to achieve secondary denoising.

[0089] The feature extraction and classification of eye movement information are discussed from three aspects:

[0090] 1) Feature extraction of eye movement information

[0091] In the fatigue driving monitoring technology based on electrooculogram signals, the work of studying eye movement signals is mainly to reflect the driver's fatigue status through the related features of eye movement signals. It mainly involves two aspects: one is to study the methods used in the process of extracting different eye movement signal features, and the other is to study the methods used in the classification of eye movement signals in eye movements. In the feature extraction of electrooculogram signals, the main method is to extract the related parameters of the electrooculogram signals in the time domain and frequency direction. Table 1 describes the classification of the original electrooculogram signals into three types of eye movement signals, namely, saccade movement signals, blink movement signals, and gaze movement signals after the preprocessing of the electrooculogram signals, as well as the eye movement features commonly used in fatigue driving monitoring of the three signals. Other eye movement features, such as pupil dilation, microsquint, vestibular ocular reflex or smooth pursuit movement, are rarely used in some applications related to EOG because they are difficult to measure.

[0092] Table 1 Common characteristic parameters related to electrooculogram

[0093]

[0094] After the signal is normalized, all signal amplitudes will be converted to the range (-1, 1). The purpose of this is to make the order of magnitude differences between dimensions not be considered, to prevent the risk of errors in the prediction process. The normalization process is as follows:

[0095] x k =(xx min ) / (x max -x min ) (5)

[0096] In the formula, x is the original signal, x k is the normalized signal, x min is the minimum value in the signal amplitude sequence; x max is the maximum value in the signal amplitude sequence.

[0097] 2) Extracting effective eye movement signals

[0098] Eye movement signals have the characteristics of non-stationary bioelectric signals, that is, from a holistic perspective, its various characteristics change with time and have time-varying characteristics. The signal is divided into several segments. From a local perspective, each segment of the signal is relatively stable, so it can be regarded as a quasi-steady-state process, namely "short-term analysis". The entire signal is divided into segments and the characteristics of each segment are studied. When the characteristics of each segment are studied, it is equivalent to completing the study of the signal. Each segment is called a "frame".

[0099] The signal can be framed by "windowing". From a mathematical point of view, a "window" is actually a mathematical function. A frame of signal is intercepted by a fixed-length mathematical function, and only the signal characteristics within the "window" are observed each time. The length of this window function is called the "frame length". When a frame of signal is observed, the window function is moved back a fixed distance to study the next part of the signal. The distance moved is called the "frame shift". The "frame length" is generally determined based on experience, and the "frame shift" is generally 1 / 2 or 1 / 3 of the frame length, that is, there will be overlapping parts between each frame, so that the transition between frames can be smooth. In order to maintain a high resolution and reduce spectrum leakage, the Hamming window is selected as the window function. The window function of the Hamming window is shown in the following formula:

[0100]

[0101] Assuming that the eye movement signal is t(n) and the window function is w(n), the signal after windowing is s(n)=t(n)×w(n), where M is the total number of signals and n is one of the data points from 1 to M. In the specific implementation, for example, the selected frame length is 15ms, the frame shift is 5ms, and the signal sampling frequency is 128Hz, that is, one frame contains 2 sampling points, and one sampling point is repeated between two frames. For the windowed signal S(n), use a three-point moving mean filter to process the one-dimensional signal, forward and then backward. Find the local minimum and local maximum and their corresponding positions, and the peak points and trough points satisfy:

[0102] The peak value P should satisfy:

[0103] Peak position L p Should meet:

[0104] The trough V should meet the following requirements:

[0105] Valley position L v Should meet:

[0106] Blink duration: T blink =L v (i+1)-L v (i)

[0107] Calculate the difference D between the nth peak and the nth trough:

[0108] D(k)=P(k)-V(k);k=1,2,···,m, (11)

[0109] In formulas (7)-(11), N is the total number of data, m is the total number of peaks, n is one of the data points from 1 to N; j represents the total number of troughs;

[0110] Use matlab to program the calculation process and calculate the value and difference D of each position of the peak and trough, such as Figure 4 The calculation results are shown in Figure 2. The solid circle represents the calculated peak P(n), the square represents the calculated trough V(n), and the difference D in different eye movement stages is also significantly different. Figure 4 It is shown that the method can accurately mark the peaks and troughs in the windowed signal as well as the relative position information.

[0111] Blinking includes two stages: closing the eyes and opening the eyes. Both stages are short, forming a fast jumping wedge. Saccades are different from blinking. There is only one rising edge or falling edge. Saccades involve movement in one direction. Different directions of eye movement will lead to differences in rising and falling edges. 100ms is a discriminative threshold to distinguish gaze from other eye activities, and its variation range is generally between 100ms and 200ms. Fatigue, eye injuries, medicine and disease can all affect the blinking speed. When adults are fatigued from blinking, their blinking frequency will unconsciously increase by 50ms. The blinking frequency of adults ranges from 6 to 30 times per minute. Individual activities will also affect the blinking interval. Reading or staring at a computer will reduce the blinking frequency. Closing the eyes for more than 1000ms is defined as microsleep. The typical characteristics of saccadic eye movement are an amplitude of 20 degrees and a duration of 10 to 100ms; fixation eye movement is the duration of two saccades, with an average of 100ms to 200ms, and the duration of blinking is between 100ms and 400ms. Traditional electrooculogram signals cannot effectively express eye closure for up to 2 seconds, which is often considered to be fixation eye movement. Since the current electrooculogram data set is text, the KNN algorithm can be used. The K-nearest neighbor algorithm processes each data point in the data set with unknown category attributes as follows:

[0112] Set five eigenvalues: the first eigenvalue is the peak position, the second eigenvalue is the peak value, the third eigenvalue is the trough position, the fourth eigenvalue is the trough value, and the fifth eigenvalue is the duration. p (i) and the trough position L v (i) Determine the duration of the eye movement signal and decompose the windowed signal into saccade, fixation and blink. The separation results of the three eye movement stages are as follows: Figure 5a-5c shown.

[0113] 3) Blink feature extraction

[0114] Differential signal: Calculate the differential signal of the vertical electrooculogram signal. The differential signal represents the instantaneous rate of change of the electrooculogram signal and can approximately represent the speed of the eyeball's up and down movement. The method for calculating the differential signal is as follows:

[0115]

[0116] Where D is the signal after differencing, V is the original signal before differencing, and R is the sampling rate in Hz.

[0117] Blink merging: Check each set of adjacent blinks to see if they should be merged. If the time interval between two adjacent blinks is less than a certain threshold T int , and the duration of at least one of the two blink processes is less than a certain threshold T short Then these two blink processes should be merged into one blink process.

[0118] Based on the above differences, the relevant characteristic data of the three stages of blinking are extracted, including: blinking duration, eye closure duration, eye opening duration, eye closure speed, eye opening speed, maximum eye closure speed, minimum eye closure speed and total energy of the blinking process.

[0119] In one embodiment, the above step S20, preprocessing the facial image information, includes:

[0120] According to the initial face frame position and the eye area positioning based on the image gradient, the image processing position is compressed to the target area and used as the input of the machine algorithm to identify and extract the eye image.

[0121] Among them, the machine algorithm performs global face detection on each input image, which will undoubtedly add a lot of redundant calculations, resulting in low detection efficiency. In theory, the input image can be shrunk into a suspicious area based on the initial face frame position, thereby reducing the input of the classifier and improving detection efficiency.

[0122] Gradient, as a vector representing the change in grayscale value between image pixels, plays a vital role in the entire field of image processing. At the edge of the image, due to the large change in the grayscale value of the image, the gradient amplitude is relatively large; in the area where the grayscale change is relatively gentle, the gradient amplitude is relatively small; the direction of the gradient is the direction in which the function grows fastest at this point. When there is a significant color difference between the target area and the background image, it is easy to separate the target area from the non-target area through threshold segmentation.

[0123] When the driver closes his eyes for a long time, the characteristics of the electrooculogram signal cannot be distinguished from the gaze stage. Therefore, the long-term eye closure cannot be separated from the gaze stage when distinguishing the signal characteristics, and the long-term eye closure behavior is considered to be a dangerous driving action. Compared with the fatigue state judgment mode using a single electrooculogram signal feature or a single image signal feature, in this embodiment, the fatigue state judgment mode is performed by fusing the image signal feature and the electrooculogram signal feature to avoid misjudgment and wrong judgment.

[0124] The specific image feature extraction is as follows:

[0125] The original eye image has a lot of noise, which will make the final binarization result image full of scattered small white blocks. In order to avoid this, the median filter is first used to process the original image to eliminate noise and make the image look smoother.

[0126] The maximum inter-class variance method (Otsu) uses the idea of ​​clustering to divide the image into two parts, background and target, according to the pixel value, so that the difference in pixel value between the two parts is the largest, and the difference in pixel value within each part is the smallest. When the grayscale values ​​of the pixels in the target area of ​​the image are unevenly distributed, the regional boundaries are not obvious, and the peaks and troughs of the image histogram are fused together, the method based on statistical learning may produce good results. First, select the grayscale 75 as the threshold, use the binarization processing with a fixed threshold, and observe the segmentation effect. Obtain two groups of pixels consisting of pixels with grayscale values ​​less than 75 and greater than or equal to 75, and make a Gaussian distribution of the two groups of pixel groups. By observing the normal distribution curve and multiple manual adjustments, for example, the appropriate threshold is finally selected as 31.

[0127] For this binary image, the length and width can be measured by the maximum number of pixels in the horizontal and vertical directions of the region. That is, when the boundary of the object is known, the length and width of its circumscribed rectangle are used to represent the length and width of the region. If 1 represents the pixel of the object and 0 represents the background pixel, then the horizontal pixel value of the eye image is defined as H, the vertical pixel value is S, and P i,j Represents the pixel value at (i, j). After the image is binarized, the value can only be 0 or 1. η i Indicates the percentage of pixels with a value of 1 in the i-th column. Just as a reference is needed to determine whether an object is in motion, a basic state is also needed as a reference when judging the state of the eyes. The reference state is denoted as R, and the maximum value of the percentage of pixels with a value of 1 in the image is taken as its value. Then it can be calculated as follows:

[0128]

[0129] After obtaining the value of R and completing the system parameter initialization, the same method can be used to calculate the eye state value G, and the state of the eye can be determined by judging the ratio of G to R. By comparing the image with eyes open, it can be clearly found that the G / R value when eyes are closed will approach a very small value. A threshold can be defined. When it is less than this threshold, it is considered to be in the closed eye state, and in other cases it is in the open eye state. One advantage of using this value to define the eye state is that the closure rate of the eyes can be easily obtained, whether it is slightly closed, half closed or open. This provides more reference data for subsequent work and different fatigue judgment accuracies can be set.

[0130] Blinking refers to the process of eyes opening, closing and then opening again. Under normal conditions, a person blinks about 16 times in one minute, an average of once every 4 to 5 seconds, and each blink lasts about 0.2 seconds. If the driver does not blink for a long time, it may be because the driver is tired, the eyes are no longer focused, and the driver is distracted. Although the eyes are open, the eyes are dull, the thinking activity is reduced, and the neuron reflex arc will also be longer. By counting the number of blinks of the driver over a period of time, the appropriate threshold is found, and it is used to determine whether the driver is in a state of fatigue.

[0131] Normal blinking takes about 0.2 seconds. When a driver is fatigued, his physiological functions will decline and his reaction to external stimuli will slow down, which will manifest in the blinking of the eyes becoming slow and weak, and lasting longer than 0.2 seconds. Therefore, monitoring the number of long blinks over a period of time can be used as a relatively objective criterion for determining eye fatigue.

[0132] When people are tired, the movement of eyes from opening to closing and then opening again will slow down. The time taken for this process is closely related to the specific degree of fatigue. The more tired a person is, the longer the process will last. Therefore, the fatigue state of the driver can be reflected by measuring the length of time the driver closes his eyes per unit time. The proportion of time that the eyes are closed to a certain degree per unit time. When this proportion is greater than a certain threshold, the driver is considered to be fatigued. That is, if the eyelid covers more than 70% of the pupil area, the eyes are recorded as closed, and the total proportion of eye closure per unit time is counted.

[0133] In one embodiment, local consistency attributes can be used as auxiliary information to improve the performance of the learning model. Based on local consistency, the regularized enhanced model is to construct a nearest neighbor graph based on some distance measurements. The feature parameters of the two information sources are formed into new parameters through the concat fusion strategy. After these new feature parameters are input into the GELM classifier, the classifier will calculate the distance information between the features of the data to be monitored and the features of the standard sample, and calculate the classification results to divide them into two states: fatigue and wakefulness. When the eye closure duration reaches 2s, the driver is in a state of severe fatigue. At this time, the system automatically alarms to avoid the safety problems caused by the misjudgment of the electrooculogram signal during long-term eye closure (the electrooculogram signal will confuse the electrooculogram signal of long-term eye closure with the electrooculogram signal of the gaze stage).

[0134] The algorithm motivation of GELM lies in the following two aspects:

[0135] (1) The local consistency property can be used as auxiliary information to improve the performance of the learning model. Recently, various researchers have considered the case when the data is extracted from a sampling of a probability distribution that has support on or near a submanifold of the environment space. The local consistency assumption generally means that nearby points should share similar properties, which emphasizes the importance of local geometric structure in the dataset. Based on local consistency, regularized enhanced models are proposed by constructing nearest neighbor graphs based on some distance measures, including local consistency Gaussian mixture models, graph regularized non-negative matrix factorization (GNMF), and graph regularized sparse coding. Therefore, GELM can be regarded as a manifold learning that aims to preserve local structure during feature learning or classification. In other words, local consistency can make the learned mapping function vary smoothly along the geodesic of the data manifold.

[0136] (2) In traditional ELM, the distance information between samples is destroyed by nonlinear mapping. The activation functions used in neural networks are usually nonlinear, such as the "sigmoid" function and the "gaussian" function. Nonlinear mapping enhances the feature extraction performance of neural networks while destroying the local consistency contained in the dataset. Therefore, the nearest neighbor graph is constructed based on the local consistency in the unsupervised samples based on the distance information.

[0137] When predicting fatigue, the classifier is trained based on the features of the standard fatigue and non-fatigue experimental data of the training sample, and then the distance between it and the feature quantity of the test sample is used to construct the nearest neighbor graph. The experimental data of fatigue driving has consistency in local features, and the same is true for non-fatigue features, so fatigue judgment is made.

[0138] Data fusion is to reasonably process the observation information of various sensors, and complement the information collected by various sensors in space and time according to some optimization criteria or combination algorithms to obtain the key information required for decision-making tasks, thereby improving the robustness and accuracy of the model. The more typical multi-sensor data fusion methods in deep learning include Addition, Average Mean, Concat, Ensemble and Mixture Experts. This embodiment takes the two information sources of electrooculogram signals and image information as examples to briefly explain the above five data fusion methods. i and m j represents two different perceptual modalities, f l is the fusion feature map of the lth layer in the neural network, f l mi and f l mj Respectively represent m i and m j In the neural network, the feature map of layer l, Gl (·) represents the feature transformation function applied in the lth layer of the neural network.

[0139] Concat: Before feature transformation, feature vectors are stacked in two dimensions: element level and depth level. This method is generally used in fully connected layers. Features are usually expanded into vectors, and the elements of the vectors are concatenated along the row direction of the feature vector. Its mathematical description is:

[0140]

[0141] In order to improve the accuracy and robustness of the fatigue driving detection model, this embodiment attempts to fuse the information in the driver's electrooculogram data and the driver's facial video data. Considering the structural differences between the devices used in data collection and the differences in the data formats output by the devices, the front fusion based on the data layer is difficult to achieve. This embodiment comprehensively considers the requirements of data fusion and the limitations of model complexity, and adopts an intermediate fusion method to fuse the driver's facial video data and the driver's electrooculogram data, that is, extracting electrooculogram features from the driver's electrooculogram data and video data and fusing them with facial features. The driver's facial feature vector extracted from the driver's facial image is denoted as V m , the electrooculogram feature vector obtained from the electrooculogram data is recorded as V y , respectively, by using the Concat operation to fuse the two feature vectors, and finally obtain the driver fatigue feature vector V p , the expression is:

[0142] V p =[V m , V y ] (15)

[0143] In this section, the performance of the GELM model for monitoring driving fatigue by fusion features of EOG signals and facial image information is evaluated. For all the following processing, the activation function of the hidden layer is the "Sigmoid" function. The fatigue driving monitoring task is treated as a multi-class classification problem. Face recognition experiments on GELM are carried out from two aspects: (1) comparing the single EOG feature recognition of fatigue driving monitoring with the GELM driving fatigue monitoring with fusion features; (2) comparing the performance of the fatigue driving monitoring system in the GELM model by downloading simple fusion features and the Concat fusion strategy.

[0144] We will first explore the accuracy of a single EOG feature in monitoring fatigue driving, then directly concatenate the EOG feature vector and the image feature vector into a larger feature vector, which is then used as input to train the detection model. We will then perform feature fusion based on the Concat fusion strategy, and finally discuss the analysis results.

[0145] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A fatigue driving monitoring method based on graph regularized extreme learning machine, characterized in that: include: S10, collecting eye contact signals and facial image information of the driver during driving by using an eye contact signal collection device and a camera respectively; S20, pre-processing the electrooculogram signal and the facial image information respectively; S30, extracting eye movement features from the preprocessed facial image information, and classifying the preprocessed electrooculogram signals into three types and extracting relevant features; S40, when the eye closing time in the eye movement feature exceeds a preset threshold, determining that the driver is driving fatigued and issuing an alarm; S50, when the eye closing duration in the eye movement feature does not exceed a preset threshold, calculating an average value of the same features extracted from the electrooculogram signal and the facial image information; The average value and the different features extracted are input into the trained graph regularized extreme learning machine classification model, and the fatigue state is discriminated by the GELM classifier.

2. The method for monitoring driving fatigue based on graph regularized extreme learning machine according to claim 1, characterized in that: In step S10, the driver's electrooculogram signal is collected once during driving at a preset time interval; And the parameters of the electrooculogram signal acquisition device are determined according to the amplitude and frequency of the eye movement signal.

3. The method for monitoring driving fatigue based on graph regularized extreme learning machine according to claim 1, characterized in that: In step S20, the electrooculogram signal is preprocessed, including: S201, denoising and reconstructing the electrooculogram signal using a wavelet function; S202, performing normalization processing on the reconstructed electrooculogram signal; S203 , using a window function to perform secondary effective extraction on the normalized data to obtain a local minimum value, a local maximum value, and their corresponding positions and differences.

4. The method for monitoring driving fatigue based on graph regularized extreme learning machine according to claim 3, characterized in that: After step S201 and before step S202, the method further includes: Unbiased risk estimation threshold is used for automatic threshold selection to achieve secondary denoising.

5. The method for monitoring driving fatigue based on graph regularized extreme learning machine according to claim 4, characterized in that: An unbiased risk estimation threshold is used for automatic threshold selection to achieve secondary denoising, including: For the signal s(n), take the absolute value of each data point, sort them from small to large, and take the square of each point as a replacement for the new signal data point. The formula is as follows: f(k)=(sort(|s|)) 2 , (k=0, 1, ···, N-1); sort(·) is the sorting function; The square root of the kth data point of f(k) is the threshold λ of the point k , The risk generated by the threshold at this point is Rish(k): Where N is the total number of data points; i represents the index value of k data points; j represents the index value of a data point from i-1 to the kth data point; The threshold corresponding to the minimum risk point is taken as the automatic threshold to achieve secondary denoising.

6. The method for monitoring driving fatigue based on graph regularized extreme learning machine according to claim 1, characterized in that: In step S20, the facial image information is preprocessed, including: According to the initial face frame position and the eye area positioning based on the image gradient, the image processing position is compressed to the target area and used as the input of the machine algorithm to identify and extract the eye image.

7. The method for monitoring driving fatigue based on graph regularized extreme learning machine according to claim 3, characterized in that: In step S30, the pre-processed electrooculogram signal is classified into three types of eye movement signals and relevant features are extracted, including: The KNN algorithm is used to divide eye movement signals into three stages according to their different characteristics: blink stage, fixation stage, and saccade stage. According to the difference, relevant characteristic data of three stages of blinking are extracted, including: blinking duration, eye closing duration, eye opening duration, eye closing speed, eye opening speed, maximum eye closing speed, minimum eye closing speed and total energy of the blinking process.

8. The method for monitoring driving fatigue based on graph regularized extreme learning machine according to claim 4, characterized in that: In step S30, extracting eye movement features from the pre-processed facial image information includes: Using a median filter to perform denoising on the pre-processed facial image information; The maximum inter-class variance method is used to extract the eye area image and perform binarization processing; Based on the binary eye area image, the blinking frequency per unit time, the blinking action time, and the proportion of eye closure time per unit time are obtained.

9. The method for monitoring driving fatigue based on graph regularized extreme learning machine according to claim 1, characterized in that: The average value and the different features extracted are input into the trained graph regularized extreme learning machine classification model, and the fatigue state is discriminated by the GELM classifier, including: Concat the average value with the electrooculogram signal and image information extracted from different features to obtain a multi-feature fusion result; The multi-feature fusion result is input into the trained graph regularized extreme learning machine classification model, and the recognition result is output.

Images