A system and method for detecting fatigue based on micro-saccade characteristics of eye movement multi-feature fusion

Abstract

The present invention provides an eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics. The eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics uses a high-speed camera based on a neural network to detect and identify human faces, accurately mark multiple key points of the face, and accurately extract the position of the eyes, thereby achieving real-time tracking of the driver's eye state, and obtaining and analyzing eye features to determine the fatigue state. The present invention uses multiple eye movement indicators to comprehensively evaluate the degree of driving fatigue, avoiding misjudgment caused by a single indicator; the comprehensive detection of the driver's fatigue state through multiple eye movement indicators and the alarm for fatigue driving are particularly important for traffic safety.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to an eye movement multi-feature fusion fatigue detection system and detection method based on micro-saccade characteristics. Background Art

[0002] Fatigue driving generally refers to the physiological and psychological dysfunction of the driver during driving, which objectively leads to a decline in driving skills. In modern life, people have increased the frequency of fatigue driving due to factors such as poor sleep quality, lack of sleep, and mental stress. If the driver continues to drive the vehicle after the fatigue state occurs, he will feel sleepy, drowsy, weak limbs, inattentive, and have reduced judgment ability. He may even become mentally confused or have momentary memory loss, and show unsafe factors such as slow or no reaction to traffic changes such as red lights, which is very likely to cause road traffic accidents and seriously endanger road traffic safety. Therefore, drivers need to detect the occurrence of fatigue driving status as early as possible and avoid fatigue driving.

[0003] In order to detect possible driver fatigue during driving, there are some methods for detecting the driver's driving status in the prior art, among which there are methods for capturing the driver's facial information by installing a camera in the car, and then extracting facial features such as the driver's eyes closing and yawning to determine whether the driver is fatigued. However, in the above existing methods, when the driver's eyes closing and other states are detected in the collected driver's facial information, it indicates that the driver has entered a deep fatigue state. At this time, the fatigue prompt may not be received and feedback by the driver. In addition, different drivers may not show abnormal expressions on the driver's face when they are in a fatigue driving state, and at this time the driver's driving reaction ability has been significantly reduced. Therefore, it is very necessary to develop a detection method that is not based on the driver's facial information, but based on the more accurate driver's reaction ability to determine whether the driver is fatigued.

[0004] In addition, the eye fatigue monitoring used in the laboratory is mainly head-mounted eye trackers, that is, the current eye movement index detection method is mainly based on the relatively bulky head-mounted eye tracker method. The head-mounted eye tracker connects two computers via wires to display simple screen signals, and detects the eye movement data generated by the subject watching the screen signal to obtain some indicators such as saccades, fixations, positions, speeds, etc. However, the head-mounted eye trackers in the laboratory have problems such as expensive equipment, bulky and inconvenient to wear, and affecting driving.

[0005] Even if the head-mounted eye tracker used in existing laboratories is directly transplanted into the driving vehicle, since most of the current fatigue detection algorithms measure fatigue by the proportion of time the eyes are closed within a certain period of time, fatigue detection still has the following problems: (1) Single indicator judgment leads to low detection accuracy and robustness. (2) Drivers are easily affected by external factors such as light and wind, which causes the human eye to have a short-term unconventional state such as squinting and rapid blinking; (3) The head-mounted eye tracker does not combine the scientific basis of the relationship between eye characteristics and fatigue, and has a very high false detection rate. Summary of the invention

[0006] In order to solve the above problems, the inventors have designed and studied for many times and provided a fatigue detection system and method based on micro-saccade characteristics with multi-feature fusion of eye movement, which obtains eye movement features through computer vision methods and then determines the driver's fatigue state through relevant algorithms.

[0007] According to the first aspect of the technical solution of the present invention, a fatigue detection system for eye movement based on micro-saccade characteristics and multi-feature fusion is provided, wherein the micro-saccade indicates a rapid eye movement behavior of moving the foveal visual field from one point to another. The micro-saccade is a fixation-type eye movement with the largest amplitude and the fastest speed, and is an unconscious eye movement generated when the human eye tries to fixate. The fatigue detection system for eye movement based on micro-saccade characteristics uses a neural network-based algorithm to detect and recognize facial information obtained by a high-speed camera: first, multiple key points of the face are accurately marked, and the position of the eyes is accurately extracted, thereby realizing real-time tracking of the driver's eye status, and then a variety of features of the micro-saccades of the eyes are acquired and analyzed to judge the fatigue status.

[0008] Among them, the eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics accurately marks more than 40 key points of the face.

[0009] Preferably, the number of key points for accurately marking a face is 68 points.

[0010] Furthermore, the fatigue state is determined based on the acquisition and analysis of eye features, and the fatigue state information of the driver is accurately identified to monitor the fatigue state in real time and issue an alarm in time.

[0011] In addition, an eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics includes a pupil detection device, a saccadic gaze judgment device, a micro-saccade detection device and a multi-index fusion device, wherein the pupil detection device is used to locate the pupil center of the human eye in real time based on video images, the saccadic gaze judgment device is used to detect the gaze sequence in the pupil data, the micro-saccade detection device is used to detect the micro-saccade characteristics, and the multi-index fusion device is used to fuse multiple features of the eye and ultimately achieve accurate classification of the fatigue state.

[0012] Preferably, the pupil detection device is used to shoot facial images through a high-speed camera, the control device stores the trained facial key point parameters in a data information storage device, and the pupil detection device is embedded in a high-frequency camera.

[0013] Preferably, the saccadic fixation determination device uses a mobile eye tracking algorithm based on a speed threshold to screen the pupil detection data, screen out sequences that meet the fixation frame standard, and exclude eye movement phenomena caused by blinking and flickering artifacts.

[0014] According to a second aspect of the technical solution of the present invention, a method for detecting fatigue by integrating multiple features of eye movement based on micro-saccade characteristics is provided, which comprises the following steps:

[0015] Step S1, pupil detection step, realizing dynamic tracking of human eyes based on face and pupil center positioning data;

[0016] Step S2, scanning gaze judgment step, based on the speed threshold and the acceleration threshold, selects the gaze frame and obtains the scanning gaze judgment;

[0017] Step S3, a microsaccade detection step, which detects microsaccades based on an eye trajectory algorithm in a two-dimensional velocity space;

[0018] Step S4, a multi-index fusion step, extracting features that can characterize driver fatigue from the driver's facial video data and micro-saccade features respectively for feature fusion;

[0019] Step S5, fatigue judgment step, integrates the eye movement multi-indicators based on micro-saccade characteristics with the PERCLOS fatigue judgment principle, and uses the LSTM network to perform multi-feature fusion to detect and judge fatigue.

[0020] Preferably, step S1, pupil detection step specifically includes the following:

[0021] Step S11, the pupil detection step uses a high-speed camera based on neural network deep learning to locate multiple key points of the face;

[0022] Step S12, real-time positioning of the pupil center of the human eye based on the video image;

[0023] Step S13, determining the gaze direction of the human eye according to the position change of the key point, i.e., dynamic tracking of the human eye;

[0024] Step S14, grayscale processing is performed on the video images obtained in step S11, step S12 and step S13, and the video images are converted from the RGB color space to the grayscale space;

[0025] Step S15, performing Gaussian filtering on the grayscale image obtained in step S14 to remove image noise;

[0026] Step S16, performing binarization processing on the image obtained in step S15 to find the human eye area, and detecting the pupil contour, and then performing corrosion and expansion operations on the pupil contour;

[0027] Step S17, positioning the pupil center to achieve dynamic tracking of the human eye.

[0028] Preferably, step S3 is a microsaccade detection step, which detects microsaccades based on an eye trajectory algorithm in a two-dimensional velocity space, and converts the time series of eye positions into velocity:

[0029]

[0030] in represents the gaze position of each frame, where represents the vector distance between the two sampling points before and after, and Δt represents the sampling frequency of the video frame.

[0031] Compared with the existing technology, the eye movement multi-feature fusion fatigue detection system and detection method based on micro-saccade characteristics have the following technical advantages:

[0032] 1. The present invention uses multiple eye movement indicators to comprehensively evaluate the degree of driving fatigue, avoiding misjudgment caused by a single indicator; comprehensively detecting the driver's fatigue status through multiple eye movement indicators and issuing an alarm for fatigue driving is particularly important for traffic safety.

[0033] 2. The visual feature-based method of the present invention utilizes a deep learning-based camera to record the driver's eye movement data instead of sampling with an eye tracker and processes it with a related algorithm, thereby realizing real-time tracking of the driver's eye status. It has the advantages of directly detecting fatigue characteristics, non-contact convenience, and economic applicability.

[0034] 3. The present invention uses a high-speed camera based on a neural network (deep learning) to extract all information from the eyes, head and body, detects micro-saccades by applying a corresponding improved adaptive judgment algorithm, and judges fatigue by combining multiple eye features, thereby significantly improving the accuracy of the detected driver fatigue condition.

[0035] 4. Based on the correlation between the visual attention mechanism and the characteristics of micro-saccades, the present invention infers that micro-saccades can be used as eye movement features for fatigue detection, providing a scientific basis for verifying the fatigue state by eye features. The accuracy of the detected fatigue state is significantly improved, and it will become a good tool for fatigue detection in the transportation engineering industry in the near future. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of pupil detection based on micro-saccade characteristics and eye movement multi-feature fusion fatigue detection technology;

[0037] Figure 2 It is the first schematic diagram of the two-dimensional velocity of pupil movement, and the connecting part represents the microsaccade sequence;

[0038] Figure 3 It is the second schematic diagram of the two-dimensional position of pupil movement, and the connecting part represents the microsaccade sequence;

[0039] Figure 4 It is a schematic diagram of a process of fusing and classifying multiple features of visual information collected by a high-speed camera according to the present invention;

[0040] Figure 5 It is a schematic diagram of the process of classifying multi-feature fusion according to the LSTM long short-term memory network of the present invention;

[0041] Figure 6 is a schematic diagram of pupil horizontal position coordinates of a video frame in the present invention, wherein the darkened portion represents a microsaccade sequence;

[0042] Figure 7 is a schematic diagram of pupil vertical position coordinates of a video frame in the present invention, wherein the darkened portion represents a microsaccade sequence;

[0043] Figure 8 It is a schematic diagram of classification by LSTM long short-term memory network using multi-feature fusion in the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the scope of protection of the present invention should not be limited to the following specific structures or components or specific parameters.

[0045] The present invention provides a fatigue detection system and method for eye movement multi-feature fusion based on microsaccade characteristics. The applicant has found through research that driver fatigue is caused by long-term driving. The microsaccades on which the present invention is based indicate a rapid eye movement behavior that moves the foveal visual field from one point to another, and are not the eye saccade behavior commonly known as "blinking" in the prior art. The microsaccades indicated by the present invention are fixation-type eye movements with the largest amplitude and the fastest speed. They are involuntary eye movements generated when the human eye tries to fixate. The frequency of this rapid small-amplitude jump is about once to several times per second. Microsaccades can be combined with eye information to detect driving mental load and fatigue status. As time goes on, the microsaccade rate and variability will gradually increase without visual stimulation. The present invention uses multiple eye movement indicators to comprehensively evaluate the degree of driving fatigue, avoiding misjudgment caused by a single indicator. Therefore, it is particularly important for traffic safety to comprehensively detect the driver's fatigue status through multiple eye movement indicators and issue an alarm for fatigue driving.

[0046] The present invention provides an eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics, which uses a high-speed camera based on a neural network to detect and identify human faces, accurately mark multiple key points of the face, preferably more than 40 points, more preferably 68 key points, and accurately extract the position of the eyes, thereby achieving real-time tracking of the driver's eye state, and obtaining and analyzing eye features to determine the fatigue state. Further, the present invention provides an eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics, which determines the fatigue state based on the acquisition and analysis of eye features, accurately identifies the driver's fatigue state information, monitors the fatigue state in real time and issues an alarm in time, and reminds the driver to take corresponding rest and adjustments to ensure the driver's health and the safety of vehicle operation.

[0047] The present invention discloses a fatigue detection method based on micro-saccade characteristics and eye movement multi-feature fusion. It is based on the relationship between the fixation time, scanning time, micro-saccade frequency and peak speed of the human eye and the fatigue state. It adopts a micro-saccade evaluation method and an eye movement multi-feature fusion method to assist the PERCLOS judgment principle to realize the judgment of driver fatigue, thereby achieving higher accuracy and practicality. The PERCLOS judgment principle refers to the judgment of the physical quantity measuring fatigue / drowsiness, which refers to the percentage of time when the degree of eye closure exceeds a certain closure value (70%, 80%) in a unit time. For example, P7O: refers to the area of ​​the pupil covered by the eyelid exceeding 70%, which is considered as eye closure, and the proportion of time when the eye is closed within a certain period of time is counted; P80: refers to the area of ​​the pupil covered by the eyelid exceeding 80%, which is considered as eye closure, and the proportion of time when the eye is closed within a certain period of time is counted; EM: refers to the area of ​​the pupil covered by the eyelid exceeding half, which is considered as eye closure, and the proportion of time when the eye is closed within a certain period of time is counted. For example, set the threshold value

[0048]

[0049] In one embodiment, the present invention provides an eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics, which includes a pupil detection device, a saccadic gaze judgment device, a micro-saccade detection device and a multi-index fusion device, wherein the pupil detection device is used to locate the pupil center of the human eye in real time based on a video image, the saccadic gaze judgment device is used to detect the gaze sequence in the pupil data, the micro-saccade detection device is used to detect the micro-saccade characteristics, and the multi-index fusion device is used to fuse multiple features of the eye and ultimately achieve accurate classification of the fatigue state.

[0050] The pupil detection device uses a high-speed camera to capture facial images, and the control device stores the trained facial key point parameters in a data information storage device, and embeds the pupil detection device into a high-frequency camera. The data information storage device is an internal storage unit of a computer device, and can also be an external storage device of a computer device, which is used to store application software and various types of data installed in the computer device, and program codes for installing the computer device, etc. Preferably, the data information is used to store a driver fatigue detection computer program. The pupil detection device further includes a data processor and a display, and the data processor is used to run the program code stored in the memory or process data, such as executing a driver fatigue detection method, etc. The display is used to display information on the computer device and to display a visual user interface.

[0051] The saccadic fixation judgment device further screens the pupil detection data using a mobile eye tracking algorithm based on a speed threshold, selects sequences that meet the fixation frame standard, and excludes eye movement phenomena caused by human factors such as blinking and flickering.

[0052] The microsaccade detection device detects microsaccade events using a detection algorithm based on a speed amplitude threshold. Once a microsaccade is detected, the following features are extracted: eye movement indicators such as the number of microsaccades, microsaccade frequency, microsaccade speed, microsaccade angular peak speed, microsaccade angular amplitude, and microsaccade duration.

[0053] The multi-index fusion device uses the micro-saccade characteristics and the PERCLOS feature to determine the eye fatigue state, and uses LSTM and RNN to learn the fused multi-feature signals, while capturing the changing rules of the feature signals in the time series and the relationship between multiple feature signals at the same time, and finally achieves accurate classification of the fatigue state. If the driver is determined to be in a moderate fatigue state, a warning prompt is activated, and if the driver is determined to be in a severe fatigue state, an alarm prompt is activated.

[0054] All or part of the processes in the above-mentioned implementation device can be completed by instructing the relevant hardware through a computer program, can be embedded in the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and run the above modules.

[0055] A method for detecting fatigue by integrating multiple features of eye movement based on micro-saccade characteristics comprises the following steps:

[0056] Step S1, pupil detection step, realizes dynamic tracking of human eyes based on face and pupil center positioning data, specifically including the following steps:

[0057] Step S11, the pupil detection step uses a high-speed camera based on a neural network (deep learning) to locate 68 key points on the face.

[0058] Step S12, real-time positioning of the pupil center of the human eye based on the video image,

[0059] Step S13, determining the gaze direction of the human eye according to the position change of the key point, i.e., dynamic tracking of the human eye;

[0060] Step S14, grayscale processing is performed on the video images obtained in steps S11, S12 and S13, and the video images are converted from the RGB color space to the grayscale space.

[0061] Step S15, performing Gaussian filtering on the grayscale image obtained in step S14 to remove image noise.

[0062] Step S16, performing binarization processing on the image obtained in step S15 to find the human eye area, and detecting the pupil contour, and then performing corrosion and expansion operations on the pupil contour;

[0063] Step S17, positioning the pupil center to achieve dynamic tracking of the human eye.

[0064] Experimental results show that the pupil detection step can track the eye movement direction in real time.

[0065] Step S2, scanning gaze judgment step, selects the gaze frame and obtains the scanning gaze judgment based on the speed threshold and the acceleration threshold, which specifically includes the following steps:

[0066] Step S21, setting a speed threshold and an acceleration threshold;

[0067] Step S22, based on the face data in step S11 and the pupil center data in step S12, the speed and acceleration of each frame of data are calculated, and each point is classified as exceeding a certain speed threshold and acceleration threshold as a saccade.

[0068] This threshold is used to exclude most of the abnormal eye movements and provide purposeful eye movement data, depending on the task (i.e., lower threshold for static tasks).

[0069] Step S23, if the speed of the frame does not reach the speed threshold, it is classified as a fixation frame. For example, any glance duration exceeding 5 frames (100ms) will be discarded because no saccade will occur within this time threshold.

[0070] In step S24, based on the obtained skip frames and fixation frames, adjacent skip points and scan points are grouped together and then classified as one event using the distance between adjacent frames.

[0071] Step S3, microsaccade detection step, which detects microsaccades based on the eye trajectory algorithm in two-dimensional velocity space. First, the time series of eye positions is converted into velocity:

[0072]

[0073] in represents the gaze position of each frame, where represents the vector distance between the two sampling points before and after, and Δt represents the sampling frequency of the video frame.

[0074] represents a weighted moving average of five data samples to remove noise. Converting the trajectory to a two-dimensional velocity space results in a scattered arrangement of the data samples around the origin of the coordinate system. In this representation, microsaccades are "outliers" with velocities much higher than what would be expected assuming a normal distribution.

[0075] Next, calculate the standard deviation σ x，y, which means a standard speed of each frame point of the fixation eye movement curve, which is used to suppress the influence of excessive speed in microsaccades:

[0076]

[0077] Where V x，y Represents the horizontal and vertical velocities of the sampling point, and <> indicates the median estimation of the variable.

[0078] The microsaccade threshold is calculated for the sampling point speed variable. If the threshold is exceeded, it is a microsaccade. The formula is as follows:

[0079] η x，y =λσ x，y

[0080] Among them, λ represents a fixed multiple and is a constant; σ x，y Represents standard deviation.

[0081] The horizontal and vertical components are analyzed separately, and the corresponding threshold η is two values. As a necessary condition for microsaccades, we require all samples:

[0082]

[0083] Where v = (v k，x , v k，y ) represents the speed of a single frame sampling point, v k，x Represents the horizontal velocity of the sampling point, v k，y represents the vertical velocity of the sampling point. Accordingly, the corresponding threshold η is two values, η x Represents the horizontal threshold of the sampling point, η y Represents the vertical threshold of the sampling point, defining an ellipse in velocity space.

[0084] In a preferred embodiment, thresholds of 6 milliseconds for microsaccade duration, 0.6 degrees for amplitude, and 60 degrees per second for peak angular velocity are used to exclude outliers and reduce noise.

[0085] Step S4, a multi-index fusion step, extracts features that can characterize driver fatigue from the driver's facial video data and micro-saccade features for feature fusion.

[0086] In the multi-index fusion step, considering that the generation of driver fatigue is a long-term dynamic physiological change process, an LSTM network is designed to process and fuse 5-dimensional feature data, namely, eye opening features, microsaccade frequency features, microsaccade speed features, microsaccade peak speed features and pupil fatigue rate coefficient features. Through the long and short-term memory function of the network, the timing characteristics and contextual information of the feature sequence are fully utilized to realize the dynamic simulation of the driver fatigue generation process, thereby realizing the classification of the driver's fatigue status.

[0087] Step S5, fatigue judgment step, integrates the eye movement multi-indicators based on micro-saccade characteristics with the PERCLOS fatigue judgment principle, and uses the LSTM network to perform multi-feature fusion to detect and judge fatigue.

[0088] Compared with the prior art, the technology of the present invention has the following technical advantages:

[0089] 1. Replacing eye trackers with high-speed cameras has the advantages of directly detecting fatigue characteristics and being non-contact. High-speed cameras based on deep learning directly learn visual features from images, have better robustness to changes in conditions such as lighting and posture, and have higher accuracy, making great progress in fatigue detection algorithms and more practical.

[0090] 2. The multi-indicator fusion fatigue detection of driver fatigue using micro-saccade characteristics is consistent across populations. The effectiveness of human eye characteristics in detecting driving fatigue makes it possible to provide a reliable scientific basis for the formulation of driver fatigue detection standards and the evaluation of workload, which has important practical significance for ensuring traffic safety.

[0091] In another embodiment, a fatigue detection method based on micro-saccade characteristics of eye movement multi-feature fusion is used. The eye movement multi-indicators based on micro-saccade characteristics are fused with the PERCLOS fatigue judgment principle, and the LSTM network is used to perform multi-feature fusion to detect fatigue. The specific method is as follows:

[0092] Step W1, use a high-speed camera for sampling. The high-speed camera is preferably a high-speed camera trained by deep learning. Through image enhancement, object detection technology yolov5 is selected. Based on Anchor classification and regression, 68 key points of the face are trained and learned. In this step, the training detection speed is faster and the detection performance is better. An algorithm that balances detection performance and speed is adopted.

[0093] Step W2, using pupil detection technology, obtains the pupil position and pupil size of the human eye, calculates the position change (unit pixel) of the x and y coordinates of the pupil in each frame of the original data, calculates the distance, speed and acceleration between them, and after calculating the speed and acceleration of each frame in the original data, the algorithm classifies each point as exceeding a certain speed threshold as a saccade. The threshold is obtained through a large number of experiments with EOG (electro-oculogram marking method) for eye tracking, and this threshold is used to exclude most abnormal eye movements and provide useful eye movement data. If the speed of the frame does not reach the speed threshold, it is classified as a fixation.

[0094] Step W3, using microsaccade detection technology, through the obtained gaze frames, calculate the changes in microsaccade frequency (the number of microsaccades in a unit frame), microsaccade speed (the average speed between sampling points), saccade peak speed (the maximum speed in microsaccade eye movement), microsaccade amplitude (the spatial distance from the initial to the end position of the microsaccade movement), and pupil fatigue rate (the ratio of the mean pupil diameter to the standard deviation of the pupil diameter) of a person when awake and tired.

[0095] When the tester is in a fatigue state, the microsaccade frequency, microsaccade speed, microsaccade peak speed, and microsaccade amplitude value of the tester will be significantly reduced, and before the reduction, the pupil diameter of the tester decreases rapidly, and the pupil fatigue rate gradually decreases, and this trend is not shown when awake. In this step, the four indicators of microsaccade frequency, microsaccade speed, microsaccade peak speed and pupil fatigue rate coefficient are introduced to comprehensively evaluate the driver's fatigue degree. It is found that as time goes by, the drift speed decreases and the microsaccade speed decreases, indicating that mental fatigue causes the instability of the eyeball to increase. Drift and microsaccade can be used to predict the activity state of the nervous system, and are very potential mental fatigue monitoring indicators. Before the driver enters moderate fatigue, the frequency, speed and peak speed of his microsaccades have a downward trend, and after entering severe fatigue, the amplitude of his variation increases significantly, indicating that before the driver enters moderate fatigue, his microsaccade frequency, speed, peak value and pupil diameter are relatively stable, while during severe fatigue and extreme fatigue, the amplitude of his variation fluctuation is large.

[0096] Step W4, the eye movement data obtained by the above calculation, such as microsaccade frequency, microsaccade speed, microsaccade peak speed and pupil fatigue rate coefficient and other related indicators are combined with the PERCLOS fatigue determination principle to comprehensively judge the fatigue state. In order to improve the accuracy and robustness of the fatigue driving detection model, the present invention fuses the information in the microsaccade data and the driver's facial video data. The facial features determine fatigue in the deep fatigue stage, and the characteristics of small feature changes and insignificant differences, and "significant feature changes in the microsaccade feature, significant feature differences", which shows that it is indeed necessary to fuse these two features, and use the microsaccade feature to improve the defects of small fatigue changes in the blink feature and difficult detection of distraction, so as to improve the accuracy and reliability of subsequent fatigue state recognition. The driver's facial features, 1D eye opening features, 2D pupil diameter and fatigue rate features were extracted from the driver's facial information collected by the camera. The 4D features of microsaccades, namely microsaccade frequency, microsaccade horizontal speed, microsaccade vertical speed and microsaccade peak speed, were extracted from the fixation frame data. According to the principle of feature layer information fusion, information fusion was carried out on the feature layer to obtain a total of 7 dimensions of driver's facial and microsaccade fusion features. Considering the structural differences between the camera and the eye tracker used in data collection and the differences between the output data formats, the consistency requirements of the fused data format require that the data measured in pixels be converted into degrees, and the driver's facial video data and microsaccade data are fused using an intermediate fusion method, that is, the corresponding feature indicators of microsaccades calculated from the eye movement data are fused with the facial features in the driver's facial video data.

[0097] The present invention will be further explained and illustrated below in conjunction with the accompanying drawings.

[0098] like Figure 1 As shown, it shows the pupil detection process in the eye movement multi-feature fusion fatigue detection technology based on micro-saccade characteristics, which uses a high-speed camera to sample the face, obtains the face frame image, obtains the data of 68 key points of the face based on the face frame image, and then obtains the pupil center coordinates based on the pupil contour detection to realize pupil positioning.

[0099] like Figure 2 and Figure 3 As shown, Figure 2 The ordinate is the vertical rate (degrees of viewing angle per second), and the abscissa is the horizontal rate (degrees of viewing angle per second). Figure 2 The first schematic diagram of the two-dimensional velocity of pupil movement, the connecting part represents the microsaccade sequence; Figure 3 The ordinate is the vertical speed (degrees of viewing angle), and the abscissa is the horizontal speed (degrees of viewing angle). Figure 3 Second schematic diagram showing the 2D position of pupil movement, where the lines represent microsaccade sequences.

[0100] Figure 4 It is a schematic diagram of the process of multi-feature fusion and classification of visual information collected by high-speed cameras;

[0101] Figure 5 This is a schematic diagram of the process of LSTM long short-term memory network classifying multi-feature fusion;

[0102] Figure 6 It is a schematic diagram of pupil horizontal position coordinates of video frames, and the darkened part represents the microsaccade sequence;

[0103] Figure 7 It is a schematic diagram of pupil vertical position coordinates of video frames, and the darkened part represents the microsaccade sequence;

[0104] Figure 8 This is a schematic diagram of multi-feature fusion classification through LSTM long short-term memory network.

[0105] The above are only preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. It can be understood by a person of ordinary skill in the art that various modifications can be made in form and details without departing from the spirit and scope of the present invention defined in the appended claims.

Claims

1. A detection method for an eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics, characterized in that: Microsaccades indicate a rapid eye movement behavior that moves the foveal visual field from one point to another. Microsaccades are an unconscious eye movement that occurs when the human eye tries to focus. The eye movement multi-feature fusion fatigue detection system based on microsaccade characteristics is based on a deep learning algorithm, uses a high-speed camera for detection and recognition, accurately marks multiple key points of the face, and accurately extracts information about the eyes and pupils, realizes real-time tracking of the driver's eye state, obtains and fuses the eye movement multi-features of microsaccades, and then analyzes and judges the driver's fatigue state; the multiple key points of the face that are accurately marked are 68 points; The fatigue status is determined by acquiring and analyzing eye features, and the fatigue status of the driver is accurately identified to monitor the fatigue status in real time and issue an alarm in time; The eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics includes a pupil detection device, a saccadic gaze judgment device, a micro-saccade detection device and a multi-index fusion device, wherein the pupil detection device is used to locate the pupil center of the human eye in real time based on a video image, the saccadic gaze judgment device is used to detect the gaze sequence in the pupil data, the micro-saccade detection device is used to detect the micro-saccade characteristics, and the multi-index fusion device is used to fuse multiple features of the eye and finally realize accurate classification of fatigue status; The pupil detection device is used to shoot a face image through a high-speed camera, the control device stores the trained face key point parameters in a data information storage device, and the pupil detection device is embedded in the high-frequency camera; The saccadic fixation judgment device uses a mobile eye tracking algorithm based on a speed threshold to filter the pupil detection data, select the sequence that meets the fixation frame standard, and exclude the eye movement phenomenon caused by blinking and flickering artificial factors; The detection method of the eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics comprises the following steps: Step S1, pupil detection step, realizing dynamic tracking of human eyes based on face and pupil center positioning data; Step S2, scanning gaze judgment step, selecting the gaze frame and obtaining the scanning gaze judgment based on the speed threshold and the acceleration threshold; Step S3, a microsaccade detection step, which detects microsaccades based on an eye trajectory algorithm in a two-dimensional velocity space; Step S4, a multi-index fusion step, extracting features that can characterize driver fatigue from the driver's facial video data and micro-saccade features respectively for feature fusion; Step S5, fatigue judgment step, integrates the eye movement multi-indicators based on micro-saccade characteristics with the PERCLOS fatigue judgment principle, and uses the LSTM network to perform multi-feature fusion to detect and judge fatigue.

2. The detection method of the eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics according to claim 1 is characterized in that: Step S1, pupil detection step specifically includes the following: Step S11, the pupil detection step uses a high-speed camera based on neural network deep learning to locate multiple key points of the face; Step S12, real-time positioning of the pupil center of the human eye based on the video image; Step S13, determining the gaze direction of the human eye according to the position change of the key point, i.e., dynamic tracking of the human eye; Step S14, grayscale processing is performed on the video images obtained in step S11, step S12 and step S13, and the video images are converted from the RGB color space to the grayscale space; Step S15, performing Gaussian filtering on the grayscale image obtained in step S14 to remove image noise; Step S16, performing binarization processing on the image obtained in step S15 to find the human eye area, and detecting the pupil contour, and then performing corrosion and expansion operations on the pupil contour; Step S17, positioning the pupil center to achieve dynamic tracking of the human eye.

3. The detection method of the eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics according to claim 1 is characterized in that: Step S3 is a microsaccade detection step, which detects the time series of eye positions in microsaccades and converts them into speed based on the eye trajectory algorithm in the two-dimensional velocity space: in represents the gaze position of each frame, where represents the vector distance between the two sampling points before and after, and Δt represents the sampling frequency of the video frame.

4. The detection method of the eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics according to claim 1 is characterized in that: Step S2, the scanning and fixation judgment step further includes step S21, setting a speed threshold and an acceleration threshold.

5. The detection method of the eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics according to claim 4 is characterized in that: After step S21, the method further includes step S22, which calculates the speed and acceleration of each frame of data based on the face data in step S11 and the pupil center data in step S12, and classifies each point as exceeding a certain speed threshold and acceleration threshold as a saccade.

6. The detection method of the eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics according to claim 5, characterized in that: The method further comprises step S23 after step S22, if the speed of the frame does not reach the speed threshold, it is classified as a fixation frame.

7. The detection method of the eye movement multi-feature fusion fatigue detection system based on micro-saccade characteristics according to claim 6, characterized in that: The method further includes step S24 after step S23, grouping adjacent viewpoints and scan points together based on the obtained viewfinder frames and gaze frames, and then classifying them into one event using the distance between adjacent frames.

Images