Eye movement anomaly detection method based on smartphone video and deep learning
Through smartphone video and deep learning technology, the problem of traditional eye movement analysis methods and equipment is expensive and subjective, and portable and easy to use efficient eye movement abnormality detection is achieved.
Patent Information
- Application Number
- CN202411750880.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-02
- Publication Date
- 2025-03-18
AI Technical Summary
Traditional eye movement analysis methods rely on expensive professional equipment and complex operating procedures, limiting their wide application in clinical practice, and are highly subjective, making it difficult to accurately capture and record eye movement abnormalities.
Using smartphone video and deep learning technology, eye-moving video data is collected through smartphone cameras, video encoding and feature extraction is used using DataVyu and Mediapipe to build a multi-channel deep learning model for abnormal pattern recognition.
It realizes portable and easy-to-use eye movement abnormality detection, improves the accuracy and efficiency of detection, reduces hardware costs, and enhances the feasibility of clinical applications.
Smart Images
Figure CN119649438A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of intelligent medical detection, and specifically relates to an eye movement abnormality detection method based on smartphone video and deep learning. Background Art
[0002] Eye movement, as an important window into the function of the nervous system, can directly reflect the cognitive processing and neural control ability of the brain. Studies have shown that abnormalities in the basic functions of eye movement (such as saccades, pursuits, and fixations) are closely related to a variety of neurological diseases, making eye movement analysis a valuable means of evaluating nervous system function. Eye movement analysis records and analyzes the trajectory characteristics of eye movements, quantitatively calculates and interprets eye movement parameters, and provides an important basis for clinical diagnosis, disease monitoring, and rehabilitation assessment. Traditional eye movement analysis mainly relies on professional eye tracker equipment. Although the measurement is accurate, the equipment is expensive, the operation is complicated, and it requires professional personnel to operate, which limits its wide application in clinical practice. With the development of computer vision and deep learning technology, video-based eye movement analysis methods have gradually become an effective alternative. This method uses video to collect eye movements, combines computer vision algorithms to extract eye movement features, and then uses deep learning models to identify abnormal patterns. It has the advantages of portability, ease of use, and low cost, and provides a more popular solution for rapid screening and continuous monitoring of eye movement abnormalities.
[0003] The key to eye movement analysis is to detect abnormal eye movement events, such as micro-saccades during fixation, saccades during pursuit, and frequent interruptions of saccades. Traditionally, eye movement abnormalities have been quantitatively evaluated mainly by paper-and-pencil rating scales used by clinicians and subjective reports from patients and their families. However, this method is susceptible to measurement errors such as recall bias, reporting bias, and responsiveness bias, which reduces the objectivity and accuracy of the assessment. At the same time, since abnormal eye movements are often short-lived and occur randomly, it is difficult to accurately capture and record these subtle abnormal features by observation and description alone.
[0004] In recent years, with the rapid development of smartphone technology, the high-definition cameras on smartphones can now capture video data comparable to that of professional cameras. This provides a portable and easy-to-use solution for detecting eye movement abnormalities. Through smartphones, doctors and patients can record abnormalities in a timely manner when they occur, or perform standardized eye movement tests on a daily basis and record the entire process. This video-based recording method not only overcomes the subjective limitations of traditional evaluation methods, but also provides objective data support for subsequent quantitative analysis. Combined with computer vision technology, these video data can be converted into accurate eye movement trajectory data, providing a more reliable basis for the identification and evaluation of abnormal patterns. Summary of the invention
[0005] In order to solve the above problems, the present invention proposes: a method for detecting eye movement anomalies based on smartphone video and deep learning, comprising the following steps:
[0006] S1. Data collection: Videos of eye movements were taken using the built-in camera of a smartphone to capture the entire face;
[0007] S2, video behavior coding: use DataVyu coding to mark normal and abnormal eye movements, extract multiple 2-second eye movement event clips, and slow down frame by frame to ensure the accuracy of timestamps;
[0008] S3, feature extraction: Through the Mediapipe facial feature point detection model, the coordinates of the facial feature points in the video clip are extracted, the movement trajectory of the key points of the eyes is retained, and the initial features of the eye movement are obtained;
[0009] S4. Constructing a dataset: randomly split the eye movement event segments into training, validation, and test sets required by the deep learning model;
[0010] S5. Model building: Building a multi-channel deep learning model based on DilatedCNN, BiLSTM and Local-global Attention;
[0011] S6. Model evaluation: The effect of the model is evaluated by the accuracy ACC of the model on the test set and the area under the receiver operating characteristic ROC curve AUC score.
[0012] Furthermore, in step S1, the smartphone is fixed by a bracket at the upper rear position of the screen directly in front of the subject to ensure that the shooting field of view completely covers the subject's face without obstruction, and three eye movement task programs of fixation, saccade and smooth pursuit are presented in sequence on the screen directly in front, and the mobile phone camera is started to record the subject's eye movement response throughout the process and collect original eye movement video data; all eye movement tasks are based on a black background, and the stimulus is a white circular pattern with a diameter of 30 pixels.
[0013] Further, the procedures for the three eye movement tasks were as follows:
[0014] (1) Smooth horizontal tracking task: The white circle moves horizontally at a constant speed from the horizontal center of the screen. Each time it reaches the edge of the screen, it pauses for 1 second and then continues to move at a constant speed in the opposite direction. The task is divided into two stages, and the movement speed of the second stage is slightly faster than that of the first stage.
[0015] (2) Fixation task: The white circle stayed at one-quarter, center, and three-quarters of the horizontal center line of the screen from left to right for 10 seconds each;
[0016] (3) Reflexive scanning task: The white circle moves randomly on the screen, appearing for 1 second each time and disappearing for 0.5 seconds, that is, it stays for 1 second each time, disappears for 0.5 seconds, and then appears at another random position.
[0017] Furthermore, in step S2, the video data is systematically encoded using DataVyu behavior analysis software:
[0018] First, preliminary annotation was performed at 4x slow motion to identify abnormal eye movement behavior. When abnormal eye movement was found, the rewind function was used and the mode was switched to 8x slow motion to accurately mark the timestamp of the abnormality. The type and degree index of the abnormality were recorded in the coding area. Finally, the video was segmented into 2-second intervals to extract normal and abnormal eye movement event segments.
[0019] Furthermore, in step S3, the Mediapipe facial feature point detection model is deployed to process all 2-second video clips. Based on a sampling rate of 60 frames per second, 120 frames of images can be obtained for each clip. The facial three-dimensional coordinate information is extracted for each frame of the image, including spatial position data in the horizontal X, vertical Y and depth Z directions, and the regional features of both eyes are extracted. 21 key points are retained for each eye, including 16 points of the eyelid contour, 4 points of the iris edge and 1 point of the pupil center, to construct a complete eye movement feature sequence.
[0020] Furthermore, in step S4, a stratified random sampling strategy is used to construct a data set required for machine learning. The specific division scheme is: 10% of normal eye movement samples and an equal number of abnormal eye movement samples are randomly selected to form a test set; 5% of normal and abnormal samples are each selected to form a validation set; and the remaining samples constitute a training set. 7. The eye movement anomaly detection method based on smartphone video and deep learning as described in claim 6 is characterized in that in step S5, a model is established: a dual-channel architecture with a dilated convolutional network DilatedCNN as the backbone; on this basis, a bidirectional long short-term memory network BiLSTM is introduced after feature extraction at each scale, and at the same time, an independent BiLSTM channel is designed to directly process the original input, and a hierarchical attention mechanism is adopted, including a local attention mechanism for the output of each channel and a global attention mechanism for feature fusion.
[0021] Furthermore, the specific steps are as follows:
[0022] (1) At the input layer, the input data is standardized, the original input dimension is maintained, and the data format is adjusted through transposition operation to adapt to subsequent processing;
[0023] (2) In the backbone network, a multi-scale feature extraction strategy is adopted, which includes three parallel dilated convolution branches to capture local details, medium-range and large-scale context features respectively, forming a multi-scale feature representation;
[0024] (3) In the dual-channel time series modeling part, the three-scale convolution features are spliced in the CNN-LSTM channel to improve the computational efficiency. The spliced features are input into the BiLSTM to enhance the time series dependency modeling through the bidirectional LSTM. The direct LSTM channel directly feeds the original input into another BiLSTM to retain the complete time series information of the original sequence.
[0025] (4) In the hierarchical attention mechanism, the outputs of three parallel dilated convolutions after BiLSTM and the direct BiLSTM output of another channel are processed through local attention to highlight the key information at each scale; global attention fuses the local attention outputs and learns the importance weights of different feature representations.
[0026] Furthermore, in step S6, model evaluation: the trained deep learning model is verified on the reserved test set, and the accuracy ACC and the area under the receiver operating characteristic ROC curve AUC scores are used as evaluation indicators of the model.
[0027] The beneficial effects of the present invention are as follows: The advantages of the present invention are as follows:
[0028] (1) Data collection is efficient and convenient. The present invention uses the built-in camera of a smartphone to collect eye movement data, breaking through the limitations of traditional eye trackers and other professional equipment in terms of portability and accessibility. This solution not only reduces the hardware cost of data collection, but also improves the flexibility of recording eye movement data, greatly improving the feasibility of clinical applications.
[0029] (2) Behavior coding is precise and controllable. The present invention uses professional DataVyu behavior analysis software for video coding, which has significant technical advantages. The software supports millisecond-level time synchronization annotation and can slow down the video to 1 / 32 of the original speed, ensuring the accuracy of abnormal eye movement type determination and timestamp annotation. Through the unified 2-second standard segment segmentation method, a standardized data basis is provided for subsequent feature extraction.
[0030] (3) Feature extraction is efficient and accurate. The present invention achieves high-precision feature extraction through the MediaPipe facial feature point detection model. The model can accurately detect 478 three-dimensional facial feature points from each frame of the image, and obtain the three-dimensional coordinate information of 42 eye key points by screening, effectively constructing a complete eye movement trajectory sequence. This method not only ensures the accuracy of feature extraction, but also improves computational efficiency.
[0031] (4) The model architecture is innovative and efficient. The present invention designs a dual-channel deep learning architecture that innovatively combines the advantages of DilatedCNN and BiLSTM. By introducing local and global attention mechanisms, the model's ability to capture temporal features and extract key information is significantly enhanced. This multi-module collaborative architecture design effectively improves the accuracy of abnormal eye movement detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 It is a flow chart of the present invention;
[0033] Figure 2 Diagram of extracting eye key points after annotating faces for Mediapipe facial feature point detection model;
[0034] Figure 3 Schematic diagram of the deep learning model structure. DETAILED DESCRIPTION
[0035] Example 1
[0036] In order to make the technical means adopted by the present invention and the purpose achieved easy to understand, the present invention is further explained in combination with specific implementation methods below. A method for detecting eye movement abnormalities based on smartphone video and deep learning is used. The video data of human eye movement is captured by a smartphone camera, the original data is encoded, abnormal eye movement events are classified, the eye movement events are processed by computer vision technology and a data set is constructed, and a deep learning model is trained to detect abnormal eye movement events.
[0037] The main steps include:
[0038] S1. Data Collection: Videos of eye movements were captured using the built-in camera of a smartphone to capture the entire face.
[0039] S2. Video behavior coding: Use DataVyu coding to mark normal and abnormal eye movements, extract multiple 2-second eye movement event clips, and slow down frame by frame to ensure the accuracy of timestamps.
[0040] S3. Feature extraction: Through the Mediapipe facial feature point detection model, the coordinates of facial feature points in the video clip are extracted, with a focus on retaining the movement trajectory of the key points of the eyes, so as to obtain the initial features of the eye movement.
[0041] S4. Build a dataset: randomly split the eye movement event segments into the training, validation, and test sets required for the deep learning model.
[0042] S5. Model building: Establish a multi-channel deep learning model based on DilatedCNN, BiLSTM and Local-global Attention.
[0043] S6. Model evaluation: The effect of the model is evaluated by the accuracy (ACC) of the model on the test set and the area under the receiver operating characteristic (ROC) curve (AUC) score.
[0044] Example 2
[0045] The present invention will be further described in detail below with reference to the accompanying drawings.
[0046] A method for detecting eye movement anomalies based on smartphone video and deep learning is designed. The smartphone shooting mode is 1080P60 frames. Figure 1 As shown, the following steps are included:
[0047] 1. Data collection: A standardized data collection scheme was used, and the smartphone was fixed to the upper back position of the screen directly in front of the subject through a bracket to ensure that the shooting field of view completely covered the subject's face and was not blocked. During the experiment, three eye movement task programs, namely fixation, saccade, and smooth pursuit, were presented in sequence on the screen directly in front. At the same time, the mobile phone camera was started to record the subject's eye movement response throughout the process and collect the original eye movement video data. All eye movement tasks were based on a black background, and the stimulus was a white circular pattern with a diameter of 30 pixels.
[0048] (1) Smooth horizontal tracking task: The white circle moves horizontally at a constant speed from the horizontal center of the screen. Each time it reaches the edge of the screen, it pauses for 1 second and then continues to move at a constant speed in the opposite direction. The task is divided into two stages, and the movement speed of the second stage is slightly faster than that of the first stage.
[0049] (2) Fixation task: The white circle stayed at one-quarter, center, and three-quarters of the horizontal center line of the screen from left to right for 10 seconds each.
[0050] (3) Reflexive scanning task: The white circle moves randomly on the screen, appearing for 1 second each time and disappearing for 0.5 seconds, that is, it stays for 1 second each time, disappears for 0.5 seconds, and then appears at another random position.
[0051] 2. Video behavior coding: DataVyu professional behavior analysis software is used to systematically encode video data. First, play it at 4x slow speed for preliminary annotation to identify abnormal eye movement behaviors (such as eye saccades, rapid eye movements, etc.). When abnormal eye movements are found, use the rewind function and switch to 8x slow mode to accurately mark the timestamp of the abnormality, and record the abnormality type and degree index in the coding area. Finally, the video is segmented at 2-second intervals to extract normal and abnormal eye movement event segments respectively.
[0052] 3. Feature extraction: such as Figure 2As shown in the figure, the Mediapipe facial feature point detection model is deployed to process all 2-second video clips. Based on a sampling rate of 60 frames per second, 120 frames of images can be obtained for each clip. The facial 3D coordinate information is extracted for each frame, including spatial position data in the horizontal (X), vertical (Y) and depth (Z) directions. The features of the eye area are extracted in particular, and 21 key points are retained for each eye, including 16 points of the eyelid contour, 4 points of the iris edge and 1 point of the pupil center, to construct a complete eye movement feature sequence.
[0053] 4. Constructing the data set A stratified random sampling strategy is used to construct the data set required for machine learning. The specific division scheme is: 10% of normal eye movement samples and an equal number of abnormal eye movement samples are randomly selected to form the test set; 5% of normal and abnormal samples are each selected to form the validation set; the remaining samples constitute the training set. This division method ensures a balanced ratio of normal and abnormal samples in each data set, providing a reliable data basis for model training and evaluation.
[0054] 5. Build the model: Figure 3 As shown in the figure, considering the feature extraction requirements of eye movement data, a dual-channel architecture with a dilated convolutional network (DilatedCNN) as the backbone is designed. DilatedCNN can expand the receptive field without increasing the number of parameters through different dilation rates, effectively capturing multi-scale spatial features. On this basis, considering the temporal nature of eye movement data, a bidirectional long short-term memory network (BiLSTM) is introduced after feature extraction at each scale. BiLSTM can effectively capture the bidirectional dependencies of the sequence and enhance the modeling ability of temporal patterns. At the same time, an independent BiLSTM channel is designed to directly process the original input to retain and utilize the complete temporal information of the original data. In order to further improve the feature selection ability of the model, a hierarchical attention mechanism is adopted, including a local attention mechanism for the output of each channel and a global attention mechanism for feature fusion. This design can automatically learn and focus on important features and improve the expression ability of the model. The specific steps are as follows:
[0055] (1) At the input layer, the input data is standardized (zero mean and unit variance), the original input dimension is maintained, and the data format is adjusted through the transposition operation to adapt to subsequent processing.
[0056] (2) In the backbone network, a multi-scale feature extraction strategy is adopted, which includes three parallel dilated convolution branches to capture local details, medium-range and large-scale context features respectively, forming a multi-scale feature representation.
[0057] (3) In the dual-channel time series modeling part, the three-scale convolution features are spliced in the CNN-LSTM channel to improve the computational efficiency. The spliced features are input into the BiLSTM to enhance the time series dependency modeling through the bidirectional LSTM. The direct LSTM channel directly feeds the original input into another BiLSTM to retain the complete time series information of the original sequence.
[0058] (4) In the hierarchical attention mechanism, the output of three parallel dilated convolutions after BiLSTM and the direct BiLSTM output of another channel are processed by local attention to highlight the key information in each scale. Global attention fuses the local attention outputs and learns the importance weights of different feature representations.
[0059] 6. Model evaluation: The trained deep learning model is verified on the reserved test set. The accuracy (ACC) and the area under the receiver operating characteristic (ROC) curve (AUC) scores are used as evaluation indicators of the model. The accuracy can intuitively and quickly reflect the overall performance of the model, while the AUC score of the ROC curve is not affected by the imbalance of sample categories. It has a high reference value in reflecting the performance of the model classification of abnormal and normal eye movements, and is particularly suitable for medical-related discrimination tasks.
[0060] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A method for detecting eye movement anomalies based on smartphone video and deep learning, characterized in that: The steps include: S1. Data collection: Videos of eye movements were taken using the built-in camera of a smartphone to capture the entire face; S2, video behavior coding: use DataVyu coding to mark normal and abnormal eye movements, extract multiple 2-second eye movement event clips, and slow down frame by frame to ensure the accuracy of timestamps; S3, feature extraction: Through the Mediapipe facial feature point detection model, the coordinates of the facial feature points in the video clip are extracted, the movement trajectory of the key points of the eyes is retained, and the initial features of the eye movement are obtained; S4. Constructing a dataset: randomly split the eye movement event segments into training, validation, and test sets required by the deep learning model; S5. Model building: Building a multi-channel deep learning model based on DilatedCNN, BiLSTM and Local-global Attention; S6. Model evaluation: The effect of the model is evaluated by the accuracy ACC of the model on the test set and the area under the receiver operating characteristic ROC curve AUC score.
2. The eye movement anomaly detection method based on smartphone video and deep learning as claimed in claim 1, characterized in that: In step S1, the smartphone is fixed to the upper rear position of the screen directly in front of the subject by a bracket to ensure that the shooting field of view completely covers the face of the subject without any obstruction, and three eye movement task programs of fixation, saccade and smooth pursuit are presented in sequence on the screen directly in front, and the camera of the mobile phone is started to record the eye movement response of the subject in the whole process, and collect the original eye movement video data; all eye movement tasks are based on a black background, and the stimulus is a white circular pattern with a diameter of 30 pixels.
3. The eye movement anomaly detection method based on smartphone video and deep learning as claimed in claim 2, characterized in that: The procedures for performing the three eye movement tasks are as follows: (1) Smooth horizontal tracking task: The white circle moves horizontally at a constant speed from the horizontal center of the screen. Each time it reaches the edge of the screen, it pauses for 1 second and then continues to move at a constant speed in the opposite direction. The task is divided into two stages, and the movement speed of the second stage is slightly faster than that of the first stage. (2) Fixation task: The white circle stayed at one-quarter, center, and three-quarters of the horizontal center line of the screen from left to right for 10 seconds each; (3) Reflexive scanning task: The white circle moves randomly on the screen, appearing for 1 second each time and disappearing for 0.5 seconds, that is, it stays for 1 second each time, disappears for 0.5 seconds, and then appears at another random position.
4. The eye movement anomaly detection method based on smartphone video and deep learning as claimed in claim 3, characterized in that: In step S2, the video data is systematically encoded using DataVyu behavior analysis software: First, the video was played at 4 times slower speed for preliminary annotation to identify abnormal eye movement behaviors; When abnormal eye movements are detected, use the rewind function and switch to 8x slow mode to accurately mark the timestamp of the abnormality, and record the abnormality type and degree index in the coding area; finally, divide the video into 2-second intervals to extract normal and abnormal eye movement event segments respectively.
5. The eye movement anomaly detection method based on smartphone video and deep learning as claimed in claim 4, characterized in that: In step S3, the Mediapipe facial feature point detection model is deployed to process all 2-second video clips. Based on a sampling rate of 60 frames per second, 120 frames of images can be obtained for each clip. The facial three-dimensional coordinate information is extracted for each frame of the image, including spatial position data in the horizontal X, vertical Y and depth Z directions, and the regional features of both eyes are extracted. 21 key points are retained for each eye, including 16 points of the eyelid contour, 4 points of the iris edge and 1 point of the pupil center, to construct a complete eye movement feature sequence.
6. The eye movement anomaly detection method based on smartphone video and deep learning as claimed in claim 5, characterized in that: In step S4, a stratified random sampling strategy is used to construct a data set required for machine learning. The specific division scheme is as follows: 10% of normal eye movement samples and an equal number of abnormal eye movement samples are randomly selected to form a test set; 5% of normal and abnormal samples are each selected to form a validation set; and the remaining samples constitute a training set.
7. The eye movement anomaly detection method based on smartphone video and deep learning as claimed in claim 6, characterized in that: In the step S5, a model is established: a dual-channel architecture with a dilated convolutional network (DilatedCNN) as the backbone; on this basis, a bidirectional long short-term memory network (BiLSTM) is introduced after feature extraction at each scale, and at the same time, an independent BiLSTM channel is designed to directly process the original input, and a hierarchical attention mechanism is adopted, including a local attention mechanism for the output of each channel and a global attention mechanism for feature fusion.
8. The eye movement anomaly detection method based on smartphone video and deep learning as claimed in claim 7, characterized in that: The specific steps are as follows: (1) At the input layer, the input data is standardized, the original input dimension is maintained, and the data format is adjusted through transposition operation to adapt to subsequent processing; (2) In the backbone network, a multi-scale feature extraction strategy is adopted, which includes three parallel dilated convolution branches to capture local details, medium-range and large-scale context features respectively, forming a multi-scale feature representation; (3) In the dual-channel time series modeling part, the three-scale convolution features are spliced in the CNN-LSTM channel to improve the computational efficiency. The spliced features are input into the BiLSTM to enhance the time series dependency modeling through the bidirectional LSTM. The direct LSTM channel directly feeds the original input into another BiLSTM to retain the complete time series information of the original sequence. (4) In the hierarchical attention mechanism, the outputs of three parallel dilated convolutions after BiLSTM and the direct BiLSTM output of another channel are processed through local attention to highlight the key information at each scale; global attention fuses the local attention outputs and learns the importance weights of different feature representations.
9. The eye movement anomaly detection method based on smartphone video and deep learning as claimed in claim 8, characterized in that: In step S6, model evaluation: the trained deep learning model is verified on the retained test set, and the accuracy ACC and the area under the receiver operating characteristic ROC curve AUC scores are used as evaluation indicators of the model.