Eye movement monitoring method based on near-eye infrared imaging of head-mounted device

By building a lightweight eye feature detection network model, using convolution and inverse residual structures, the problem of difficult to achieve high real-time and high precision in devices with limited computing resources in the prior art is solved, and high efficiency in wearable devices is achieved Eye movement monitoring.

CN118918630AActive Publication Date: 2024-11-08SHENZHEN UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410827292.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-25
Publication Date
2024-11-08
Estimated Expiration
2044-06-25

AI Technical Summary

Technical Problem

Existing deep learning eye feature detection methods are difficult to meet the needs of high real-time and high precision in wearable devices with limited computing resources, especially the semantic segmentation model is time-consuming and difficult to achieve real-time monitoring.

Method used

Build a lightweight eye feature detection network model, adopt convolution, deep convolution and inverse residual structures, optimize network parameters through data enhancement and loss functions, reduce the calculation amount and parameter amount, and improve the robustness and generalization of the model , realize real-time eye movement monitoring.

Benefits of technology

High-precision and real-time eye movement monitoring is realized in wearable devices, which can effectively detect eye feature points, balance accuracy and real-time requirements, and is suitable for scenarios with limited computing resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918630A_ABST
    Figure CN118918630A_ABST
Patent Text Reader

Abstract

The invention discloses an eye movement monitoring method based on near-eye infrared imaging of a head-mounted device, and belongs to the technical field of eye detection. The eye movement monitoring method comprises the steps that an existing eye movement infrared image data set is collected, and label standardization processing is carried out; constructing a lightweight eye feature detection network model of the head-mounted equipment; training and testing the lightweight eye feature detection network model based on the eye movement infrared image data set, and extracting eye features; and monitoring eye movement state parameters based on the eye feature result in combination with the time sequence information. According to the method, the light-weight eye feature detection network model is constructed, and the important information of the image is efficiently extracted by using the light-weight convolution strategy, so that the calculation amount and the parameter amount are reduced, and the real-time application requirement of the wearable device under the condition that the calculation resources are limited is met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of eye detection technology, and in particular to an eye movement monitoring method based on near-eye infrared imaging of a head-mounted device. Background Art

[0002] Eye movement monitoring technology uses advanced image processing and analysis methods to detect and identify important eye features (such as eyelids, pupils, irises, etc.) from eye images. Combined with timing information, it can monitor the user's eye movement state parameters (such as fatigue state, gaze direction, etc.) and is widely used in medical diagnosis, assisted driving, virtual reality and other fields.

[0003] Eye feature detection is the core content of eye movement monitoring, which directly determines the accuracy and robustness of advanced functions such as fatigue detection and gaze estimation. Traditional eye feature detection methods usually use thresholding to extract pupil pixels or ellipse fitting to extract pupil contours. The principle is simple, but it can only detect pupil features and has poor robustness. The detection accuracy is low in environments such as light reflection, illumination and occlusion. Compared with traditional methods, the advantage of deep learning lies in its powerful feature learning ability. It can extract higher-level and more abstract features from the input image, capture important information such as eye shape and texture, thereby improving detection accuracy and robustness, and more accurately locating eye features. However, existing deep learning eye feature detection methods usually focus on improving accuracy, resulting in an increasingly large scale of network models and a sharp increase in computational complexity. In practical engineering applications, such as wearable devices, the trade-off between accuracy and real-time performance is a factor that needs to be considered, and existing methods are difficult to meet the requirements of high real-time tasks. Therefore, it is still challenging and difficult to ensure high real-time performance and high accuracy at the same time when the computing resources of wearable devices are limited.

[0004] Compared with large-model deep learning algorithms, lightweight deep learning algorithms aim to further reduce the number of model parameters and complexity while maintaining model accuracy, and have gradually become a research hotspot in computer vision. The patent with application number 202311547575.0 proposes a real-time binocular pupil inspection system. This method uses two sets of binocular infrared cameras to shoot the left and right eyes of the tester in real time, and uses the lightweight target detection model Yolo to detect the pupil positions of the left and right eyes of the tester in real time. The semantic segmentation model FastSCNN is used to extract the pupil area near the detected pupil position, and finally the pupil diameter and pupil shape are calculated, and the pupil line of sight is tracked and blink detection is performed to achieve binocular detection with reliable results. However, the two-stage strategy of this method has a large amount of calculation, especially the semantic segmentation model takes a long time. In the case of limited computing resources such as wearable devices, it is difficult to meet real-time requirements. Summary of the invention

[0005] The purpose of the present invention is to provide an eye movement monitoring method based on near-eye infrared imaging of a head-mounted device. By constructing a lightweight eye feature detection network model and utilizing a lightweight convolution strategy to efficiently extract important information from the image, the amount of calculation and the amount of parameters are reduced, thus meeting the real-time application requirements of wearable devices in scenarios with limited computing resources.

[0006] To achieve the above object, the present invention provides an eye movement monitoring method based on near-eye infrared imaging of a head-mounted device, the steps comprising:

[0007] S1, collect existing eye movement infrared image datasets and perform label normalization processing;

[0008] S2. Build a lightweight eye feature detection network model for head-mounted devices;

[0009] S3, training and testing a lightweight eye feature detection network model based on the eye movement infrared image dataset processed in step S1 to extract eye features;

[0010] S4. Based on the eye feature results and combined with the timing information, monitor the eye movement state parameters.

[0011] Preferably, the lightweight eye feature detection network model includes a Conv module, a DepthwiseConv module, a Block module and a fully connected layer, the Conv module includes a convolution layer and a batch normalization layer, the DepthwiseConv module is a deep convolution layer, the Block module is an inverted residual structure, and the fully connected layer is arranged at the tail of the lightweight eye feature detection network.

[0012] Preferably, in step S3, a loss function is used to train the lightweight eye feature detection network model, and the training process includes:

[0013] S31, setting network parameters of a lightweight eye feature detection network model;

[0014] S32, inputting a near-infrared eye image, gradually extracting features of the near-infrared eye image using convolution, deep convolution, and inverse residual, and obtaining feature maps of different scales through a branching strategy;

[0015] S33. Aggregate feature maps of different scales, and use a fully connected layer and an inverted residual structure to map the feature maps into the x and y coordinates of the eye feature points.

[0016] Preferably, a data enhancement strategy is used to increase near-infrared eye image training data during the training process, and the data enhancement strategy includes but is not limited to cropping, scaling, Gaussian filtering and median filtering.

[0017] Preferably, during the training process, network parameters are optimized by adjusting parameters and loading pre-trained weights.

[0018] Preferably, the loss function is the mean value of the sum of squared errors (MSE), which is the mean of the sum of squares of the corresponding points of the statistical prediction data and the original data, and the formula is:

[0019]

[0020] In the formula, x i Indicates the true value of the x coordinate of the eye feature point, y i Indicates the true value of the y coordinate of the eye feature point, Indicates the predicted value of the x coordinate of the eye feature point; represents the predicted value of the y coordinate of the eye feature point, and m represents the number of eye feature detection points.

[0021] Preferably, the eye movement state parameters in step S4 include but are not limited to eyelid opening and closing degree, PERCLOS, blinking frequency, average eye closing speed, average eye opening degree, and maximum continuous eye closing time.

[0022] Therefore, the present invention adopts the above-mentioned eye movement monitoring method based on near-eye infrared imaging of a head-mounted device, which has the following beneficial effects:

[0023] (1) Construct a lightweight eye feature detection network, use the inverted residual structure to improve the nonlinear expression ability of the network, reduce the number of channels through expansion layers and projection layers to reduce the amount of calculation and the number of parameters, and use depthwise separable convolution to independently convolve each input channel with the corresponding convolution kernel, greatly reducing the amount of calculation and the number of parameters;

[0024] (2) Aggregating multi-scale feature modules can effectively integrate feature information and context features of different scales, improve the positioning accuracy of eye feature points, and improve the robustness and generalization of the lightweight eye feature detection network model;

[0025] (3) In actual engineering application scenarios where computing resources of wearable devices are limited, it is possible to balance the requirements of accuracy and real-time performance and monitor eye movement state parameters in real time.

[0026] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0028] Figure 2 is an inverted residual structure diagram of an embodiment of the present invention;

[0029] Figure 3A shortcut connection diagram of the inverted residual structure of an embodiment of the present invention;

[0030] Figure 4 This is a diagram of a lightweight eye feature detection network structure according to an embodiment of the present invention;

[0031] Figure 5 is a flow chart of blink detection according to an embodiment of the present invention;

[0032] Figure 6 are eyelid feature points according to an embodiment of the present invention;

[0033] Figure 7 This is a test result diagram of a lightweight eye feature detection network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] Example

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0036] Reference Figure 1 , an eye movement monitoring method based on near-eye infrared imaging of a head-mounted device, the steps comprising:

[0037] S1. Collect existing eye movement infrared image datasets and perform label normalization processing.

[0038] S2. Build a lightweight eye feature detection network model for head-mounted devices.

[0039] Specifically, the lightweight eye feature detection network model includes multiple Conv modules, a DepthwiseConv module, multiple Block modules and three fully connected layers.

[0040] The Conv module includes convolutional layers and batch normalization layers. Batch normalization normalizes the input of each batch to keep the mean and variance of the input stable, accelerate the convergence of the network, improve the generalization ability of the model, and has a certain regularization effect.

[0041] The DepthwiseConv module is a deep convolution layer. Deep convolution convolves each input channel independently with the corresponding convolution kernel to generate the corresponding output channel, thereby reducing the amount of calculation and the number of parameters.

[0042] The Block module is an inverted residual structure, which uses the ReLU6 activation function to limit the output value between 0 and 6, improving the numerical stability of the model, enhancing the anti-noise performance, and helping to reduce the cost of calculation and storage. The inverted residual structure is increased in dimension by 1×1 convolution, and then reduced in dimension by 1×1 convolution, such as Figure 2 As shown in the figure, the expansion layer and projection layer correspond to the dimension increase and dimension reduction operations of 1×1 convolution respectively. The former improves the network's expressiveness by increasing the number of channels, thereby providing more feature information, while the latter reduces the amount of calculation and the number of parameters by reducing the number of channels, while helping the network to focus on learning and expressing the main features. Inverted residual structure When stride = 1 and the shape of the input feature matrix is ​​the same as that of the output matrix, a shortcut connection is performed, such as Figure 3 shown.

[0043] S3. Based on the eye movement infrared image dataset processed in step S1, a lightweight eye feature detection network model is trained and tested to extract eye features.

[0044] The training process includes:

[0045] S31. Setting network parameters of a lightweight eye feature detection network model.

[0046] S32. Input a near-infrared eye image, gradually extract near-infrared eye image features using convolution, deep convolution, and inverse residual, and obtain feature maps of four different scales through a branching strategy.

[0047] S33. Aggregate the feature maps of four different scales, and use the fully connected layer and the inverted residual structure to map the feature maps into the x and y coordinates of 50 eye feature points, indicating the positions of the eyelids, pupils, and irises. The number of feature points of the eyelids, pupils, and irises are 34, 8, and 8, respectively.

[0048] like Figure 4 As shown in the figure, it is the network structure of the lightweight eye feature detection network model. The "3×3Conv 64,2" form of the Conv module in the figure indicates that the size of the convolution kernel is 3×3, the number of convolution kernels is 64, and the step size is 2.

[0049] During training, X is the input image, which first passes through a 3×3 convolution layer, a depth convolution layer, and three block inverted residual convolution modules. Then it is divided into two paths, one path passes through a 3×3 convolution layer and a 1×1 convolution layer to form the first feature block; the other path passes through four block inverted residual convolution modules. It is divided into two paths again, one path passes through a 3×3 convolution layer and two 1×1 convolution layers to form the second feature block; the other path passes through five block inverted residual convolution modules. Finally, one path passes through a 3×3 convolution layer and three 1×1 convolution layers to form the third feature block; the other path passes through a 3×3 convolution layer and four 1×1 convolution layers to form the fourth feature block. The four feature blocks are all reduced to 1 through the convolution layer before the fully connected layer, and are aggregated to combine cross-channel and cross-scale context features. Three fully connected layers are set at the end. The first fully connected layer integrates the feature information in the receptive fields of different sizes extracted by the previous convolutional layer and maps this information to the sample label space so that the network can better understand and process the input data and increase the representation ability of the model. The second and third fully connected layers gradually reduce the output dimension of the network and finally map it to the x, y coordinates of 50 eye feature points. Among them, the number of feature points of the eyelid, pupil, and iris are 34, 8, and 8 respectively.

[0050] Specifically, during the training process, data enhancement strategies such as cropping, scaling, Gaussian filtering and median filtering are used to increase the near-infrared eye image training data.

[0051] Specifically, during the training process, network parameters are optimized by adjusting parameters and loading pre-trained weights.

[0052] Specifically, the loss function used is the mean value of the sum of squared errors (MSE), which is the mean of the sum of squares of the corresponding points of the predicted data and the original data, reflecting the distribution of the prediction error. The formula is:

[0053]

[0054] In the formula, x i Indicates the true value of the x coordinate of the eye feature point, y i Indicates the true value of the y coordinate of the eye feature point, Indicates the predicted value of the x coordinate of the eye feature point; represents the predicted value of the y coordinate of the eye feature point, and m represents the number of eye feature detection points.

[0055] S4, such as Figure 5 As shown, based on the eye feature results and combined with the timing information, eye movement state parameters such as eyelid opening and closing degree, PERCLOS, blinking frequency, average eye closure speed, average eye opening degree, and maximum continuous eye closure time are monitored.

[0056] Eyelid opening: the degree to which the eyes are open. Figure 6 As shown in the figure, based on the eyelid feature points, the eyelid opening and closing degree is expressed as:

[0057]

[0058] In the formula, p 1 Indicates the upper left edge of the eyelid, p 2 Indicates the lower left edge of the eyelid, p 3 Indicates the upper right edge of the eyelid, p 4 The lower right edge of the eyelid, p 5 Indicates the inner corner of the eyelid, p 6 Indicates the end of the eye

[0059] PERCLOS (Percentage of Eyelid Closure Over the Pupil Over Time): refers to the percentage of time spent with eyes closed over a period of time. When the eyelid opening and closing degree is less than the set threshold t, it is considered as a closed eye state. In a fatigued state, PERCLOS will increase significantly. PERCLOS is expressed as:

[0060]

[0061] Blinking frequency: the number of times you blink per unit time. When the eyes go through the state of open-closed-open, it is considered as one blink. In a fatigued state, the blinking frequency will increase or decrease significantly. The blinking frequency is expressed as:

[0062]

[0063] Average eye closing speed: It is represented by the average time required for the eyes to open and close. The longer the time, the slower the eye closing speed. When you are tired, the eye closing speed will be significantly slower. The average eye closing speed is expressed as:

[0064] Average eye closing speed = mean (time from eye opening to eye closing)

[0065] Average eye opening degree: refers to the percentage of time that the eyes are open within a period of time. When the eyelid opening degree is greater than the set threshold t, it is considered to be an open state. In a fatigued state, the average eye opening degree will be significantly reduced. The average eye opening degree can be expressed as:

[0066]

[0067] Maximum continuous eye closure time: Under fatigue conditions, the maximum continuous eye closure time will be significantly prolonged. The maximum continuous eye closure time is expressed as:

[0068] Maximum continuous eye closure time = max(continuous eye closure time)

[0069] In order to verify the effectiveness of the solution of the present invention, the existing near-infrared eye image data set is classified into 4:1 for network training test. The hardware information of the test equipment is: GPU, model NVIDIA 2080Ti, memory is 12GB. The test results are as follows Figure 7 As shown in the figure, it can be seen that the present invention can well detect the positions of eye features such as eyelids, pupils and irises in head-mounted near-eye scenes.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for monitoring eye movement based on near-eye infrared imaging of a head-mounted device, characterized in that the steps include: S1, collect existing eye movement infrared image datasets and perform label normalization processing; S2. Build a lightweight eye feature detection network model for head-mounted devices; S3, training and testing a lightweight eye feature detection network model based on the eye movement infrared image dataset processed in step S1 to extract eye features; S4. Based on the eye feature results and combined with the timing information, monitor the eye movement state parameters.

2. The eye movement monitoring method based on near-eye infrared imaging of a head-mounted device according to claim 1, characterized in that: The lightweight eye feature detection network model includes a Conv module, a DepthwiseConv module, a Block module and a fully connected layer. The Conv module includes a convolution layer and a batch normalization layer. The DepthwiseConv module is a deep convolution layer. The Block module is an inverted residual structure. The fully connected layer is arranged at the tail of the lightweight eye feature detection network.

3. The eye movement monitoring method based on near-eye infrared imaging of a head-mounted device according to claim 2, characterized in that: In step S3, the loss function is used to train the lightweight eye feature detection network model, and the training process includes: S31, setting network parameters of a lightweight eye feature detection network model; S32, inputting a near-infrared eye image, gradually extracting features of the near-infrared eye image using convolution, deep convolution, and inverse residual, and obtaining feature maps of different scales through a branching strategy; S33. Aggregate feature maps of different scales, and use a fully connected layer and an inverted residual structure to map the feature maps into the x and y coordinates of the eye feature points.

4. The eye movement monitoring method based on near-eye infrared imaging of a head-mounted device according to claim 3, characterized in that: During the training process, a data enhancement strategy is used to increase near-infrared eye image training data, and the data enhancement strategy includes but is not limited to cropping, scaling, Gaussian filtering and median filtering.

5. The eye movement monitoring method based on near-eye infrared imaging of a head-mounted device according to claim 3, characterized in that: During the training process, network parameters are optimized by adjusting parameters and loading pre-trained weights.

6. The eye movement monitoring method based on near-eye infrared imaging of a head-mounted device according to claim 3, characterized in that: The loss function is the mean value of the sum of squared errors (MSE), which is the mean of the sum of squares of the corresponding points of the statistical prediction data and the original data. The formula is: In the formula, x i Indicates the true value of the x coordinate of the eye feature point, y i Indicates the true value of the y coordinate of the eye feature point, Indicates the predicted value of the x coordinate of the eye feature point; represents the predicted value of the y coordinate of the eye feature point, and m represents the number of eye feature detection points.

7. The eye movement monitoring method based on near-eye infrared imaging of a head-mounted device according to claim 1, characterized in that: The eye movement state parameters in step S4 include but are not limited to eyelid opening and closing degree, PERCLOS, blinking frequency, average eye closing speed, average eye opening degree, and maximum continuous eye closing time.

Citation Information

Patent Citations

  • Real-time double-eye pupil examination system and detection method

    CN117530654A

  • Driver fatigue detection method based on CNN eye state recognition

    CN108294759A

  • Sight line tracking model training method, and sight line tracking method and device

    CN110058694A

  • Eye movement interaction system based on visual image information

    CN114821753A

  • Eye tracking latency enhancements

    US20220301217A1