Eye movement tracking interaction method and device

Abstract

The invention provides an eye movement tracking interaction method. The method comprises the following steps: S1, collecting an eye image and a scene image; the eye images are classified through a neural network, one type is an eye-open image, and the other type is an eye-closed image and an undetected pupil image; s2, constructing an eye movement model and a YOLO target recognition model, and training the eye movement model and the YOLO target recognition model; s3, preprocessing the eye opening image and the scene image to obtain eye image data and scene image data; respectively inputting the eye image data and the scene image data into an eye movement model and a YOLO target recognition model to obtain a fixation point coordinate, a target type and a pixel coordinate of the target type; and S4, performing fixation target judgment, and if the target is judged to be fixation, executing a target corresponding program. The application of eye movement tracking is further expanded through a method of selecting a real object after the real object is watched by combining the eye movement model and the yolk.

Description

Technical Field

[0001] The present application relates to the field of eye tracking technology, and in particular to an eye tracking interaction method and device. Background Art

[0002] Eye tracking devices can be divided into remote and wearable types. The structure of a wearable eye tracking device is similar to that of glasses. An eye camera and a scene camera are installed on the wearable eye tracking device. The eye camera mainly captures images of the human eye, and the scene camera mainly captures images in front of the human eye. Constructing a data set with a wearable eye tracking device usually requires placing a calibration object within the shooting range of the scene camera. During the data set construction process, the human eye needs to keep staring at the calibration point to ensure the accuracy of the data set.

[0003] The current eye tracking dataset construction procedures have the following problems:

[0004] ① The recognition of the calibration object uses the traditional image algorithm to extract the contour. This limitation comes from the fact that when the scene camera captures a similar contour structure, noise will be introduced, causing recognition errors. In addition, the calibration object cannot be replaced. If the calibration object is replaced, the recognition algorithm needs to be modified again, which is a cumbersome implementation process.

[0005] ② The current eye movement dataset construction program does not have the eye closure detection function, and the data collection process requires continuous gaze of the human eye, which will cause eye fatigue and save pictures in a non-gaze state to the dataset. This will bring difficulties to the dataset collection process. Saving erroneous images will increase the error of the dataset. Therefore, how to reduce the error in the dataset collection process is also an existing problem.

[0006] (2) Problems with traditional image processing eye tracking

[0007] Traditional image processing algorithms are mainly based on the tools provided in the OpenCV toolkit. The accuracy of pupil detection in eye tracking algorithms based on traditional image processing will be reduced due to environmental factors. In traditional image processing algorithms, pupil capture mostly requires binarization, and then canny edge detection is used to extract pupil contours.

[0008] There are two problems with this:

[0009] ① The placement of the camera will have a significant impact on the accuracy of the algorithm. When the camera is too close, the brightness around the pupil will dim, making it difficult to ensure that the pupil contour is not affected by the surrounding shadows after binarization, making it impossible for Canny to extract a complete and correct pupil contour image.

[0010] ② The influence of external light sources will also cause the pixel values ​​around the pupil to change, thus affecting the failure of contour detection. As shown in the figure below, the pupil contour is connected to the surrounding shadows, and the contour shape will be misjudged after canny.

[0011] (3) At present, eye tracking technology is mainly used in psychology, market research and advertising, medical diagnosis, learning behavior analysis and other fields. However, research in the field of human-computer interaction is relatively lacking. Summary of the invention

[0012] In order to solve the above technical problems, in the first aspect of the present application, an eye tracking interaction method is proposed, comprising:

[0013] S1, collecting eye images and scene images; classifying the eye images through a neural network, one of which is an open-eye image and the other is an image of a closed eye and an image without pupil detection;

[0014] S2. Build an eye movement model and a YOLO target recognition model, and train the eye movement model and the YOLO target recognition model;

[0015] S3, preprocessing the open-eye image and the scene image to obtain eye image data and scene image data; inputting the eye image data and the scene image data into an eye movement model and a YOLO target recognition model respectively to obtain the gaze point coordinates, the target type and the pixel coordinates of the target type;

[0016] S4. Determine the gaze target. If the target is determined to be gazed at, execute the target corresponding program.

[0017] Through the above-mentioned technical means, the applicant built an eye tracking interaction system to collect the gaze points of open eyes in real time, obtain the user's gaze points on the real scene through the eye tracking deep learning model, and determine whether the user is looking at a specific real target, thereby realizing the control of real objects through eye movements.

[0018] Preferably,

[0019] S101, collecting eye images and scene images;

[0020] S102. Use pytorh to build a two-layer simple neural network to classify eye images, one of which is open-eye images and the other is closed-eye images with undetected pupils.

[0021] Preferably,

[0022] S201, build an eye movement model through pytorch;

[0023] S202. Build a YOLO target recognition model based on pytorch and adopt the YOLOv7 model;

[0024] S203, constructing a data set containing a recognition target, and using the data set containing the target to train the YOLO target recognition model;

[0025] S204, using the YOLO target recognition model trained with the target data set to generate an eye movement model data set;

[0026] S205: Train a YOLO target recognition model using a data set containing the recognition target, and train an eye movement model using an eye movement model data set.

[0027] Preferably,

[0028] S301, converting the eye-opening image and the scene image into grayscale images;

[0029] S302, normalizing the grayscale image size;

[0030] S303, converting the format of the grayscale image after the size transformation into tensor type data, thereby obtaining open-eye image data and scene image data;

[0031] S304, the eye image data is input into the eye movement model to obtain the gaze point coordinates, and the scene image data is input into the YOLO target recognition model to obtain the target type and the pixel coordinates of the target type.

[0032] Preferably, the target gaze determination is specifically as follows:

[0033] Set the gaze state of each target type to 0;

[0034] Determine the pixel coordinates and gaze point coordinates of each target type one by one;

[0035] When the gaze point coordinates are within the pixel coordinate area of ​​any target type, the gaze state of the target type is modified to 1.

[0036] Preferably, the eye movement model adopts a double-layer CNN convolutional structure.

[0037] Preferably, the data set containing the identified target is completed by using LabelImg; the data set containing the identified target includes: the type of the target and the corresponding position.

[0038] In a second aspect of the present application, an eye tracking interaction device is provided, which includes a frame and a main control module. The frame is provided with a scene camera, an eye nose pad, an eye camera and an infrared light source. The eye camera and the infrared light source are arranged on a side of the frame close to the temple, and the scene camera is arranged on a side of the frame away from the temple. The scene camera, the eye camera and the infrared light source are all connected to the main control module by signal. The main control module is used to realize the control of real objects triggered by eye movements through the method of any one of claims 1-7.

[0039] Preferably, the scene camera adopts the 135° wide-angle camera of the OV2659 model; the eye camera adopts the ov5640 camera module.

[0040] Preferably, the infrared light source is an infrared LED light source with a light emission wavelength of 850 nm.

[0041] Compared with the prior art, the beneficial results of this application are:

[0042] 1. Based on the Pupil project, the application of eye tracking is further expanded, especially providing a trigger architecture and interface for controlling real objects.

[0043] 2. Provide users with a flexible trigger architecture and interface. Users can customize trigger programs and effects according to specific needs, thereby achieving personalized device control and human-computer interaction. This innovative interaction method not only improves the user experience, but also expands the application scope of eye tracking technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present application. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.

[0045] Figure 1 is a flow chart of an eye tracking interaction method according to an embodiment of the present application;

[0046] Figure 2 It is a flowchart of a YOLO target recognition model training of an eye tracking interaction method according to an embodiment of the present application;

[0047] Figure 3 is a structural schematic diagram of an eye movement model of an eye movement tracking interaction method according to an embodiment of the present application;

[0048] Figure 4is a schematic diagram of a process of collecting images according to an eye tracking interaction method according to an embodiment of the present application;

[0049] Figure 5 is a schematic diagram of the structure of an eye tracking interaction device according to an embodiment of the present application;

[0050] Figure 6 is a schematic diagram of module connections of an eye tracking interaction device according to an embodiment of the present application;

[0051] Figure 7 A power supply circuit for an infrared light source of an eye tracking interactive device according to an embodiment of the present application Figure 1 ;

[0052] Figure 8 A power supply circuit for an infrared light source of an eye tracking interactive device according to an embodiment of the present application Figure 2 .

[0053] Explanation of the accompanying drawings: 1. Infrared light source; 2. PCB board No. 1; 3. Scene camera; 4. Nose pad; 5. Eye camera; 6. PCB board No. 2; 7. Main control module. DETAILED DESCRIPTION

[0054] In the following detailed description, reference is made to the accompanying drawings, which form a part of the detailed description and are shown by illustrative specific embodiments in which the present application can be practiced. To this end, directional terms, such as "top", "bottom", "left", "right", "up", "down", etc., are used with reference to the orientation of the figures described. Because the components of the embodiments can be positioned in several different orientations, directional terms are used for the purpose of illustration and are by no means limiting. It should be understood that other embodiments may be utilized or logical changes may be made without departing from the scope of the present application. Therefore, the following detailed description should not be adopted in a limiting sense, and the scope of the present application is defined by the appended claims.

[0055] Figure 1 is a flow chart of an eye tracking interaction method according to an embodiment of the present application, such as Figure 1 As shown, the method disclosed in the embodiments of the present application comprises the following steps:

[0056] S1, collecting eye images and scene images; classifying the eye images through a neural network, one of which is an open-eye image and the other is an image of a closed eye and an image without pupil detection;

[0057] Wherein step S1 also includes:

[0058] S101, collecting eye images and scene images;

[0059] S102. Use pytorh to build a two-layer simple neural network to classify eye images, one of which is images with closed eyes and the other is images with closed eyes and undetected pupils.

[0060] S2. Build an eye movement model and a YOLO target recognition model, and train the eye movement model and the YOLO target recognition model;

[0061] Wherein step S2 also includes:

[0062] S201, build an eye movement model through pytorch;

[0063] S202. Build a YOLO target recognition model based on pytorch and adopt the YOLOv7 model;

[0064] S203, constructing a data set containing the recognized target, and using the data set containing the target to train the YOLO target recognition model. ;

[0065] S204, using the YOLO target recognition model trained with the target data set to generate an eye movement model data set; specifically, the YOLO target recognition model identifies the target pixel coordinate points in the scene image captured by the front camera, and the eye camera saves the eye image, and the pixel coordinates in the eye image and the scene image (wherein the coordinates are the gaze point coordinates) are correspondingly saved);

[0066] S205: Train a YOLO target recognition model using a data set containing the recognition target, and train an eye movement model using an eye movement model data set.

[0067] S3, preprocessing the open-eye image and the scene image to obtain eye image data and scene image data; inputting the eye image data and the scene image data into an eye movement model and a YOLO target recognition model respectively to obtain the gaze point coordinates, the target type and the pixel coordinates of the target type;

[0068] Wherein step S3 further comprises:

[0069] S301, converting the eye-opening image and the scene image into grayscale images;

[0070] S302, normalizing the grayscale image size;

[0071] S303, converting the format of the grayscale image after the size transformation into tensor type data, thereby obtaining open-eye image data and scene image data;

[0072] S304, the eye image data is input into the eye movement model to obtain the gaze point coordinates, and the scene image data is input into the YOLO target recognition model to obtain the target type and the pixel coordinates of the target type.

[0073] S4. Determine the gaze target. If the target is determined to be gazed at, execute the target corresponding program.

[0074] Among them, the target gaze determination is specifically as follows:

[0075] Set the gaze state of each target type to 0;

[0076] Determine the pixel coordinates and gaze point coordinates of each target type one by one;

[0077] When the gaze point coordinates are within the pixel coordinate area of ​​any target type, the gaze state of the target type is modified to 1.

[0078] In a specific embodiment, Figure 2 As shown in the figure, when collecting image data, eye images and scene images are collected separately, and the scene images are put into the pre-trained YOLO target recognition model for calibration target search. When a calibration is detected, the program will start to generate a data set, and when a blink is detected, the program will skip the frame. The blink detection is based on pytorch, and a two-layer simple neural network is built through pytorch to classify the pictures. By constructing a 2-classification data set, one of the categories is open-eye photos, and the other category is closed-eye photos with undetected pupils.

[0079] In a specific embodiment, Figure 3 As shown in the figure, the structure and data flow of the eye movement model are as follows: First, the input eye image is 640x480, which is resized to a grayscale image of 64x64x1, and then sent to the neural network model for operation. The first convolution uses a 5x5 convolution kernel to generate 16 feature maps. After the convolution operation, the ReLU activation function is used to increase the nonlinear expression ability of the model. The size of the output feature map is 60x60x16. Then it is pooled again. The 2x2 maximum pooling (MaxPooling) operation is used to further reduce the size of the feature map, reduce the amount of calculation, and extract more important features. The output size after pooling is 30x30x16. Then it enters the second convolution layer, which uses a 5x5 convolution kernel to generate 32 feature maps. The ReLU activation function is used again. The size of the output feature map is 30x30x32. Finally, it passes through the fully connected layer, which flattens the pooled feature map and converts it into a one-dimensional vector. Then it is input into the fully connected layer for further feature combination and calculation. The output of the fully connected layer is the pixel coordinates (x, y) of the target object in the front camera.

[0080] In a specific embodiment, a dataset containing the identified targets needs to be constructed in advance. This can be done by using the open source labeling software LabelImg. In the process of constructing the dataset, we need to mark the targets to be identified in the image, including the type of target and the corresponding position. The labeling process requires accurately marking the bounding box and label of each target, and using the constructed dataset to train the YOLOv7 model. Through repeated training, the model will gradually learn and improve its ability to recognize various targets, such as Figure 4 As shown in the figure, the training process of the YOLO target recognition model is as follows: import the external data path, create an instance of the dataset type, and also create a subclass of nn.module to store the structure of the model;

[0081] After connecting through DataLoader, import the loop main program, which includes: 1. Forward propagation; 2. Calculate loss; 3. Backward propagation; 4. Store model;

[0082] At the same time, you can also select various parameters during training, including but not limited to: loss function, optimizer and loss calculation formula. When the number of loops is met, the loop ends, otherwise it continues.

[0083] In this method, customized requirements can be achieved by adjusting various parameters during training.

[0084] In a second aspect, the present application also discloses an eye tracking interaction device, referring to Figure 5 and Figure 6 The device includes a frame and a main control module 7. The frame is provided with a scene camera 3, an eye nose pad 4, an eye camera 5 and an infrared light source 1. The eye camera 5 and the infrared light source 1 are arranged on a side of the frame close to the temple, and the scene camera 3 is arranged on a side of the frame away from the temple. The scene camera 3, the eye camera 5 and the infrared light source 1 are all connected to the main control module 7 by signal.

[0085] In a specific embodiment, two cameras are mounted on the glasses, one is a scene camera and the other is an infrared camera. In addition, infrared LEDs are required around the infrared camera to irradiate 850nm infrared light to enhance the infrared image. The scene camera uses a 135° wide-angle camera of the OV2659 model, which is fixed in front of the glasses frame to collect video streams outside the frame. The eye camera uses a 2 million camera of 850nm, which is fixed at the lower right corner of the frame to collect video streams of human eye movements. The focal lengths of the eye camera and the OV2659 model scene camera can be focused manually. The refresh rate and frame rate of the two cameras can be adjusted according to the pixel size of the picture, and support a variety of operating systems (windows, Linux, Android, Mac, os). The size of the camera is small, which is convenient for system integration. The working current of the two cameras is moderate, which reduces the pressure of system heat dissipation problems. The scene camera and the eye camera all use a USB driver-free interface design, and transmit video data to the video processing unit in real time via a USB data cable. The infrared LED is powered by the 5V power provided by the USB interface, which is converted into 1.5V output by the AMS1117 linear regulator to provide power for our 850nm infrared LED.

[0086] In addition, reference Figure 7 and Figure 8 The power of the entire device is provided by a USB cable. The USB2 interface is used for data transmission and power supply of the infrared eye camera, while the USB2 interface is used for power supply of the scene camera and the 850nm infrared LED.

[0087] Obviously, those skilled in the art can make various modifications and changes to the embodiments of the present application without departing from the spirit and scope of the present application. In this way, if these modifications and changes are within the scope of the claims of the present application and their equivalents, the present application is also intended to cover these modifications and changes. The word "comprising" does not exclude the presence of other elements or steps not listed in the claims. The simple fact that certain measures are recorded in mutually different dependent claims does not indicate that the combination of these measures cannot be used to profit. Any figure mark in the claims should not be considered to limit the scope.

Claims

1. An eye tracking interaction method, characterized in that: include: S1, collecting eye images and scene images; classifying the eye images through a neural network, one of which is an open-eye image and the other is an image of a closed eye and an image without a pupil detected; S2, constructing an eye movement model and a YOLO target recognition model, and training the eye movement model and the YOLO target recognition model; S3, preprocessing the open-eye image and the scene image to obtain eye image data and scene image data; Inputting the eye image data and the scene image data into the eye movement model and the YOLO target recognition model respectively to obtain the gaze point coordinates, the target type and the pixel coordinates of the target type; S4. Determine the gaze target. If the target is determined to be gazed at, execute the target corresponding program.

2. The eye tracking interaction method according to claim 1, characterized in that: The S1 includes: S101, collecting eye images and scene images; S102, constructing a two-layer simple neural network through pytorh to classify the eye images, one of which is the open-eye images, and the other is the closed-eye images and the images without pupil detection.

3. The eye tracking interaction method according to claim 1, characterized in that: The S2 includes: S201, constructing the eye movement model through pytorch; S202. Build a YOLO target recognition model based on pytorch and adopt the YOLOv7 model; S203, constructing a data set containing a recognition target, and using the data set containing the target to train the YOLO target recognition model; S204, using the trained YOLO target recognition model to generate an eye movement model data set YOLO target recognition model; S205: Training the eye movement model using the eye movement model dataset.

4. The eye tracking interaction method according to claim 1, characterized in that: The S3 includes: S301, converting the eye-opening image and the scene image into grayscale images; S302, normalizing the grayscale image size; S303, converting the format of the grayscale image after size transformation into tensor type data, thereby obtaining the eye-opening image data and the scene image data; S304, the eye image data is input into the eye movement model to obtain the gaze point coordinates, and the scene image data is input into the YOLO target recognition model to obtain the target type and the pixel coordinates of the target type.

5. The eye tracking interaction method according to claim 1, characterized in that: The target gaze determination is specifically as follows: Setting the gaze state of each target type to 0; Determine the pixel coordinates of each target type and the gaze point coordinates one by one; When the gaze point coordinates are located within the pixel coordinate area of ​​any target type, the gaze state of the target type is modified to 1.

6. The eye tracking interaction method according to claim 1, characterized in that: The eye movement model adopts a double-layer CNN convolution structure.

7. The eye tracking interaction method according to claim 3, characterized in that: The data set containing the identified target is completed by using LabelImg; the data set containing the identified target includes: the type of the target and the corresponding position.

8. An eye tracking interactive device, characterized in that: The device includes a frame and a main control module, the frame is provided with a scene camera, an eye nose pad, an eye camera and an infrared light source, the eye camera and the infrared light source are arranged on the side of the frame close to the temple, and the scene camera is arranged on the side of the frame away from the temple, the scene camera, the eye camera and the infrared light source are all connected to the main control module signal, and the main control module is used to realize the control of real objects triggered by eye movements through the method described in any one of claims 1-7.

9. An eye tracking interaction device according to claim 8, characterized in that: The scene camera adopts the 135° wide-angle camera of the OV2659 model; the eye camera adopts the ov5640 camera module.

10. The eye tracking interaction device according to claim 8, characterized in that: The infrared light source adopts an infrared LED light source with a light emission wavelength of 850nm.

Images