An eye tracking method and an eye tracking device

Abstract

The application relates to an eye movement tracking method and an eye movement tracking device. The method builds an eye movement tracking system of a human eye, collects a fixation point of the human eye in real time, and calculates a trajectory graph and a hotspot graph of the fixation point of the human eye when observing an object. According to the result, the key points of attention and the key parts of interest of the human eye when browsing the external object are evaluated. The application can be used not only for improving and adjusting the product design style, but also for adapting to the subjective feelings of the public and attracting the eyeballs of the audience. Meanwhile, the application can also be used for evaluating the emotions and fatigue states of the human eye and clinically treating emotional disorders in the field of nerves.

Description

Technical Field

[0001] This application relates to the technical field of eye tracking, and in particular to an eye tracking method and an eye tracking device. Background Technology

[0002] Current research on eye-tracking technology primarily focuses on the application of near-eye eye-tracking technology to analyze the emotions and assess fatigue levels of test subjects. The main research emphasis is on changes in the human eye's gaze point, without addressing the external environment as perceived by the eye. This makes it difficult to assess the focus of attention and key areas of interest when people browse external objects, thus hindering the improvement and adjustment of product (webpage, book, painting, architectural specifications) design styles. Summary of the Invention

[0003] To address the technical problems mentioned above, this application proposes an eye-tracking method and an eye-tracking device, employing the following technical solution:

[0004] In a first aspect, this application proposes an eye-tracking method, comprising the following steps:

[0005] S1: Capture an image of the eye using an infrared eye camera, and perform image processing on the eye image to obtain the center of the pupil;

[0006] S2: Illuminate the eye with an infrared light source to form a bright spot on the surface of the eyeball. Then, detect the bright spot by performing bright spot detection on the eye image captured by the infrared eye camera, calculate the centroid of the bright spot, and then construct the eye vector using the centroid and the pupil center.

[0007] S3: Construct the mapping equation that maps the eye vector to scene coordinates;

[0008] S4: Use an infrared eye camera and a distant camera to capture the user's eye image and scene image respectively. Obtain the eye movement vector through the eye image, and then substitute the eye movement vector into the mapping equation in step S3 to obtain the screen gaze coordinates.

[0009] S5: Use the screen gaze coordinates to draw a heatmap and overlay it onto the scene image to obtain an image fused with the heatmap and the scene image.

[0010] By adopting the above technical solution, this application constructs an eye-tracking system to collect real-time eye gaze points and calculates the trajectory and heatmap of people's gaze points when observing an object (webpage, book, building, landscape photo, etc.). Based on these results, the focus of people's attention and key areas of interest when browsing external objects can be assessed. This invention can not only be used to improve and adjust the design style of products (webpages, books, paintings, architectural specifications) to better suit the subjective feelings of the public and attract viewers' attention, but also for assessing people's emotions and fatigue states, and for the clinical treatment of emotional disorders in the neurological field of medicine.

[0011] Preferably, S1 includes:

[0012] S11: Convert the eye image to a grayscale image;

[0013] S12: Binarize the grayscale image;

[0014] S13: Perform contour detection on the binarized grayscale image to obtain the contour center, and use the contour center as the pupil center.

[0015] S14: Use the Hough circle detection function to find the circle whose center is closest to the center in a grayscale image;

[0016] S15: Display the detected outline on the original icon.

[0017] Preferably, S1 includes:

[0018] S11: Convert the eye image to a grayscale image;

[0019] S12: Binarize the grayscale image;

[0020] S13: Perform two iterative opening operations on the binarized grayscale image using a 3x3 convolution kernel;

[0021] S14: After the opening operation, the image is used to perform contour detection by ellipse fitting. The second largest contour among all the fitted contours is taken as the pupil contour, and the center of the fitted ellipse is taken as the pupil center.

[0022] Preferably, S2 specifically includes:

[0023] S21: Perform grayscale conversion on the eye image;

[0024] S22: Binarize the grayscale image processed in step S21, wherein a high threshold is set for the binarization function;

[0025] S23: Perform an opening operation on the image using a 3x3 convolution kernel;

[0026] S24: Perform contour detection on the binarized image processed in step S23 to obtain bright spot contours;

[0027] S25: Calculate the centroid of the bright spot using the outline of the bright spot, and then construct the eye vector using the centroid and the pupil center.

[0028] Preferably, S3 specifically includes:

[0029] S31: Acquire multiple sets of eye images and scene images on user gaze calibration cards to obtain multiple sets of eye movement vectors and scene camera coordinates;

[0030] S32: Substitute multiple sets of eye-tracking vectors and scene camera coordinates into the transformation equation from eye-tracking vectors to scene camera coordinates to solve for the parameters;

[0031] The transformation equation from the eye-tracking vector to the scene camera coordinates is:

[0032]

[0033] Where Xe and Ye are eye-tracking vectors, and Xs and Ys are scene camera coordinates.

[0034] Preferably, S5 includes:

[0035] S51: Read the screen gaze coordinates identified by eye tracking;

[0036] S52: Put all screen gaze coordinates into a list variable named data, where data = [[x1, y1][x2, y2]...];

[0037] S53: Draw a heatmap and overlay the weighted heatmap onto the original scene image.

[0038] Preferably, in S53: if the user gazes once at position (x, y), the user's interest value is set to be the largest at the center (x, y) and decreases linearly outwards, and then different colors are assigned according to different interest values.

[0039] Secondly, this application also proposes an eye-tracking device, characterized in that: the device includes a frame and a main control module; the frame is equipped with a distant-view camera, an infrared eye camera, and an infrared light source; the infrared eye camera and the infrared light source are located on the side of the frame closer to the temples, and the distant-view camera is located on the side of the frame away from the temples; the distant-view camera, the infrared eye camera, and the infrared light source are all signal-connected to the main control module; the main control module is used to obtain an image fused from a heat map and a scene map using the method described in the first aspect and to transmit the image to a communication terminal via wireless communication.

[0040] Preferably, the infrared light source is an infrared LED light source with an emission wavelength of 940nm.

[0041] Preferably, the main control module uses a USB to Type-C circuit structure to power the infrared LED light source, and the circuit structure includes a 6-pin Type-C female connector slot.

[0042] In summary, this application includes at least one of the following beneficial technical effects:

[0043] 1. This application comprehensively applies the pupil-corneal reflection method and the ellipse fitting method for pupil center position to obtain the eye movement vector. An infrared near-field camera, fixed to the frame of eyeglasses, captures a video stream of human eye movements. Based on the pupil-corneal reflection method, an infrared eye camera captures images of the eye, and then the pupil center position is obtained through ellipse fitting. An external infrared light source illuminates the eye, forming a bright spot on the surface of the eyeball. Since the formation of this bright spot is based on the position of the light source and the camera, the spot remains in a fixed position during eye movement. The eye movement vector is obtained through the pupil center and the center of the bright spot; this vector represents the eye movement characteristics.

[0044] 2. The eye-tracking system proposed in this invention not only has a simple overall circuit structure design and is easy to wear, but also has a small size and is easy to carry. Since it only collects video signals, the computational requirements of the main control module are reduced. By adjusting the appropriate algorithm, it is possible to achieve a better improvement in the accuracy of gaze point estimation.

[0045] 3. The eye-tracking device described in this application requires a calibration process before it can be used normally. After completing the device calibration, the user can move freely and achieve remote eye tracking within the WIFI coverage area by wearing our monocular or binocular portable eye-tracking device. Image data can be displayed via a personal computer. The PC can view the scene camera images on the wearable device and mark the calculated gaze point position in the scene camera, allowing the PC to detect the object the user is currently looking at in real time. Attached Figure Description

[0046] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of this application. Other embodiments and many anticipated advantages of these embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0047] Figure 1This is a flowchart of an eye-tracking method according to an embodiment of this application.

[0048] Figure 2 This is a schematic diagram of the pupil center detection process in one embodiment of this application.

[0049] Figure 3 This is a schematic diagram of the pupil center detection process in one embodiment of this application.

[0050] Figure 4 This is a schematic diagram of the pupil result identified by ellipse fitting in one embodiment of this application.

[0051] Figure 5 This is a schematic diagram of calibration data acquisition in one embodiment of this application.

[0052] Figure 6 This is a schematic diagram of the structure of an eye-tracking device according to an embodiment of this application.

[0053] Figure 7 This is a schematic diagram of the structure of an eye-tracking device in one embodiment of this application.

[0054] Figure 8 This is a schematic diagram of the main control module in one embodiment of this application.

[0055] Figure 9 This is a schematic diagram of the main control module in one embodiment of this application.

[0056] Figure 10 This is a schematic diagram of the working mode of the OV7251 global shutter image sensor in one embodiment of this application.

[0057] Figure 11 This is a schematic diagram of using the LT9211 chip to convert MIPI signals into TTL signals in one embodiment of this application.

[0058] Explanation of reference numerals in the attached diagram: 1. Frame; 2. Main control module; 3. Long-range camera; 4. Infrared eye camera; 5. Infrared light source; 6. PCB board No. 1; 7. PCB board No. 2. Detailed Implementation

[0059] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0060] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0061] Firstly, referring to Figure 1 An eye-tracking method disclosed in this application includes the following steps:

[0062] S1: Capture an image of the eye using an infrared eye camera, and perform image processing on the eye image to obtain the center of the pupil;

[0063] In an optional implementation, S1 includes:

[0064] S11: Convert the eye image to a grayscale image;

[0065] S12: Binarize the grayscale image;

[0066] S13: Perform contour detection on the binarized grayscale image to obtain the contour center, and use the contour center as the pupil center.

[0067] S14: Use the Hough circle detection function to find the circle whose center is closest to the center in a grayscale image;

[0068] S15: Display the detected outline on the original icon.

[0069] In another alternative implementation, S1 includes:

[0070] S11: Convert the eye image to a grayscale image;

[0071] S12: Binarize the grayscale image;

[0072] S13: Perform two iterative opening operations on the binarized grayscale image using a 3x3 convolution kernel;

[0073] S14: After the opening operation, the image is fitted with an ellipse for contour detection. The second largest contour among all fitted contours is taken as the pupil contour, and the center of the fitted ellipse is taken as the pupil center. S2: The eye is illuminated by an infrared light source to form a bright spot on the surface of the eyeball. Then, the eye image captured by the infrared eye camera is used to detect the bright spot, calculate the centroid of the bright spot, and then construct the eye vector using the centroid and the pupil center.

[0074] In an optional implementation, S2 specifically includes:

[0075] S21: Perform grayscale conversion on the eye image;

[0076] S22: Binarize the grayscale image processed in step S21, wherein a high threshold is set for the binarization function;

[0077] S23: Perform an opening operation on the image using a 3x3 convolution kernel;

[0078] S24: Perform contour detection on the binarized image processed in step S23 to obtain bright spot contours;

[0079] S25: Calculate the centroid of the bright spot using the outline of the bright spot, and then construct the eye vector using the centroid and the pupil center.

[0080] S3: Construct the mapping equation that maps the eye vector to scene coordinates;

[0081] In an optional implementation, S3 specifically includes:

[0082] S31: Acquire multiple sets of eye images and scene images on user gaze calibration cards to obtain multiple sets of eye movement vectors and scene camera coordinates;

[0083] S32: Substitute multiple sets of eye-tracking vectors and scene camera coordinates into the transformation equation from eye-tracking vectors to scene camera coordinates to solve for the parameters;

[0084] The transformation equation from the eye-tracking vector to the scene camera coordinates is:

[0085]

[0086] Where Xe and Ye are eye-tracking vectors, and Xs and Ys are scene camera coordinates.

[0087] S4: Use an infrared eye camera and a distant camera to capture the user's eye image and scene image respectively. Obtain the eye movement vector through the eye image, and then substitute the eye movement vector into the mapping equation in step S3 to obtain the screen gaze coordinates.

[0088] S5: Use the screen gaze coordinates to draw a heatmap and overlay it onto the scene image to obtain an image fused with the heatmap and the scene image.

[0089] In an optional implementation, S5 includes:

[0090] S51: Read the screen gaze coordinates identified by eye tracking;

[0091] S52: Put all screen gaze coordinates into a list variable named data, where data = [[x1, y1][x2, y2]...];

[0092] S53: Draw a heatmap and overlay the weighted heatmap onto the original scene image.

[0093] In S53: if the user gazes once at position (x, y), the user's interest value is set to be the largest at the center (x, y) and decreases linearly outwards, and then different colors are assigned according to different interest values.

[0094] In a specific embodiment, the eye-tracking method of this application will be described in detail below:

[0095] This application first captures an image of the eye using an infrared eye camera, and then obtains the pupil center through computer image processing. Next, an external infrared light source illuminates the eye, creating a bright spot on the surface of the eyeball. Since the formation of this bright spot is based on the positions of the light source and the camera, the spot remains in a fixed position during eye movement. Therefore, the vector obtained by subtracting the coordinates of the pupil center and the bright spot center can be called the eye movement vector, which essentially represents the eye movement vector coordinates (Xe, Ye).

[0096] (1) Pupil detection:

[0097] Pupil center detection was performed using both the Hough circle method and the ellipse fitting method:

[0098] Hough circle test: refer to Figure 2 This method is prone to false detection of other contours in grayscale images, and when the circles in the image are not smooth enough, the detection method is prone to false detection and thus error. Even if the grayscale image is binarized and then the Hough circle detection function is applied, the function will still fail to detect due to the irregularity of the circles obtained from the grayscale image.

[0099] Ellipse fitting: Reference Figure 3 After binarization, the binarized image undergoes an opening operation. This operation uses a 3x3 convolution kernel and performs two iterations. This step ensures that the contour detection function avoids detecting noisy contours (contours unrelated to the pupil contour). This resulting image can be called the image after the opening operation. The image after the opening operation then enters the contour detection function. The function places all detected contours into a matrix. We use a contour area calculation function to calculate and sort the area of ​​all contours. Then, we substitute the second largest contour in the matrix (which is actually the pupil contour) into the ellipse fitting function. In this function, we can calculate the center of the ellipse, and finally, we use the fitted ellipse center as the pupil center.

[0100] To extract the pupil contour, this application employs an area calculation method. The area of ​​all contours within the defined range is calculated, and then the calculated areas are sorted. The largest contour within the defined range is always the bounding box defined in this application; therefore, the largest contour among all calculated contours should be excluded. The pupil contour should always be the second largest in area among the contours. Therefore, the index of the second largest contour is extracted. Results show that ellipse fitting is more effective because during Hough circle detection, pupil movement causes deformation, making it impossible for the Hough circle detection function to detect a circular contour in the eye camera image. Therefore, ellipse fitting is ultimately chosen. The ellipse fitting function can fit the deformed pupil into an ellipse, thereby determining the pupil center. The pupil result identified by ellipse fitting is shown below. Figure 4 As shown.

[0101] (2) Highlight detection

[0102] The bright spot detection process first converts the input eye image to grayscale, then performs binarization thresholding. Because the brightness of the bright spots in the bright spot detection is very high, a higher threshold is set for the binarization function to improve the accuracy of bright spot extraction, allowing the bright spots to be better detected by the contours. After binarization, a 3x3 convolution kernel is used to perform an opening operation on the image to remove some noise points and improve contour detection. Due to the high threshold and the use of the opening operation, the processed image ultimately retains only the bright spots. The bright spot contours can be obtained directly from the contour detection image, and then the centroid of the bright spot is calculated for the calculation of the eye vector. Once the pupil center and the bright spot centroid are determined, the eye vector can be calculated.

[0103] (3) Calibration steps

[0104] Reference Figure 5 During the calibration process, this application uses a portable calibration card for calibration. When the user gazes at the calibration point on the card, this application will collect the user's eye image and the scene image. After gazing at the five calibration points respectively, this application will obtain five sets of data. Each set of data has an eye movement vector (Xe, Ye) and scene camera coordinates (Xs, Ys). Based on these five sets of data, the ai and bi of the system of equations are solved, and then the conversion equation from eye movement vector to scene camera coordinates can be obtained.

[0105]

[0106] (4) Mapping principle

[0107] A polynomial transformation function is used to map the vector between the pupil center and corneal scintillation to the corresponding gaze coordinates on the forehead screen. This mapping function can be expressed as: f:(Xe,Ye)→(Xs,Ys), where Xe,Ye and Xs,Ys represent the relationship between the eye movement vector coordinates and the screen coordinates, respectively.

[0108] The user inputs the points obtained during the calibration process into the equation, solves for the coefficients a and b, and finally obtains the mapping equation. Inputting the eye vector into this mapping equation maps the eye vector to screen coordinates. This application's portable calibration process involves inserting a portable calibration card into the eye-tracking device, allowing the user to focus on the calibration points marked on the card.

[0109] Heat map:

[0110] After eye tracking of the user, this application needs to visualize the identified gaze coordinates. To obtain the user's gaze information more clearly and intuitively, this application considers using heatmaps for visualization. Heatmaps can be easily drawn in Python; therefore, this application draws a heatmap based on the gaze coordinates in Python and overlays it onto the scene image. The final result is a semi-transparent image fused with the scene image. To achieve the above effect, the following three steps are required:

[0111] Step 1: Read the gaze coordinates identified by eye tracking.

[0112] Step 2: Put all the coordinates into a list variable named data, i.e., data = [[x1, y1][x2, y2]...]

[0113] Step 3: Draw a heat map (heat map size and scene) Figure 1 (This is done by overlaying the heatmap onto the original scene image with weighted weights).

[0114] The following method was used in step three:

[0115] (1) Bresenham's line algorithm and circle algorithm

[0116] To render a 2D heatmap, lines, circles, and rectangles need to be drawn, with drawing lines and circles being the most complex. The pyHeatMap library uses the Bresenham algorithm. The Bresenham line algorithm is used to draw a straight line determined by two points; it calculates the closest point on an n-dimensional raster. This algorithm uses only relatively fast integer addition, subtraction, and bit shifting, and is commonly used to draw straight lines in computer graphics. It is one of the earliest algorithms developed in computer graphics. The Bresenham circle algorithm, also known as the midpoint circle algorithm, is similar to the Bresenham line algorithm. Its basic method is to use a discriminant variable to determine the nearest pixel. The value of the discriminant variable can be calculated using only some addition, subtraction, and shifting operations. For simplicity, consider a circle centered at the origin, and only calculate the points on the eighth of the circumference; the points on the remaining circumference can be obtained using symmetry.

[0117] (2) Mapping

[0118] Based on the gaze coordinates, this application aims to understand which areas on the scene are the user's focus or ignored areas. To do this, it needs to infer the user's region of interest from the coordinate information and understand how the user's level of interest changes within this area. If the user gazes once at position (x, y), this application assumes that the user is interested in a circular area on the page centered at point (x, y) with radius r. This r can be 10 pixels or other reasonable values. Regarding the user's level of interest, it can be assumed that the user's interest value is maximum at the center (x, y), for example, a value of r, and decreases linearly outwards, decreasing by 1 for every pixel further from the center, reaching zero at the edge of the circle.

[0119] (3) Heat map color wheel

[0120] The mapping rule was established: each gaze point coordinate is mapped to a circle, with the center being the hottest and the heat decreasing outwards. Then, each heat level needs to be assigned a suitable color. The pyHeatMap library uses HSL colors, with the color wheel showing a transparency transition range of 61.8% on the left and a color transition range of 61.8% on the right, with the left 38.2% being pure blue.

[0121] Secondly, embodiments of this application also disclose an eye-tracking device, referring to... Figure 6 and Figure 7The device includes a frame 1 and a main control module 2. The frame 1 is equipped with a distant view camera 3, an infrared eye camera 4, and an infrared light source 5. The infrared eye camera 4 and the infrared light source 5 are located on the side of the frame 1 closer to the temples, while the distant view camera 3 is located on the side of the frame 1 away from the temples. The distant view camera 3, the infrared eye camera 4, and the infrared light source 5 are all connected to the main control module 2 via signal. The main control module 2 is used to obtain an image fused from a heat map and a scene map using the method described in the first aspect and to transmit the image to a communication terminal via wireless communication.

[0122] The infrared light source uses an infrared LED with a wavelength of 940nm. The main control module uses a USB to Type-C converter to power the infrared LED light source, and the circuit structure includes a 6-pin Type-C female connector slot.

[0123] In one specific embodiment, the eye-tracking device of this application will be described in detail below:

[0124] Reference Figure 6 and Figure 7 The eye-tracking device in this embodiment includes a frame 1 and a main control module 2. The frame 1 is equipped with a PCB board 6 and a PCB board 7. The PCB board 6 is mainly responsible for the construction of peripheral circuits such as scene cameras and infrared cameras, as well as power supply. The PCB board 7 connects all data to the host computer for data processing via a data cable. The data cable is led out through the PCB board 7.

[0125] In one implementation, the circuit diagram of the main control module is shown in Figure 9. The main control module adopts a USB 2.0 / USB 3.0 communication interface, which not only facilitates the transmission of image data but also provides power to the remote camera, infrared camera, and infrared light source. This ensures a stable power supply for the infrared light source. The main control module is developed using Python, and the OpenCV library integrates a large number of image processing packages, making it easier to call and port algorithms, thus reducing the product development cycle.

[0126] The telephoto camera uses an OV2659 model with a 135° wide-angle lens. This camera can capture all objects within the user's field of vision. It is fixed to the top front of the glasses frame to capture video streams from outside the frame. The infrared near-field camera uses a 2-megapixel 940nm lens, fixed to the lower right of the frame, to capture video streams of eye movements. Both the infrared near-field camera and the OV2659 telephoto camera can be manually focused. The refresh rate and frame rate of both cameras can be adjusted according to the image pixel size, and they support multiple operating systems (Windows, Linux, Android, Mac, OS). The cameras are small in size, facilitating system integration. The operating current of both cameras is moderate, reducing the burden on system heat dissipation. Both the telephoto and infrared near-field cameras use a driverless USB interface design, transmitting video data to the main control module in real time via USB cable. The main control module further processes and analyzes the captured video streams using software algorithms. The eye-tracking heatmap and trajectory map obtained after algorithm processing are transmitted to the mobile APP terminal via wireless communication protocol.

[0127] The LED light source uses infrared LEDs with a 940nm emission wavelength. This infrared LED light source uses a 0603-sized package, resulting in a small structural size that facilitates integration and further saves space in the eyeglass frame. Eight infrared LEDs are placed symmetrically on each of the left and right frames, encircling the glasses. Experimental tests were conducted to compare the performance of the eight LED light sources. The LED in the most suitable position was selected as the sole light source for subsequent experimental testing.

[0128] The Type-C interface circuit uses a 6-pin female connector. To ensure the stability and compatibility of the Type-C interface, the CC1 and CC2 ports of the Type-C connector are connected to the bottom through a 5.1K resistor. The 6-pin Type-C interface design not only has a simple circuit structure but is also easy to solder.

[0129] The eye-tracking device of this embodiment has the following technical advantages:

[0130] 1. This system uses infrared LEDs with a wavelength of 940nm as the light source. Currently, most commonly used infrared light sources on the market have a wavelength concentrated around 850nm. However, the 850nm near-infrared band still contains some visible light, which can affect human vision. The 940nm infrared wavelength essentially does not contain visible light signals, further improving system performance. This infrared LED light source uses a 0603 package, resulting in a smaller structural size and saving space in the eyeglass frame.

[0131] 2. A 940nm infrared camera and an OV2659 135° wide-angle camera are used as video acquisition modules for the human eye and outdoor scenes, respectively. Both cameras offer adjustable resolution and frame rate. The internal DSP chips in these cameras feature low latency video acquisition, with DSP processors achieving millisecond-level frame processing speeds, up to 0.1 seconds per frame. While maintaining automatic white balance, they also allow for focus adjustment. These cameras are compatible with multiple operating systems (including Windows, Linux, Android, Mac OS), and operate over a wide temperature range, making them suitable for various complex external environments. The moderate operating current reduces the burden on system heat dissipation.

[0132] 3. The main control module powers the 940nm infrared LEDs via a USB-to-Type-C circuit. The camera uses a driverless USB interface, which unifies the power supply of the entire system and improves the system's portability and stability. Since the main control module is developed using Python, the OpenCV library contains a wealth of image processing algorithms. Using OpenCV as the main control module makes it easier to process subsequent video stream algorithms.

[0133] 4. It adopts a 6-pin Type-C female connector slot, which has a high penetration rate and fewer pins, making the circuit design relatively simple and facilitating product adoption.

[0134] 5. The portable calibration device allows for eye-tracking calibration anytime, anywhere. Users simply plug in the device to begin the calibration process when needed.

[0135] In another implementation, refer to Figure 9 This solution uses an FPGA as the main control chip and an OV7251 as the infrared camera. The FPGA drives the OV7251 infrared near-field camera to acquire human eye video streams based on the SCCB protocol, and converts the MIPI protocol data stream output by the OV7251 into a TTL data stream that is easy for the FPGA to read through an LT9211. An OV7670 is used as the far-field camera, and the FPGA drives the OV7670 according to the DVP communication protocol. SDRAM is used as memory to cache the data acquired by the near-field and far-field cameras. An AP1117 is used as a voltage regulator chip to provide power to the infrared LED light source. A TMI3108 step-down chip provides 2.8V voltage to the OV7251.

[0136] In this embodiment, the OV7251 global shutter image sensor is selected. This is a 3.0-micron sensor with a 1 / 7.5-inch optical format, which can provide 120fps 640×480 resolution images, 180fps 320×240 resolution images, and 360fps 160×120 resolution images. The high frame rate provided by the OV7251 makes it an ideal solution for low-latency machine vision applications.

[0137] Since the OV7251 needs to output high frame rate images, this embodiment chooses to use an FPGA for image processing and is responsible for configuring the OV7251 to capture and output 60fps 640x480 resolution black and white eye images for subsequent processing needs.

[0138] Because the OV7251 has a single-channel MIPI serial output interface that conforms to the MIPI CSI2 interface standard, the standard's operating mode is as follows: Figure 10 The transmission modes are divided into two types: 1. HS high-speed transmission mode, used for transmitting burst data, synchronous transmission, the signal is a differential signal, and the level range is 100mV-300mV; 2. LP low-power mode, used for transmitting control commands, asynchronous transmission, the signal line is single-ended, the level range is 0-1.2V, there is no clock line, the clock is obtained by XORing the two data lines, and the speed is only 10Mbps.

[0139] For FPGAs, most FPGA chips cannot directly recognize MIPI signals with two different level standards. Therefore, before the FPGA reads image data, a dedicated chip, LT9211, is used to convert the MIPI signal into a TTL signal for the FPGA to receive. Figure 11 As shown. The following points should be noted in the circuit design:

[0140] 1. Before routing the traces, do a good job of layout, plan the placement of inputs, outputs and power supplies to reduce PCB size;

[0141] 2. Drill more holes in the empty space of the circuit board to facilitate heat dissipation;

[0142] 3. The crystal oscillator traces should be appropriately widened to ensure clock signal quality. The clock traces should ideally be grounded to reduce EMI.

[0143] 4. For the differential signal lines output by OV7251, the traces should be of equal length to ensure that they reach LT9211 at the same time. The reference plane should preferably be a complete ground plane to avoid the differential lines crossing the power plane.

[0144] 5. Because the LT9211 chip is packaged in a QFN package, the chip pin pitch is too small, resulting in the proximity of the adjacent pin traces to the vias being too close under a 10mil aperture, which cannot guarantee the safe distance between the vias and the traces.

[0145] The glasses-type eye-tracking device described in this application requires a calibration process before it can be used normally. After completing the device calibration, users can move freely and achieve remote eye tracking within the WIFI coverage area by wearing our monocular or binocular portable eye-tracking device. Image data can be displayed via a personal computer. The PC can view the scene camera images on the wearable device and mark the calculated gaze point position in the scene camera, allowing the PC to detect the object the user is currently looking at in real time.

[0146] The eye-tracking device proposed in this invention not only has a simple overall circuit structure and is easy to wear, but is also small in size and easy to carry. Since it only collects video signals, the computational requirements of the main control module are reduced. By adjusting the appropriate algorithm, it is possible to achieve a better improvement in the accuracy of gaze point estimation.

[0147] The specific embodiments of this application have been described above, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0148] In the description of this application, it should be understood that the terms "upper," "lower," "inner," "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this application and for simplification, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The simple fact that certain measures are recited in mutually different dependent claims does not indicate that combinations of these measures cannot be used for improvement. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. An eye-tracking method, characterized in that: The method includes the following steps: S1: Capture an image of the eye using an infrared eye camera, and perform image processing on the eye image to obtain the pupil center; S2: Illuminate the eye with an infrared light source to form a bright spot on the surface of the eyeball. Then, detect the bright spot by performing bright spot detection on the eye image captured by the infrared eye camera, calculate the centroid of the bright spot, and then construct the eye vector using the centroid and the pupil center. S3: Construct a mapping equation that maps eye vectors to scene coordinates. The mapping equation is a quadratic polynomial transformation equation from eye movement vectors to scene camera coordinates. Multiple sets of eye images and scene images are collected from multiple user gaze calibration cards to obtain multiple sets of eye movement vectors and scene camera coordinates. Substitute the multiple sets of eye movement vectors and scene camera coordinates into the transformation equation from eye movement vectors to scene camera coordinates to solve for the parameters. The transformation equation from the eye-tracking vector to the scene camera coordinates is: ； ； Where Xe and Ye are eye-tracking vectors, and Xs and Ys are scene camera coordinates; S4: Use an infrared eye camera and a distant camera to capture the user's eye image and scene image respectively. Obtain the eye movement vector through the eye image, and then substitute the eye movement vector into the mapping equation in step S3 to obtain the screen gaze coordinates. S5: Use the screen gaze coordinates to draw a heat map and overlay it onto the scene image to obtain a fused image of the heat map and the scene image. The method for drawing the fused image is as follows: if the user gazes once at position (x, y), the user's interest value is set to be the maximum at the center (x, y) and decreases linearly outwards. Then, different colors are assigned according to different interest values.

2. The eye-tracking method according to claim 1, characterized in that: S1 includes: S11: Convert the eye image to a grayscale image; S12: Binarize the grayscale image; S13: Perform contour detection on the binarized grayscale image to obtain the contour center, and use the contour center as the pupil center. S14: Use the Hough circle detection function to find the circle whose center is closest to the center in a grayscale image; S15: Display the detected outline on the original icon.

3. The eye-tracking method according to claim 1, characterized in that: S1 includes: S11: Convert the eye image to a grayscale image; S12: Binarize the grayscale image; S13: Perform two iterative opening operations on the binarized grayscale image using a 3x3 convolution kernel; S14: After the opening operation, the image is used to perform contour detection by ellipse fitting. The second largest contour among all the fitted contours is taken as the pupil contour, and the center of the fitted ellipse is taken as the pupil center.

4. The eye-tracking method according to claim 1, characterized in that: S2 specifically includes: S21: Perform grayscale conversion on the eye image; S22: Binarize the grayscale image processed in step S21, wherein a threshold is set for the binarization function; S23: Perform an opening operation on the image using a 3x3 convolution kernel; S24: Perform contour detection on the binarized image processed in step S23 to obtain bright spot contours; S25: Calculate the centroid of the bright spot using the outline of the bright spot, and then construct the eye vector using the centroid and the pupil center.

5. The eye-tracking method according to claim 4, characterized in that: S5 includes: S51: Read the screen gaze coordinates identified by eye tracking; S52: Put all screen gaze coordinates into a list variable named data, where data = [[x1, y1][x2, y2]...]; S53: Draw a heatmap and overlay the weighted heatmap onto the original scene image.

6. An eye-tracking device, characterized in that: The device includes a frame and a main control module. The frame is equipped with a distant-view camera, an infrared eye camera, and an infrared light source. The infrared eye camera and the infrared light source are located on the side of the frame closer to the temples, and the distant-view camera is located on the side of the frame away from the temples. The distant-view camera, the infrared eye camera, and the infrared light source are all signal-connected to the main control module. The main control module is used to obtain an image fused from a heat map and a scene map using the method described in any one of claims 1-5 and to transmit the image to a communication terminal via wireless communication.

7. An eye-tracking device according to claim 6, characterized in that: The infrared light source is an infrared LED light source with an emission wavelength of 940nm.

8. An eye-tracking device according to claim 6, characterized in that: The main control module uses a USB to Type-C circuit structure to power the infrared LED light source. The circuit structure includes a 6-pin Type-C female connector slot.

Images