Eye tracking method, device, electronic device and readable storage medium

Abstract

The application discloses an eyeball tracking method and device, electronic equipment and readable storage medium, and is used for improving the accuracy of eyeball positioning. The method comprises the following steps: acquiring an eyeball image in real time through a camera; determining an eyeball tracking mode according to an initial frame of the eyeball image; and performing eyeball positioning on the eyeball image based on the eyeball tracking mode to obtain eyeball position information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an eyeball tracking method and device, electronic equipment and readable storage medium. BACKGROUND

[0002] Eyeball tracking technology is a key technology widely used in augmented reality, virtual reality, medical detection and human-computer interaction scenarios in recent years. One of its core tasks is to accurately obtain the spatial position information of the eyeball in the image. Existing eyeball positioning methods rely on traditional image processing algorithms, such as gray distribution analysis, shape fitting, Hough circle detection, etc. to estimate the eyeball center. These methods can achieve certain accuracy in ideal environments with high-quality static images and uniform illumination, but in actual applications, they often face the influence of complex factors such as image noise interference, illumination changes, partial occlusion, and insufficient image resolution, resulting in unstable eyeball positioning results and decreased accuracy.

[0003] In addition, to improve processing efficiency, some systems only analyze global features of the image, lacking deep mining and dynamic response to local features, making it difficult to adapt to small changes in the position of the eyeball in consecutive frames, and prone to drift, jitter or misjudgment. Especially in the case where no effective ROI region is set or the initial position estimation is not accurate, the calculation result of the eyeball center fluctuates greatly, affecting the response accuracy of the subsequent tracking or control system.

[0004] In summary, the existing technology is difficult to achieve stable and high-precision eyeball positioning in complex environments, and there is an urgent need for an eyeball image processing scheme that can enhance the positioning accuracy in multiple dynamic scenes to improve the overall performance of the eyeball tracking system. SUMMARY

[0005] The present application provides an eyeball tracking method, device, electronic equipment and readable storage medium, which can improve the accuracy of eyeball positioning.

[0006] The first aspect of the present application provides an eyeball tracking method, comprising:

[0007] obtaining an eyeball image in real time through a camera;

[0008] determining an eyeball tracking mode according to an initial frame of the eyeball image;

[0009] performing eyeball positioning on the eyeball image based on the eyeball tracking mode to obtain eyeball position information.

[0010] Optionally, the step of determining an eyeball tracking mode according to an initial frame of the eyeball image comprises:

[0011] Determine an eye tracking mode according to whether initial position information is acquired from an initial frame of the eye image, the initial position information being information of a position of an eye in the initial frame.

[0012] Optionally, when the initial position information is not acquired, the eye positioning of the eye image based on the eye tracking mode is performed to acquire the eye position information, and the eye positioning includes:

[0013] Separate RGB channels of the eye image and perform blue channel extraction to acquire a blue channel image;

[0014] Determine an ROI based on the blue channel image;

[0015] Calculate a gradient vector of each pixel point in the ROI;

[0016] Calculate a first displacement vector of each pixel point in the ROI according to a center point of the ROI and the gradient vector;

[0017] Calculate a voting value of each pixel point in the ROI according to the gradient vector and the first displacement vector;

[0018] Determine eye position information according to the voting value.

[0019] Optionally, when the initial position information is acquired, the eye positioning of the eye image based on the eye tracking mode is performed to acquire the eye position information, and the eye positioning includes:

[0020] Extract multi-channel features of the eye image;

[0021] Call a preset filter to perform processing on the multi-channel features to generate a response map;

[0022] Determine a maximum response point according to the response map;

[0023] Calculate a second displacement vector according to the maximum response point;

[0024] Calculate eye position information according to the second displacement vector and the initial position information.

[0025] Optionally, after the maximum response point is determined according to the response map, the method further includes:

[0026] Calculate a tracking value according to the maximum response point;

[0027] Determine whether tracking is successful according to the tracking value and a preset threshold;

[0028] The calculation of the second displacement vector according to the maximum response point includes:

[0029] When it is determined that the tracking is successful, a second displacement vector is calculated according to the maximum response point.

[0030] Optionally, after the eye positioning is performed on the eye image based on the eye tracking manner to obtain the eye position information, the method further comprises:

[0031] The eye position information of the current frame is compared with the eye position information of a previous frame to determine whether the eye position information of the current frame is updated.

[0032] Optionally, after the eye positioning is performed on the eye image based on the eye tracking manner to obtain the eye position information, the method further comprises:

[0033] The eye position information is sent to a light machine control system, so that the light machine control system calculates an irradiation position according to the eye position information and irradiates the eye according to the irradiation position.

[0034] The second aspect of the present application provides an eye tracking device, comprising:

[0035] An acquisition unit is configured to acquire eye images in real time through a camera;

[0036] A determination unit is configured to determine an eye tracking manner according to an initial frame of the eye images;

[0037] A positioning unit is configured to perform eye positioning on the eye images based on the eye tracking manner to obtain eye position information.

[0038] Optionally, the determination unit is specifically configured to:

[0039] Determine the eye tracking manner according to whether initial position information of an eye position in the initial frame of the eye images is acquired.

[0040] Optionally, when the initial position information is not acquired, the positioning unit comprises:

[0041] A separation module is configured to separate RGB channels of the eye images and perform blue channel extraction to obtain a blue channel image;

[0042] A first determination module is configured to determine an ROI based on the blue channel image;

[0043] A first calculation module is configured to calculate a gradient vector of each pixel point in the ROI;

[0044] A second calculation module is configured to calculate a first displacement vector of each pixel point in the ROI according to a center point of the ROI and the gradient vector;

[0045] a third calculation module, configured to calculate a voting value of each pixel in the ROI according to the gradient vector and the first displacement vector;

[0046] The second determining module is used to determine the eye position information according to the voting value.

[0047] Optionally, when the initial position information is acquired, the positioning unit includes:

[0048] An extraction module, configured to extract multi-channel features of the eye image;

[0049] A calling module, configured to call a preset filter to process the multi-channel features to generate a response map;

[0050] a third determining module, configured to determine a maximum response point according to the response graph;

[0051] a fourth calculation module, configured to calculate a second displacement vector according to the maximum response point;

[0052] A fifth calculation module is used to calculate eye position information based on the second displacement vector and the initial position information.

[0053] Optionally, the positioning unit further includes:

[0054] a sixth calculation module, configured to calculate a tracking value according to the maximum response point;

[0055] A judgment module, configured to judge whether the tracking is successful based on the tracking value and a preset threshold;

[0056] The fourth calculation module is specifically configured to:

[0057] When it is determined that the tracking is successful, a second displacement vector is calculated according to the maximum response point.

[0058] Optionally, the device further includes an updating unit, configured to:

[0059] The eye position information of the current frame is compared with the eye position information of the previous frame to determine whether to update the eye position information of the current frame.

[0060] Optionally, the device further includes a sending unit, wherein the sending unit is configured to:

[0061] The eyeball position information is sent to an optical-mechanical control system, so that the optical-mechanical control system calculates an irradiation position according to the eyeball position information and irradiates the eye according to the irradiation position.

[0062] A third aspect of the present application provides an electronic device, including:

[0063] processor, memory, input and output units, and buses;

[0064] The processor is connected to the memory, the input and output unit, and the bus;

[0065] A program is stored in the memory, and the processor calls the program to execute the method in the first aspect and any possible implementation of the first aspect.

[0066] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the computer executes the method in the first aspect and any possible implementation of the first aspect.

[0067] It can be seen from the above technical solutions that this application has the following advantages:

[0068] In this embodiment, by determining the most appropriate eye tracking method based on the initial frame, a strategy that closely matches the current environment and image quality can be selected during subsequent positioning, significantly improving eye positioning accuracy. Furthermore, this embodiment can reduce data acquisition latency and continuously and stably output eye position information, providing reliable data for subsequent interactions and enhancing the smoothness and accuracy of the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a flowchart of an embodiment of the eye tracking method in this application;

[0070] Figure 2 A flowchart of an embodiment of determining an eye tracking method in this application;

[0071] Figure 3 This is a flowchart of an embodiment of obtaining eye position information in this application;

[0072] Figure 4 This is a flowchart of another embodiment of obtaining eye position information in this application;

[0073] Figure 5 This is a flowchart of an embodiment of determining whether tracking is successful in this application;

[0074] Figure 6 This is a flowchart of an embodiment of updating the eye position information of the current frame in this application;

[0075] Figure 7 This is a schematic structural diagram of an embodiment of an eye tracking device in this application;

[0076] Figure 8This is a schematic structural diagram of an embodiment of an electronic device in this application. DETAILED DESCRIPTION

[0077] Embodiments of the present application provide an eye tracking method, device, electronic device, and readable storage medium for improving the accuracy of eye positioning.

[0078] The method of the present application can be applied to a server, a terminal or other devices with logic processing capabilities, and the present application does not limit this. For the convenience of description, the following description is based on the example of a terminal as the execution subject.

[0079] The embodiments of the present application will be described below with reference to the accompanying drawings.

[0080] See also Figure 1 , Figure 1 An embodiment of the eye tracking method provided by the present application includes:

[0081] 101. Obtain eyeball images in real time through a camera;

[0082] The terminal calls the camera driver interface, opens the camera, and sets the appropriate resolution (e.g., 640×480 or higher) and frame rate (e.g., 30 fps). It then enters looped acquisition mode, continuously reading each frame from the camera buffer. Furthermore, to improve subsequent processing quality, the terminal can preprocess each frame, including using Gaussian filtering to remove noise, automatic exposure, or brightness correction to stabilize image brightness.

[0083] 102. Determine an eye tracking method according to an initial frame of the eye image;

[0084] The terminal analyzes the overall position and image quality of the eyeball in the first frame image, determines which eye tracking method is suitable for the current environment, and saves the information of the method locally in the terminal so that the same strategy can be used for subsequent images.

[0085] 103. Eye positioning is performed on the eye image based on an eye tracking method to obtain eye position information.

[0086] The terminal determines and locates the eye position in each frame of the image according to the previously determined eye tracking method, and continuously updates the eye position information of the current frame.

[0087] In this embodiment, by determining the most appropriate eye tracking method based on the initial frame, the terminal can select a strategy that closely matches the current environment and image quality during subsequent positioning, significantly improving eye positioning accuracy. Furthermore, this embodiment can reduce data acquisition latency and continuously and stably output eye position information, providing reliable data for subsequent interactions and enhancing the smoothness and accuracy of the user experience.

[0088] Referring to Figure 2 In some embodiments of the present application, step 102 in the above embodiment determines the eye tracking mode according to the initial frame of the eye image, which can include the following steps:

[0089] 201. Determine the eye tracking mode according to whether the initial position information is obtained from the initial frame of the eye image, and the initial position information is the information of the eye position in the initial frame.

[0090] The terminal analyzes the initial frame of the eye image and attempts to extract the initial position information of the eye from it. If the eye position in the initial frame is successfully extracted, it indicates that the image is clear and the features are obvious, and the terminal selects the tracking mode with higher precision and relying on the initial position information accordingly; if the initial position information is not extracted, it indicates that the image quality is low or the features are not obvious, and the terminal selects the tracking mode with higher robustness and adaptability to ensure the stability of the subsequent eye positioning process. Among them, the eye tracking mode includes the initial ROI-free mode and the initial ROI mode.

[0091] In actual application, the terminal can use a pre-trained model to extract the initial position information of the initial frame of the eye image, or use a manually labeled method to extract the initial position information. If it is extracted, it can be preliminarily determined to select the initial ROI mode for eye tracking in the subsequent. At this time, the algorithm corresponding to the initial ROI mode can be initialized, which can include saving the initial ROI, creating a Hanning window, extracting multi-channel features (including gray, HOG and color features), initializing the channel weight as a uniform distribution, creating a Gaussian response function, and creating an initial filter model. If the algorithm initialization is successful, the initial ROI mode is finally determined to be selected for eye tracking. Otherwise, the initial ROI-free mode is used for eye tracking.

[0092] In this embodiment, the terminal matches the appropriate eye tracking mode by judging whether the initial frame contains eye position information, so that the subsequent eye positioning process can maintain high accuracy under different image conditions. At the same time, the terminal dynamically switches the strategy according to the image quality, which helps to improve the overall stability and adaptability of eye tracking.

[0093] Referring to Figure 3 In some embodiments of the present application, when the initial ROI-free mode is selected for eye tracking, step 103 in the above embodiment performs eye positioning on the eye image based on the eye tracking mode to obtain eye position information, which can include the following steps:

[0094] 301. Separate the RGB channels of the eye image and perform blue channel extraction to obtain a blue channel image;

[0095] After acquiring the eye image, the terminal first splits the image into three channels: red, green, and blue. The blue channel image is then extracted for subsequent processing. The blue channel provides more distinct contrast for the edges and structures of the eye region, enhancing the effectiveness of subsequent gradient calculations. This operation can be performed directly using the image channel separation function, resulting in an image containing only the blue component.

[0096] 302. Determine ROI based on the blue channel image;

[0097] The terminal determines the ROI within the image based on the blue channel results of the current image, reducing computational complexity and improving positioning accuracy. Specifically, if the approximate eye position coordinates are known, the ROI is set to a local rectangular area centered on that point, with a width and height typically set to 20% of the image size. If the approximate eye position coordinates are unknown, the ROI defaults to the central area of ​​the image (e.g., 20%–80% horizontally and 10%–90% vertically), and a median filter (e.g., a 5×5 window) is used to process the ROI to reduce noise.

[0098] 303. Calculate the gradient vector of each pixel in the ROI;

[0099] For each pixel in the ROI, the terminal calculates the horizontal and vertical variation (i.e., gradient) and further calculates the gradient magnitude and direction. To eliminate interference, the terminal sets a dynamic threshold based on histogram analysis, retaining only pixels with gradient magnitudes greater than the threshold and normalizing the gradient vectors of these pixels to unit vectors.

[0100] The terminal calculates the horizontal and vertical image gradients for each pixel in the ROI area to obtain the gradient vector of each point. The calculation method can use the central difference method and normalize the gradient amplitude, retaining only the amplitude greater than the dynamic threshold. The gradient information of the pixel point.

[0101] Formula 1

[0102] Formula 2

[0103] Formula 3

[0104] Formula 4

[0105] Formula 5

[0106] in, is the blue channel image value; is the gradient in the x direction; is the gradient in the y direction; is the gradient magnitude; is the dynamic threshold; 、 is the normalized effective gradient component.

[0107] 304. Calculate a first displacement vector for each pixel in the ROI according to the center point and the gradient vector of the ROI;

[0108] The terminal sets the center point of the ROI as the candidate eye center, calculates the displacement vector from the center point to each pixel with a valid gradient, and performs normalization processing to prepare for subsequent voting.

[0109] 305. Calculate the voting value of each pixel in the ROI according to the gradient vector and the first displacement vector;

[0110] The terminal calculates the voting value of each pixel for the candidate eye center by taking the dot product of the gradient vector and the normalized displacement vector, taking the square of the non-negative part, multiplying it by the weight of the pixel, and finally adding it to the voting value corresponding to the candidate center point. The voting value calculation formula is as follows:

[0111] Formula 6

[0112] in, It is the set of all pixels in the ROI whose normalized gradient vector is not zero; , for point The displacement vector relative to the candidate center; is the length of the first displacement vector; for point The weight of the image is usually the result of inverse Gaussian blur. Candidate Center Cumulative voting.

[0113] 306. Determine eyeball position information according to the voting value.

[0114] The terminal searches for the pixel with the largest cumulative voting value in the voting map, considers it to be the most likely eye center position, and outputs its coordinates as the eye position information in the current frame image.

[0115] In this embodiment, the terminal extracts enhanced image features through the blue channel, defines the ROI region based on a priori or default strategies, calculates effective gradients, constructs displacement vectors for gradient voting, and ultimately locates the eye center using the accumulated voting values. This achieves a technical approach for efficient eye localization without the need for template matching or complex models. The close coordination of these steps makes eye localization more robust and more adaptable to lighting changes and image noise, thereby improving the accuracy and stability of eye position information.

[0116] See also Figure 4 In some embodiments of the present application, when the initial ROI method is selected for eye tracking, step 103 in the above embodiment performs eye positioning on the eye image based on the eye tracking method to obtain eye position information, which may include the following steps:

[0117] 401. Extract multi-channel features of the eye image;

[0118] The terminal extracts multiple feature channels from the current eye image frame. Each feature channel reflects the image information in different perceptual dimensions, such as texture, edge or color, to prepare for subsequent filtering and matching.

[0119] 402. Calling a preset filter to process the multi-channel features to generate a response map;

[0120] The terminal calls a preset filter (i.e., the filter model when the algorithm is initialized in step 201), convolves each channel feature with the corresponding filter, and weighted sums the convolution results of all channels to obtain a comprehensive response graph.

[0121] Formula 7

[0122] in, represents the image features of the i-th channel; represents the corresponding filter; Represents the weight coefficient of the i-th channel; Represents a two-dimensional convolution operation; n represents the number of channels; Indicates a point The response value at is used to reflect the degree of matching.

[0123] 403. Determine the maximum response point based on the response graph;

[0124] Terminal traversal response graph , find the position with the maximum response value as the most likely position of the target in the current frame.

[0125] Formula 8

[0126] in, Indicates the coordinates of the point with the largest response value in the response graph; Used to return The largest Location.

[0127] 404. Calculate a second displacement vector according to the maximum response point;

[0128] The terminal calculates the offset of the maximum response point relative to the initial position as the second displacement vector, and limits it to within the maximum allowable displacement range.

[0129] Formula 9

[0130] Formula 10

[0131] Formula 11

[0132] in, Indicates the width of the response graph; Indicates the height of the response graph; Indicates the offset from the center point of the response graph to the maximum response point; Indicates the x-axis offset; Indicates the y-axis offset; represents the maximum allowable displacement, usually set to half the size of the target area.

[0133] 405. Calculate eyeball position information according to the second displacement vector and the initial position information.

[0134] The terminal sets the eyeball position information of the current frame to the sum of the position of the previous frame and the second displacement vector, thereby updating the eyeball center position. The calculation formula is as follows:

[0135] Formula 12

[0136] in, Indicates the eye center coordinates of the previous frame; Indicates the eye center coordinates of the current frame; indicates the second displacement vector.

[0137] In this embodiment, the terminal achieves precise positioning based on image response by extracting multi-channel features, applying filters to generate a response map, and calculating the position information of the eyeball based on the maximum response point; among them, the maximum response point provides the position indication with the highest intensity, and the displacement vector calculated based on this point combined with the initial position information can dynamically correct the coordinates of the center of the eyeball, so that the system can maintain strong target recognition capabilities and positioning stability even under a certain degree of image blur, occlusion or lighting changes, thereby improving the accuracy and real-time performance of eye tracking.

[0138] See also Figure 5In some embodiments of the present application, after step 403 in the above embodiment determines the maximum response point according to the response map, the eye tracking method can further include the following steps:

[0139] 501. Calculate a tracking value according to the maximum response point;

[0140] After the terminal obtains the maximum response point, it calculates the contrast between the peak value of the response map and the surrounding background area as the tracking value of the frame. The PSR (Peak to Sidelobe Ratio) calculation method is adopted. The larger the PSR value, the more obvious the difference between the peak value and the background, and the better the tracking effect. The calculation formula is as follows:

[0141] Formula 13

[0142] wherein, represents the maximum value in the response map, corresponding to the maximum response point; represents the average value of the sidelobe area (i.e. the remaining area after excluding the peak value area); represents the standard deviation of the sidelobe area.

[0143] The terminal defines a peak value area (usually 5*5 in size according to requirements) centered on the maximum response point. After excluding this area, the average value and standard deviation of the remaining sidelobe area pixels are calculated to obtain the PSR value.

[0144] 502. Determine whether tracking is successful according to the tracking value and a preset threshold;

[0145] The terminal compares the calculated PSR value with the preset threshold. If it is greater than the threshold, it is determined that the tracking of the current frame is successful, otherwise it is considered that the tracking fails, thereby triggering a repositioning mechanism or using the position of the previous frame for correction.

[0146] When it is determined that the tracking is successful, the terminal continues to execute step 404.

[0147] In this embodiment, the terminal calculates the PSR value after obtaining the maximum response point and compares it with the preset threshold. Only when the PSR is higher than the threshold is the tracking considered successful, thereby automatically filtering low-quality positioning before the result is output, improving the accuracy of the eye position information. At the same time, the threshold judgment enables the terminal to trigger repositioning or use the position of the previous frame for correction in a timely manner when the tracking fails, reducing the jitter caused by displacement jumps and improving the stability and robustness of the overall tracking.

[0148] Please refer to Figure 6 In some embodiments of the present application, after step 103 in the above embodiment performs eye positioning on the eye image based on the eye tracking method to obtain eye position information, the eye tracking method can further include the following steps:

[0149] 601. Compare the eye position information of the current frame with the eye position information of the previous frame to determine whether to update the eye position information of the current frame.

[0150] After obtaining the eye position information of the current frame, the terminal compares this information with the eye position of the previous frame to determine whether to perform an update operation. Specifically, the terminal uses a position smoothing strategy. In each frame, the current detection value is not directly used as the eye position. Instead, it is combined with the smoothing result of the previous frame and the changing trend of the positioning result is controlled through an update mechanism. If the difference between the current detection result and the smoothing value of the previous frame exceeds the set threshold, it is considered a valid displacement and the terminal will update the eye position of the current frame; if the difference is small, it is considered to be a detection error or jitter, and the terminal will continue to maintain the result of the previous frame to reduce the error accumulation or picture jitter caused by high-frequency noise.

[0151] For example, the terminal can calculate the smoothing value of the eye position information of the current frame based on the eye position information of the previous frame and the eye position information of the current frame, and then calculate the pixel offset between the smoothing value and the eye position information of the previous frame. If the offset is greater than a set threshold, the smoothing value is used as the latest eye position information. The smoothing calculation formula is as follows:

[0152] Formula 14

[0153] Formula 15

[0154] in, is the current smoothed value; is the smoothed value of the previous moment; is the current measurement value; is the impact factor, which can be adjusted according to actual conditions; The preset pixel offset threshold.

[0155] 602. Send the eyeball position information to the optical control system, so that the optical control system calculates the irradiation position according to the eyeball position information and irradiates the eye according to the irradiation position.

[0156] After obtaining accurate eye position information, the terminal transmits this information as a control parameter to the optomechanical control system to guide the optomechanical illumination operation. To ensure real-time and accurate control, the terminal uses a specific transmission mechanism to promptly transmit the pixel coordinates of the eye center position and trigger the optomechanical control system to perform position conversion. After receiving these coordinates, the optomechanical control system maps the pixel coordinates to the control coordinates required by the optomechanical control system based on the conversion relationship between the camera coordinates and the optomechanical coordinates. It then adjusts the illumination position and direction accordingly to ensure that the light spot accurately falls on the target area.

[0157] For example, the terminal obtains the eyeball center position as pixel coordinates (x, y) in the current frame image, and transmits it to the light machine control system in a form of less than 5 millisecond delay. The light machine control system converts the pixel coordinates into target control coordinates (x', y') using the coordinate transformation matrix in the configuration file, and the conversion formula is as follows:

[0158] Formula 16

[0159] Where (x, y) is the pixel position in the camera coordinate system (that is, the eyeball position information sent by the terminal); (x', y') is the position required by the light machine control system (that is, the irradiation information calculated by the light machine control system); 、 、 、 、 、 is a preset value for describing the translation, rotation, and scaling relationship of affine transformation.

[0160] It should be noted that in the embodiment, the terminal can only execute any one of steps 601 or 602, or can execute both steps 601 and 602. Moreover, there is no strict execution order between steps 601 and 602. If the terminal executes both, the terminal can execute step 601 first and then execute step 602, or can execute step 602 first and then execute step 601.

[0161] In the embodiment, the terminal compares the previous frame data when judging whether to update the current frame eyeball position information, to ensure that the positioning result of the current frame is more reliable, thereby improving the accuracy of the final eyeball positioning. At the same time, the terminal transmits the updated eyeball position information to the light machine control system in a timely manner, and cooperates with coordinate transformation and fast response, to realize real-time linkage between eyeball tracking and light control.

[0162] Referring to Figure 7 , Figure 7 An embodiment of an eyeball tracking device provided in the present application includes:

[0163] The acquisition unit 701 is configured to acquire an eyeball image in real time through a camera.

[0164] The determination unit 702 is configured to determine an eyeball tracking mode according to an initial frame of the eyeball image.

[0165] The positioning unit 703 is configured to perform eyeball positioning on the eyeball image based on the eyeball tracking mode, to obtain eyeball position information.

[0166] In this embodiment, by determining the most suitable eye tracking mode according to the initial frame, the eye tracking device can select a strategy that is highly matched with the current environment and image quality in the subsequent positioning process, thereby significantly improving the accuracy of eye positioning. At the same time, this embodiment can also reduce data acquisition delay and continuously and stably output eye position information, providing reliable data for subsequent interaction and improving the smoothness and accuracy of the use experience.

[0167] Optionally, the determination unit 702 is specifically configured to:

[0168] Determine the eye tracking mode according to whether the initial position information is acquired from the initial frame of the eye image, the initial position information being information of a position of the eye in the initial frame.

[0169] Optionally, when the initial position information is not acquired, the positioning unit 703 includes:

[0170] A separation module configured to separate RGB channels of the eye image and perform blue channel extraction to acquire a blue channel image;

[0171] A first determination module configured to determine an ROI based on the blue channel image;

[0172] A first calculation module configured to calculate a gradient vector of each pixel point in the ROI;

[0173] A second calculation module configured to calculate a first displacement vector of each pixel point in the ROI according to a center point of the ROI and the gradient vector;

[0174] A third calculation module configured to calculate a voting value of each pixel point in the ROI according to the gradient vector and the first displacement vector;

[0175] A second determination module configured to determine the eye position information according to the voting value.

[0176] Optionally, when the initial position information is acquired, the positioning unit 703 includes:

[0177] An extraction module configured to extract multi-channel features of the eye image;

[0178] A calling module configured to call a preset filter to perform processing on the multi-channel features to generate a response map;

[0179] A third determination module configured to determine a maximum response point according to the response map;

[0180] A fourth calculation module configured to calculate a second displacement vector according to the maximum response point;

[0181] A fifth calculation module configured to calculate the eye position information according to the second displacement vector and the initial position information.

[0182] Optionally, the positioning unit 703 further comprises:

[0183] The sixth calculation module is configured to calculate the tracking value according to the maximum response point.

[0184] The judging module is configured to judge whether the tracking is successful according to the tracking value and the preset threshold.

[0185] The fourth calculation module is specifically configured to:

[0186] When it is determined that the tracking is successful, the second displacement vector is calculated according to the maximum response point.

[0187] Optionally, the device further comprises an updating unit, and the updating unit is configured to:

[0188] The eye position information of the current frame is compared with the eye position information of the previous frame to determine whether the eye position information of the current frame is updated.

[0189] Optionally, the device further comprises a sending unit, and the sending unit is configured to:

[0190] The eye position information is sent to the light machine control system, so that the light machine control system calculates the irradiation position according to the eye position information, and irradiates the eye according to the irradiation position.

[0191] In the embodiment, the functions of each unit and module correspond to the steps in the foregoing Figures 1 to 6 embodiments, and thus will not be described herein.

[0192] Please refer to Figure 8 , Figure 8 An embodiment of an electronic device provided in the present application includes:

[0193] The processor 801, the memory 802, the input / output unit 803, and the bus 804;

[0194] The processor 801 is connected with the memory 802, the input / output unit 803, and the bus 804.

[0195] The memory 802 stores a program, and the processor 801 invokes the program to perform the steps in the foregoing Figures 1 to 6 embodiments.

[0196] In the embodiment, the functions of the processor 801 correspond to the steps in the foregoing Figures 1 to 6 embodiments, and thus will not be described herein.

[0197] The present application further provides a computer readable storage medium, and the computer readable storage medium stores a program, and the program causes the computer to perform the method in the foregoing Figures 1 to 6 any one of the possible embodiments when the program is executed on the computer.

[0198] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0199] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0200] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0201] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0202] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, read-only memory), a random access memory (RAM, random access memory), a magnetic disk or an optical disk, and various program code storage media.

Claims

1. An eye tracking method, characterized in that: include: Acquire eyeball images in real time through a camera; determining an eye tracking mode according to an initial frame of the eye image; Performing eye positioning on the eye image based on the eye tracking method to obtain eye position information; The determining of the eye tracking mode according to the initial frame of the eye image includes: determining an eye tracking mode according to whether initial position information is obtained from an initial frame of the eye image, wherein the initial position information is information about the eye position in the initial frame; When the initial position information is not obtained, performing eye positioning on the eye image based on the eye tracking method to obtain eye position information includes: Separating the RGB channels of the eyeball image and performing blue channel extraction to obtain a blue channel image; Determine a ROI based on the blue channel image; Calculate the gradient vector of each pixel in the ROI; Calculate a first displacement vector for each pixel in the ROI according to the center point of the ROI and the gradient vector; Calculate the voting value of each pixel in the ROI according to the gradient vector and the first displacement vector; determining eyeball position information according to the voting value; When the initial position information is obtained, performing eye positioning on the eye image based on the eye tracking method to obtain eye position information includes: Extracting multi-channel features of the eyeball image; Calling a preset filter to process the multi-channel features to generate a response map; determining a maximum response point according to the response graph; Calculating a second displacement vector according to the maximum response point; Eyeball position information is calculated based on the second displacement vector and the initial position information.

2. The method according to claim 1, characterized in that After determining the maximum response point according to the response graph, the method further includes: Calculating a tracking value according to the maximum response point; Determining whether the tracking is successful based on the tracking value and a preset threshold; Calculating the second displacement vector according to the maximum response point includes: When it is determined that the tracking is successful, a second displacement vector is calculated according to the maximum response point.

3. The method according to claim 1 or 2, characterized in that After performing eye positioning on the eye image based on the eye tracking method to obtain eye position information, the method further includes: The eye position information of the current frame is compared with the eye position information of the previous frame to determine whether to update the eye position information of the current frame.

4. The method according to claim 1 or 2, characterized in that After performing eye positioning on the eye image based on the eye tracking method to obtain eye position information, the method further includes: The eyeball position information is sent to an optical-mechanical control system, so that the optical-mechanical control system calculates an irradiation position according to the eyeball position information and irradiates the eye according to the irradiation position.

5. An eye tracking device, characterized in that: include: An acquisition unit, used for acquiring an eyeball image in real time through a camera; a determining unit, configured to determine an eye tracking mode according to an initial frame of the eye image; a positioning unit, configured to perform eye positioning on the eye image based on the eye tracking method to obtain eye position information; The determining unit is specifically configured to: determining an eye tracking mode according to whether initial position information is obtained from an initial frame of the eye image, wherein the initial position information is information about the eye position in the initial frame; When the initial position information is not obtained, the positioning unit includes: A separation module, used for separating the RGB channels of the eyeball image and performing blue channel extraction to obtain a blue channel image; A first determining module, configured to determine a ROI based on the blue channel image; A first calculation module is used to calculate the gradient vector of each pixel in the ROI; A second calculation module is used to calculate a first displacement vector of each pixel in the ROI according to the center point of the ROI and the gradient vector; a third calculation module, configured to calculate a voting value of each pixel in the ROI according to the gradient vector and the first displacement vector; A second determining module is used to determine eyeball position information according to the voting value; When the initial position information is acquired, the positioning unit includes: An extraction module, configured to extract multi-channel features of the eye image; A calling module, configured to call a preset filter to process the multi-channel features to generate a response map; a third determining module, configured to determine a maximum response point according to the response graph; a fourth calculation module, configured to calculate a second displacement vector according to the maximum response point; A fifth calculation module is used to calculate eye position information based on the second displacement vector and the initial position information.

6. An electronic device, characterized in that: include: processor, memory, input and output units, and buses; The processor is connected to the memory, the input and output unit, and the bus; A program is stored in the memory, and the processor calls the program to execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, and when the program is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 4.

Images