Self-adaptive eye movement tracking method and device

Abstract

The invention provides a self-adaptive eye movement tracking method and device. The method comprises the steps that a calibration target position matrix and a historical center position are initialized; obtaining state information of the contact lenses, a calibration eye image sequence and a calibration sensing position sequence by using the intelligent head-mounted equipment, the intelligent contact lenses and the calibration target position matrix; processing the calibration target position matrix, the historical center position, the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence to obtain a prediction target matrix, a prediction pupil center position matrix, a post-processing mark and an updated historical center position and calibration target position matrix; the post-processing mark is judged, and the calibration target position matrix is updated; and extracting the calibration target position matrix to obtain M predicted fixation point positions. Therefore, according to the method provided by the invention, the eye movement tracking mode can be adaptively selected, the defect of a single tracking method is overcome, and the accuracy and smoothness of eye movement tracking are improved.

Description

Technical Field

[0001] The present invention belongs to the field of human-computer interaction, and in particular relates to an adaptive eye tracking method and device. Background Art

[0002] At present, smart head-mounted devices are becoming more and more popular, and eye tracking has become a standard feature, providing users with an accurate interactive experience. The underlying algorithm in the field of eye tracking is the most important commercial core technology, which is usually built into the device in the form of a black box. There are currently four major categories of eye tracking algorithms: algorithms based on two-dimensional feature mapping, algorithms based on three-dimensional eyeball models, algorithms based on deep learning, and algorithms based on smart contact lenses. The first three of these four algorithms use near-eye images as the main input, and all algorithms ultimately output the user's screen gaze point. Although their inputs and outputs are similar, due to the different principles of reliance, the advantages and disadvantages of each algorithm are significantly different, so they are suitable for different application scenarios. In daily user life, due to the various scenarios involved, some limitations may be encountered.

[0003] In common eye trackers, the algorithm details are generally not disclosed, and at most they only state which type of algorithm is used. In the pupil core device, a well-known open source platform in the field of eye movement, users are provided with the option to select the type of algorithm, and users can choose the algorithm to be used according to their needs. However, it is not possible to achieve an algorithm that can cover all eye movement scenarios. At present, all eye movement algorithms are independent and parallel, and effective fusion has not yet been achieved.

[0004] These four types of eye tracking algorithms have typical differences. Algorithms based on two-dimensional feature mapping can achieve high accuracy when the head is fixed and in a small range. The principle is to establish a mapping model based on the two-dimensional human eye feature parameters extracted from the captured image and the gaze direction or gaze point of the user's glasses at the time of acquisition. This method achieves the highest eye movement interaction accuracy after the user is calibrated in scenarios without slippage and changes in wearing posture. However, after the user leaves the calibration position, the line of sight accuracy will drop significantly, and it is highly dependent on the unchanged spatial position.

[0005] The algorithm based on the three-dimensional eyeball model studies the imaging model and spatial geometry model of the human eye, and calculates the three-dimensional line of sight direction from the line of sight feature parameters of the image and the coordinates of the light source position. When the system calibrates the screen position, the line of sight intersects with the screen to obtain the screen line of sight landing point. It relies on complex geometric models, needs to solve nonlinear equations, or assumes some invariant parameters of the eye, so the accuracy is relatively low. Although complex system calibration is required, it simplifies the user calibration process and can compensate for changes in spatial geometric relationships caused by slippage and wearing posture.

[0006] The deep learning-based algorithm directly uses human eye image information as input, and no longer performs manual feature extraction. It uses a deep learning-based method to train a mapping model between human eye appearance and screen sight points from a large number of training samples. This method has a large sample statistic, strong robustness, and low dependence on calibration, but its accuracy is lower than that of the first two algorithms, and it is suitable for use when eye features are unclear.

[0007] Algorithms based on smart contact lenses do not rely on images, but use additional eye movement hardware for assistance. This type of hardware is usually expensive to produce and uses a contact method to fit the glasses. For example, when the glasses move, the contact lenses will follow the movement of the glasses, and the electronic components inside the smart glasses will reflect the movement of the glasses, thereby obtaining accurate eye movement information. This type of algorithm can achieve the highest level of accuracy, but it requires additional hardware and is less comfortable for users to wear. It is only considered for use in situations and special environments where high accuracy is particularly required.

[0008] Since these four algorithms are very different in principle, there is no possibility of internal fusion. However, they all follow a similar process during operation, that is, they first need multiple fixation points for calibration. The more fixation points, the higher the accuracy. When the fixation points are limited, different algorithms show different accuracy performance differences, and then the normal algorithm can be used. Summary of the invention

[0009] The technical problem to be solved by the present invention is that the invention aims to solve the problem of method adaptability of smart head-mounted devices in different scenarios, can adaptively select eye tracking methods, overcome the defects of a single tracking method, and improve the accuracy and smoothness of eye tracking.

[0010] In order to solve the above technical problems, a first aspect of an embodiment of the present invention discloses an adaptive eye tracking method, the method comprising:

[0011] S1, initializing a calibration target position matrix and a historical center position; the calibration target position matrix is ​​an N×2 matrix; N is an integer greater than 1;

[0012] S2, using the smart head mounted device, the smart contact lens and the calibration target position matrix, obtaining contact lens state information, a calibration eye image sequence and a calibration sensing position sequence;

[0013] The smart head-mounted device is worn on the user's head and is provided with a near-eye camera and a plurality of infrared light sources; the near-eye camera is used to obtain eye images in real time; the smart contact lens is worn on the left eye or right eye of the user and is used to detect the user's gaze point in real time, obtain the sensed position and send it to the head display of the smart head-mounted device;

[0014] The calibration eye image sequence and the calibration sensing position sequence include N eye calibration images and N sensing positions, respectively;

[0015] S3, processing the calibration target position matrix, the historical center position, the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence to obtain a predicted target matrix, a predicted pupil center position matrix and a post-processing flag, as well as an updated historical center position and the calibration target position matrix;

[0016] S4, judging the post-processing flag and updating the calibration target position matrix;

[0017] When the post-processing flag is no, setting the updated calibration target position matrix as the predicted target matrix;

[0018] When the post-processing flag is yes, calculating the sum of the predicted target matrix and the predicted pupil center position matrix to obtain the updated calibration target position matrix;

[0019] S5, extracting the last M rows of the calibration target position matrix to obtain M predicted gaze point positions; wherein M is greater than or equal to 1 and less than N;

[0020] S6, repeat S3 to S5 until the user turns off the smart head mounted device.

[0021] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the use of the smart head mounted device, the smart contact lens and the calibration target position matrix to obtain the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence includes:

[0022] S21, using the smart head mounted device to detect the working state of the smart contact lens to obtain the contact lens state information;

[0023] S22, using the calibration target position matrix, sequentially displaying the N gaze points on the screen of the head display;

[0024] S23, sequentially acquiring the eye image and the sensing position when the user gazes at each of the gaze points, to obtain the calibration eye image sequence and the calibration sensing position sequence.

[0025] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the calibration target position matrix, the historical center position, the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence are processed to obtain a predicted target matrix, a predicted pupil center position matrix and a post-processing flag, and the updated historical center position, including:

[0026] S31, processing the historical center position, the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence to obtain a calibration tracking position matrix, a predicted tracking position matrix, a predicted pupil center position matrix and the post-processing flag, and the updated historical center position;

[0027] S32, establishing an eye movement calibration model using the calibration target position matrix and the calibration tracking position matrix;

[0028] The eye movement calibration model is:

[0029] P=TS

[0030] Wherein, P is the calibration target position matrix, S is the calibration tracking position matrix, and T is the homography transformation matrix;

[0031] S33, solving the eye movement calibration model using the least squares method to obtain the homography transformation matrix;

[0032] S34, multiplying the homography transformation matrix by the predicted tracking position matrix to obtain a predicted target matrix.

[0033] As an optional implementation, in the first aspect of the embodiment of the present invention, the processing of the historical center position, the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence to obtain a calibration tracking position matrix, a predicted tracking position matrix, a predicted pupil center position matrix and the post-processing flag, as well as the updated historical center position and the calibration target position matrix includes:

[0034] S311, obtaining a tracking sensing position sequence and a tracking eye image sequence; the tracking sensing position sequence and the tracking eye image sequence respectively include M sensing positions and M eye images;

[0035] S312, adding the tracking sensing position sequence and the tracking eye image sequence to the end of the calibration sensing position sequence and the calibration eye image sequence, respectively, to obtain a global sensing position sequence and an eye global image sequence;

[0036] S313, processing each of the eye images in the eye global image sequence to obtain a corresponding pupil ellipse image, pupil center position, pupil effective area ratio, Purchin's spot position set and Purchin's spot center position; the pupil ellipse image, pupil center position, pupil effective area ratio, Purchin's spot position set and Purchin's spot center position respectively constitute a pupil ellipse image sequence, a pupil center position sequence, a pupil effective area ratio sequence, a Purchin's spot position set sequence and a Purchin's spot center position sequence;

[0037] S314, processing each corresponding group of the pupil ellipse image and the Purkinje spot position set in the pupil ellipse image sequence and the Purkinje spot position set sequence to obtain a corresponding optical axis vector; the optical axis vectors constitute an optical axis vector sequence;

[0038] S315, processing the Purchin spot position set sequence, the Purchin spot center position sequence and the historical center position to obtain head display position change information and the updated historical center position;

[0039] S316, obtaining the Qth pupil effective area ratio in the pupil effective area ratio sequence to obtain a current pupil effective area ratio; Q is greater than N and less than M+N;

[0040] S317, processing the contact lens state information, the current sensing position, the calibration sensing position sequence, the current pupil effective area ratio, the head display position change information, and the optical axis vector sequence to obtain a tracking position sequence and the post-processing flag;

[0041] S318, processing the tracking position sequence and the pupil center position sequence to obtain the predicted tracking position matrix, the calibrated tracking position matrix and the predicted pupil center position matrix.

[0042] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of each eye image in the global eye image sequence to obtain a corresponding pupil ellipse image, a pupil center position, a pupil effective area ratio, a Purkinje spot position set and a Purkinje spot center position includes:

[0043] S3131, performing grayscale conversion on the eye image to obtain an eye grayscale image;

[0044] S3132, using a preset first threshold, binarizing the eye grayscale image to obtain a first eye binarized image;

[0045] S3133, using a preset second threshold, binarizing the eye grayscale image to obtain a second eye binarized image;

[0046] S3134, processing the first eye binary image to obtain the pupil ellipse image, the pupil center position and the pupil effective area ratio;

[0047] S3135, processing the pupil center position and the second eye binary image to obtain the Purkinje spot position set and the Purkinje spot center position.

[0048] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the Purchin spot position set sequence, the Purchin spot center position sequence and the historical center position to obtain the head display position change information and the updated historical center position includes:

[0049] S3151, extracting the Qth Purchin spot position set in the Purchin spot position set sequence to obtain a current Purchin spot position set;

[0050] S3152, extracting the Qth center position of the Purchin spot in the Purchin spot center position sequence to obtain the current center position;

[0051] S3153, using a maximum distance difference calculation model, processing the current Purchin spot position set and the current center position to obtain a maximum distance difference;

[0052] The maximum distance difference calculation model is:

[0053]

[0054] Where, d n is the maximum distance difference, t is the preset displacement coefficient, cc n is the current center position, cc n-1 is the historical center location, c i is the i-th element in the current Purchin spot position set, where i is an integer between 1 and NN, and NN is the number of elements in the current Purchin spot position set;

[0055] S3154, determining the maximum distance difference to obtain the position change information of the head display;

[0056] When the maximum distance difference is greater than 0, the head display position change information is set to yes; when the maximum distance difference is less than or equal to 0, the head display position change information is set to no;

[0057] S3155: Set the historical center position as the current center position to obtain the updated historical center position.

[0058] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the contact lens state information, the current sensing position, the calibration sensing position sequence, the current pupil effective area ratio, the head display position change information and the optical axis vector sequence to obtain the tracking position sequence and the post-processing flag includes:

[0059] S3171, setting the post-processing flag to negative;

[0060] S3172, determining the contact lens status information;

[0061] When the contact lens status information is normal, the calibration target position matrix and the post-processing flag remain unchanged, and the current sensing position is added to the end of the calibration sensing position sequence to obtain a tracking position sequence;

[0062] When the contact lens status information is abnormal, judging the current pupil effective area ratio;

[0063] When the current pupil effective area ratio is greater than the effective area threshold, the calibration target position matrix and the post-processing flag remain unchanged, and the tracking position sequence is set to the pupil center position sequence;

[0064] When the current pupil effective area ratio is less than or equal to the effective area threshold, determining the head display position change information;

[0065] When the head display position change information is no, the calibration target position matrix and the pupil center position sequence are processed to obtain an updated calibration target position matrix, and the post-processing flag and the tracking position sequence are set to yes and the optical axis vector sequence, respectively;

[0066] When the head display position change information is yes, the calibration target position matrix and the post-processing flag remain unchanged, and the tracking eye image sequence is processed using a preset eye image feature extraction model to obtain the tracking position sequence.

[0067] A second aspect of an embodiment of the present invention discloses an adaptive eye tracking device, the device comprising:

[0068] The calibration module 201 is used to initialize the calibration target position matrix and the historical center position, and obtain the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence by using the head display, the smart contact lens and the calibration target position matrix;

[0069] A prediction module 202, for processing the calibration target position matrix, the historical center position, the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence to obtain a predicted target matrix, a predicted pupil center position matrix and a post-processing flag, as well as an updated historical center position and the calibration target position matrix;

[0070] The gaze point acquisition module 203 is used to determine the post-processing flag, update the calibration target position matrix, and extract the last M rows of the calibration target position matrix to obtain M predicted gaze point positions.

[0071] A third aspect of an embodiment of the present invention discloses another adaptive eye tracking device, the device comprising:

[0072] A memory storing executable program code;

[0073] a processor coupled to the memory;

[0074] The processor calls the executable program code stored in the memory to execute part or all of the steps in the adaptive eye tracking method disclosed in the first aspect of the embodiment of the present invention.

[0075] A fourth aspect of an embodiment of the present invention discloses a computer storable medium, the device comprising:

[0076] A memory storing executable program code;

[0077] a processor coupled to the memory;

[0078] The processor calls the executable program code stored in the memory to execute part or all of the steps in the adaptive eye tracking method disclosed in the first aspect of the embodiment of the present invention.

[0079] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0080] The present invention provides an adaptive eye tracking method and device, which utilizes intelligent head-mounted devices and intelligent contact lenses to detect changes in the external environment in real time and adaptively select an eye tracking method, thereby overcoming the defects of a single tracking method and improving the accuracy of eye tracking. In addition, the tracking process combines historical tracking data with the current external environment, thereby improving the smoothness of the eye tracking results. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0082] Figure 1 is a flow chart of an adaptive eye tracking method according to an embodiment of the present invention;

[0083] Figure 2 A schematic diagram of an eye grayscale image, a first eye binary image, and a second eye binary image according to an adaptive eye tracking method according to an embodiment of the present invention;

[0084] Figure 3 is an eye contour image and a Purkinje spot position set according to an adaptive eye tracking method according to an embodiment of the present invention;

[0085] Figure 4 A schematic diagram of a three-dimensional eyeball model of an adaptive eye tracking method according to an embodiment of the present invention;

[0086] Figure 5 is a schematic structural diagram of an adaptive eye tracking device according to an embodiment of the present invention;

[0087] Figure 6 Schematic diagram of the structure of another adaptive eye tracking device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0088] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0089] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0090] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0091] The present invention discloses an adaptive eye tracking method and device, which is beneficial to improving the processing efficiency and accuracy of visual information, while significantly reducing the design cost of existing equipment, reducing the hardware requirements of the equipment, solving the hardware cost, shortening the user's use cost, and improving the user experience. The following are detailed descriptions.

[0092] Embodiment 1

[0093] See also Figure 1 , Figure 1 is a flow chart of an adaptive eye tracking method disclosed in an embodiment of the present invention. Figure 1 The described adaptive eye tracking method is applied to the field of human-computer interaction, such as virtual reality / augmented reality, eye tracking technology, etc., which is not limited in the embodiments of the present invention. Figure 1 As shown, the adaptive eye tracking method may include the following operations:

[0094] S1, initializing a calibration target position matrix and a historical center position; the calibration target position matrix is ​​an N×2 matrix; N is an integer greater than 1;

[0095] Optionally, the above N is 3, 5, 9 or 13;

[0096] Optionally, the initial values ​​of the horizontal and vertical coordinates of the above historical center position are negative values;

[0097] It should be noted that each row of the calibration target position matrix corresponds to one calibration target position; the calibration target position includes the horizontal coordinate and the vertical coordinate of the gaze point;

[0098] S2, using the smart head mounted device, the smart contact lens and the calibration target position matrix, obtaining contact lens state information, a calibration eye image sequence and a calibration sensing position sequence;

[0099] The smart head-mounted device is worn on the user's head and is provided with a near-eye camera and a plurality of infrared light sources; the near-eye camera is used to obtain eye images in real time; the smart contact lens is worn on the left eye or right eye of the user and is used to detect the user's gaze point in real time, obtain the sensed position and send it to the head display of the smart head-mounted device;

[0100] It should be noted that under the illumination of multiple infrared light sources preset in the headset, the pupil appears black, which contrasts with the iris and can highlight the pupil characteristics;

[0101] Optionally, the smart head-mounted device and the smart contact lens are connected via Bluetooth;

[0102] Optionally, the above-mentioned intelligent head-mounted device is a virtual reality glasses;

[0103] The calibration eye image sequence and the calibration sensing position sequence include N eye calibration images and N sensing positions, respectively;

[0104] S3, processing the calibration target position matrix, the historical center position, the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence to obtain a predicted target matrix, a predicted pupil center position matrix and a post-processing flag, and an updated historical center position;

[0105] S4, judging the post-processing flag and updating the calibration target position matrix;

[0106] When the post-processing flag is no, setting the updated calibration target position matrix as the predicted target matrix;

[0107] When the post-processing flag is yes, calculating the sum of the predicted target matrix and the predicted pupil center position matrix to obtain the updated calibration target position matrix;

[0108] S5, extract the last M rows of the calibration target position matrix to obtain M predicted gaze point positions; where M is greater than or equal to 1 and less than N

[0109] S6, repeat S3 to S5 until the user turns off the smart head-mounted device.

[0110] In another optional embodiment, the above-mentioned use of the smart head mounted device, the smart contact lens and the calibration target position matrix to obtain the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence includes:

[0111] S21, using the smart head mounted device to detect the working state of the smart contact lens to obtain the contact lens state information;

[0112] It should be noted that when the head display cannot obtain the sensing position, it indicates that the smart contact lens does not exist or cannot work normally, and the contact lens status information is set to abnormal; when the head display can obtain the sensing position, it indicates that the smart contact lens works normally, and the contact lens status information is set to normal;

[0113] S22, using the calibration target position matrix, sequentially displaying the N gaze points on the screen of the head display;

[0114] It should be noted that the mth row of the calibration target position matrix corresponds to the mth fixation point, and the m is greater than or equal to and less than or equal to N;

[0115] S23, sequentially acquiring the eye image and the sensing position when the user gazes at each of the gaze points, to obtain the calibration eye image sequence and the calibration sensing position sequence.

[0116] It should be noted that when the contact lens status information is normal, the sensing position obtained by the head display comes from the smart contact lens; when the contact lens status information is abnormal, the sensing position obtained by the head display is an arbitrary position.

[0117] In yet another optional embodiment, the calibration target position matrix, the historical center position, the contact lens state information, the calibration eye image sequence, and the calibration sensing position sequence are processed to obtain a predicted target matrix, a predicted pupil center position matrix and a post-processing flag, and an updated historical center position, including:

[0118] S31, processing the historical center position, the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence to obtain a calibration tracking position matrix, a predicted tracking position matrix, a predicted pupil center position matrix and the post-processing flag, and the updated historical center position;

[0119] S32, establishing an eye movement calibration model using the calibration target position matrix and the calibration tracking position matrix;

[0120] The eye movement calibration model is:

[0121] P=TS

[0122] Wherein, P is the calibration target position matrix, S is the calibration tracking position matrix, and T is the homography transformation matrix;

[0123] S33, solving the eye movement calibration model using the least squares method to obtain the homography transformation matrix;

[0124] S34, multiplying the homography transformation matrix by the predicted tracking position matrix to obtain a predicted target matrix.

[0125] In yet another optional embodiment, the above processing of the historical center position, the contact lens state information, the calibration eye image sequence, and the calibration sensing position sequence to obtain a calibration tracking position matrix, a predicted tracking position matrix, a predicted pupil center position matrix, and the post-processing flag, as well as the updated historical center position, includes:

[0126] S311, obtaining a tracking sensing position sequence and a tracking eye image sequence; the tracking sensing position sequence and the tracking eye image sequence respectively include M sensing positions and M eye images;

[0127] It should be noted that the above tracking sensing position sequence includes M consecutive sensing positions obtained through the head display; the above tracking eye image sequence includes M consecutive eye images obtained through the near-eye camera of the head display;

[0128] S312, adding the tracking sensing position sequence and the tracking eye image sequence to the end of the calibration sensing position sequence and the calibration eye image sequence, respectively, to obtain a global sensing position sequence and an eye global image sequence;

[0129] It should be noted that the above global sensing position sequence includes M+N sensing positions; the above global eye image sequence includes M+N eye images;

[0130] S313, processing each of the eye images in the eye global image sequence to obtain a corresponding pupil ellipse image, pupil center position, pupil effective area ratio, Purchin's spot position set and Purchin's spot center position; the pupil ellipse image, pupil center position, pupil effective area ratio, Purchin's spot position set and Purchin's spot center position respectively constitute a pupil ellipse image sequence, a pupil center position sequence, a pupil effective area ratio sequence, a Purchin's spot position set sequence and a Purchin's spot center position sequence;

[0131] S314, processing each corresponding group of the pupil ellipse image and the Purkinje spot position set in the pupil ellipse image sequence and the Purkinje spot position set sequence to obtain a corresponding optical axis vector; the optical axis vectors constitute an optical axis vector sequence;

[0132] S315, processing the Purchin spot position set sequence, the Purchin spot center position sequence and the historical center position to obtain head display position change information and the updated historical center position;

[0133] S316, obtaining the Qth pupil effective area ratio in the pupil effective area ratio sequence to obtain a current pupil effective area ratio; Q is greater than N and less than M+N;

[0134] S317, processing the contact lens state information, the current sensing position, the calibration sensing position sequence, the current pupil effective area ratio, the head display position change information, and the optical axis vector sequence to obtain a tracking position sequence and the post-processing flag;

[0135] S318, processing the tracking position sequence and the pupil center position sequence to obtain the predicted tracking position matrix, the calibrated tracking position matrix and the predicted pupil center position matrix.

[0136] In yet another optional embodiment, the above-mentioned processing of each eye image in the eye global image sequence to obtain the corresponding pupil ellipse image, pupil center position, pupil effective area ratio, Purkinje spot position set and Purkinje spot center position includes:

[0137] S3131, performing grayscale conversion on the eye image to obtain an eye grayscale image;

[0138] S3132, using a preset first threshold, binarizing the eye grayscale image to obtain a first eye binarized image;

[0139] It should be noted that if Figure 2 As shown, the first eye binary image removes most of the interference information such as the white of the eye, iris, eyelashes, etc. in the eye grayscale image, and can highlight the pupil features;

[0140] S3133, using a preset second threshold, binarizing the eye grayscale image to obtain a second eye binarized image;

[0141] It should be noted that if Figure 2 As shown, the second eye binary image retains the bright spots with pixel values ​​higher than the second threshold in the eye grayscale image, thereby retaining the Purkinje spot information;

[0142] S3134, processing the first eye binary image to obtain the pupil ellipse image, the pupil center position and the pupil effective area ratio;

[0143] S3135, processing the pupil center position and the second eye binary image to obtain the Purkinje spot position set and the Purkinje spot center position.

[0144] In yet another optional embodiment, the processing of the first eye binary image to obtain the pupil ellipse image, the pupil center position and the pupil effective area ratio includes:

[0145] S31341, processing the first eye binary image to obtain an eye contour image;

[0146] Optionally, OpenCV is used to perform contour detection and screening on the first eye binary image to obtain an eye contour image; Figure 3 As shown, the above eye contour image is Figure 3 The area inside the medium green wireframe.

[0147] It should be noted that the above screening includes: (1) removing contours with less than 25 contour points, thereby removing noise points such as Purkinje spots; (2) removing contours with a contour distance greater than a preset maximum pupil diameter, or less than a preset minimum pupil diameter; the above contour distance is the maximum Euclidean distance of the contour inner point set of the eye contour image, which is the maximum value of the distance between any two points in a given contour point set, and is usually used to estimate the maximum linear size of the contour; (3) removing tiny noise points and illuminated outer contours formed due to lighting reasons;

[0148] S31342, processing the eye contour image to obtain a pupil ellipse image and the pupil center position;

[0149] Optionally, use OpenCV to perform ellipse fitting and contour screening on the eye contour image to obtain the pupil ellipse image and the pupil center position;

[0150] It should be noted that the above-mentioned contour screening can ensure that the fitted ellipse meets the conditions of the ratio of the short axis to the long axis, the diameter range and the uniform distribution of the original contour, and selects the ellipse with the best fitting degree as the pupil to be detected.

[0151] S31343, calculate the average pixel value of the pupil ellipse image to obtain the pupil effective area ratio.

[0152] It should be noted that the lower the pupil effective area ratio, the larger the pupil area accounts for; the higher the pupil effective area ratio, the larger the proportion of interference areas such as surface eyelids and eye bags.

[0153] In yet another optional embodiment, the processing of the pupil center position and the second eye binary image to obtain the Purkinje spot position set and the Purkinje spot center position includes:

[0154] S31351, using OpenCV to perform two dilation processes on the second eye binary image to obtain an eye dilated image;

[0155] S31352, using OpenCV to filter out connected domains whose maximum size is smaller than a diameter threshold in the eye dilation image, to obtain an initial eye filtered image;

[0156] S31353, using the gradient filtering algorithm in OpenCV, processing the initial eye filtered image to obtain the eye filtered image.

[0157] It should be noted that the eye-filtered image retains the small bright spots in the initial eye-filtered image;

[0158] S31354, processing the eye filtered image to obtain a first position set; the first position set includes more than one connected domain center position;

[0159] Optionally, the eye dilation image is processed using a connected domain detection algorithm of OpenCV to obtain the connected domain center position of each connected domain;

[0160] S31355, using the distribution pattern of Purkinje spots, screening the first position set to obtain a second position set;

[0161] For example, assuming that there are 20 near-eye cameras distributed in a circle, the first position set above will ideally contain 20 connected domain center positions distributed in an elliptical shape. However, if it contains 22 connected domain center positions, and 20 of them form an ellipse, and the other two connected domain center positions are near the center of the ellipse, then these two connected domain center positions are outliers and need to be removed.

[0162] S31356, deleting the connected domain center position whose distance from the pupil center position is less than a distance threshold in the second position set, to obtain a Purchin's spot position set; the Purchin's spot position set is as follows: Figure 3 As shown in red mark;

[0163] S31357, calculating the mean of the center positions of all the connected domains in the Purkinje spot position set to obtain the current center position.

[0164] In another optional embodiment, in the above-mentioned pupil ellipse image sequence and the Purkinje spot position set sequence, each group of corresponding pupil ellipse images and Purkinje spot position sets are processed to obtain a corresponding optical axis vector, based on Figure 4 The three-dimensional eyeball model shown includes:

[0165] S3141, using the optical center position O and the Purkinje spot position set, establish a reflection point calculation model;

[0166] The above reflection point calculation model is:

[0167]

[0168] In the formula, G i is the position of the i-th reflection point, ki is the preset proportionality factor, g i is the i-th element in the Purkinje spot position set, NN is the number of elements in the Purkinje spot position set, i∈{1,2,...,NN};

[0169] It should be noted that the optical center position O is obtained by pre-calibrating the near-eye camera; each element in the Purkinje spot position set corresponds to one Purkinje spot, so the reflection point calculation model contains NN equations;

[0170] S3142, solving the reflection point calculation model using the least square method to obtain a reflection point position set;

[0171] S3143, solving the corneal center calculation model, processing the reflection point position set, the optical center position and the light source position, and obtaining the corneal center position;

[0172] The above corneal center calculation model is:

[0173]

[0174] Where C is the center of the cornea, L ii is the position of the infrared light source corresponding to the ii-th element in the Purkinje spot position set, which is also obtained by pre-calibration, and ii is any integer greater than or equal to 1 and less than or equal to NN;

[0175] S3144, using the corneal curvature radius calculation model, the reflection point position G ii The corneal center position C is processed to obtain the corneal curvature radius R;

[0176] The above corneal curvature radius calculation model is:

[0177] R=||G i -C||2

[0178] S3145, solving the pupil center position calculation model to obtain the pupil center position;

[0179] The above pupil center position calculation model is:

[0180]

[0181] Where n1 and n2 are the known corneal refractive index and air refractive index, P is the pupil center position, and p j is the position of the pupil edge imaging point on the imaging plane; p j is the pixel value of the jth pixel at the contour position in the pupil ellipse image, j and k are any integers greater than or equal to 1 and less than or equal to T, j is not equal to k, and T is the number of pixels at the contour position in the pupil ellipse image; P jand F j are unknown quantities, respectively p j The corresponding real pupil edge position and the refraction point position on the corneal surface;

[0182] S3146, subtract the pupil center position from the cornea center position to obtain the optical axis vector.

[0183] In yet another optional embodiment, the processing of the Purchin spot position set sequence, the Purchin spot center position sequence, and the historical center position to obtain the head display position change information and the updated historical center position includes:

[0184] S3151, extracting the Qth Purchin spot position set in the Purchin spot position set sequence to obtain a current Purchin spot position set;

[0185] S3152, extracting the Qth center position of the Purchin spot in the Purchin spot center position sequence to obtain the current center position;

[0186] S3153, using a maximum distance difference calculation model, processing the current Purchin spot position set and the current center position to obtain a maximum distance difference;

[0187] The maximum distance difference calculation model is:

[0188]

[0189] Where, d n is the maximum distance difference, t is the preset displacement coefficient, cc n is the current center position, cc n-1 is the historical center location, c i is the i-th element in the current Purchin spot position set, where i is an integer between 1 and NN, and NN is the number of elements in the current Purchin spot position set;

[0190] S3154, determining the maximum distance difference to obtain the position change information of the head display;

[0191] When the maximum distance difference is greater than 0, the head display position change information is set to yes; when the maximum distance difference is less than or equal to 0, the head display position change information is set to no;

[0192] S3155: Set the historical center position as the current center position to obtain the updated historical center position.

[0193] In yet another optional embodiment, the contact lens state information, the current sensing position, the calibration sensing position sequence, the current pupil effective area ratio, the head display position change information, and the optical axis vector sequence are processed to obtain a tracking position sequence and the post-processing flag, including:

[0194] S3171, setting the post-processing flag to negative;

[0195] S3172, determining the contact lens status information;

[0196] When the contact lens status information is normal, the calibration target position matrix and the post-processing flag remain unchanged, and the current sensing position is added to the end of the calibration sensing position sequence to obtain a tracking position sequence;

[0197] When the contact lens status information is abnormal, judging the current pupil effective area ratio;

[0198] When the current pupil effective area ratio is greater than the effective area threshold, the calibration target position matrix and the post-processing flag remain unchanged, and the tracking position sequence is set to the pupil center position sequence;

[0199] When the current pupil effective area ratio is less than or equal to the effective area threshold, determining the head display position change information;

[0200] When the head display position change information is no, the calibration target position matrix and the pupil center position sequence are processed to obtain an updated calibration target position matrix, and the post-processing flag and the tracking position sequence are set to yes and the optical axis vector sequence, respectively;

[0201] When the head display position change information is yes, the calibration target position matrix and the post-processing flag remain unchanged, and the tracking eye image sequence is processed using a preset eye image feature extraction model to obtain the tracking position sequence.

[0202] In yet another optional embodiment, the above processing of the calibration target position matrix and the pupil center position sequence to obtain the updated calibration target position matrix includes:

[0203] Sequentially splicing the pupil center positions in the pupil center position sequence into a pupil center position matrix;

[0204] It should be noted that the above pupil center position matrix is ​​a (M+N)×2 matrix;

[0205] Delete the last M rows of the pupil center position matrix to obtain the target pupil center position matrix;

[0206] The target pupil center position matrix is ​​subtracted from the calibration target position matrix to obtain an updated calibration target position matrix.

[0207] In yet another optional embodiment, the above-mentioned processing of the tracking position sequence and the pupil center position sequence to obtain the predicted tracking position matrix, the calibrated tracking position matrix and the predicted pupil center position matrix includes:

[0208] S3181, sequentially splicing the tracking positions in the tracking position sequence into a tracking position matrix;

[0209] S3182, extracting the first N rows and the last N rows of the tracking position matrix respectively to obtain the predicted tracking position matrix and the calibrated tracking position matrix;

[0210] S3183, extracting the last N pupil center positions in the pupil center position sequence to obtain a predicted pupil center position sequence;

[0211] S3184, sequentially splicing the pupil center positions in the predicted pupil center position sequence into a predicted pupil center position matrix.

[0212] It should be noted that the above-mentioned tracking position matrix is ​​a (M+N)×2 matrix; the above-mentioned predicted tracking position matrix, calibrated tracking position matrix and predicted pupil center position matrix are all N×2 matrices.

[0213] In yet another optional embodiment, the eye image feature extraction model is a computer vision model based on the Resnet-18 network architecture;

[0214] It should be noted that the above-mentioned eye image feature extraction model has been trained on a large-scale eye tracking dataset in advance; during the training phase, the above-mentioned eye image feature extraction model uses the original eye diagram and gaze point position in the large-scale eye tracking dataset as input and supervision labels, respectively, and uses the L2 loss function for supervised training; when the above-mentioned eye image feature extraction model is working, each eye image in the tracking eye image sequence is processed in turn to obtain the corresponding image feature position; the above-mentioned image feature positions constitute the tracking position sequence in turn;

[0215] Preferably, the above-mentioned eye image feature extraction model first compresses the size of the input eye image to 128×128 before performing feature extraction;

[0216] Preferably, the output dimension of the fully connected layer of the above-mentioned eye image feature extraction model is 2.

[0217] It can be seen that the implementation of the adaptive eye tracking method described in the embodiment of the present invention utilizes smart head-mounted devices and smart contact lenses to detect changes in the external environment in real time and adaptively selects an eye tracking method, which can overcome the defects of a single tracking method and improve the accuracy of eye tracking; in addition, the tracking process combines historical tracking data with the current external environment, which can improve the smoothness of the eye tracking results.

[0218] Embodiment 2

[0219] See also Figure 5 , Figure 5 is a schematic diagram of the structure of an adaptive eye tracking device disclosed in an embodiment of the present invention. Figure 5 The adaptive eye tracking device described can be applied to the field of human-computer interaction, such as virtual reality / augmented reality, eye tracking technology, etc., which is not limited in the embodiments of the present invention. Figure 2 As shown, the device may include:

[0220] The calibration module 201 is used to initialize the calibration target position matrix and the historical center position, and obtain the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence by using the head display, the smart contact lens and the calibration target position matrix;

[0221] A prediction module 202, for processing the calibration target position matrix, the historical center position, the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence to obtain a predicted target matrix, a predicted pupil center position matrix and a post-processing flag, as well as an updated historical center position and the calibration target position matrix;

[0222] The gaze point acquisition module 203 is used to determine the post-processing flag, update the calibration target position matrix, and extract the last M rows of the calibration target position matrix to obtain M predicted gaze point positions.

[0223] In an optional embodiment, if Figure 5 As shown, the test connection relationship acquisition module 201 uses the head display, the smart contact lens and the calibration target position matrix to obtain contact lens state information, a calibration eye image sequence and a calibration sensing position sequence, including:

[0224] Using the head display to detect the working state of the contact lens to obtain the contact lens state information;

[0225] Using the calibration target position matrix, sequentially displaying the N gaze points on the screen of the head display;

[0226] Sequentially acquiring the eye image and the sensing position when the user gazes at each of the focus points to obtain the calibration eye image sequence and the calibration sensing position sequence;

[0227] In another optional embodiment, Figure 5 As shown, the prediction module 202 processes the calibration target position matrix, the historical center position, the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence to obtain a predicted target matrix, a predicted pupil center position matrix and a post-processing flag, as well as the updated historical center position and the calibration target position matrix, including:

[0228] Processing the historical center position, the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence to obtain a calibration tracking position matrix, a predicted tracking position matrix, a predicted pupil center position matrix and the post-processing flag, as well as the updated historical center position;

[0229] Establishing an eye movement calibration model using the calibration target position matrix and the calibration tracking position matrix;

[0230] Solving the eye movement calibration model using the least squares method to obtain the homography transformation matrix;

[0231] The homography transformation matrix is ​​multiplied by the predicted tracking position matrix to obtain a predicted target matrix.

[0232] It can be seen that implementation Figure 2 The described adaptive eye tracking device uses a smart head-mounted device and smart contact lenses to detect changes in the external environment in real time and adaptively selects an eye tracking method, which can overcome the defects of a single tracking method and improve the accuracy of eye tracking; in addition, the tracking process combines historical tracking data and the current external environment, which can improve the smoothness of the eye tracking results.

[0233] Embodiment 3

[0234] See also Figure 6 , Figure 6 is a schematic diagram of the structure of another adaptive eye tracking device disclosed in an embodiment of the present invention. Figure 6 The adaptive eye tracking device described can be applied to the field of human-computer interaction, such as virtual reality / augmented reality, eye tracking technology, etc., which is not limited in the embodiments of the present invention. Figure 3 As shown, the adaptive eye tracking device may include the following operations:

[0235] A memory 301 storing executable program codes;

[0236] a processor 302 coupled to the memory 301;

[0237] The processor 302 calls the executable program code stored in the memory 301 to execute the steps in the adaptive eye tracking method described in the first embodiment.

[0238] Embodiment 4

[0239] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps of the adaptive eye tracking method described in the first embodiment.

[0240] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0241] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0242] Finally, it should be noted that the adaptive eye tracking method and device disclosed in the embodiments of the present invention disclose only the preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive eye tracking method, characterized in that: include: S1, initializing a calibration target position matrix and a historical center position; the calibration target position matrix is ​​an N×2 matrix; Said N is an integer greater than 1; S2, using the smart head mounted device, the smart contact lens and the calibration target position matrix, obtaining contact lens state information, a calibration eye image sequence and a calibration sensing position sequence; The smart head-mounted device is worn on the user's head and is provided with a near-eye camera and a plurality of infrared light sources; the near-eye camera is used to obtain eye images in real time; the smart contact lens is worn on the left eye or right eye of the user and is used to detect the user's gaze point in real time, obtain the sensed position and send it to the head display of the smart head-mounted device; The calibration eye image sequence and the calibration sensing position sequence include N eye calibration images and N sensing positions, respectively; S3, processing the calibration target position matrix, the historical center position, the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence to obtain a predicted target matrix, a predicted pupil center position matrix and a post-processing flag, as well as an updated historical center position and the calibration target position matrix; S4, judging the post-processing flag and updating the calibration target position matrix; When the post-processing flag is no, setting the updated calibration target position matrix as the predicted target matrix; When the post-processing flag is yes, calculating the sum of the predicted target matrix and the predicted pupil center position matrix to obtain the updated calibration target position matrix; S5, extracting the last M rows of the calibration target position matrix to obtain M predicted gaze point positions; wherein M is greater than or equal to 1 and less than N; S6, repeat S3 to S5 until the user turns off the smart head mounted device.

2. The adaptive eye tracking method according to claim 1, characterized in that: The method of using the smart head mounted device, the smart contact lens and the calibration target position matrix to obtain contact lens state information, a calibration eye image sequence and a calibration sensing position sequence includes: S21, using the smart head mounted device to detect the working state of the smart contact lens to obtain the contact lens state information; S22, using the calibration target position matrix, sequentially displaying the N gaze points on the screen of the head display; S23, sequentially acquiring the eye image and the sensing position when the user gazes at each of the gaze points, to obtain the calibration eye image sequence and the calibration sensing position sequence.

3. The adaptive eye tracking method according to claim 1, characterized in that: The step of processing the calibration target position matrix, the historical center position, the contact lens state information, the calibration eye image sequence, and the calibration sensing position sequence to obtain a predicted target matrix, a predicted pupil center position matrix, a post-processing flag, and an updated historical center position includes: S31, processing the historical center position, the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence to obtain a calibration tracking position matrix, a predicted tracking position matrix, a predicted pupil center position matrix and the post-processing flag, and an updated historical center position; S32, establishing an eye movement calibration model using the calibration target position matrix and the calibration tracking position matrix; The eye movement calibration model is: P=TS Wherein, P is the calibration target position matrix, S is the calibration tracking position matrix, and T is the homography transformation matrix; S33, solving the eye movement calibration model using the least squares method to obtain the homography transformation matrix; S34, multiplying the homography transformation matrix by the predicted tracking position matrix to obtain a predicted target matrix.

4. The adaptive eye tracking method according to claim 3, characterized in that: The processing of the historical center position, the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence to obtain a calibration tracking position matrix, a predicted tracking position matrix, a predicted pupil center position matrix and the post-processing flag, as well as the updated historical center position and the calibration target position matrix includes: S311, obtaining a tracking sensing position sequence and a tracking eye image sequence; the tracking sensing position sequence and the tracking eye image sequence respectively include M sensing positions and M eye images; S312, adding the tracking sensing position sequence and the tracking eye image sequence to the end of the calibration sensing position sequence and the calibration eye image sequence, respectively, to obtain a global sensing position sequence and an eye global image sequence; S313, processing each of the eye images in the eye global image sequence to obtain a corresponding pupil ellipse image, pupil center position, pupil effective area ratio, Purchin's spot position set and Purchin's spot center position; the pupil ellipse image, pupil center position, pupil effective area ratio, Purchin's spot position set and Purchin's spot center position respectively constitute a pupil ellipse image sequence, a pupil center position sequence, a pupil effective area ratio sequence, a Purchin's spot position set sequence and a Purchin's spot center position sequence; S314, processing each corresponding group of the pupil ellipse image and the Purkinje spot position set in the pupil ellipse image sequence and the Purkinje spot position set sequence to obtain a corresponding optical axis vector; the optical axis vectors constitute an optical axis vector sequence; S315, processing the Purchin spot position set sequence, the Purchin spot center position sequence and the historical center position to obtain head display position change information and the updated historical center position; S316, obtaining the Qth pupil effective area ratio in the pupil effective area ratio sequence to obtain a current pupil effective area ratio; Q is greater than N and less than M+N; S317, processing the contact lens state information, the current sensing position, the calibration sensing position sequence, the current pupil effective area ratio, the head display position change information, and the optical axis vector sequence to obtain a tracking position sequence and the post-processing flag; S318, processing the tracking position sequence and the pupil center position sequence to obtain the predicted tracking position matrix, the calibrated tracking position matrix and the predicted pupil center position matrix.

5. The adaptive eye tracking method according to claim 4, characterized in that: The step of processing each eye image in the eye global image sequence to obtain a corresponding pupil ellipse image, a pupil center position, a pupil effective area ratio, a Purkinje spot position set and a Purkinje spot center position includes: S3131, performing grayscale conversion on the eye image to obtain an eye grayscale image; S3132, using a preset first threshold, binarizing the eye grayscale image to obtain a first eye binarized image; S3133, using a preset second threshold, binarizing the eye grayscale image to obtain a second eye binarized image; S3134, processing the first eye binary image to obtain the pupil ellipse image, the pupil center position and the pupil effective area ratio; S3135, processing the pupil center position and the second eye binary image to obtain the Purkinje spot position set and the Purkinje spot center position.

6. The adaptive eye tracking method according to claim 4, characterized in that: The processing of the Purchin spot position set sequence, the Purchin spot center position sequence and the historical center position to obtain the head display position change information and the updated historical center position includes: S3151, extracting the Qth Purchin spot position set in the Purchin spot position set sequence to obtain a current Purchin spot position set; S3152, extracting the Qth center position of the Purchin spot in the Purchin spot center position sequence to obtain the current center position; S3153, using a maximum distance difference calculation model, processing the current Purchin spot position set and the current center position to obtain a maximum distance difference; The maximum distance difference calculation model is: Where, d n is the maximum distance difference, t is the preset displacement coefficient, cc n is the current center position, cc n-1 is the historical center location, c i is the i-th element in the current Purchin spot position set, where i is an integer between 1 and NN, and NN is the number of elements in the current Purchin spot position set; S3154, determining the maximum distance difference to obtain the position change information of the head display; When the maximum distance difference is greater than 0, the head display position change information is set to yes; when the maximum distance difference is less than or equal to 0, the head display position change information is set to no; S3155: Set the historical center position as the current center position to obtain the updated historical center position.

7. The adaptive eye tracking method according to claim 4, characterized in that: The processing of the contact lens state information, the current sensing position, the calibration sensing position sequence, the current pupil effective area ratio, the head display position change information and the optical axis vector sequence to obtain a tracking position sequence and the post-processing flag includes: S3171, setting the post-processing flag to negative; S3172, determining the contact lens status information; When the contact lens status information is normal, the calibration target position matrix and the post-processing flag remain unchanged, and the current sensing position is added to the end of the calibration sensing position sequence to obtain a tracking position sequence; When the contact lens status information is abnormal, judging the current pupil effective area ratio; When the current pupil effective area ratio is greater than the effective area threshold, the calibration target position matrix and the post-processing flag remain unchanged, and the tracking position sequence is set to the pupil center position sequence; When the current pupil effective area ratio is less than or equal to the effective area threshold, determining the head display position change information; When the head display position change information is no, the calibration target position matrix and the pupil center position sequence are processed to obtain an updated calibration target position matrix, and the post-processing flag and the tracking position sequence are set to yes and the optical axis vector sequence, respectively; When the head display position change information is yes, the calibration target position matrix and the post-processing flag remain unchanged, and the tracking eye image sequence is processed using a preset eye image feature extraction model to obtain the tracking position sequence.

8. An adaptive eye tracking device, characterized in that: The device comprises a calibration module, a prediction module and a gaze point acquisition module; The calibration module is used to initialize the calibration target position matrix and the historical center position, and obtain contact lens state information, calibration eye image sequence and calibration sensing position sequence using the head display, the smart contact lens and the calibration target position matrix; The prediction module is used to process the calibration target position matrix, the historical center position, the contact lens state information, the calibration eye image sequence and the calibration sensing position sequence to obtain a predicted target matrix, a predicted pupil center position matrix and a post-processing flag, and an updated historical center position and the calibration target position matrix; The gaze point acquisition module is used to judge the post-processing flag, update the calibration target position matrix, and extract the last M rows of the calibration target position matrix to obtain M predicted gaze point positions.

9. An adaptive eye tracking device, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the adaptive eye tracking method according to any one of claims 1 to 7.

10. A computer storable medium, characterized in that: The computer storable medium stores computer instructions, and when the computer instructions are called, they are used to execute the adaptive eye tracking method according to any one of claims 1 to 7.

Images