Method and device for sight tracking, electronic equipment and storage medium

Abstract

The invention relates to a sight tracking method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining light reflection point information, which is obtained based on a light reflection point signal of light emitted by a light source and reflected by the surface of a cornea sphere captured by an event camera; estimating a corneal sphere center point position and an eyeball rotation center point position based on the light reflection point information; and determining three-dimensional sight line related information according to the corneal ball center point position and the eyeball rotation center point position.

Description

Technical Field

[0001] The present disclosure relates to the field of gaze tracking, and in particular to a method, device, electronic device and storage medium for gaze tracking. Background Art

[0002] Gaze tracking is an important way for augmented reality systems to achieve natural human-computer interaction. For example, the results of gaze tracking can be used for human-computer interaction of head-mounted devices or applications such as gaze-based rendering. Since the user's gaze needs to be tracked at a high frequency in human-computer interaction, it is expected to achieve gaze tracking with low latency and low cost. However, current three-dimensional gaze tracking methods always need to extract and use pupil information for gaze tracking, and additional hardware and software are required to perform additional calculations in order to obtain pupil information, resulting in low efficiency and high cost of gaze tracking. Summary of the invention

[0003] The present disclosure provides a method, an electronic device and a storage medium for line of sight tracking, so as to at least solve the above-mentioned problems in the related art.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for line of sight tracking is provided, comprising: acquiring reflection point information, wherein the reflection point information is obtained based on a reflection point signal of light emitted by a light source captured by an event camera and reflected by the surface of a corneal ball; estimating a corneal ball center point position and an eyeball rotation center point position based on the reflection point information; and determining three-dimensional line of sight related information based on the corneal ball center point position and the eyeball rotation center point position.

[0005] Optionally, the estimating the position of the corneal ball center point and the eyeball rotation center point based on the reflection point information includes: estimating the position of the corneal ball center point at each moment based on the reflection point information at that moment; estimating the position of the eyeball rotation center point based on the estimated corneal ball center point positions at multiple moments and the distance between the calibrated eyeball rotation center point and the corneal ball center point.

[0006] Optionally, the reflection point information includes the reflection point image position, wherein the reflection point image position is the position of the reflection point corresponding to the light source in the image coordinate system of the event camera, and the estimating the corneal sphere center point position at the moment based on the reflection point information at each moment includes: based on the reflection point image position at each moment, using a regressor to estimate the corneal sphere center point position at the moment; or, based on the reflection point image position at each moment, the parameters of the event camera, the position of the light source and the radius of the corneal sphere, estimating the corneal sphere center point position at the moment through a numerical solver.

[0007] Optionally, the method of estimating the position of the center point of the corneal ball at each moment by using a regressor based on the image position of the reflection point at each moment includes: transforming the image position of the reflection point at each moment from the image coordinate system of the event camera to the image coordinate system of the virtual camera based on the intrinsic parameters in the parameters of the event camera; estimating the position of the center point of the corneal ball at the moment in the three-dimensional coordinate system of the virtual camera by using the regressor based on the image position of the reflection point at the moment in the image coordinate system of the virtual camera; and determining the position of the center point of the corneal ball at the moment in the three-dimensional coordinate system of the event camera according to the position of the center point of the corneal ball at the moment in the three-dimensional coordinate system of the virtual camera.

[0008] Optionally, the method estimates the position of the center point of the corneal sphere at each moment through a numerical solver based on the image position of the reflection point at each moment, the parameters of the event camera, the position of the light source and the radius of the corneal sphere, including: transforming the image position of the reflection point at each moment into the calibrated image coordinate system of the event camera according to the intrinsic parameters in the parameters of the event camera; transforming the position of the light source into the three-dimensional coordinate system of the event camera according to the extrinsic parameters in the parameters of the event camera; estimating the position of the center point of the corneal sphere at each moment through the numerical solver based on the image position of the reflection point at each moment in the calibrated image coordinate system, the position of the light source in the three-dimensional coordinate system of the event camera, and the radius of the corneal sphere.

[0009] Optionally, estimating the eyeball rotation center position based on the estimated corneal ball center position at multiple moments and the calibrated distance between the eyeball rotation center and the corneal ball center includes: estimating the center point position of a rotating sphere as the eyeball rotation center point position, wherein the rotating sphere is a sphere with the distance as a radius formed based on the motion trajectory of the corneal ball center point position at the multiple moments.

[0010] Optionally, estimating the position of the eyeball rotation center point includes: if the range of the movement trajectory of the corneal ball center point position at the multiple moments exceeds a predetermined range threshold, estimating the position of the eyeball rotation center point.

[0011] Optionally, the method also includes: when it is detected that the head-mounted device is sliding, using the estimated eye rotation center point position to update the current eye rotation center point position, wherein the head-mounted device includes the event camera and the light source, wherein the three-dimensional line of sight related information based on the corneal ball center point position and the eye rotation center point position includes: determining the three-dimensional line of sight related information based on the corneal ball center point position and the updated eye rotation center point position.

[0012] Optionally, when the head-mounted device is detected to be sliding, the current eye rotation center point position is updated using the estimated eye rotation center point position, including: when the change of the estimated eye rotation center point position exceeds a predetermined threshold, the current eye rotation center point position is updated using the estimated eye rotation center point position.

[0013] Optionally, the estimated eye rotation center point position includes an estimated monocular eye rotation center point position, and when a change in the estimated eye rotation center point position exceeds a predetermined threshold, updating the current eye rotation center point position using the estimated current eye rotation center point position comprises: comparing the estimated monocular eye rotation center point position with the current monocular eye rotation center point position; if the difference between the estimated monocular eye rotation center point position and the current monocular eye rotation center point position exceeds a first threshold, updating the current monocular eye rotation center point position using the estimated monocular eye rotation center point position.

[0014] Optionally, the estimated eye rotation center point position includes: the left eye eye rotation center point position and the right eye eye rotation center point position, and when the change of the estimated eye rotation center point position exceeds a predetermined threshold, the current eye rotation center point position is updated using the estimated eye rotation center point position, including: estimating the posture of the head relative to the head-mounted device based on the estimated left eye eye rotation center point position and the right eye eye rotation center point position; comparing the estimated posture of the head relative to the head-mounted device with the current posture of the head relative to the head-mounted device; if the difference between the estimated posture and the current posture exceeds a second threshold, updating the current posture with the estimated posture, and updating the current left eye eye rotation center point position and the current right eye rotation center point position based on the updated posture.

[0015] Optionally, determining the three-dimensional line of sight related information based on the corneal ball center point position and the eyeball rotation center point position includes: determining the eye optical axis based on the corneal ball center point position and the eyeball rotation center point position; based on the eye optical axis and the Kappa angle, determining the related information of the eye visual axis as the three-dimensional line of sight related information, wherein the Kappa angle is the angle between the calibrated eye optical axis and the eye visual axis.

[0016] Optionally, obtaining the reflective point information includes: using the event camera to capture the reflective point signal to obtain event data; processing the event data to obtain the reflective point information, wherein the reflective point information includes the reflective point image position, wherein the reflective point image position is the position of the reflective point corresponding to the light source in the image coordinate system of the event camera.

[0017] Optionally, the processing of the event data to obtain the reflective point information includes: performing frequency filtering on the event data to obtain a binary code sequence of reflective points, wherein the binary code sequence is used to drive the light source; determining a predefined code in the binary code sequence, the predefined code including a code that can be uniquely determined in the binary code sequence when a preset condition is met; determining a timestamp corresponding to a start bit of the binary code sequence according to the determined predefined code; determining a reflective point label according to the timestamp and a predefined coding rule of the light source; and obtaining the reflective point image position of the reflective point corresponding to the determined reflective point label as the reflective point information.

[0018] Optionally, key bits in the codes of the light source corresponding to the predefined code and the light source adjacent to the predefined code are different from each other.

[0019] Optionally, the light sources corresponding to the predefined codes are arranged such that the distances therebetween meet a predetermined distance requirement.

[0020] According to a second aspect of an embodiment of the present disclosure, there is provided a device for line of sight tracking, comprising: a reflective point information acquisition unit, configured to acquire reflective point information, wherein the reflective point information is obtained based on a reflective point signal of light emitted by a light source captured by an event camera and reflected by the surface of a corneal ball; a position estimation unit, configured to estimate the position of the center point of the corneal ball and the position of the center point of eyeball rotation based on the reflective point information; and a line of sight determination unit, configured to determine three-dimensional line of sight related information based on the position of the center point of the corneal ball and the position of the center point of eyeball rotation.

[0021] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, characterized in that it includes: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the method for eye tracking as described above.

[0022] According to a fifth aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, characterized in that when the instructions are executed by at least one processor, the at least one processor is prompted to execute the method for line of sight tracking as described above.

[0023] According to the method and device for line of sight tracking of the embodiments of the present disclosure, since the reflection point information is obtained by capturing the reflection point signal of the light emitted by the light source reflected by the surface of the corneal ball based on the event camera, the position of the center point of the corneal ball and the position of the center point of eyeball rotation are estimated based on the reflection point information, and the three-dimensional line of sight related information is determined according to the position of the center point of the corneal ball and the position of the center point of eyeball rotation. Therefore, the three-dimensional line of sight related information can be determined without using pupil information, thereby realizing line of sight tracking more efficiently and at a low cost.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate exemplary embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0026] Figure 1 is a flowchart of a method for eye tracking according to an embodiment of the present disclosure.

[0027] Figure 2 is a schematic diagram showing the working principle of an event camera.

[0028] Figure 3 It is a schematic diagram of the optical path of the light source reflected by the reflective point on the surface of the cornea of ​​the eye to form an image on the camera imaging plane.

[0029] Figure 4 It is a schematic diagram of the position of the LED light source.

[0030] Figure 5 It is a schematic diagram of the signal for controlling the LED light source according to the coding method of Table 2.

[0031] Figure 6 A schematic diagram showing an example of obtaining reflective point labels.

[0032] Figure 7 is a schematic diagram showing a first example of estimating the position of the center point of the corneal sphere.

[0033] Figure 8 is a schematic diagram showing a second example of estimating the position of the center point of the corneal sphere.

[0034] Fig. 9 A schematic diagram showing the relationship between the eyeball rotation center and the corneal ball center point position at K moments.

[0035] Fig.10 Schematic diagram showing the estimated position of the eyeball rotation center point.

[0036] Fig.11 Schematic diagram showing how the three-dimensional line of sight is determined using the position of the corneal sphere center point and the position of the eyeball rotation center point.

[0037] Fig.12 It is a schematic diagram of monocular-based sliding detection and compensation.

[0038] Fig.13 It is a schematic diagram of the relationship between the eye positions in the head coordinate system and the device coordinate system.

[0039] Fig.14 It is a schematic diagram of binocular-based sliding detection and compensation.

[0040] Fig.15 is a schematic diagram showing an example of a method for gaze tracking according to an embodiment of the present disclosure.

[0041] Fig.16 It is a block diagram showing an apparatus for implementing tracking according to an embodiment of the present disclosure.

[0042] Fig.17 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0044] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0045] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0046] As mentioned in the background of this application, current three-dimensional (3D) gaze tracking methods always need to extract and use pupil information for gaze tracking, but additional hardware and software are required to obtain pupil signals. To this end, the present disclosure proposes a 3D gaze tracking method that does not rely on pupil information, allowing the 3D gaze to be determined using only the reflective point information obtained by the event camera, so that the 3D gaze can be determined more efficiently and at a low cost.

[0047] Hereinafter, a method for gaze tracking according to an exemplary embodiment of the present disclosure will be described with reference to the accompanying drawings.

[0048] Figure 1 is a flow chart showing a method for eye tracking according to an exemplary embodiment of the present disclosure. Figure 1 , in step S110, the reflection point information is obtained, wherein the reflection point information is obtained based on the reflection point signal of the light emitted by the light source reflected by the corneal surface captured by the event camera. An event camera is a special camera based on the principle of biological retina that only generates signals for local brightness changes. With the support of a specially designed light source, the event data captured by the event camera only contains the reflection point information reflected by the light source on the surface of the cornea, which can greatly reduce the computing load so that the line of sight tracking can achieve low power consumption and low latency performance. For example, event cameras bring good news to extended reality devices (e.g., head-mounted display devices) with limited speed and power. An event camera is a novel sensor, such as Figure 2 As shown in (a), compared with standard frame cameras (here, frame cameras refer to all cameras that capture image frames, such as RGB cameras, grayscale cameras, etc.), event cameras only measure changes in the scene and output event streams. Pixels in an event camera operate asynchronously and independently, reporting changes as events—tuples of position (x, y), polarity s, and timestamp t—with microsecond resolution. For example, an event camera can operate at low power (5mW) and respond to changes in the scene with microsecond delays. Figure 2 As shown in (b), if logI(x, t)-logI(x, t-Δt)=±C, an event is triggered and the event camera captures the event. Figure 2 In (b), the arrow direction indicates polarity, C indicates the threshold of change, t indicates time, and I(x, t) indicates the signal intensity of the reflective point. If the intensity change exceeds the threshold, the event is triggered. These characteristics enable event cameras to be used as gaze tracking sensors on mobile platforms such as augmented / virtual reality (AR / VR) devices, as these systems have strict real-time and power consumption constraints.

[0049] Figure 3 It is a schematic diagram of the optical path of the light source reflected by the reflective point on the surface of the cornea of ​​the eye and imaged on the camera imaging plane. Figure 3As shown, the event camera captures the reflected light point signal of the light emitted by the light source reflected by the corneal sphere surface. The present disclosure determines the three-dimensional line of sight by using the reflected light point information obtained based on the reflected light point signal of the light emitted by the light source reflected by the corneal sphere surface captured by the event camera. The three-dimensional line of sight is Figure 3 The visual axis of the eye is shown in .

[0050] According to an embodiment, step S110 may include using an event camera to capture a reflective point signal to obtain event data; processing the event data to obtain the reflective point information, wherein the reflective point information includes a reflective point image position, wherein the reflective point image position is the position of the reflective point corresponding to the light source in the image coordinate system of the event camera.

[0051] For example, in order to obtain the image position of a reflective point, it is necessary to know the number of the reflective point. Since the light source corresponds to the reflective point, the number of the light source is the number of the reflective point. Usually, a multi-bit binary code is used to encode the numbers of multiple different light sources. However, in practice, when the synchronization between the camera and the light source is not available, it is difficult to identify the starting bit of the binary coding sequence, which affects the determination of the number of the reflective point. Optionally, in order to reduce the ambiguity in determining the starting bit of the binary coding sequence, the present disclosure further proposes an anti-ambiguous coding method for the light source.

[0052] The following describes the anti-ambiguity encoding method by taking 10 light sources (e.g., LED lights) as an example. Specifically, if 10 light sources are to be used, 4 binary digits are required to encode them. Since 4 binary digits can generate a total of 16 different codes, 10 of them can be selected as the codes for the 10 light sources.

[0053] For example, the 10 codes shown in the following Table 1 may be selected.

[0054]

[0055]

[0056] Table 1

[0057] In Table 1, an "unambiguous" code means that even if the starting bit is unknown, the code number (e.g., the number of the light source corresponding to the code, or the number of the reflective point corresponding to the code, etc.) can still be identified. For example, if the reflective point has four consecutive bits containing three zeros and one 1 (e.g., in any order) without knowing the starting bit, it may come from "0001", "0010", "0100" and "1000". If the codes "0001", "0010", and "0100" are excluded from the 10 codes, for example, the codes "0001", "0010", and "0100" are not used, then if the reflective point has four consecutive bits containing three zeros and one 1 (e.g., in any order) without knowing the starting bit, it must come from the code "1000". Therefore, "1000" is an unambiguous code. That is to say, if the codes "0001", "0010", and "0100" are not used, the code "1000" can be easily identified, so as to determine the number of the light source corresponding to the code (for example, the number of the reflective point corresponding to the code) without being confused with other codes and causing ambiguity.

[0058] The start position of the code is "unambiguous" if the code's corresponding label is known, and the start position of the code can be easily identified. For example, for "1000", the start position can be identified by detecting the first position 1.

[0059] According to the above rules, the two codes "1000" and "0111" in the light source code in Table 1 have neither code ambiguity nor start bit ambiguity, and thus have good label recognition and start bit recognition performance at the same time. Such codes can be called "unambiguous codes". Such "unambiguous codes" can be predefined, so the "unambiguous codes" in this article can also be called "predefined codes", wherein the predefined codes are codes that can be uniquely determined in a binary code sequence under the condition that predetermined conditions are met. For example, the predetermined conditions can be the conditions mentioned in the example of Table 1 above that the codes are unambiguous and the start bits are unambiguous. After a light source code including a predefined code is designed, such as shown in Table 1, the designed binary code sequence of the light source can be used to control the light source, wherein 1 means the light source is on and 0 means the light source is off.

[0060] In the case of adopting the above-mentioned anti-ambiguous coding method, after using an event camera to capture the reflective point signal of the light emitted by the light source reflected by the corneal ball surface to obtain event data, the event data is processed to obtain the reflective point information, which may include: performing frequency filtering on the event data to obtain a binary code sequence of the reflective point, wherein the binary code sequence is used to drive the light source; determining a predefined code in the binary code sequence, the predefined code is a code that can be uniquely determined in the binary code sequence when preset conditions are met; determining a timestamp corresponding to the start bit of the binary code sequence according to the determined predefined code; determining the reflective point label according to the timestamp and the pre-defined coding rule of the light source; finally, obtaining the reflective point image position of the reflective point corresponding to the determined reflective point label as the reflective point information.

[0061] For example, in the case where the light source coding is designed as shown in Table 1, after obtaining the binary coding sequence of the reflective point, the code "1000" or the code "0111" can be determined first. Any of them can be used to determine the starting bit of the binary coding sequence. After the predefined code "1000" or "0111" is determined, the starting bit of the predefined code is the starting bit of the binary coding sequence. The timestamp of the start bit can be used to find the starting bits of other codes and determine their labels. For example, based on the timestamp of the start bit and the coding rule of the light source, the starting bit of the coding of other light sources except the light source corresponding to the unambiguous code can be determined, and the coding of other light sources can be determined based on the determined starting bit, and then the table (for example, Table 1) can be looked up to determine the label of the light source. One light source corresponds to one reflective point, and the label of the light source is the label of the reflective point, so the label of the reflective point can be determined. After the label of the reflective point is determined, the reflective point image position of the reflective point corresponding to the determined reflective point label can be obtained as the reflective point information. For example, Figure 3 As shown, after the reflective point is determined according to the reflective point label, the position of the reflective point in the image plane, for example, the reflective point image position, can be determined based on the reflective point and the camera optical center.

[0062] Through the above method, the ambiguity in determining the starting position of the binary code sequence can be reduced, thereby improving the accuracy of the acquired reflective point information.

[0063] According to an embodiment of the present disclosure, in order to further reduce errors in the process of identifying the starting bit, optionally, the key bits in the encoding of the light source corresponding to the predefined code and the light source adjacent to the light source may be different from each other. According to an embodiment, the key bit is a predetermined coding bit in the encoding of the light source. In the above manner, it can be made that for the key bit in the predefined code, its adjacent LED lights always have different states, thereby facilitating better identification of the starting bit. As an example, the predetermined coding bit can be the first bit in the code. For example, for example, the first bit of LED-9 is 0, while its adjacent LED-8 and LED-10 are both different 1s. Based on the above considerations, the encoding method shown in Table 1 can be further improved to the encoding method shown in Table 2.

[0064]

[0065] Table 2

[0066] As shown above, the predefined code (also referred to as the "key code") can be used to identify the starting position of the binary code sequence. Therefore, it is expected that at least one of the light sources corresponding to the predefined code is visible. This is because when the eye looks to the side, the corneal area may lose some flicker. In order to ensure that the flicker of at least one light source corresponding to the predefined code is not lost, the present disclosure further proposes that the light sources corresponding to the predefined code are arranged so that the distance between each other meets the predetermined distance requirement. For example, if all the light sources form a circle, the light sources corresponding to the predefined code can be separated from each other by a diameter. For example, in the coding design of Table 1, "1000" and "0111" are both predefined codes. The light source corresponding to "1000" (LED-4) and the light source corresponding to "0111" (LED-9) can be arranged so that the distance between each other meets the predetermined distance requirement. For example, Figure 4 As shown, if all light sources are surrounded by a circle, LED-4 and LED-9 can be separated by a diameter.

[0067] After the coding of the light source is designed as shown in Table 2, the designed coding can be used to control the light source, where 1 means the light source is on and 0 means the light source is off. For example, the schematic diagram of the signal for controlling the LED light source according to the coding method of Table 2 is as follows: Figure 5 As shown. Figure 5In the example, each LED light source includes an LED-primary light source and an LED-secondary light source that are close to each other, wherein the LED-primary light source and the LED-secondary light source have opposite states, that is, when the LED-primary light source is on, the LED-secondary light source is off, and when the LED-secondary light source is off, the LED-primary light source is on. The codes of each LED light source in Table 2 are the codes of the LED-primary light source. Figure 5 As shown, for example, if in the first clock cycle and the second clock cycle of four clock (CLK) cycles it is identified that the light source is on for the first one-third of the time and is off for the last two-thirds of the time, and in the third clock cycle and the fourth clock cycle it is identified that the light source is on for the first two-thirds of the time and is off for the last one-third of the time, then it can be identified that the code of the light source is 0011, and the light source is LED-1, thereby determining that the label of the reflective point corresponding to the light source is 1.

[0068] Figure 6 A schematic diagram showing an example of obtaining the reflective point label. If the encoding method in Table 2 is used, then Figure 6 The reflective point number is obtained in the manner shown. Figure 6 As shown, first, the binary code sequence (also called "01 state sequence") of the reflective point is obtained from the event data stream through frequency filtering, and then it is identified whether there are three consecutive 0 bits or three consecutive 1 bits. If there are three consecutive 0 bits, the code "1000" is identified, thereby determining that the starting bit of the binary code sequence is the position where 1 is located, and if there are three consecutive 1 bits, the code "0111" is identified, thereby determining that the starting bit of the binary code sequence is the position where 0 is located. Subsequently, the timestamp corresponding to the starting bit can be determined. Finally, according to the timestamp corresponding to the starting bit and the 4-bit encoding rule, the label of the reflective point can be obtained by looking up Table 2.

[0069] Return to reference Figure 1 After obtaining the reflective point information, the method for eye tracking according to the embodiment of the present disclosure does not need to obtain the through-hole information to determine the three-dimensional eye-line related information. Specifically, the method for eye tracking according to the embodiment of the present disclosure does not estimate the pupil center, but determines the three-dimensional eye-line related information based on the eyeball rotation center and the corneal ball center.

[0070] Specifically, in step S120, the position of the center point of the corneal ball and the position of the center point of eyeball rotation are estimated based on the reflection point information.

[0071] According to an embodiment, step S120 may include: estimating the position of the corneal ball center point at the moment based on the reflective point information at each moment; estimating the position of the eyeball rotation center point based on the estimated corneal ball center point positions at multiple moments and the distance between the calibrated eyeball rotation center point and the corneal ball center point. It should be noted that the "moment" here can be a real-time moment or a non-real-time moment. In addition, the "moment" can be a periodic moment (that is, a moment every predetermined time period) or a non-periodic moment (that is, the duration between moments is not fixed). The present disclosure has no restrictions on this. For example, the "moment" can be set according to the application scenario of the method for line of sight tracking disclosed in the present disclosure, for example, the "moment" can be set to be real-time or non-real-time, periodic or non-periodic. For example, it can be set to obtain reflective point information every 10 seconds, but it is not limited to this.

[0072] The following first introduces how to estimate the position of the center point of the corneal sphere.

[0073] According to an embodiment, the reflective point information includes a reflective point image position, wherein the reflective point image position is the position of the reflective point corresponding to the light source in the image coordinate system of the event camera. Here, the position of the reflective point corresponding to the light source in the image coordinate system is the position of the reflective point in the image coordinate system. Figure 3 The position in the image plane shown in , for example, the position where the reflection point is projected onto the image plane through the optical center of the camera.

[0074] According to an embodiment, estimating the position of the center point of the corneal sphere at each moment based on the reflection point information at each moment includes: estimating the position of the center point of the corneal sphere at each moment by using a regressor based on the reflection point image position at each moment; or estimating the position of the center point of the corneal sphere at each moment by using a numerical solver based on the reflection point image position at each moment, the parameters of the event camera, the position of the light source and the radius of the corneal sphere.

[0075] Figure 7 is a schematic diagram showing a first example of estimating the position of the center point of the corneal sphere.

[0076] exist Figure 7 In the example, based on the image position of the reflection point at each moment, a regressor is used to estimate the center point position of the corneal sphere at that moment. According to an embodiment, the regressor is trained based on the simulated image position of the reflection point, wherein the simulated image position of the reflection point is simulated based on the light source position, the parameters of the virtual camera corresponding to the event camera, and the corneal sphere radius.

[0077] For example, Figure 7As shown, the position of the reflection point image can be simulated in advance according to the light source position, the parameters of the virtual camera corresponding to the event camera, and the corneal sphere radius. Then, the regressor can be trained using the simulated reflection point image position, so that the trained regressor can estimate the position of the corneal sphere center point in the three-dimensional coordinate system of the virtual camera based on the simulated reflection point image position. As an example, the regressor can be a linear regressor, a polynomial-based regressor, a neural network-based regressor (such as a multi-layer perceptron model MLP), but is not limited thereto.

[0078] After the regressor is trained, when the image position of the reflection point at each moment is obtained, the position of the center point of the corneal ball at the moment can be estimated using the pre-trained regressor based on the image position of the reflection point at each moment.

[0079] For example, Figure 7 As shown, after obtaining the image position Gi(x, y) of the reflection point at each moment (where i is the number of the reflection point, the reflection point corresponds to the light source, and therefore, the number of the light source can be understood as the number of the reflection point), first, based on the intrinsic parameters in the parameters of the event camera, the image position of the reflection point at each moment can be transformed from the image coordinate system of the event camera to the image coordinate system of the virtual camera. The intrinsic parameters of the camera are parameters related to the characteristics of the camera itself, such as the focal length and pixel size of the camera. Secondly, based on the image position of the reflection point at the moment in the image coordinate system of the virtual camera, the position of the center point of the cornea ball at the moment in the three-dimensional coordinate system of the virtual camera is estimated by the regressor. Finally, according to the position of the center point of the cornea ball at the moment in the three-dimensional coordinate system of the virtual camera, the position of the center point of the cornea ball at the moment in the three-dimensional coordinate system of the event camera is determined.

[0080] As an example, the image position of a reflective point can be transformed from the image coordinate system of the event camera to the image coordinate system of the virtual camera using the following formula:

[0081] a*[Gix',Giy',1]=K_vir*inv(K_cam)*[Gix,Giy,1],

[0082] Among them, a is a scale factor used to ensure that the last item of the homogeneous coordinate vector [Gix', Giy', 1] is 1, K_cam is the intrinsic parameter of the event camera, K_vir is the intrinsic parameter of the virtual camera, [Gix, Giy] is the image position of the reflection point in the image coordinate system of the event camera, and [Gix', Giy'] is the image position of the reflection point in the image coordinate system of the virtual camera. Through the above transformation, the domain difference caused by the lens used in the real event camera is eliminated. Figure 7The example allows the same regressor to be trained on simulated data from event cameras in different head-mounted devices. The method of estimating the position of the corneal sphere center point is not limited to Figure 7 For example, you can also use Figure 8 Example of estimating the location of the center point of the corneal sphere.

[0083] Figure 8 is a schematic diagram showing a second example of estimating the position of the center point of the corneal sphere.

[0084] according to Figure 8For example, the position of the center point of the corneal ball at each moment can be estimated by a numerical solver based on the image position of the reflection point at each moment, the parameters of the event camera, the position of the light source and the radius of the corneal ball. For example, after obtaining the image position of the reflection point at each moment Gi (x, y), the image position of the reflection point at each moment can be first transformed into the corrected image coordinate system of the event camera according to the internal parameters in the parameters of the event camera. Since the eye may have lens distortion similar to a fisheye camera, in order to correct the lens distortion, the image position of the reflection point is transformed into the corrected image coordinate system of the event camera. Secondly, the position of the light source can be transformed into the three-dimensional coordinate system of the event camera according to the external parameters in the parameters of the event camera. The camera external parameters are parameters in the world coordinate system, such as the position of the camera, the posture of the camera (for example, the rotation direction), etc. Finally, based on the image position of the reflection point at each moment in the corrected image coordinate system, the position of the light source in the three-dimensional coordinate system of the event camera, and the radius of the corneal ball, the center point position of the corneal ball at the moment can be estimated by the numerical solver. For example, based on the reflection points of two light sources captured by the event camera, the three-dimensional position coordinates of the center point of the corneal sphere (the center of the corneal sphere) can be calculated according to the following method. This method utilizes the geometric constraints between the incident light, the reflected light, and the normal direction of the corneal sphere surface. When the radius of the corneal sphere is known, the center of the corneal sphere can be solved by a numerical search algorithm. Specifically, considering two light source points, point light source 1 and point light source 2, according to the camera optical center, the position of the light source calibrated in advance (for example, the position of the light source in the three-dimensional coordinate system of the event camera), the image position of the reflection point in the corrected image coordinate system, and the camera internal parameters, the plane 1 determined by the incident light, the reflection point, and the reflected light of the light source 1, and the plane 2 determined by the incident light, the reflection point, and the reflected light of the light source 2 can be determined. The straight line obtained by the intersection of plane 1 and plane 2 is the line connecting the optical center of the camera and the center point of the corneal sphere. Therefore, as long as the distance k from the optical center of the camera to the center of the corneal sphere is additionally calculated, the three-dimensional position of the center of the corneal sphere can be determined. It can be seen that the three-dimensional coordinates of the center of the corneal sphere can be expressed as a parameterized representation with k as the only unknown parameter. According to the law of optical reflection, the incident angle is equal to the reflection angle, and a constraint equation will be obtained, which contains only one unknown number k. By iteratively searching for k to find a numerical solution that satisfies the constraint equation, k can be solved, and after knowing k, the three-dimensional coordinates of the center of the corneal sphere can be obtained. After estimating the position of the center of the corneal sphere at multiple times based on the position of the image of the reflective point, the parameters of the event camera, the position of the light source, and the radius of the corneal sphere, the position of the center of eye rotation can be estimated based on the estimated position of the center of the corneal sphere at multiple times and the distance between the calibrated center of eye rotation and the center of the corneal sphere.For example, the center point position of a rotating sphere (hereinafter also referred to as a "corneal ball center point rotating sphere") can be estimated as the center point position of the eyeball rotation, wherein the rotating sphere is a sphere with the distance as the radius formed based on the motion trajectory of the corneal ball center point position at the multiple moments.

[0085] Fig. 9 A schematic diagram showing the relationship between the eyeball rotation center and the corneal ball center position at K moments. Fig. 9 As shown, when the eye moves, since the corneal ball moves around the eyeball rotation center, the movement trajectory of the corneal ball center should be on the surface of a three-dimensional sphere centered on the eyeball rotation center. This sphere is the corneal ball center point rotation sphere. The corneal ball center point rotation sphere has the eyeball rotation center d as the center, and its radius is the distance between the eyeball rotation center d and the corneal ball center c. This distance DC = ||dc|| is user-specific and can be calibrated during the user's eye calibration process and then saved in advance. For example, Fig. 9 As shown, the corneal sphere center point rotating sphere may be a sphere with a distance DC as a radius formed based on the movement trajectory of the corneal sphere center points c1 to ck from time T1 to TK.

[0086] When the positions (x, y, z) of the center of the cornea sphere and the radius R (e.g., distance DC) of the rotation sphere of the center of the cornea sphere are obtained at multiple moments, the center position (x, y, z) of the rotation sphere of the center of the cornea sphere can be fitted by, for example, the following method: c ,y c , z c ):

[0087] Assume that the spherical function of the rotation sphere at the center of the corneal sphere is:

[0088] (xx c ) 2 +(yy c ) 2 +(zz c ) 2 =R 2 ,

[0089] This is equivalent to:

[0090]

[0091] Assume the unknown vector p is:

[0092]

[0093] Assume that Ap = b, where the Ai in the i-th row of A is: [-2xi,-2yi,-2zi,1]', and the bi in the i-th row of b is [-xi*xi-yi*yi-zi*zi], then p can be solved as follows: p = (A T A) -1 A T b.

[0094] In order to avoid digital degradation in fitting, the eye movement can be checked before fitting to ensure that the trajectory crosses a predetermined range threshold, for example, across a certain angle range. Optionally, estimating the position of the eyeball rotation center point at the multiple moments includes: if the range of the movement trajectory of the corneal ball center point position at the multiple moments exceeds a predetermined range threshold, then estimating the position of the eyeball rotation center point. For example, Fig. 9 As shown, if the motion trajectory from c1 to ck spans a predetermined angle range (eg, 30 degrees), the center point position of the corneal sphere center rotating sphere is estimated, otherwise the corneal sphere center point position at the next moment is continuously acquired.

[0095] Fig.10 is a schematic diagram showing the estimated position of the center point of eyeball rotation. Assuming that the position of the center point of the cornea ball is represented by CSC, the positions of the center point of the cornea ball at multiple moments constitute a CSC set. Fig.10 As shown, first, the corneal ball center point position CSC at a new moment is read and added to the CSC set, and then, it is determined whether the range of the motion trajectory corresponding to the CSC set exceeds the predetermined range threshold. If so, the center point of the rotating sphere of the corneal ball center point is fitted on the CSC set. For example, the coordinates of the center point position of the rotating sphere of the corneal ball center point can be fitted based on the previously calibrated radius of the rotating sphere of the corneal ball center (for example, the distance DC in the above text) and the CSC set in the manner described above, and then the coordinates of the eyeball rotation center point position are determined; if not, the corneal ball center point position CSC at the next moment is read and added to the CSC set.

[0096] Return to reference Figure 1, after estimating the position of the corneal ball center point and the position of the eyeball rotation center point, in step S130, the three-dimensional sight line related information can be determined according to the position of the corneal ball center point and the position of the eyeball rotation center point. According to an embodiment, the eye optical axis can be first determined according to the position of the corneal ball center point and the position of the eyeball rotation center point, and then, based on the eye optical axis and the Kappa angle, the information related to the eye visual axis can be determined as the three-dimensional sight line related information, wherein the Kappa angle is the angle between the calibrated eye optical axis and the eye visual axis. As an example, the three-dimensional sight line related information can be the angle information between the eye visual axis (e.g., three-dimensional sight line) and the three spatial axes in the three-dimensional space, but is not limited thereto, and the three-dimensional sight line related information can be different according to the representation form of the three-dimensional sight line.

[0097] like Figure 3 As shown, the optical axis of the glasses can be determined by connecting the corneal spherical center and the eyeball rotation center. Let the corneal spherical center be c and the eyeball rotation center be d, then the horizontal angle and vertical angle (φ and θ respectively) related to the optical axis of the eye can be calculated by solving the equation:

[0098]

[0099] After determining the horizontal angle and vertical angle associated with the optical axis of the eye, the visual axis of the eye can be determined based on the horizontal angle and vertical angle associated with the optical axis of the eye based on the Kappa angle. The Kappa angle is the angle between the optical axis of the eye and the visual axis of the eye (e.g., three-dimensional line of sight), which is fixed for a specific user and can therefore be pre-calibrated. Assuming that the Kappa angle is represented by the horizontal angle α and the vertical angle β, the angle information of the three spatial axes associated with the three-dimensional line of sight can be calculated, for example, in the following manner:

[0100] g=[cos(θ+β)sin(θ+α),sin(φ+β),-cos(φ+β)sin(θ+α)]

[0101] The method for determining three-dimensional sight-line related information according to an embodiment of the present disclosure has been described above. Fig.11 is a schematic diagram showing an example of determining three-dimensional sight-related information using the position of the corneal ball center point and the eyeball rotation center point position. Fig.11 Provide a brief description.

[0102] like Fig.11As shown, first, the event data is acquired by shooting with an event camera, and then the number and position of the reflective point are obtained by processing the event data, and then the image position Gi(x, y) of the reflective point is obtained, where i is the number of the reflective point, i=1,…,N. Next, for each cycle, the position of the center point of the corneal ball is estimated based on the image position of the reflective point at each moment, the parameters of the event camera, the position of the light source, and the radius of the corneal ball. The parameters of the event camera and the position of the light source can be obtained by system calibration of a device (e.g., a head-mounted device including an event camera and a light source). The radius of the corneal ball can be obtained based on the calibration of the wearer's eye parameters, for example, by calibrating the eye parameters of the wearer of the head-mounted device. After estimating the center point position of the corneal ball at K moments, the position of the eyeball rotation center point can be estimated based on the center point position of the corneal ball at K moments. Subsequently, the eyeball optical axis can be determined based on the estimated corneal ball center point position and the eyeball rotation center point position, and the eye visual axis can be determined based on the Kappa angle and the determined eye optical axis as a three-dimensional line of sight. The Kappa angle can be obtained based on the wearer's eye parameter calibration. Fig.11 In the previous content Fig.11 The details of each operation designed in have been described, so they will not be repeated here.

[0103] According to the method for line of sight tracking of the embodiment of the present disclosure, since the reflection point information is obtained by capturing the reflection point signal of the light emitted by the light source reflected by the surface of the corneal ball based on the event camera, the position of the center point of the corneal ball and the position of the center point of eyeball rotation are estimated based on the reflection point information, and the three-dimensional line of sight related information is determined according to the position of the center point of the corneal ball and the position of the center point of eyeball rotation. Therefore, the three-dimensional line of sight can be determined without using pupil information, thereby realizing line of sight tracking more efficiently and at a low cost.

[0104] The method for eye tracking according to an embodiment of the present disclosure can be applied to a head-mounted device. If the device slides during use of the head-mounted device (for example, unexpected head movement between the user and the head-mounted device), errors may occur in the three-dimensional line of sight determination. After a slide occurs, a recalibration process is usually required, but the recalibration process requires the user to look at one / multiple points on the display screen. The requirement for recalibration after each slide reduces the user experience. In response to this, the present disclosure further proposes a method for detecting slides and compensating for errors caused by the slide. Specifically, the slide can be detected by detecting changes in the center of rotation of the eyeball. When a slide is detected, the last determined position of the center of rotation of the eyeball is updated to compensate for the gaze error caused by the slide.

[0105] Optionally, Figure 1The method for eye tracking shown may also include: when a head mounted device is detected to be sliding, using the estimated current eye rotation center point position to update the last determined eye rotation center point position, wherein the head mounted device includes the event camera and the light source. Accordingly, step S130 may determine the three-dimensional eye-related information based on the estimated corneal ball center point position and the updated eye rotation center point position.

[0106] As an example, it is possible to detect whether the head mounted device has slipped based on the change in the estimated position of the eye rotation center point. If the change in the estimated position of the eye rotation center point exceeds a predetermined threshold, it can be considered that the head mounted device has slipped, otherwise it is considered that no slip has occurred. That is to say, when the change in the estimated position of the eye rotation center point exceeds a predetermined threshold (it is considered that slipping has occurred at this time), the current position of the eye rotation center point is updated using the estimated position of the eye rotation center point. It should be noted that the current position of the eye rotation center point can be the calibrated position of the eye rotation center point, or it can be the position of the eye rotation center point obtained by the last update. For example, if the head mounted device is detected to slip for the first time after the eye rotation center point position is calibrated, the calibrated position of the eye rotation center point is updated with the estimated position of the eye rotation center point. After that, if the head mounted device slips again, the estimated position of the eye rotation center point continues to be used to update the position of the eye rotation center point obtained by the last update.

[0107] According to the embodiment, the sliding detection and compensation may be performed based on a single eye or based on both eyes.

[0108] Fig.12 This is a schematic diagram of monocular-based sliding detection and compensation. Fig.12 As shown, the corneal ball center position is estimated first, and then the eye rotation center (ERC) position is estimated. The estimation of the corneal ball center position and the estimation of the eye rotation center position have been described above and will not be repeated here. According to an embodiment, the eye rotation center position estimated in step S120 may include an estimated monocular eye rotation center position. In this case, when the change in the estimated eye rotation center position exceeds a predetermined threshold, updating the current eye rotation center position using the estimated eye rotation center position includes: Fig.12As shown, the estimated position of the monocular eyeball rotation center point is compared with the current position of the monocular eyeball rotation center point. If the difference between the estimated position of the monocular eyeball rotation center point and the current position of the monocular eyeball rotation center point exceeds the first threshold, the estimated position of the monocular eyeball rotation center point is used to update the current position of the monocular eyeball rotation center point. Otherwise, the comparison step described above is returned to be executed. Finally, the three-dimensional sight-line related information is determined according to the estimated corneal ball center point position and the updated ERC position.

[0109] Fig.13 It is a schematic diagram of the relationship between the eye positions in the head coordinate system and the device coordinate system. Assuming that the eye rotation center of the left eye and the right eye is fixed in the user's head coordinate system, the ERC position can be updated based on the transformation of the head to the device posture. Fig.14 This is a schematic diagram of sliding detection and compensation based on both eyes. Fig.14 As shown, the corneal ball center position is estimated first, and then the eye rotation center (ERC) position is estimated. The estimation of the corneal ball center position and the estimation of the eye rotation center position have been described above and will not be repeated here. According to an embodiment, the eye rotation center position estimated in step S120 may include the left eye rotation center position and the right eye rotation center position. In this case, when the change of the estimated eye rotation center position exceeds a predetermined threshold, the current eye rotation center position is updated using the estimated eye rotation center position, including: estimating the posture of the head relative to the head-mounted device based on the estimated left eye rotation center position and the right eye rotation center position; comparing the estimated posture of the head relative to the head-mounted device with the current posture of the head relative to the head-mounted device; if the difference between the estimated posture and the current posture exceeds a second threshold, updating the current posture with the estimated posture, and updating the current left eye rotation center position and the current right eye rotation center position based on the updated posture. For example, as Fig.14As shown, the 6-DOF head posture relative to the head-mounted device (referred to as "head-to-device posture") can be estimated based on the estimated left-eye ERC position and the right-eye ERC position. Subsequently, the estimated 6-DOF head-to-device posture and the current head-to-device posture are compared. If the difference between the two exceeds a second threshold, the current head-to-device posture is updated with the estimated 6-DOF head-to-device posture, and the current ERC position is updated based on the updated head-to-device posture. Otherwise, return to perform the comparison step as described above. Finally, based on the estimated corneal sphere center point position and the updated ERC position, the three-dimensional line of sight related information is determined. For example, the eye light axis is calculated based on the estimated corneal sphere center point position and the updated ERC position, and the three-dimensional line of sight related information is determined based on the eye light axis and the Kappa angle. Here, 6 degrees of freedom refers to translation in three directions and rotation around three axes. As Fig.13 As shown, according to the conversion between the device coordinate system and the head coordinate system, the updated ERC position can be determined based on the updated head-to-device posture.

[0110] Fig.15 is a schematic diagram showing an example of a method for eye tracking according to an embodiment of the present disclosure. Fig.15 In the example, the anti-ambiguous coding method mentioned above can be used to encode the light source, and the encoded binary code sequence can be used to control the light source. The event camera can capture the reflective point signal. When the aforementioned anti-ambiguous coding method is used, the reflective point information can be obtained based on the anti-ambiguous coding. Afterwards, the position of the center point of the corneal ball and the position of the center point of the eyeball rotation can be estimated based on the reflective point information to determine the three-dimensional line of sight related information. In addition, optionally, in order to more accurately determine the three-dimensional line of sight related information, sliding detection and compensation can be performed to determine the three-dimensional line of sight related information.

[0111] exist Fig.15 In the previous content, we have described light source control, obtaining reflection point information, estimating the position of the corneal ball center and the eye rotation center to determine the three-dimensional line of sight, and sliding detection and compensation. We will not repeat them here. For relevant details, please refer to the description above.

[0112] The above has been combined Figures 1 to 15A method for line of sight tracking according to an embodiment of the present disclosure and an example thereof are described. According to the method, based on the reflective point signal of the light emitted by the light source captured by the event camera and reflected by the surface of the corneal ball, reflective point information is obtained, the position of the center point of the corneal ball and the position of the center point of the eyeball rotation are estimated based on the reflective point information, and the three-dimensional line of sight related information is determined according to the position of the center point of the corneal ball and the position of the center point of the eyeball rotation. Therefore, the three-dimensional line of sight related information can be determined without using pupil information, so that line of sight tracking can be achieved more efficiently and at a low cost. In addition, optionally, the accuracy of the obtained reflective point information can be improved by the anti-ambiguous encoding method described above, and then the accuracy of determining the three-dimensional line of sight related information can be improved. In addition, optionally, the accuracy of determining the three-dimensional line of sight related information can be improved by performing sliding detection and compensation as described above, thereby improving the user experience.

[0113] The following briefly describes a device for eye tracking according to an embodiment of the present disclosure. Fig.16 Detailed description is a block diagram showing a device for gaze tracking according to an embodiment of the present disclosure (hereinafter referred to as “gaze tracking device” for convenience of description).

[0114] Reference Fig.16 , the sight tracking device 1600 may include a reflective point information acquisition unit 1601, a position estimation unit 1602 and a sight determination unit 1603. Specifically, the reflective point information acquisition unit 1601 may be configured to acquire reflective point information, wherein the reflective point information is obtained based on the reflective point signal of the light emitted by the light source captured by the event camera and reflected by the surface of the corneal ball. The position estimation unit 1602 may be configured to estimate the position of the corneal ball center point and the position of the eyeball rotation center point based on the reflective point information. The sight determination unit 1603 may be configured to determine the three-dimensional sight related information according to the position of the corneal ball center point and the position of the eyeball rotation center point.

[0115] because Figure 1 The method for eye tracking shown can be obtained by Fig.16 The sight tracking device 1600 shown in the figure is used to perform, and the reflection point information acquisition unit 1601, the position estimation unit 1602 and the sight line determination unit 1603 respectively perform steps S110, S120 and S130. Therefore, regarding Fig.16 Any details on the operations performed by the various units in Figure 1 The description of the corresponding steps will not be repeated here.

[0116] In addition, it should be noted that although the gaze tracking device 1600 is divided into units for respectively performing corresponding processing when it is introduced above, it is clear to those skilled in the art that the processing performed by the above-mentioned units can also be performed when the gaze tracking device 1600 does not perform any specific unit division or when there is no clear boundary between the units. In addition, the gaze tracking device 1600 may also include other units, such as a storage unit.

[0117] Fig.17 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

[0118] Reference Fig.17 The electronic device 1700 may include at least one memory 1701 and at least one processor 1702, wherein the at least one memory stores computer executable instructions, and when the computer executable instructions are executed by the at least one processor, the at least one processor 1702 executes the method for line of sight tracking according to an embodiment of the present disclosure.

[0119] As an example, the electronic device may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instruction (or instruction set) individually or in combination. The electronic device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0120] The processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0121] The processor can execute instructions or codes stored in the memory, wherein the memory can also store data. Instructions and data can also be sent and received through the network via the network interface device, wherein the network interface device can adopt any known transmission protocol.

[0122] The memory may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor or the like. In addition, the memory may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the memory.

[0123] In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0124] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the method for eye tracking according to the embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The instructions or computer programs in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0125] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for eye tracking, comprising: Acquiring reflection point information, wherein the reflection point information is obtained based on the reflection point signal of light emitted by the light source captured by the event camera and reflected by the surface of the cornea; Estimate the position of the center point of the corneal ball and the position of the center point of eyeball rotation based on the reflection point information; The three-dimensional sight-line related information is determined according to the position of the corneal ball center point and the position of the eyeball rotation center point.

2. The method according to claim 1, wherein: The method of estimating the position of the center point of the corneal ball and the position of the center point of eyeball rotation based on the reflection point information includes: Estimate the position of the center point of the corneal sphere at each moment based on the reflection point information at each moment; The position of the eyeball rotation center point is estimated based on the estimated positions of the corneal ball center point at multiple moments and the distance between the calibrated eyeball rotation center point and the corneal ball center point.

3. The method according to claim 2, wherein: The light reflection point information includes a light reflection point image position, wherein the light reflection point image position is a position of the light reflection point corresponding to the light source in the image coordinate system of the event camera, The estimating the position of the center point of the corneal sphere at each moment based on the reflection point information at that moment comprises: Based on the position of the reflection point image at each moment, using a regressor to estimate the position of the corneal sphere center point at that moment; or Based on the position of the reflection point image at each moment, the parameters of the event camera, the position of the light source and the radius of the corneal sphere, the position of the center point of the corneal sphere at that moment is estimated by a numerical solver.

4. The method according to claim 3, wherein: The method of estimating the position of the center point of the corneal sphere at each moment by using a regressor based on the position of the reflection point image at each moment comprises: Based on the intrinsic parameters in the parameters of the event camera, transform the image position of the reflection point at each moment from the image coordinate system of the event camera to the image coordinate system of the virtual camera; Based on the image position of the reflection point at the moment in the image coordinate system of the virtual camera, the position of the center point of the corneal sphere at the moment in the three-dimensional coordinate system of the virtual camera is estimated by the regressor; The position of the center point of the cornea sphere at the moment in the three-dimensional coordinate system of the event camera is determined according to the position of the center point of the cornea sphere at the moment in the three-dimensional coordinate system of the virtual camera.

5. The method according to claim 3, wherein: The method of estimating the position of the center point of the corneal sphere at each moment through a numerical solver based on the position of the reflection point image at each moment, the parameters of the event camera, the position of the light source and the radius of the corneal sphere comprises: According to the intrinsic parameters in the parameters of the event camera, the image position of the reflection point at each moment is transformed into the calibrated image coordinate system of the event camera; According to the extrinsic parameters in the parameters of the event camera, transforming the position of the light source into the three-dimensional coordinate system of the event camera; Based on the image position of the reflection point at each moment in the corrected image coordinate system, the position of the light source in the three-dimensional coordinate system of the event camera, and the radius of the corneal sphere, the position of the center point of the corneal sphere at that moment is estimated by the numerical solver.

6. The method according to claim 2, wherein: The method of estimating the position of the eyeball rotation center point based on the estimated positions of the corneal ball center point at multiple moments and the calibrated distance between the eyeball rotation center point and the corneal ball center point comprises: The center point position of the rotating sphere is estimated as the center point position of the eyeball rotation, wherein the rotating sphere is a sphere with the distance as the radius formed based on the motion trajectory of the center point position of the corneal ball at the multiple moments.

7. The method according to claim 2 or 6, wherein: The estimating the position of the eyeball rotation center point comprises: If the range of the movement trajectory of the corneal ball center point position at the multiple moments exceeds a predetermined range threshold, the eyeball rotation center point position is estimated.

8. The method according to claim 1, further comprising: When a head mounted device is detected to be sliding, the current position of the eyeball rotation center point is updated using the estimated position of the eyeball rotation center point, wherein the head mounted device includes the event camera and the light source, Among them, determining the three-dimensional line of sight related information according to the corneal ball center point position and the eyeball rotation center point position includes: determining the three-dimensional line of sight related information according to the corneal ball center point position and the updated eyeball rotation center point position.

9. The method according to claim 8, wherein: When the head mounted device is detected to be sliding, updating the current position of the eyeball rotation center point by using the estimated position of the eyeball rotation center point comprises: When the change of the estimated position of the eyeball rotation center point exceeds a predetermined threshold, the current position of the eyeball rotation center point is updated using the estimated position of the eyeball rotation center point.

10. The method according to claim 9, wherein: The estimated eyeball rotation center position includes an estimated monocular eyeball rotation center position, and when a change in the estimated eyeball rotation center position exceeds a predetermined threshold, updating the current eyeball rotation center position using the estimated current eyeball rotation center position includes: Compare the estimated position of the monocular eye rotation center point with the current position of the monocular eye rotation center point; If the difference between the estimated position of the monocular eye rotation center point and the current position of the monocular eye rotation center point exceeds a first threshold, the estimated position of the monocular eye rotation center point is used to update the current position of the monocular eye rotation center point.

11. The method according to claim 9, wherein: The estimated eyeball rotation center position includes: a left eyeball rotation center position and a right eyeball rotation center position, and when a change in the estimated eyeball rotation center position exceeds a predetermined threshold, using the estimated eyeball rotation center position to update the current eyeball rotation center position includes: Estimate the posture of the head relative to the head-mounted device based on the estimated positions of the left eye rotation center and the right eye rotation center; Comparing the estimated head posture relative to the head mounted device with the current head posture relative to the head mounted device; If the difference between the estimated posture and the current posture exceeds the second threshold, the current posture is updated with the estimated posture, and the current left eye eyeball rotation center point position and the current right eye eyeball rotation center point position are updated based on the updated posture.

12. The method according to claim 1, wherein: Determining the three-dimensional sight-line related information according to the corneal ball center point position and the eyeball rotation center point position includes: Determine the optical axis of the eye according to the position of the center point of the corneal ball and the position of the center point of the eyeball rotation; Based on the optical axis of the eye and the Kappa angle, relevant information of the visual axis of the eye is determined as the three-dimensional line of sight related information, wherein the Kappa angle is the angle between the calibrated optical axis of the eye and the visual axis of the eye.

13. The method according to claim 1, wherein: The obtaining of the reflective point information includes: Using the event camera to capture the reflected light point signal to obtain event data; The event data is processed to obtain the reflection point information, wherein the reflection point information includes the reflection point image position, wherein the reflection point image position is the position of the reflection point corresponding to the light source in the image coordinate system of the event camera.

14. The method according to claim 13, wherein: The processing of the event data to obtain the reflective point information includes: Performing frequency filtering on the event data to obtain a binary code sequence of the light reflection points, wherein the binary code sequence is used to drive the light source; Determine a predefined code in the binary code sequence, wherein the predefined code includes a code that can be uniquely determined in the binary code sequence when a preset condition is met; Determine a timestamp corresponding to a start bit of the binary code sequence according to the determined predefined code; Determine the reflective point number according to the timestamp and the predefined coding rule of the light source; The reflective point image position of the reflective point corresponding to the determined reflective point number is obtained as the reflective point information.

15. The method according to claim 14, wherein: The key bits in the codes of the light source corresponding to the predefined code and the light source adjacent to the predefined code are different from each other.

16. The method according to claim 14 or 15, wherein: The light sources corresponding to the predefined codes are arranged such that the distances between them meet a predetermined distance requirement.

17. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to perform the method of any one of claims 1 to 16.

18. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one processor, the at least one processor is prompted to perform the method according to any one of claims 1 to 16.

Images