Eye tracking method and eye tracking device

Abstract

The present invention provides an eye tracking method and an eye tracking device. The method includes: obtaining a reference pupil distance value; capturing a plurality of images of a user of a 3D display through a plurality of imaging components, and finding first eye pixel coordinates and second eye pixel coordinates corresponding to the user's first eye and second eye in each image; detecting first eye spatial coordinates and second eye spatial coordinates of the user's first eye and second eye, and determining a plurality of projection coordinates based on the first eye spatial coordinates, the second eye spatial coordinates, and a plurality of optical parameters of the plurality of imaging components; determining an optimization condition associated with the first eye spatial coordinates and the second eye spatial coordinates based on the first eye pixel coordinates, the second eye pixel coordinates, the plurality of projection coordinates, and the reference pupil distance value of each image; and optimizing the first eye spatial coordinates and the second eye spatial coordinates based on the optimization condition. Thereby, the viewing experience of the user can be improved.

Description

Technical Field

[0001] The present invention relates to an eye tracking method and an eye tracking device, and particularly to an eye tracking method and an eye tracking device that can be used in conjunction with a naked-eye 3D display. Background Art

[0002] Current naked-eye 3D displays first place the pixels of the left and right eyes at the corresponding pixel positions on the display panel, and then control the light path through the liquid crystal in the 3D lens to project the left-eye and right-eye images onto the corresponding eyes respectively. Since it is necessary to focus on both eyes, the 3D lens usually has an arc design so that the left (right)-eye image can be focused and projected into the left (right) eye. However, limited by the refraction light path, some light rays may be projected into the wrong eye. That is to say, the left (right)-eye image goes wrong into the right (left) eye, and this phenomenon is called 3D crosstalk.

[0003] Generally, a naked-eye 3D display usually configures an eye tracking system to provide corresponding images to both eyes after obtaining the positions of the user's two eyes. Currently, the commonly used eye tracking methods mostly use a dual-pupil camera for face recognition and use triangulation to obtain the positions of the two eyes. However, this measurement method is often limited by the hardware capabilities and cannot maintain the measurement accuracy at a relatively far distance. In this case, the 3D crosstalk may be more serious when the distance between the user and the 3D display is relatively large. Summary of the Invention

[0004] In view of this, the present invention provides an eye tracking method and an eye tracking device, which can be used to solve the above technical problems.

[0005] The present invention provides an eye tracking method suitable for an eye tracking device including a plurality of imaging components, including: obtaining a reference interpupillary distance value; taking a plurality of images of a user of a 3D display through the plurality of imaging components, and finding a first-eye pixel coordinate and a second-eye pixel coordinate corresponding to a first eye and a second eye of the user in each image; detecting a first-eye spatial coordinate and a second-eye spatial coordinate of the first eye and the second eye of the user, and determining a plurality of projection coordinates based on the first-eye spatial coordinate, the second-eye spatial coordinate, and a plurality of optical parameters of the plurality of imaging components; determining an optimization condition associated with the first-eye spatial coordinate and the second-eye spatial coordinate based on the first-eye pixel coordinate, the second-eye pixel coordinate, the plurality of projection coordinates, and the reference interpupillary distance value in each image; and optimizing the first-eye spatial coordinate and the second-eye spatial coordinate based on the optimization condition.

[0006] The present invention provides an eye tracking device, including a plurality of imaging components and a processor. The processor is coupled to the eye tracking components and is configured to: obtain a reference interpupillary distance value; capture a plurality of images of a user of a 3D display through the plurality of imaging components, and find a first eye pixel coordinate and a second eye pixel coordinate corresponding to a first eye and a second eye of the user in each image; detect a first eye spatial coordinate and a second eye spatial coordinate of the first eye and the second eye of the user, and determine a plurality of projection coordinates based on the first eye spatial coordinate, the second eye spatial coordinate, and a plurality of optical parameters of the plurality of imaging components; determine an optimization condition associated with the first eye spatial coordinate and the second eye spatial coordinate based on the first eye pixel coordinate, the second eye pixel coordinate, the plurality of projection coordinates, and the reference interpupillary distance value of each image; and optimize the first eye spatial coordinate and the second eye spatial coordinate based on the optimization condition. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings are included to provide a further understanding of the present invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the present invention.

[0008] Figure 1A and Figure 1B are schematic diagrams of an eye tracking device and a 3D display shown in accordance with an embodiment of the present invention;

[0009] Figure 2 is a flowchart of an eye tracking method shown in accordance with an embodiment of the present invention;

[0010] Figure 3 is an application scenario diagram shown in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0011] Reference will now be made in detail to exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0012] Please refer to Figure 1A and Figure 1B , which are schematic diagrams of an eye tracking device and a 3D display shown in accordance with an embodiment of the present invention. In Figure 1A , the eye tracking device 100 includes N (N is a positive integer) imaging components 111 to 11N and a processor 104. In an embodiment of the present invention, the eye tracking device 100 can be, for example, externally connected to a 3D display 199 (which is, for example, a naked-eye 3D display) to provide relevant eye tracking information for the 3D display 199. In addition, as Figure 1B shows, the eye tracking device 100 can also be integrated into the 3D display 199 to provide relevant eye tracking information.

[0013] After obtaining the eye tracking information provided by the eye tracking device 100, the 3D display 199 can accordingly adjust the display content, so that the user viewing the 3D display 199 can view the display content of this 3D display under the condition of feeling less 3D crosstalk. The relevant details will be described in detail later.

[0014] In different embodiments, the imaging components 111-11N are, for example, any imaging devices having a charge-coupled device (CCD) lens, a complementary metal oxide semiconductor transistor (CMOS) lens, but are not limited thereto.

[0015] The processor 104 is coupled to the imaging components 111-11N and can be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array circuit (FPGA), any other type of integrated circuit, a state machine, a processor based on an advanced reduced instruction set machine (ARM), and the like.

[0016] In an embodiment of the present invention, the processor 104 accesses relevant modules and program codes to implement the eye tracking method proposed by the present invention, and the details are described in detail below.

[0017] Please refer to Figure 2 , which is a flowchart of the eye tracking method shown according to an embodiment of the present invention. The method of this embodiment can be executed by the Figure 1A eye tracking device 100, and the details of each step are described below in conjunction with the Figure 1A components shown. Figure 2 Details of each step.

[0018] First, in step S210, the processor 104 obtains a reference interpupillary distance value (denoted as D1). In the first embodiment, the processor 104 can directly use the average binocular interpupillary distance value of humans (for example, 63.5 mm) as the reference interpupillary distance value.

[0019] In the second embodiment, the processor 104 may, for example, request the above-mentioned user to move to a designated position and measure the current pupil distance value of the user as the above-mentioned reference pupil distance value, wherein the distance between the above-mentioned designated position and each image capturing component 111-11N is less than a preset distance threshold value. In different embodiments, the above-mentioned preset distance threshold value may be set to a distance value that allows the processor 104 to measure the pupil distance accurately enough. In short, the processor 104 may request the user to move to a position closer to each image capturing component 111-11N to measure the pupil distance of the user more accurately. In addition, in other embodiments, the above concept may also be understood as requiring the user to move to a position close enough to the 3D display 199 to allow the processor 104 to measure the pupil distance accurately enough, but it is not limited thereto.

[0020] In different embodiments, the processor 104 may, for example, control the 3D display 199 to guide the user to the above-mentioned designated position through sound or images, but it is not limited thereto.

[0021] In addition, in the third embodiment, the processor 104 may also directly measure the current pupil distance value of the user, correct this current pupil distance value based on multiple historical pupil distance values, and use the corrected current pupil distance value as the above-mentioned reference pupil distance value. In one embodiment, the processor 104 may input the above-mentioned historical pupil distance value and the current pupil distance value into a Kalman filter to correct the current pupil distance value by this Kalman filter, but it is not limited thereto.

[0022] In the third embodiment, the processor 104 may provide a preset starting pupil distance value for the Kalman filter. In some embodiments, this starting pupil distance value may be set to any value. In some embodiments, this starting pupil distance value may be set to the average binocular pupil distance value of humans (for example, 63.5 mm), but it is not limited thereto. In addition, during the process of obtaining the above-mentioned reference pupil distance value using the Kalman filter, overly deviated values will be filtered out accordingly.

[0023] Next, in step S220, the processor 104 captures a plurality of images of the user of the 3D display 199 through the image capturing components 111-11N, and finds the first eye pixel coordinates and the second eye pixel coordinates corresponding to the user's first eye and second eye in each image.

[0024] To make the concept of the present invention easier to understand, the following is supplemented with Figure 3 for further illustration. Please refer to Figure 3 , which is an application scenario diagram shown according to an embodiment of the present invention. In this embodiment, it is assumed that the eye tracking device 100 includes a total of 2 (i.e., N is 2) image capturing components 111 and 112, and they are respectively located at Figure 3 the positions shown.

[0025] In one embodiment, the processor 104 may control the imaging components 111 and 112 to capture images IM1 and IM2 of the user of the 3D display 199, respectively. In Figure 3 , it is assumed that the coordinates of the user's first eye and second eye in three-dimensional space are represented as the first eye spatial coordinates X1 and the second eye spatial coordinates X2, respectively. In this case, the processor 104 may find the first eye pixel coordinates and the second eye pixel coordinates corresponding to the user's first eye and second eye in the images IM1 and IM2, respectively.

[0026] For example, in the image IM1, the processor 104 may find the first eye pixel coordinates u 11 and the second eye pixel coordinates u 21 corresponding to the user's first eye and second eye, respectively. Additionally, in the image IM2, the processor 104 may find the first eye pixel coordinates u 12 and the second eye pixel coordinates u 22 corresponding to the user's first eye and second eye, respectively.

[0027] In Figure 3 this scenario, the imaging components 111 and 112 can be understood as a binocular camera. When performing eye tracking using a binocular camera, generally, image analysis is performed on the images IM1 and IM2 respectively to obtain the corresponding information of the two eyes and the face, and only feature matching and bundle adjustment are performed on the eye or face information. Thereby, the first eye pixel coordinates u 11 , the second eye pixel coordinates u 21 in the image IM1 and the first eye pixel coordinates u 12 and the second eye pixel coordinates u 22 in the image IM2 can be found, but it is not limited thereto.

[0028] In step S230, the processor 104 detects the first eye spatial coordinates X1 and the second eye spatial coordinates X2 of the user's first eye and second eye, and determines a plurality of projection coordinates based on the first eye spatial coordinates X1, the second eye spatial coordinates X2, and the plurality of optical parameters of the plurality of imaging components. In an embodiment of the present invention, the plurality of optical parameters (such as exposure value, focal length, etc.) of the jth (j is an integer between 1 and N) imaging component among the imaging components 111 to 11N can be represented as C j , and the projection coordinates generated based on X i and C j can be represented as f(X i , C j ) (i is 1 or 2).

[0029] In some embodiments, the details of steps S220 and S230 can refer to the content of "Chen, Yu & Chen, Yisong & Wang, Guoping. (2019). Bundle Adjustment Revisited.", which will not be elaborated here.

[0030] In step S240, the processor 104 determines the optimization conditions associated with the first-eye spatial coordinate X1 and the second-eye spatial coordinate X2 based on the first-eye pixel coordinates, second-eye pixel coordinates, projection coordinates, and reference pupil distance value of each image.

[0031] In the prior art, although there are means to determine the optimization conditions associated with the first-eye spatial coordinate X1 and the second-eye spatial coordinate X2, the optimization conditions used do not involve any parameters related to the pupil distance value.

[0032] For example, the optimization conditions adopted by the prior art are, for example: It can be seen from this formula that no parameters related to the user's pupil distance value are considered when optimizing the first-eye spatial coordinate X1 and the second-eye spatial coordinate X2. Therefore, errors may occur in the first-eye spatial coordinate X1 and the second-eye spatial coordinate X2 optimized by this formula, resulting in the 3D crosstalk mentioned above.

[0033] However, in the embodiments of the present invention, the optimization conditions obtained in step S240 are, for example: where norm(X1 - X2) is the distance between X1 and X2, and β is the weight. In different embodiments, β can be selected as any value according to the designer's requirements.

[0034] After that, in step S250, the processor 104 optimizes the first-eye spatial coordinate X1 and the second-eye spatial coordinate X2 based on the above optimization conditions. In different embodiments, the processor 104 can perform step S250 based on methods such as Stochastic Gradient Descent (SGD), Momentum Gradient Descent, Adagrad, RMSProp, Adaptive Moment Estimation, etc., but is not limited thereto.

[0035] In one embodiment, the processor 104 may also provide the optimized first-eye spatial coordinates X1 and second-eye spatial coordinates X2 to the 3D display 199, where the 3D display 199 determines at least one first pixel for projecting to the user's first eye and at least one second pixel for projecting to the user's second eye among the multiple pixels of the 3D display based on the optimized first-eye spatial coordinates X1 and second-eye spatial coordinates X2.

[0036] In one embodiment, after obtaining the optimized first-eye spatial coordinates X1 and second-eye spatial coordinates X2, the 3D display 199 may turn on the Lenticular lens and adjust the pixel positions on the 3D display 199, so that the user can see a three-dimensional image. For relevant details, reference can be made to the literature related to 3D rendering in the prior art, and the details are not elaborated here.

[0037] By adding the reference pupil distance value to the optimization conditions, the optimization results of the subsequently obtained first-eye spatial coordinates X1 and second-eye spatial coordinates X2 can be made more accurate, so that the 3D display 199 can then provide display content with lower 3D crosstalk for the user to view.

[0038] In summary, the embodiments of the present invention can take the pupil distance value into account in the optimization conditions related to the first-eye spatial coordinates and second-eye spatial coordinates, so as to obtain more accurate optimization results of the first-eye spatial coordinates and second-eye spatial coordinates. In this case, even if the distance between the user and the 3D display is relatively far, the 3D display can still provide display content with lower 3D crosstalk for the user to view based on the optimization results of the first-eye spatial coordinates and second-eye spatial coordinates, thereby improving the user's viewing experience.

[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An eye tracking method, suitable for an eye tracking device including a plurality of imaging components, characterized in that, Comprising: Obtaining a reference interpupillary distance value; Taking multiple images of a user of a 3D display by the multiple imaging components, and finding first-eye pixel coordinates and second-eye pixel coordinates corresponding to the user's first eye and second eye in each of the images; Detecting first-eye spatial coordinates and second-eye spatial coordinates of the user's first eye and second eye, and determining multiple projection coordinates based on the first-eye spatial coordinates, the second-eye spatial coordinates, and multiple optical parameters of the multiple imaging components; Determining an optimization condition associated with the first-eye spatial coordinates and the second-eye spatial coordinates based on the first-eye pixel coordinates, the second-eye pixel coordinates, the multiple projection coordinates, and the reference interpupillary distance value of each of the images; Optimizing the first-eye spatial coordinates and the second-eye spatial coordinates based on the optimization condition; And Providing the optimized first-eye spatial coordinates and second-eye spatial coordinates to the 3D display, wherein the 3D display determines at least one first pixel for projecting to the user's first eye and at least one second pixel for projecting to the user's second eye among multiple pixels of the 3D display based on the optimized first-eye spatial coordinates and second-eye spatial coordinates.

2. The method according to claim 1, wherein the step of obtaining the reference interpupillary distance value comprises: Requesting the user to move to a specified position, and measuring the user's current interpupillary distance value as the reference interpupillary distance value, wherein the distance between the specified position and the multiple imaging components is less than a default distance threshold value.

3. The method according to claim 1, wherein the step of obtaining the reference interpupillary distance value comprises: Measuring the user's current interpupillary distance value, and correcting the current interpupillary distance value based on multiple historical interpupillary distance values; And Taking the corrected current interpupillary distance value as the reference interpupillary distance value.

4. The method according to claim 3, wherein the step of correcting the current interpupillary distance value based on the multiple historical interpupillary distance values comprises: Inputting the multiple historical interpupillary distance values and the current interpupillary distance value into a Kalman filter to correct the current interpupillary distance value by the Kalman filter.

5. The method according to claim 1, wherein the number of the plurality of imaging components and the plurality of images is N, and the first eye pixel coordinates and the second eye pixel coordinates on the j-th image among the plurality of images are respectively represented as u 1j and u 2j , the first eye spatial coordinates and the second eye spatial coordinates are respectively represented as X1 and X2, and the plurality of optical parameters of the j-th imaging component among the plurality of imaging components are represented as C j , and the optimization condition is represented as: where norm(X1 - X2) is the distance between X1 and X2, β is the weight, D1 is the reference pupillary distance value, and f(X i , C j ) is the projection coordinate generated based on X i and C j .

6. The method according to claim 1, wherein the step of optimizing the first-eye spatial coordinates and the second-eye spatial coordinates based on the optimization condition comprises: Adopting a gradient descent method to optimize the first-eye spatial coordinates and the second-eye spatial coordinates based on the optimization condition.

7. An eye tracking device, characterized in that, Comprising: Multiple imaging components; A processor, coupled to the multiple imaging components and configured to: Obtain a reference interpupillary distance value; Take multiple images of a user of a 3D display by the multiple imaging components, and find first-eye pixel coordinates and second-eye pixel coordinates corresponding to the user's first eye and second eye in each of the images; Detect first-eye spatial coordinates and second-eye spatial coordinates of the user's first eye and second eye, and determine multiple projection coordinates based on the first-eye spatial coordinates, the second-eye spatial coordinates, and multiple optical parameters of the multiple imaging components; Determine an optimization condition associated with the first-eye spatial coordinates and the second-eye spatial coordinates based on the first-eye pixel coordinates, the second-eye pixel coordinates, the plurality of projection coordinates, and the reference pupil distance value of each of the images; Optimize the first-eye spatial coordinates and the second-eye spatial coordinates based on the optimization condition; and Provide the optimized first-eye spatial coordinates and second-eye spatial coordinates to the 3D display, wherein the 3D display determines at least one first pixel for projecting to the first eye of the user and at least one second pixel for projecting to the second eye of the user among a plurality of pixels of the 3D display based on the optimized first-eye spatial coordinates and second-eye spatial coordinates.

8. The eye tracking device according to claim 7, wherein the processor performs: Request the user to move to a specified position, and measure the current pupil distance value of the user as the reference pupil distance value, wherein the distance between the specified position and the plurality of imaging components is less than a default distance threshold.

9. The eye tracking device according to claim 7, wherein the processor performs: Measure the current pupil distance value of the user, and correct the current pupil distance value based on a plurality of historical pupil distance values; and Use the corrected current pupil distance value as the reference pupil distance value.

10. The eye tracking device according to claim 9, wherein the processor performs: Input the plurality of historical pupil distance values and the current pupil distance value into a Kalman filter to correct the current pupil distance value by the Kalman filter.

11. The eye tracking device according to claim 7, wherein the number of the plurality of imaging components and the plurality of images is N, and the first eye pixel coordinates and the second eye pixel coordinates on the j-th image of the plurality of images are respectively represented as u 1j and u 2j , the first eye spatial coordinates and the second eye spatial coordinates are respectively represented as X1 and X2, and the plurality of optical parameters of the j-th imaging component of the plurality of imaging components are represented as C j , and the optimization condition is represented as: where norm(X1 - X2) is the distance between X1 and X2, β is the weight, D1 is the reference pupil distance value, and f(X i , C j ) is the projection coordinate generated based on X i and C j .

12. The eye tracking device according to claim 7, wherein the processor performs: Adopt the gradient descent method to optimize the first-eye spatial coordinates and the second-eye spatial coordinates based on the optimization condition.

Images