Method and host for determining binocular fixation point

Abstract

A method for determining a binocular gaze point and a host are provided. The method includes obtaining a first gaze point of a first eye on a reference plane and obtaining a second gaze point of a second eye on the reference plane; and determining a binocular gaze point by combining the first gaze point and the second gaze point.

Description

Technical Field

[0001] The present invention is generally related to a mechanism for determining a gaze point, and more particularly to a method and host for determining a binocular gaze point. Background Technology

[0002] Today, in order to provide a more immersive experience, some hosts (e.g., head-mounted displays (HMDs)) typically perform eye tracking on the user's eyes when the user is using a reality service provided by the host (e.g., augmented reality (AR) service, virtual reality (VR) service, etc.).

[0003] Typically, the host computer determines the user's gaze point during eye tracking. For example, the host (e.g., HMD) can determine two rays (or directions) originating from the user's two eyes respectively, and thereby determine a fusion ray (or fusion direction), which can be considered to originate from / begin somewhere between the two eyes. Once the fusion ray is determined, the host computer can determine several reference planes in the virtual world of the real-world service, and determine the intersection of the fusion rays on each reference plane as the user's gaze point on each reference plane.

[0004] For example, suppose there are two reference planes in front of the user in the virtual world, one 1 meter away and the other 2 meters away. The host can determine the intersection of the fusion rays on the reference plane 1 meter away from the user as the 1-meter binocular gaze point. Similarly, the host can determine the intersection of the fusion rays on the reference plane 2 meters away from the user as the 2-meter binocular gaze point.

[0005] In other words, the conventional method for a host computer to determine the gaze point of both eyes is to first determine the fusion ray (or fusion direction) based on the rays corresponding to the user's two eyes, and then use the fusion ray to determine the gaze point of both eyes on each reference plane.

[0006] However, conventional techniques have failed to provide a solution for determining binocular gaze points based on the gaze point of each eye. Summary of the Invention

[0007] Therefore, the present invention relates to a method for determining the gaze point of both eyes and a host computer, which can be used to solve the above-mentioned technical problems.

[0008] An embodiment of the present invention provides a method for determining binocular gaze points, the method being adapted to a host computer, the method comprising: obtaining a first gaze point of a first eye on a reference plane, and obtaining a second gaze point of a second eye on the reference plane; and determining binocular gaze points by combining the first gaze point and the second gaze point.

[0009] Embodiments of the present invention provide a host computer including a storage circuit and a processor. The storage circuit stores program code. The processor is coupled to the storage circuit and accesses the program code to execute: obtaining a first gaze point of a first eye on a reference plane, and obtaining a second gaze point of a second eye on the reference plane; and determining a binocular gaze point by combining the first gaze point and the second gaze point. Attached Figure Description

[0010] The accompanying drawings are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0011] Figure 1 A schematic diagram of a host computer according to an embodiment of the present invention is shown.

[0012] Figure 2 A flowchart illustrating a method for determining binocular gaze points according to an embodiment of the present invention is shown.

[0013] Figure 3 A schematic diagram illustrating the determination of the gaze point according to an embodiment of the present invention is shown.

[0014] Explanation of icon numbers

[0015] 100: Host;

[0016] 102: Storage circuit;

[0017] 103: Eye-tracking circuit;

[0018] 104: Processor;

[0019] 311: First Eyes;

[0020] 312: The Second Eye;

[0021] G1: First gaze point;

[0022] G2: Second gaze point;

[0023] G3: Binocular gaze point;

[0024] PL: Reference plane;

[0025] R1: First ray;

[0026] R2: Second ray;

[0027] S210: Steps;

[0028] S220: Steps. Detailed Implementation

[0029] Reference will now be made in detail to preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings. Where possible, the same reference numerals are used in the drawings and description to refer to the same or similar parts.

[0030] See Figure 1 It illustrates a schematic diagram of a host computer according to an embodiment of the present invention. Figure 1 In this context, host 100 can be any electronic device and / or smart device capable of performing eye tracking on its users.

[0031] In one embodiment, host 100 may be an HMD worn by a user, and host 100 may include eye-tracking circuitry 103 for capturing images of each of the user's eyes to determine the condition / movement / motion / position / state of each eye, but the invention is not limited thereto.

[0032] exist Figure 1 In this context, the storage circuit 102 is one or a combination of a static or mobile random access memory (RAM), a read-only memory (ROM), a flash memory, a hard disk, or any other similar device, and it records multiple modules and / or program code that can be executed by the processor 104.

[0033] In one embodiment, eye-tracking circuitry 103 may include a camera for capturing images of a user's eyes to analyze the state of the user's eyes (e.g., gaze point). In one embodiment, eye-tracking circuitry 103 may be implemented based on the description in U.S. Patent Publication US2019 / 0056599A1, where a laser can be used to scan the user's eyes to obtain the corneal surface curvature distribution of the user's eyes. In this case, the corneal surface of the user's eyes can be mapped as contour lines, and these contour lines will change if the corresponding eyes gaze in different directions. Therefore, the changes in the contour lines can be used to track the state of the user's eyes.

[0034] In another embodiment, such as US Patent Publications US 2016 / 0353988A1 and US2017 / 0205876A1, the eye-tracking circuit 103 may include a proximity sensor or an array of proximity sensors positioned in front of the user's eyes to detect the above-mentioned contour lines, and the detected changes in the contour lines can be used to track the state of the user's eyes, but the present invention is not limited thereto.

[0035] Processor 104 may be coupled to storage circuit 102 and eye-tracking circuit 103, and processor 104 may be, for example, a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc.

[0036] In embodiments of the present invention, processor 104 may access modules and / or program code stored in storage circuit 102 to implement a method for determining binocular gaze points provided in the present invention, which will be further discussed below.

[0037] See Figure 2 The diagram illustrates a flowchart of a method for determining binocular fixation points according to an embodiment of the present invention. This method can be derived from... Figure 1 The host 100 in the middle executes, and will be referred to below as Figure 1 The components shown in the figure are used to describe Figure 2 Details of each step in the process.

[0038] In step S210, the processor 104 obtains the first gaze point of the first eye on the reference plane and the second gaze point of the second eye on the reference plane.

[0039] In one embodiment, the processor 104 provides visual content of a virtual world corresponding to a real-world service provided to a user of the host 100, wherein the virtual world contains user-representative objects and the reference plane is at a predetermined distance from the user-representative objects.

[0040] In different embodiments, the predetermined distance can be determined based on the designer's requirements. In some embodiments, the processor 104 can determine a plurality of reference planes in the virtual world, and each reference plane can be at a different predetermined distance in front of a user-representative object, but the invention is not limited thereto.

[0041] In various embodiments, the processor 104 can determine the first and second gaze points on the reference plane via any existing eye-tracking technology.

[0042] See Figure 3Figure 3 illustrates a schematic diagram of determining a gaze point according to an embodiment of the present invention. In Figure 3, the processor 104 can obtain a first image of a first eye 311 and thereby determine a first ray R1 in the virtual world, wherein the first ray R1 originates from (or is emitted from) the first eye 311. In one embodiment, the processor 104 can determine the center of the first eye 311 and a cross-section corresponding to the center on the first eye 311. Next, the processor 104 can determine that the ray originating from the center of the first eye 311 is the first ray R1 corresponding to the first eye 311, but the invention is not limited thereto.

[0043] After determining the first ray R1 corresponding to the first eye 311, the processor 104 can determine the first intersection point of the first ray R1 on the reference plane PL, and determine the first intersection point as the first gaze point G1 of the first eye 311 on the reference plane PL, but the present invention is not limited thereto.

[0044] In one embodiment, processor 104 may determine a second gaze point G2 of the second eye 312 on the reference plane PL based on a similar mechanism taught above. For example, processor 104 may obtain a second image of the second eye 312 and thereby determine a second ray R2 in the virtual world, wherein the second ray R2 originates from (or is emitted from) the second eye 312. In one embodiment, processor 104 may determine the center of the second eye 312 and a cross-section corresponding to the center on the second eye 312. Next, processor 104 may determine that the ray originating from the center of the second eye 312 is the second ray R2 corresponding to the second eye 312.

[0045] In an embodiment of the present invention, the first ray R1 and the second ray R2 may intersect at the location where the object being gazed at by the user is positioned, but the present invention is not limited thereto.

[0046] After determining the second ray R2 corresponding to the second eye 312, the processor 104 can determine the second intersection point of the second ray R2 on the reference plane PL, and determine the second intersection point as the second gaze point G2 of the second eye 312 on the reference plane PL, but the present invention is not limited thereto.

[0047] In step S220, the processor 104 determines the binocular gaze point G3 by combining the first gaze point G1 and the second gaze point G2.

[0048] In one embodiment, the processor 104 determines the binocular gaze point G3 by performing a weighted combination of the first gaze point G1 and the second gaze point G2.

[0049] In various embodiments, processor 104 may perform any combination (e.g., linear or nonlinear combination) of the first gaze point G1 and the second gaze point G2 to obtain a binocular gaze point G3.

[0050] In one embodiment, the processor 104 determines a first weight corresponding to a first gaze point G1 and a second weight corresponding to a second gaze point G2.

[0051] In some embodiments, a first weight is determined based on at least one of the following quality factors: a first image recognition confidence score (e.g., for identifying the pupil position in a first image capturing the first eye 311), a first degree of variation in the continuous estimation result of the first gaze point G1, a first flatness of the pupil shape in the first image capturing the first eye 311, a first signal-to-noise ratio of the first image capturing the first eye 311, and a first sharpness of the first image capturing the first eye 311.

[0052] In one embodiment, the first image recognition confidence score is the confidence score used to identify the pupil position in the first image capturing the first eye 311, but the invention is not limited thereto.

[0053] In one embodiment, once the processor 104 obtains a first image of the first eye 311, the processor 104 can input the first image into a pre-trained machine learning model to determine the first gaze point G1 of the first eye 311. That is, the processor 104 can use the machine learning model to determine the first gaze point G1 based on the first image of the first eye 311, without using... Figure 3 The method taught herein is not limited thereto. In this case, the machine learning model can provide a definite confidence score with respect to the first gaze point G1, and the processor 104 can determine this confidence score as a first image recognition confidence score with respect to the first gaze point G1, but the invention is not limited thereto.

[0054] In an exemplary embodiment, to enable the machine learning model to possess the above capabilities, during the training process of the machine learning model, the designer can feed specially designed training data into the machine learning model for corresponding learning. For example, after obtaining an eye image labeled as corresponding to a certain gaze point, the processor 104 can transform this eye image into a corresponding feature vector and feed this feature vector into the machine learning model. Therefore, the machine learning model can learn the features of a certain gaze point from the feature vector. In this case, when the machine learning model receives another feature vector corresponding to the above feature vector in the future (which can be transformed from another eye image), the machine learning model can thus determine the corresponding gaze point and provide a corresponding confidence score, but the invention is not limited thereto.

[0055] In one embodiment, the processor 104 may (continuously) collect several images of the first eye 311 and determine several gaze points based on these images. Next, the processor 104 may determine the degree of change in the position of these gaze points, and determine this degree of change as the first degree of change of the first gaze point G1, but the invention is not limited thereto.

[0056] In one embodiment, processor 104 may perform any existing image analysis on the first image to determine a first flatness of the first pupil shape in the first image capturing the first eye 311, a first signal-to-noise ratio of the first image capturing the first eye 311, and a first sharpness of the first image capturing the first eye 311, but the invention is not limited thereto.

[0057] In one embodiment, the processor 104 may determine the second weight of the second gaze point G2 of the second eye 312 using a similar mechanism as taught above.

[0058] In some embodiments, the second weight is determined based on at least one of the following quality factors: second image recognition (e.g., for identifying the pupil position in the first image capturing the second eye 312) confidence score, second degree of variation of the continuous estimation result of the second gaze point G2, second flatness of the pupil shape in the second image capturing the second eye 312, second signal-to-noise ratio of the second image capturing the second eye 312, and second sharpness of the second image capturing the second eye 312.

[0059] In one embodiment, once the processor 104 obtains a second image of the second eye 312, the processor 104 can input the second image into a pre-trained machine learning model to determine the second gaze point G2 of the second eye 312. That is, the processor 104 can use the machine learning model to determine the second gaze point G2 based on the second image of the second eye 312, without using... Figure 3 The method taught herein is not limited thereto. In this case, the machine learning model can provide a definite confidence score with respect to the second gaze point G2, and the processor 104 can determine this confidence score as a second image recognition confidence score with respect to the second gaze point G2, but the invention is not limited thereto.

[0060] In one embodiment, the processor 104 may (continuously) collect several images of the second eye 312 and determine several gaze points based on these images. Next, the processor 104 may determine the degree of change in the position of these gaze points, and thereby determine this degree of change as a second degree of change of the second gaze point G2, but the invention is not limited thereto.

[0061] In one embodiment, processor 104 may perform any existing image analysis on the second image to determine a second flatness of the second pupil shape in the second image capturing the second eye 312, a second signal-to-noise ratio of the second image capturing the second eye 312, and a second sharpness of the second image capturing the second eye 312, but the invention is not limited thereto.

[0062] In one embodiment, the first weight is positively correlated with the first image recognition confidence score, the first flatness of the first pupil shape in the first image capturing the first eye 311, the first signal-to-noise ratio of the first image capturing the first eye 311, and the first sharpness of the first image capturing the first eye 311. Additionally, the first weight is negatively correlated with the first degree of change of the first continuously estimated gaze point G1.

[0063] In one embodiment, the second weight is positively correlated with the second image recognition confidence score, the second flatness of the second pupil shape in the second image capturing the second eye 312, the second signal-to-noise ratio of the second image capturing the second eye 312, and the second sharpness of the second image capturing the second eye 312. Additionally, the second weight is negatively correlated with the second degree of change of the second continuously estimated gaze point G2.

[0064] In one embodiment, after determining the first weight and the second weight, the processor 104 performs a weighted combination of the first gaze point G1 and the second gaze point G2 based on the first weight and the second weight.

[0065] In one embodiment, the first gaze point G1 and the second gaze point G2 are characterized by a first coordinate and a second coordinate on the reference plane PL, respectively. In this case, the processor 104 can perform a weighted combination of the first coordinate (referred to as POG1) and the second coordinate (referred to as POG2).

[0066] For example, assuming the first weight is w1 and the second weight is w2, the weighted combination of the first coordinate and the second coordinate can be represented by "(w1*POG1+w2*POG2) / (w1+w2)", but the present invention is not limited thereto.

[0067] In one embodiment, the processor 104 considers the weighted combination of the first and second coordinates as the coordinates of the binocular gaze point G3 on the reference plane PL, but the invention is not limited thereto.

[0068] In one embodiment, the processor 104 may further display visual cues (e.g., dots, crosses, or the like) corresponding to the binocular gaze point G3 in the visual content for the user to see, but the invention is not limited thereto.

[0069] In summary, embodiments of the present invention provide a novel way to determine binocular gaze points based on the gaze points of each eye on a reference plane.

[0070] It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the invention without departing from its scope or spirit. In view of the foregoing, it is intended that this invention cover modifications and variations thereof, provided they fall within the scope of the appended claims and their equivalents.

Claims

1. A method for determining a vergence point of both eyes, the method being adapted for a host, characterized in that, The method includes: Obtaining the first gaze point of the first eye on the reference plane includes: A first image of the first eye is obtained, and the first image is input into a pre-trained machine learning model to determine the first gaze point of the first eye, wherein the machine learning model further provides a first image recognition confidence score about the first gaze point. Obtaining the second gaze point of the second eye on the reference plane includes: A second image of the second eye is obtained, and the second image is input into the pre-trained machine learning model to determine the second gaze point of the second eye, wherein the machine learning model further provides a second image recognition confidence score regarding the second gaze point; and Determining the binocular gaze point by combining the first gaze point and the second gaze point includes: Determining the binocular gaze point by performing a weighted combination of the first gaze point and the second gaze point includes: A first weight corresponding to the first gaze point is determined, and a second weight corresponding to the second gaze point is determined, wherein the first weight includes the first image recognition confidence score, and the second weight includes the second image recognition confidence score. The weighted combination is applied to the first gaze point and the second gaze point based on the first weight and the second weight.

2. The method for determining binocular gaze points according to claim 1, wherein the first gaze point and the second gaze point are characterized by a first coordinate and a second coordinate on the reference plane, respectively, and the step of determining the binocular gaze point by performing the weighted combination on the first gaze point and the second gaze point includes: The binocular gaze point is determined by performing the weighted combination on the first coordinate and the second coordinate.

3. The method for determining binocular fixation points according to claim 1, further comprising: Provide visual content corresponding to a virtual world, wherein the virtual world includes user-representative objects, and the reference plane is at a predetermined distance from the user-representative objects.

4. The method for determining binocular gaze points according to claim 3, further comprising: Visual cues corresponding to the binocular gaze points are displayed in the visual content.

5. A host, characterized by, include: Storage circuitry stores program code; as well as The processor is coupled to the storage circuitry and accesses the program code for execution. Obtaining the first gaze point of the first eye on the reference plane includes: A first image of the first eye is obtained, and the first image is input into a pre-trained machine learning model to determine the first gaze point of the first eye, wherein the machine learning model further provides a first image recognition confidence score about the first gaze point. Obtaining the second gaze point of the second eye on the reference plane includes: A second image of the second eye is obtained, and the second image is input into the pre-trained machine learning model to determine the second gaze point of the second eye, wherein the machine learning model further provides a second image recognition confidence score regarding the second gaze point; and Determining the binocular gaze point by combining the first gaze point and the second gaze point includes: Determining the binocular gaze point by performing a weighted combination of the first gaze point and the second gaze point includes: A first weight corresponding to the first gaze point is determined, and a second weight corresponding to the second gaze point is determined, wherein the first weight includes the first image recognition confidence score, and the second weight includes the second image recognition confidence score. The weighted combination is applied to the first gaze point and the second gaze point based on the first weight and the second weight.

Images