Sight line estimation method and related equipment

Abstract

The invention provides a sight line estimation method and related equipment. The method comprises the following steps: determining three-dimensional position information of an eyeball rotation center under an equipment coordinate system and a distance from the eyeball rotation center to a pupil center under the equipment coordinate system; obtaining a target eye image; determining target position information of the pupil center in the target eye image; according to the target position information, in combination with the distance from the eyeball rotation center to the pupil center in the equipment coordinate system, determining three-dimensional position information of the pupil center in the equipment coordinate system; and according to the three-dimensional position information of the eyeball rotation center in the equipment coordinate system and the three-dimensional position information of the pupil center in the equipment coordinate system, determining the sight line direction.

Description

Technical Field

[0001] The present disclosure relates to the field of extended reality technology, and in particular, to a method for estimating line of sight and related devices. Background Art

[0002] Extended Reality (XR) refers to combining the real and the virtual through a computer to create a virtual environment for human-computer interaction. XR (Extended Reality) technology can further include Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR), which use hardware devices and a variety of technical means to integrate virtual content with the real scene.

[0003] Generally, an extended reality system provides a wearable device for a user to achieve human-computer interaction, and the wearable device can be a head-mounted wearable device. In some scenarios, the wearable device can collect eye images for calculation to achieve functions such as line of sight tracking or pupil distance estimation.

[0004] However, the inventors of the present disclosure have found that in the related art, the method for line of sight tracking is relatively complex. Summary of the Invention

[0005] The present disclosure provides a method for estimating line of sight and related devices to solve or partially solve the above problems.

[0006] In a first aspect of the present disclosure, there is provided a method for estimating line of sight, including:

[0007] Determining three-dimensional position information of the center of rotation of the eyeball in the device coordinate system and the distance from the center of rotation of the eyeball to the center of the pupil in the device coordinate system;

[0008] Obtaining a target eye image;

[0009] Determining target position information of the center of the pupil in the target eye image;

[0010] According to the target position information, and in combination with the distance from the center of rotation of the eyeball to the center of the pupil in the device coordinate system, determining three-dimensional position information of the center of the pupil in the device coordinate system;

[0011] According to the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system and the three-dimensional position information of the center of the pupil in the device coordinate system, determining the line of sight direction.

[0012] In a second aspect of the present disclosure, there is provided a line of sight estimation device, including:

[0013] The first determination module is configured to: determine the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system and the distance from the center of rotation of the eyeball to the center of the pupil in the device coordinate system;

[0014] The acquisition module is configured to: acquire a target eye image;

[0015] The second determination module is configured to: determine the target position information of the center of the pupil in the target eye image;

[0016] The third determination module is configured to: based on the target position information and in combination with the distance from the center of rotation of the eyeball to the center of the pupil in the device coordinate system, determine the three-dimensional position information of the center of the pupil in the device coordinate system;

[0017] The fourth determination module is configured to: based on the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system and the three-dimensional position information of the center of the pupil in the device coordinate system, determine the line-of-sight direction.

[0018] In a third aspect of the present disclosure, there is provided a computer device, including one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and are executed by the one or more processors, and the programs include instructions for executing the method according to the first aspect.

[0019] In a fourth aspect of the present disclosure, there is provided a non-volatile computer-readable storage medium containing a computer program, which, when executed by one or more processors, causes the processors to execute the method according to the first aspect.

[0020] In a fifth aspect of the present disclosure, there is provided a computer program product, including computer program instructions, which, when run on a computer, cause the computer to execute the method according to the first aspect.

[0021] The line-of-sight estimation method and related devices provided by the embodiments of the present disclosure can pre-calculate some fixed parameters for subsequent line-of-sight estimation tasks, can achieve fast calculation of the line-of-sight direction, thereby improving the calculation efficiency, saving computing power, and can avoid the occurrence of frame loss, and improve the accuracy of the algorithm. Description of the Drawings

[0022] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following will briefly introduce the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings in the following description are only the embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0023] Figure 1A Shows a schematic diagram of an exemplary system provided by an embodiment of the present disclosure.

[0024] Figure 1B Shows a schematic diagram of an exemplary head-mounted wearable device.

[0025] Figure 1C Shows a schematic diagram of an exemplary eyeball model.

[0026] Figure 2 Shows a schematic flowchart of an exemplary method provided by an embodiment of the present disclosure.

[0027] Figure 3A Shows a schematic flowchart of another exemplary method provided by an embodiment of the present disclosure.

[0028] Figure 3B Shows a schematic flowchart of an exemplary method for determining the three-dimensional position information of the eyeball rotation center in the device coordinate system according to an embodiment of the present disclosure.

[0029] Figure 3C Shows a schematic flowchart of an exemplary method for determining the three-dimensional position information of the historical pupil center in the device coordinate system according to an embodiment of the present disclosure.

[0030] Figure 3D Shows a schematic flowchart of an exemplary method for determining the three-dimensional position information of the pupil center in the device coordinate system according to an embodiment of the present disclosure.

[0031] Figure 3E Shows a schematic flowchart of another exemplary method for determining the three-dimensional position information of the pupil center in the device coordinate system according to an embodiment of the present disclosure.

[0032] Figure 4 Shows a schematic diagram of an exemplary device provided by an embodiment of the present disclosure.

[0033] Figure 5 Shows a schematic diagram of the hardware structure of an exemplary computer device provided by an embodiment of the present disclosure. Detailed implementation manners

[0034] To make the objectives, technical solutions, and advantages of the present disclosure more clear and understandable, the present disclosure will be further described in detail below with reference to specific embodiments and the accompanying drawings.

[0035] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the ordinary meanings understood by those of ordinary skill in the field to which the present disclosure belongs. The terms "first", "second" and similar terms used in the embodiments of the present disclosure do not denote any order, quantity or importance, but are only used to distinguish different components. Words such as "including" or "comprising" mean that the elements or objects appearing before this word cover the elements or objects listed after this word and their equivalents, without excluding other elements or objects. Words such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Upper", "lower", "left", "right", etc. are only used to represent relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0036] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0037] For example, when responding to receiving an active request from the user, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server or a storage medium that performs the operations of the technical solutions of the present disclosure according to the prompt message.

[0038] As an optional but non-limiting implementation manner, the manner of sending a prompt message to the user in response to receiving an active request from the user may be, for example, in the form of a pop-up window, and the prompt message may be presented in text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0039] It can be understood that the above process of notifying and obtaining the user's authorization is only illustrative and does not limit the implementation manner of the present disclosure, and other manners that meet relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0040] It can be understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the corresponding laws, regulations and related provisions.

[0041] Figure 1A FIG. shows a schematic diagram of an exemplary extended reality system 100 provided by an embodiment of the present disclosure.

[0042] Extended Reality (XR) refers to the combination of the real and the virtual through a computer to create a virtual environment for human-computer interaction. XR (Extended Reality) technology can further include augmented reality (AR), virtual reality (VR), and mixed reality (MR), which use hardware devices and various technical means to integrate virtual content with real scenes.

[0043] As Figure 1A shown, the system 100 may include various types of wearable devices. For example, a head-mounted wearable device (e.g., VR / AR glasses or a head-mounted display (HMD)) 104, an operation handle 108, etc. In some scenarios, a camera 110 for taking pictures of the operator (user) 102 may also be provided. In some embodiments, when the foregoing devices do not have a processing function, the system 100 may further include an external control device 112 for providing a processing function. The control device 112 may be, for example, a computer device such as a mobile phone or a computer. In some embodiments, when any one of the foregoing devices serves as a control device or a main control device, it may communicate with other devices in the system 100 through wired or wireless communication means to achieve information interaction.

[0044] In the system 100, the user 102 may use the head-mounted wearable device 104 and the operation handle 108 to interact with the extended reality system 100. In some scenarios, the system 100 may use the images collected by the camera 110 to identify the posture, gestures, etc. of the user 102, and then complete the interaction with the user 102 based on the identified posture and gestures. In some embodiments, the user 130 may also achieve gesture input with bare hands. The head-mounted wearable device 104 may collect real-time images in front through a camera or a camera provided in front of the head-mounted wearable device 104, and identify the gestures of the user 130 by recognizing the image.

[0045] In some embodiments, as Figure 1A shown, the system 100 may also communicate with the server 114, and may obtain data from the server 114, such as pictures, audio, video, etc., and may output these data through the head-mounted wearable device 104. For example, display pictures or videos on the display screen of the head-mounted wearable device 104, play the audio carried by the audio and video using the speaker of the head-mounted wearable device 104, and so on. In some embodiments, as Figure 1A shown, the server 114 may retrieve the required data, such as pictures, audio, video, etc., from the database server 116 for storing data.

[0046] In some embodiments, a collection unit for collecting information may be provided on the head-mounted wearable device 104. The types of the collection unit can be various.

[0047] In some embodiments, the collection unit may further include an environment acquisition unit and a positioning and tracking unit. The environment acquisition unit may be used to collect environment information around (e.g., in front of) the wearable device 104, and the positioning and tracking unit may be used to perform positioning and tracking on the wearable device 104. Optionally, the environment acquisition unit may include, but is not limited to, photosensitive elements such as a three-color camera (e.g., an RGB camera), a depth camera, a binocular camera, a laser, etc., and the positioning and tracking unit may include, but is not limited to, modules such as Visual Simultaneous Localization and Mapping (Visual SLAM), an Inertial Measurement Unit (IMU), a Global Positioning System (GPS), an Ultra-Wideband Wireless Communication Technology (UWB), a laser, etc.

[0048] In some embodiments, the head-mounted wearable device 104 may also be provided with a speed sensor, an acceleration sensor, an angular velocity sensor (e.g., a gyroscope), etc. for collecting the speed information or acceleration information of the head-mounted wearable device 104. Also, for example, the operation handle 108 may also be provided with a speed sensor, an acceleration sensor, an angular velocity sensor (e.g., a gyroscope), etc. for collecting the speed information or acceleration information of the wearable glove 106. It should be noted that in addition to being provided on the head-mounted wearable device 104 and the operation handle 108, the aforementioned collection unit may also be directly attached to the body part of the interactive user 102 without relying on hardware devices, so as to collect relevant information of the body part, such as speed or acceleration or angular velocity information, or information collected by other sensors or collection units (e.g., an eye image (including a pupil image), etc.).

[0049] In some embodiments, the head-mounted wearable device 104 may also be provided with a camera or a camera for taking photos of the operator (user) 102 (e.g., photos of the hand or foot) and environmental images.

[0050] In some embodiments, the system 100 can identify the posture, gestures, etc. of the user 102 based on the collected information, and then perform corresponding interactions according to the identified user postures and gestures.

[0051] Figure 1B A schematic diagram of an exemplary head-mounted wearable device 104 is shown.

[0052] As Figure 1BAs shown, the head-mounted wearable device 104 may include a barrel 1042, inside which a display screen 1044 for displaying images and an optical component 1046 for processing the optical path may be provided. Optionally, the optical component 1046 may further include a plurality of lenses (e.g., lenses 1046A and 1046B), and the combination of the plurality of lenses may project the light emitted by the display screen 1044 into the human eye 1022, so that the human eye 1022 can view the picture displayed on the display screen 1044. It can be understood that Figure 1B only the unilateral structure of the head-mounted wearable device 104 is exemplarily shown herein. To achieve binocular display, the head-mounted wearable device 104 may include two barrel structures arranged side by side.

[0053] In some embodiments, as Figure 1B shown, the head-mounted wearable device 104 may further be provided with a camera 1048 for collecting eye images, and the camera 1048 may be a charge-coupled device (CCD) image sensor, a complementary metal-oxide-semiconductor (CMOS) image sensor, and so on.

[0054] Optionally, the camera 1048 may be an eye tracking (ET) camera, and the eye images collected by it may be used to implement functions such as interpupillary distance estimation and eye tracking.

[0055] As Figure 1B shown, as an alternative implementation, the camera 1048 may be disposed outside the barrel. And, in order to be able to better collect complete eye images and not affect the user's viewing of the picture on the display screen 1022, the common deployment positions of the camera 1048 are generally at the outer corner of the eye or the nose wing position. Referring to Figure 1B shown, if the camera 1048 is close to the outside of the device, then Figure 1B the shown camera deployment position is at the outer corner of the eye. If the camera 1048 is close to the inside of the device, then Figure 1B the shown camera deployment position is at the nose wing position. In addition to the foregoing implementation, the camera may also be disposed inside the barrel, so as to avoid being blocked by foreign objects (e.g., glasses worn by the user) and may have a better viewing angle to obtain better captured images.

[0056] Eye tracking is usually performed by acquiring the eye images collected by the camera 1048 and then estimating the eye gaze based on the eye images. The task of eye gaze estimation is to estimate the orientation of the eye gaze based on the input eye images.

[0057] There are two methods for eye gaze estimation: the solution method based on the eyeball model and the estimation method based on appearance.

[0058] The calculation method based on the eyeball model simplifies the eye into an eyeball model. For example, Figure 1C as shown, the eyeball is approximated as a sphere that rotates around the eyeball rotation center (abbreviated as the eye rotation center), and the cornea is also approximated as a sphere. The eyeball rotation center, the cornea center, and the pupil center are collinear, and this line is the line of sight. When estimating the line of sight, the line of sight direction can be determined by determining the ray passing through any two of the eyeball rotation center, the cornea center, and the pupil center.

[0059] The appearance-based estimation method is data-driven and uses a deep learning model to learn the mapping from appearance features to the line of sight direction through a large amount of data.

[0060] Generally, the required frame rate for line of sight estimation is relatively high, usually around 90Hz, and the calculation for each frame of image usually needs to be completed within 10ms. For the calculation method based on the eyeball model, it usually needs to use a light source (e.g., LED) to form a light spot on the cornea, and then calculate the cornea center based on the position of the light spot, and further combine the pupil center to determine the line of sight direction, resulting in a large amount of calculation. The appearance-based estimation method also requires a deep learning model and has a very large amount of calculation. Therefore, in the case of high load, it is easy to cause frame loss (i.e., the computing power is insufficient to support frame-by-frame calculation), affecting downstream tasks. Therefore, a faster line of sight estimation method is needed that can quickly calculate the line of sight direction without using light spots when frame loss occurs.

[0061] Figure 2 The flowchart of the exemplary method 200 provided by the embodiments of the present disclosure is shown.

[0062] The method 200 can be used to estimate the line of sight for line of sight tracking. And, as Figure 2 shown, the method 200 can further include the following steps.

[0063] In step 202, a first eye image can be obtained first. For example, the first eye image is acquired by the camera 1048.

[0064] In step 204, determine the first position information O = [O x , O y of the pupil center in the first eye image.

[0065] Optionally, through image recognition technology, pupil detection can be performed in the first eye image to obtain the first position information [O x , O y of the pupil center in the first eye image. This first position information can be two-dimensional coordinates in the imaging plane of the camera.

[0066] In step 206, according to the first position information [Ox , O y , determine the second position information of the center of eye rotation in the first eye image x , E y .

[0067] Since the center of eye rotation can generally be considered fixed, some parameters calculated from historical data can be used to further determine the second position information x , E y .

[0068] Therefore, in some embodiments, step 206 may further include the following steps:

[0069] Obtain a plurality of historical line-of-sight data corresponding to a plurality of historical eye images;

[0070] According to the plurality of historical line-of-sight data, combined with the first position information O = [O x , O y , determine the second position information x , E y . Wherein, the historical line-of-sight data may be the line-of-sight direction pre-calculated from the historical eye image (which can be directly called when performing this step without further calculation), and the historical eye image may be the eye image captured before the first eye image. Optionally, the plurality of historical eye images may be a plurality of eye images adjacent to the capture time node of the first eye image. For example, if the first eye image is the nth eye image, the plurality of historical eye images may be the (n - 1)th, (n - 2)th, (n - 3)th eye images, and so on.

[0071] As an alternative embodiment, the line of sight passing through the pupil center can be projected onto the imaging plane of the camera 1048, and let the line-of-sight direction passing through the pupil center be The external parameters of the camera 1048 are R and T, and the internal parameter of the camera 1048 is P. Then the starting point of the line of sight is [O x , O y , and the orientation is Wherein, The external parameters R, T and the internal parameter P of the camera 1048 are known data that have been calibrated for the camera 1048.

[0072] Then, combined with a plurality of historical line-of-sight data, find the second position information of the center of eye rotation x , E y .

[0073] Specifically, since the center of eye rotation is the intersection point of multiple lines of sight, let the center of eye rotation be E = [E x , Ey , then it satisfies: (E - O) T y , -v x = 0.

[0074] Let A = [v y , -v x , b = O T [v y , -v x , then, through the least squares method, combining multiple historical line-of-sight data, solve the following formula:

[0075] E = (A T A) -1 A T b.

[0076] Then the second position information [E x , E y of the eye rotation center can be obtained.

[0077] In step 208, determine the interpupillary distance of the user's binocular eyes.

[0078] Since the user needs to adjust the relative position between the lens barrel and the pupil when wearing the wearable device 104 to prevent dizziness, therefore, when the adjustment is completed, the interpupillary distance of the user's binocular eyes can be determined according to the position information of the lens barrel.

[0079] In step 210, according to the second position information [E x , E y , determine the connection line between the optical center of the camera 1048 and the eye rotation center in the device coordinate system.

[0080] Specifically, since the second position information [E x , E y of the eye rotation center has been obtained in step 206, combined with the external parameters R, T and internal parameter P of the camera 1048, according to the imaging model of the camera, the connection line E l , E r between the optical center of the camera 1048 and the eye rotation center in the three-dimensional space of the device coordinate system can be determined, where the connection line E l is the connection line between the optical center of the left eye and the eye rotation center, and the connection line E r is the connection line between the optical center of the right eye and the eye rotation center.

[0081] In step 212, according to the connection line between the optical center of the camera 1048 and the eye rotation center in the device coordinate system, combined with the interpupillary distance of the user's binocular eyes, determine the three-dimensional position information of the eye rotation center in the device coordinate system.

[0082] ​In this step, assuming that the depths of the user's two eyes are the same in the device coordinate system, after rotating the plane of depth z to the device coordinate system, the line segment E can be obtained. l and E r have two intersection points with the plane of depth z.

[0083] Since the center of eye rotation and the center of pupil are on the line of sight, it can be considered that the distance between the centers of eye rotation of the two eyes is the interpupillary distance. Furthermore, the distance between the two intersection points of the aforementioned line segment E l and E r with the plane of depth z (i.e., the distance between the centers of eye rotation of the two eyes) can be considered as the interpupillary distance IPD.

[0084] Let the optical centers of the left and right cameras be S l and S r . Then:

[0085]

[0086] The only unknown in the above formula is the depth z, so the depth z can be calculated.

[0087] After obtaining the depth z, determine the plane where the depth z is located. Combining the line segment E l and E r , the three-dimensional position information M l and M r of the centers of eye rotation of the two eyes in the device coordinate system can be obtained. Among them, the three-dimensional position information M l is the three-dimensional coordinates of the center of eye rotation of the left eye, and the three-dimensional position information M r is the three-dimensional coordinates of the center of eye rotation of the right eye.

[0088] In step 214, according to the historical line-of-sight data, the three-dimensional position information of the center of eye rotation in the device coordinate system, and the position information of the historical center of pupil in the imaging plane of the camera (obtained by detecting the pupil in the historical eye image), determine the three-dimensional position information of the historical center of pupil in the device coordinate system.

[0089] According to the imaging model of the camera and the eye model, it can be known that the line-of-sight ray passing through the center of eye rotation and the imaging ray of the center of pupil in the imaging plane of the camera and the optical center of the camera can intersect at a point, and this intersection point is the three-dimensional position of the center of pupil in the device coordinate system.

[0090] First, according to the historical line-of-sight data and the three-dimensional position information of the center of eye rotation in the device coordinate system, the line-of-sight ray passing through the center of eye rotation can be determined;

[0091] The imaging ray of the center of pupil is: the starting point is S l , and the direction is R-1 (P -1 [O x ,O y -T);

[0092] Find the intersection point of two lines (i.e., the three-dimensional coordinates of the pupil center): Let the intersection point be p, the starting point of line l be r1, the direction be l1, the starting point of line 2 be r2, and the direction be l2, then it satisfies:

[0093] p = r1 + k1ι1 = r2 + k2ι2

[0094] Then the solution result of p is as follows:

[0095]

[0096] Among them, line l can be the line of sight ray passing through the center of rotation of the eyeball, its direction is l1, and the starting point r1 is the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system; line 2 can be the imaging light ray of the pupil center, its direction is l2, and the starting point is r2, which can be determined according to the calibrated camera parameters. The intersection point of line l and line 2 is the pupil center.

[0097] In this way, according to the above formula, the three-dimensional positions of the pupil centers of both eyes in the device coordinate system can be calculated. Since the above imaging light ray of the pupil center is calculated based on the position information of the historical pupil center in the imaging plane of the camera, the three-dimensional position of the pupil center obtained in the above steps is the three-dimensional position information of the historical pupil center in the device coordinate system.

[0098] In step 216, according to the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system and the three-dimensional position information of the historical pupil center in the device coordinate system, determine the distance from the center of rotation of the eyeball to the pupil center in the device coordinate system.

[0099] In the above steps, the three-dimensional position information M l 、M r of the center of rotation of the eyeball in the device coordinate system and the three-dimensional position information p l 、p r of the historical pupil center in the device coordinate system have been obtained, and the distance of "center of rotation of the eyeball - pupil center" of each eye can be obtained:

[0100]

[0101] It can be understood that steps 202 to 216 can be regarded as preprocessing steps. In other words, when performing the line-of-sight estimation task each time, steps 202 to 216 do not have to be fully executed. Some fixed parameters calculated in steps 202 to 216 (for example, the distance from the center of eye rotation to the center of the pupil in the device coordinate system, the three-dimensional position information of the center of eye rotation in the device coordinate system, etc.) can be directly used for the line-of-sight estimation task without secondary calculation.

[0102] In step 218, a target eye image is acquired. Optionally, the target eye image can be a real-time acquired eye image or a currently acquired eye image.

[0103] Next, the line of sight can be calculated in the following manner.

[0104] In step 220, the target position information of the center of the pupil in the target eye image is determined.

[0105] Optionally, through image recognition technology, pupil detection can be performed in the target eye image, so as to obtain the target position information of the center of the pupil in the target eye image. The target position information can be two-dimensional coordinates in the imaging plane of the camera.

[0106] In step 222, according to the target position information, the imaging ray of the center of the pupil is determined, that is, the connection line between the optical center of the camera and the target position of the center of the pupil in the imaging plane of the camera.

[0107] Specifically, according to the internal parameter P of the camera 1048 and the external parameters R and T of the camera 1048, combined with the target position information, the imaging ray of the center of the pupil can be determined.

[0108] In step 224, according to the three-dimensional position information of the center of eye rotation in the device coordinate system and the imaging ray of the center of the pupil, the three-dimensional position information of the center of the pupil in the device coordinate system is determined.

[0109] Since the distance d between the center of the pupil and the center of eye rotation was obtained in the previous steps l 、d r , and for the eyes of the same user, this distance is a fixed value. Therefore, with the center of eye rotation as the center of the sphere and the radius d l 、d r a sphere is drawn, and the intersection point of the imaging ray of the center of the pupil and this sphere is calculated. The one closest to the starting point of the imaging ray is the three-dimensional coordinates of the center of the pupil in the device coordinate system.

[0110] Optionally, the calculation method of the intersection point of the imaging ray and the sphere is as follows:

[0111] Assume that x is a point on the sphere with a radius of r, and the ray Passing through x, there is an equation:

[0112] ||x - c|| 2 = r 2 And

[0113] We get: where d is d l , d r ;

[0114] According to the solutions of the quadratic equation of one variable In this equation:

[0115] c = (o - c) 2 - r 2 , then it is possible to determine whether there is an intersection point (i.e., the corresponding ray length) by judging Δ.

[0116] Since the ray direction (i.e., the imaging light ray) is known, the intersection point can be calculated. Since the required point is on the side closer to the origin of the ray, the sign of the solution of the quadratic equation of one variable can be determined.

[0117] In step 226, according to the three-dimensional position information of the eye rotation center in the device coordinate system and the three-dimensional position information of the pupil center in the device coordinate system, the line-of-sight direction is determined.

[0118] After obtaining the three-dimensional coordinates of the pupil center and knowing the three-dimensional coordinates of the eye rotation center, the line connecting the two is the line-of-sight direction.

[0119] As can be seen from the above embodiments, the embodiments of the present disclosure pre-calculate some fixed parameters for subsequent line-of-sight estimation tasks, which can achieve fast calculation of the line-of-sight direction, thereby improving the calculation efficiency, saving computing power, and avoiding frame loss, and improving the accuracy of the algorithm.

[0120] It should be noted that the method of the embodiments of the present disclosure can be executed by a single device, such as a computer or a server, etc. The method of this embodiment can also be applied to a distributed scenario, and completed by multiple devices cooperating with each other. In this case of a distributed scenario, one of the multiple devices can only execute one or more steps of the method of the embodiments of the present disclosure, and these multiple devices will interact with each other to complete the described method.

[0121] It should be noted that some embodiments of the present disclosure have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the above embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0122] The embodiments of the present disclosure also provide a method for gaze estimation. Figure 3A The flowchart of an exemplary method 300 provided by the embodiments of the present disclosure is shown. This method 300 can be applied to Figure 1A or Figure 1B the head-mounted wearable device 104, and can also be applied to Figure 1A the external device 112. As Figure 3A shown, this method 300 may further include the following steps.

[0123] In step 302, the three-dimensional position information of the center of eye rotation in the device coordinate system and the distance from the center of eye rotation to the center of the pupil in the device coordinate system can be determined.

[0124] In some embodiments, as Figure 3B shown, to determine the three-dimensional position information of the center of eye rotation in the device coordinate system, it may further include:

[0125] In step 3022, a first eye image is acquired.

[0126] For example, the first eye image is acquired by the camera 1048.

[0127] In step 3024, the first position information O = [O x , O y of the center of the pupil in the first eye image is determined.

[0128] Optionally, through image recognition technology, pupil detection can be performed in the first eye image to obtain the first position information [O x , O y of the center of the pupil in the first eye image. This first position information can be two-dimensional coordinates in the imaging plane of the camera.

[0129] In step 3026, according to the first position information [O x , O y , the second position information [E x , E y of the center of eye rotation in the first eye image is determined.

[0130] Since the center of eye rotation can generally be considered fixed, some parameters calculated from historical data can be used to further determine the second position information. x , y .

[0131] Therefore, in some embodiments, determining the second position information of the center of eye rotation in the first eye image according to the first position information may further include:

[0132] Obtaining a plurality of historical line-of-sight data corresponding to a plurality of historical eye images;

[0133] Determining the second position information according to the plurality of historical line-of-sight data in combination with the first position information.

[0134] As an alternative embodiment, the line of sight passing through the pupil center can be projected onto the imaging plane of the camera 1048, and the direction of the line of sight passing through the pupil center is The external parameters of the camera 1048 are R and T, and the internal parameters of the camera 1048 are P. Then the starting point of the line of sight is [O x , O y , and the orientation is where The external parameters R and T and the internal parameters P of the camera 1048 are known data that have been calibrated for the camera 1048.

[0135] Then, in combination with a plurality of historical line-of-sight data, the second position information [E x , E y of the center of eye rotation is obtained.

[0136] Specifically, since the center of eye rotation is the intersection point of multiple lines of sight, let the center of eye rotation be E = [E x , E y , then it satisfies: (E - O) T [v y , -v x = 0.

[0137] Let A = [v y , -v x , b = O T [v y , -v x , then, by the least squares method, in combination with a plurality of historical line-of-sight data, the following formula is solved:

[0138] E = (A T A) -1 A T b.

[0139] the second position information of the center of rotation of the eyeball can be obtained x , E y .

[0140] In step 3028, according to the second position information, a connection line between the optical center of the camera and the center of rotation of the eyeball in the device coordinate system is determined.

[0141] Specifically, since the second position information of the center of rotation of the eyeball has been obtained in step 206 x , E y . By combining the external parameters R, T and the internal parameter P of the camera 1048, according to the imaging model of the camera, a connection line E l , E r in the three-dimensional space between the optical center of the camera 1048 and the center of rotation of the eyeball in the device coordinate system can be determined, where the connection line E l is the connection line between the optical center of the left eye and the center of rotation of the eyeball, and the connection line Er is the connection line between the optical center of the right eye and the center of rotation of the eyeball.

[0142] In step 3030, the interpupillary distance of the user's both eyes is determined.

[0143] Since the user needs to adjust the relative position between the lens barrel and the pupil when wearing the wearable device 104 to prevent dizziness, therefore, when the adjustment is completed, the interpupillary distance of the user's both eyes can be determined according to the position information of the lens barrel.

[0144] In step 3032, according to the connection line between the optical center of the camera and the center of rotation of the eyeball in the device coordinate system, and in combination with the interpupillary distance of the user's both eyes, the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system is determined.

[0145] In some embodiments, determining the distance from the center of rotation of the eyeball to the center of the pupil in the device coordinate system includes:

[0146] Determining the three-dimensional position information of the historical center of the pupil in the device coordinate system;

[0147] According to the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system and the three-dimensional position information of the historical center of the pupil in the device coordinate system, determining the distance from the center of rotation of the eyeball to the center of the pupil in the device coordinate system.

[0148] In some embodiments, as Figure 3C shown, determining the three-dimensional position information of the historical center of the pupil in the device coordinate system may further include:

[0149] In step 30342, obtaining a historical eye image and historical gaze data corresponding to the historical eye image;

[0150] In step 30344, determine the historical position information of the pupil center in the historical eye image;

[0151] In step 30346, determine the imaging light ray of the pupil center according to the historical position information of the pupil center in the historical eye image;

[0152] In step 30348, determine the line of sight ray passing through the center of eye rotation according to the historical line of sight data and the three-dimensional position information of the center of eye rotation in the device coordinate system;

[0153] In step 30350, determine the three-dimensional position information of the historical pupil center in the device coordinate system according to the imaging light ray of the pupil center and the line of sight ray passing through the center of eye rotation.

[0154] In step 3032, assuming that the depths of the user's two eyes are the same, after converting to the device coordinate system, the line E l and E r with the two intersection points of the plane of depth z can be obtained.

[0155] Since the center of eye rotation and the pupil center are in the same line of sight direction, it can be considered that the distance between the centers of eye rotation of the two eyes is the interpupillary distance. Furthermore, the distance between the two intersection points of the aforementioned line E l and E r with the plane of depth z (i.e., the distance between the centers of eye rotation of the two eyes) can be considered as the interpupillary distance IPD.

[0156] Let the optical centers of the left and right cameras be S l 、S r , then:

[0157]

[0158] The only unknown in the above formula is the depth z, so the depth z can be calculated.

[0159] After obtaining the depth z, determine the plane where the depth z is located, and combine the line E l and E r , then the three-dimensional position information of the centers of eye rotation of the two eyes in the device coordinate system M l 、M r can be obtained. Among them, the three-dimensional position information M l is the three-dimensional coordinates of the center of eye rotation of the left eye, and the three-dimensional position information M r is the three-dimensional coordinates of the center of eye rotation of the right eye.

[0160] According to the imaging model of the camera and the eyeball model, it can be known that the line of sight ray passing through the center of rotation of the eyeball and the position of the pupil center in the imaging plane of the camera and the imaging light ray between the camera optical center can intersect at a point, and this intersection point is the three-dimensional position of the pupil center in the device coordinate system.

[0161] First, according to the historical line of sight data and the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system, the line of sight ray passing through the center of rotation of the eyeball can be determined;

[0162] The imaging light ray of the pupil center is: the starting point is S l , and the direction is R -1 (P -1 [O x , O y -T);

[0163] Find the intersection point of the two lines (i.e., the three-dimensional coordinates of the pupil center): Let the intersection point be p, the starting point of line 1 is r1, the direction is l1, the starting point of line 2 is r2, and the direction is 12, then it satisfies:

[0164] p = r1 + k1ι1 = r2 + k2ι2

[0165] Then the solution result of p is as follows:

[0166]

[0167] According to the above formula, the three-dimensional positions of the pupil centers of both eyes in the device coordinate system can be calculated. Since the above imaging light ray of the pupil center is calculated based on the position information of the historical pupil center in the imaging plane of the camera, the three-dimensional position of the pupil center obtained in the above steps is the three-dimensional position information of the historical pupil center in the device coordinate system.

[0168] In the above steps, the three-dimensional position information M l , M r of the center of rotation of the eyeball in the device coordinate system and the three-dimensional position information p l , p r of the historical pupil center in the device coordinate system have been obtained, and the distance between the "center of rotation of the eyeball - pupil center" of each eye can be obtained:

[0169]

[0170] It can be understood that steps 202 to 216 can be regarded as preprocessing steps. In other words, when performing the gaze estimation task each time, steps 202 to 216 do not have to be fully executed. Some fixed parameters calculated in steps 202 to 216 (for example, the distance from the eye rotation center to the pupil center in the device coordinate system, the three-dimensional position information of the eye rotation center in the device coordinate system, etc.) can be directly used for the gaze estimation task without secondary calculation.

[0171] In step 304, a target eye image can be obtained.

[0172] Optionally, the target eye image can be a real-time captured eye image or a currently captured eye image.

[0173] Next, the gaze can be calculated in the following manner.

[0174] In step 306, the target position information of the pupil center in the target eye image can be determined.

[0175] Optionally, through image recognition technology, pupil detection can be performed in the target eye image to obtain the target position information of the pupil center in the target eye image. This target position information can be two-dimensional coordinates in the imaging plane of the camera.

[0176] In step 308, based on the target position information and in combination with the distance from the eye rotation center to the pupil center in the device coordinate system, the three-dimensional position information of the pupil center in the device coordinate system can be determined.

[0177] In some embodiments, as Figure 3D shown, based on the target position information and in combination with the distance from the eye rotation center to the pupil center in the device coordinate system, determining the three-dimensional position information of the pupil center in the device coordinate system can further include:

[0178] In step 3082, based on the target position information, the imaging light ray of the pupil center is determined;

[0179] Specifically, based on the internal parameters P of camera 1048 and the external parameters R, T of camera 1048, in combination with the target position information, the imaging light ray of the pupil center can be determined.

[0180] In step 3084, based on the three-dimensional position information of the eye rotation center in the device coordinate system and the imaging light ray of the pupil center, in combination with the distance from the eye rotation center to the pupil center in the device coordinate system, the three-dimensional position information of the pupil center in the device coordinate system is determined.

[0181] In some embodiments, asFigure 3E As shown, based on the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system and the imaging light ray of the pupil center, and combining the distance from the center of rotation of the eyeball to the pupil center in the device coordinate system, to determine the three-dimensional position information of the pupil center in the device coordinate system, it may further include:

[0182] In step 30842, draw a spherical surface with the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system as the center of the sphere and the distance from the center of rotation of the eyeball to the pupil center in the device coordinate system as the radius;

[0183] In step 30844, determine the intersection point of the imaging light ray of the pupil center and the spherical surface;

[0184] In step 30846, determine the three-dimensional position information of the pupil center in the device coordinate system according to the intersection point of the imaging light ray of the pupil center and the spherical surface.

[0185] Since the distance d between the pupil center and the center of rotation of the eyeball was obtained in the previous step l 、d r , and for the eyes of the same user, this distance is a fixed value. Therefore, draw a sphere with the center of rotation of the eyeball as the center of the sphere and the radii d l 、d r , calculate the intersection point of the imaging light ray of the pupil center and this sphere, and the one closest to the starting point of the imaging light ray is the three-dimensional coordinates of the pupil center in the device coordinate system.

[0186] Optionally, the calculation method of the intersection point of the imaging light ray and the sphere is as follows:

[0187] Assume that x is a point on the sphere with a radius of r, and the ray passes through x, then there is an equation:

[0188] ||x - c|| 2 = r 2 And

[0189] Get: where d is d l 、d r ;

[0190] According to the solution of the quadratic equation In this equation:

[0191] c = (o - c) 2 - r 2 , then it is possible to determine whether there is an intersection point (i.e., the corresponding ray length) by judging Δ.

[0192] Since the ray direction (i.e., the imaging light ray) is known, the intersection point can be calculated. Since the point to be determined is on the side closer to the origin of the ray, the sign of the solution of the quadratic equation can be determined.

[0193] In step 310, the line-of-sight direction can be determined based on the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system and the three-dimensional position information of the center of the pupil in the device coordinate system.

[0194] After obtaining the three-dimensional coordinates of the center of the pupil and knowing the three-dimensional coordinates of the center of rotation of the eyeball, the line connecting the two is the line-of-sight direction.

[0195] As can be seen from the above embodiments, the embodiments of the present disclosure pre-calculate some fixed parameters for subsequent line-of-sight estimation tasks, which can achieve fast calculation of the line-of-sight direction, thereby improving the calculation efficiency, saving computing power, and avoiding frame loss, and improving the accuracy of the algorithm.

[0196] It should be noted that the method of the embodiments of the present disclosure can be executed by a single device, such as a computer or a server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In this case of a distributed scenario, one of the multiple devices can only execute one or more steps of the method of the embodiments of the present disclosure, and these multiple devices will interact with each other to complete the described method.

[0197] It should be noted that some embodiments of the present disclosure have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the above embodiments and still achieve the desired results. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0198] The embodiments of the present disclosure also provide a line-of-sight estimation device. Figure 4 The schematic diagram of the exemplary device 400 provided by the embodiments of the present disclosure is shown. As Figure 4 shown, the device 400 can be used to implement the method 200 or 300 and can further include the following modules.

[0199] The first determination module 402 is configured to: determine the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system and the distance from the center of rotation of the eyeball to the center of the pupil in the device coordinate system;

[0200] The acquisition module 404 is configured to: acquire a target eye image;

[0201] The second determination module 406 is configured to: determine the target position information of the pupil center in the target eye image;

[0202] The third determination module 408 is configured to: determine the three-dimensional position information of the pupil center in the device coordinate system according to the target position information and in combination with the distance from the center of rotation of the eyeball to the pupil center in the device coordinate system;

[0203] The fourth determination module 410 is configured to: determine the line of sight direction according to the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system and the three-dimensional position information of the pupil center in the device coordinate system.

[0204] In some embodiments, the first determination module 402 is configured to:

[0205] Obtain a first eye image;

[0206] Determine the first position information of the pupil center in the first eye image;

[0207] According to the first position information, determine the second position information of the center of rotation of the eyeball in the first eye image;

[0208] According to the second position information, determine the connection line between the optical center of the camera and the center of rotation of the eyeball in the device coordinate system;

[0209] Determine the interpupillary distance of the user's binoculars;

[0210] According to the connection line between the optical center of the camera and the center of rotation of the eyeball in the device coordinate system, and in combination with the interpupillary distance of the user's binoculars, determine the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system.

[0211] In some embodiments, the first determination module 402 is configured to:

[0212] Obtain a plurality of historical line-of-sight data corresponding to a plurality of historical eye images;

[0213] According to the plurality of historical line-of-sight data, and in combination with the first position information, determine the second position information.

[0214] In some embodiments, the first determination module 402 is configured to:

[0215] Determine the three-dimensional position information of the historical pupil center in the device coordinate system;

[0216] According to the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system and the three-dimensional position information of the historical pupil center in the device coordinate system, determine the distance from the center of rotation of the eyeball to the pupil center in the device coordinate system.

[0217] In some embodiments, the first determination module 402 is configured to:

[0218] Obtain a historical eye image and historical gaze data corresponding to the historical eye image;

[0219] Determine historical position information of the pupil center in the historical eye image;

[0220] Determine the imaging light ray of the pupil center according to the historical position information of the pupil center in the historical eye image;

[0221] Determine a line of sight ray passing through the center of rotation of the eyeball according to the historical gaze data and the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system;

[0222] Determine the three-dimensional position information of the historical pupil center in the device coordinate system according to the imaging light ray of the pupil center and the line of sight ray passing through the center of rotation of the eyeball.

[0223] In some embodiments, the third determination module 408 is configured to:

[0224] Determine the imaging light ray of the pupil center according to the target position information;

[0225] Determine the three-dimensional position information of the pupil center in the device coordinate system according to the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system, the imaging light ray of the pupil center, and the distance from the center of rotation of the eyeball to the pupil center in the device coordinate system.

[0226] In some embodiments, the third determination module 408 is configured to:

[0227] Draw a spherical surface with the three-dimensional position information of the center of rotation of the eyeball in the device coordinate system as the center of the sphere and the distance from the center of rotation of the eyeball to the pupil center in the device coordinate system as the radius;

[0228] Determine the intersection point of the imaging light ray of the pupil center and the spherical surface;

[0229] Determine the three-dimensional position information of the pupil center in the device coordinate system according to the intersection point of the imaging light ray of the pupil center and the spherical surface.

[0230] For convenience of description, when describing the above device, various modules are described separately according to their functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0231] The device of the above embodiment is used to implement the corresponding method 200 or 300 in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated herein.

[0232] An embodiment of the present disclosure also provides a computer device for implementing the above method 200 or 300. Figure 5 The schematic diagram of the hardware structure of an exemplary computer device 500 provided by an embodiment of the present disclosure is shown. The computer device 500 can be used to implement Figure 1A or Figure 1B the head-mounted wearable device 104, or can be used to implement Figure 1A the external device 112, or can be used to implement Figure 1A the server 114. In some scenarios, the computer device 500 can also be used to implement Figure 1A the database server 116.

[0233] As Figure 5 shown, the computer device 500 may include: a processor 502, a memory 504, a network module 506, a peripheral interface 508, and a bus 510. Among them, the processor 502, the memory 504, the network module 506, and the peripheral interface 508 are communicatively connected to each other inside the computer device 500 through the bus 510.

[0234] The processor 502 may be a central processing unit (CPU), an image processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. The processor 502 may be used to execute functions related to the technology described in the present disclosure. In some embodiments, the processor 502 may further include multiple processors integrated into a single logic component. For example, as Figure 5 shown, the processor 502 may include multiple processors 502a, 502b, and 502c.

[0235] The memory 504 may be configured to store data (for example, instructions, computer code, etc.). As Figure 5As shown, the data stored in the memory 504 may include program instructions (e.g., program instructions for implementing the method 200 or 300 of the embodiments of the present disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). The processor 502 can also access the program instructions and data stored in the memory 504 and execute the program instructions to operate on the data to be processed. The memory 504 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 504 may include a random access memory (RAM), a read-only memory (ROM), an optical disc, a magnetic disk, a hard disk, a solid state drive (SSD), a flash memory, a memory stick, etc.

[0236] The network interface 506 can be configured to provide communication with other external devices to the computer device 500 via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC), etc.), a cellular network, the Internet, or a combination of the above. It can be understood that the type of the network is not limited to the above specific examples.

[0237] The peripheral interface 508 can be configured to connect the computer device 500 to one or more peripheral devices to implement information input and output. For example, the peripheral devices may include input devices such as a keyboard, a mouse, a touchpad, a touch screen, a microphone, various sensors, etc. and output devices such as a display, a speaker, a vibrator, an indicator light, etc.

[0238] The bus 510 can be configured to transmit information between various components of the computer device 500 (e.g., the processor 502, the memory 504, the network interface 506, and the peripheral interface 508), such as an internal bus (e.g., a processor-memory bus), an external bus (a USB port, a PCI-E bus), etc.

[0239] It should be noted that although the architecture of the above computer device 500 only shows the processor 502, the memory 504, the network interface 506, the peripheral interface 508, and the bus 510, in the specific implementation process, the architecture of the computer device 500 may further include other components necessary for normal operation. In addition, those skilled in the art can understand that the architecture of the above computer device 500 may also only include the components necessary for implementing the solution of the embodiments of the present disclosure, and do not have to include all the components shown in the figure.

[0240] Based on the same inventive concept, corresponding to any of the above method embodiments, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method 200 or 300 as described in any of the above embodiments.

[0241] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0242] The computer instructions stored in the storage medium of the above embodiment are used to cause the computer to execute the method 200 or 300 described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0243] Based on the same inventive concept, corresponding to the method 200 or 300 in any of the above embodiments, the present disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processor to execute the method 200 or 300. Corresponding to the execution subject of each step in the embodiments of the method 200 or 300, the processor that executes the corresponding step can belong to the corresponding execution subject.

[0244] The computer program product of the above embodiment is used to cause the computer and / or the processor to execute the method 200 or 300 described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0245] Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples; under the concept of the present disclosure, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of brevity.

[0246] In addition, for simplicity of explanation and discussion, and so as not to render the embodiments of the present disclosure difficult to understand, well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Further, the devices may be shown in block diagram form in order to avoid rendering the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that details regarding the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the understanding of those skilled in the art). In cases where specific details (such as circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be practiced without these specific details or with variations of these specific details. Accordingly, these descriptions are to be regarded as illustrative rather than restrictive.

[0247] Although the present disclosure has been described in connection with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description. For example, other memory architectures (such as dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0248] Embodiments of the present disclosure are intended to cover all such alternatives, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A method for estimating line of sight, comprising: Determine the three-dimensional position information of the eye rotation center in the device coordinate system and the distance from the eye rotation center to the pupil center in the device coordinate system; Obtain a target eye image; Determine the target position information of the pupil center in the target eye image; According to the target position information and in combination with the distance from the eye rotation center to the pupil center in the device coordinate system, determine the three-dimensional position information of the pupil center in the device coordinate system; According to the three-dimensional position information of the eye rotation center in the device coordinate system and the three-dimensional position information of the pupil center in the device coordinate system, determine the line of sight direction.

2. The method according to claim 1, wherein, Determine the three-dimensional position information of the eye rotation center in the device coordinate system, including: Obtain a first eye image; Determine the first position information of the pupil center in the first eye image; According to the first position information, determine the second position information of the eye rotation center in the first eye image; According to the second position information, determine the connection line between the optical center of the camera and the eye rotation center in the device coordinate system; Determine the interpupillary distance of the user's two eyes; According to the connection line between the optical center of the camera and the eye rotation center in the device coordinate system and in combination with the interpupillary distance of the user's two eyes, determine the three-dimensional position information of the eye rotation center in the device coordinate system.

3. The method according to claim 2, wherein, According to the first position information, determine the second position information of the eye rotation center in the first eye image, including: Obtain multiple historical eye images corresponding to multiple historical line of sight data; According to the multiple historical line of sight data and in combination with the first position information, determine the second position information.

4. The method according to claim 2, wherein, Determine the distance from the eye rotation center to the pupil center in the device coordinate system, including: Determine the three-dimensional position information of the historical pupil center in the device coordinate system; According to the three-dimensional position information of the eye rotation center in the device coordinate system and the three-dimensional position information of the historical pupil center in the device coordinate system, determine the distance from the eye rotation center to the pupil center in the device coordinate system.

5. The method according to claim 4, wherein, Determine the three-dimensional position information of the historical pupil center in the device coordinate system, including: Obtain a historical eye image and the corresponding historical line of sight data; Determine the historical position information of the pupil center in the historical eye image; According to the historical position information of the pupil center in the historical eye image, determine the imaging light ray of the pupil center; According to the historical line of sight data and the three-dimensional position information of the eye rotation center in the device coordinate system, determine the line of sight ray passing through the eye rotation center; According to the imaging light ray of the pupil center and the line of sight ray passing through the eye rotation center, determine the three-dimensional position information of the historical pupil center in the device coordinate system.

6. The method according to claim 1, wherein, According to the target position information and in combination with the distance from the eye rotation center to the pupil center in the device coordinate system, determine the three-dimensional position information of the pupil center in the device coordinate system, including: According to the target position information, determine the imaging light ray of the pupil center; Based on the three-dimensional position information of the center of eye rotation in the device coordinate system and the imaging light of the pupil center, and combining the distance from the center of eye rotation to the pupil center in the device coordinate system, determine the three-dimensional position information of the pupil center in the device coordinate system.

7. The method according to claim 6, wherein, Based on the three-dimensional position information of the center of eye rotation in the device coordinate system and the imaging light of the pupil center, and combining the distance from the center of eye rotation to the pupil center in the device coordinate system, determining the three-dimensional position information of the pupil center in the device coordinate system includes: Taking the three-dimensional position information of the center of eye rotation in the device coordinate system as the center of the sphere and the distance from the center of eye rotation to the pupil center in the device coordinate system as the radius, draw a spherical surface; Determine the intersection point of the imaging light of the pupil center and the spherical surface; Based on the intersection point of the imaging light of the pupil center and the spherical surface, determine the three-dimensional position information of the pupil center in the device coordinate system.

8. A device for estimating line of sight, comprising: The first determination module is configured to: determine the three-dimensional position information of the center of eye rotation in the device coordinate system and the distance from the center of eye rotation to the pupil center in the device coordinate system; The acquisition module is configured to: acquire a target eye image; The second determination module is configured to: determine the target position information of the pupil center in the target eye image; The third determination module is configured to: based on the target position information and combining the distance from the center of eye rotation to the pupil center in the device coordinate system, determine the three-dimensional position information of the pupil center in the device coordinate system; The fourth determination module is configured to: based on the three-dimensional position information of the center of eye rotation in the device coordinate system and the three-dimensional position information of the pupil center in the device coordinate system, determine the line-of-sight direction.

9. A computer device, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and are executed by the one or more processors, and the programs include instructions for performing the method according to any one of claims 1-7.

10. A non-volatile computer-readable storage medium containing a computer program, which, when executed by one or more processors, causes the processors to perform the method according to any one of claims 1-7.

11. A computer program product, comprising computer program instructions which, when run on a computer, cause the computer to perform the method according to any one of claims 1-7.

Images