Eye movement tracking method, device, and apparatus for near-eye display device, and storage medium

Abstract

The application relates to the technical field of eye movement tracking, and provides an eye movement tracking method and device of a near-eye display device, an apparatus and a storage medium, wherein the near-eye display device is provided with a shooting module and a preset light source; the method comprises the following steps: acquiring a geometric eyeball model corresponding to a user wearing the near-eye display device; the geometric eyeball model is used for indicating geometric size features of the user's eyes; acquiring a target image obtained by shooting the user's eyes by the shooting module; the target image comprises a target light spot formed by the user's eyes reflecting light emitted by the preset light source, and a target pupil region corresponding to the pupil in the target image; and obtaining eye movement information of the user according to the geometric size features, the target light spot and the target pupil region, so as to improve the eye movement tracking accuracy of the near-eye display device.

Description

Technical Field

[0001] This application relates to the field of eye-tracking technology, and in particular to an eye-tracking method, apparatus, device, and storage medium for a near-eye display device. Background Technology

[0002] In related technologies, near-eye display devices typically employ either a pupil center-corneal reflection (PCCR) scheme or a single-lamp floodlight eye-tracking scheme for eye tracking. However, when using the PCCR scheme, if the near-eye display device uses only a single light source, the accuracy of eye tracking can easily become poor. Similarly, when using a single-lamp floodlight eye-tracking scheme, individual differences in the eyes of different users can also lead to poor eye tracking accuracy. Therefore, there is an urgent need to improve the eye-tracking accuracy of near-eye display devices. Summary of the Invention

[0003] The main objective of this application is to provide an eye-tracking method, apparatus, device, and storage medium for a near-eye display device, aiming to solve the technical problem of poor eye-tracking accuracy in near-eye display devices due to the presence of only a single light source or individual differences in the eyes of different users.

[0004] In a first aspect, this application provides an eye-tracking method for a near-eye display device, wherein the near-eye display device is equipped with a shooting module and a preset light source;

[0005] The eye-tracking method includes:

[0006] Obtain a geometric eye model corresponding to the user wearing the near-eye display device; the geometric eye model is used to indicate the geometric size characteristics of the user's eye.

[0007] The camera module captures a target image of the user's eye. The target image includes a target light spot formed by the light emitted from the preset light source reflected from the user's eye, and a target pupil area corresponding to the pupil in the user's eye on the target image.

[0008] The user's eye movement information is obtained based on the geometric size characteristics, the target light spot, and the target pupil region.

[0009] Secondly, this application provides an eye-tracking device for a near-eye display device, the eye-tracking device comprising:

[0010] The model acquisition module is used to acquire a geometric eyeball model corresponding to the user wearing the near-eye display device; the geometric eyeball model is used to indicate the geometric size features of the user's eye.

[0011] The target image acquisition module is used to acquire a target image of the user's eye obtained by the imaging module of the near-eye display device; the target image includes a target light spot formed by the light emitted by the preset light source of the near-eye display device reflected by the user's eye, and a target pupil area corresponding to the pupil in the user's eye on the target image;

[0012] An eye-tracking module is used to obtain the user's eye movement information based on the geometric size features, the target light spot, and the target pupil area.

[0013] Thirdly, this application provides a near-eye display device, which is equipped with a shooting module and a preset light source; the near-eye display device includes a memory and a processor;

[0014] The memory is used to store computer programs;

[0015] The processor is configured to execute the computer program and, in executing the computer program, implement the steps of the eye-tracking method for the near-eye display device as described above.

[0016] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the eye-tracking method for a near-eye display device as described above.

[0017] This application provides an eye-tracking method, apparatus, device, and storage medium for a near-eye display device. The near-eye display device is equipped with a shooting module and a preset light source. The eye-tracking method includes: acquiring a geometric eye model corresponding to a user wearing the near-eye display device; the geometric eye model is used to indicate the geometric size characteristics of the user's eye; acquiring a target image of the user's eye captured by the shooting module; the target image includes a target light spot formed by light emitted from the preset light source reflected from the user's eye, and a target pupil area corresponding to the pupil in the user's eye on the target image; and obtaining the user's eye movement information based on the geometric size characteristics, the target light spot, and the target pupil area.

[0018] Having acquired a geometric eye model corresponding to the user, the near-eye display device can use this model to indicate the geometric dimensions of the user's eyes. By combining these geometric dimensions with the target image's light spot and pupil region, the device can perform eye tracking to obtain the user's eye movement information. Furthermore, if different users have different geometric dimensions, the near-eye display device can utilize the individual geometric eye models of each user to determine their respective eye movement information, thereby improving the accuracy of eye tracking. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic flowchart of an eye-tracking method for a near-eye display device provided in an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of the structure of a geometric eyeball model according to an embodiment of this application;

[0022] Figure 3 This is a schematic diagram illustrating the geometric positional relationship between the imaging module and the preset light source of a near-eye display device according to an embodiment of this application and the user's eye.

[0023] Figure 4 This is a schematic diagram of a first pupil circle or a second pupil circle according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the pupil center position corresponding to the target image or the pupil center position corresponding to the preset image in an embodiment of this application;

[0025] Figure 6 This is a partial schematic diagram of a geometric eyeball model according to an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the normal vector projection line corresponding to the preset pupil circle in one embodiment of this application;

[0027] Figure 8 This is a schematic block diagram of an eye-tracking device for a near-eye display provided in an embodiment of this application;

[0028] Figure 9This is a schematic block diagram of a near-eye display device provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0031] This application provides an eye-tracking method, apparatus, device, and storage medium for a near-eye display device. The eye-tracking method for this near-eye display device can be applied to such devices. Near-eye display devices may include augmented reality (AR) glasses, virtual reality (VR) glasses, mixed reality (MR) glasses, AR headsets, VR headsets, MR headsets, etc., and are not limited thereto. The eye-tracking method can also be applied to a server, which can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0032] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0033] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating an eye-tracking method for a near-eye display device according to an embodiment of this application. It should be noted that the eye-tracking method for a near-eye display device provided in this embodiment can be used in a near-eye display device or on a server, and is not limited thereto.

[0034] For example, a near-eye display device is equipped with a camera module and a preset light source. The camera module can be used to capture images of the user's eyes while wearing the near-eye display device, i.e., to capture the user's eyes. The preset light source can be used to emit light towards the user's eyes for eye tracking.

[0035] like Figure 1 As shown, the display method of the near-eye display device includes steps S101 to S103.

[0036] S101. Obtain the geometric eye model corresponding to the user wearing the near-eye display device; the geometric eye model is used to indicate the geometric size characteristics of the user's eyes.

[0037] In some implementations, the geometric eye model can be obtained by fitting the geometry of the user's eye.

[0038] A geometric eye model can include an eye sphere and a corneal sphere. The eye sphere can be used to indicate the main body of the user's eye. The corneal sphere can be used to indicate the cornea located in front of the user's eye, such as the transparent corneal region in the user's eye. The eye sphere can have a corresponding eyeball radius to simulate the radius of the main body of the user's eye.

[0039] like Figure 2 As shown, the cross-section of the geometric eye model in a preset direction can include the spherical surface of the eyeball corresponding to the spherical body and the spherical surface of the cornea corresponding to the spherical body. The center of the spherical body can be represented as the eyeball center E. The center of the spherical body can be represented as the cornea center C. The radius of the eyeball can be represented as the eyeball radius R. In the geometric eye model, the iris and pupil of the user's eye can be fitted as two concentric circles included in the geometric eye model, namely the iris circle corresponding to the iris and the pupil circle corresponding to the pupil. The iris circle and the pupil circle are located on the same plane, and the normal directions of the iris circle and the pupil circle are parallel to the line connecting the eyeball center E and the cornea center C. The common center of the iris circle and the pupil circle can be regarded as the geometric center of the pupil. The geometric center of the pupil can be represented as the pupil center P. The pupil center P is located on the surface of the spherical body of the eyeball. The pupil center P can be the center point where the pupil surface is tangent to the spherical body of the eyeball.

[0040] A geometric eye model can be used to simulate the movement of a user's eyes. For example, when simulating eye rotation using a geometric eye model, the rotation can be represented as the eyeball in the geometric eye model rotating around its center E. During eye movement, if the position of the pupil changes, the position of the pupil center P in the geometric eye model will also change accordingly. If the position of the eyeball itself does not change, the position of the eyeball center E and the eyeball radius R in the geometric eye model will also remain unchanged. The position of the eyeball center E can also be called the eyeball center position. The position of the pupil center P can also be called the pupil center position. Based on this, a geometric eye model can be used to indicate the geometric dimensions of a user's eyes. These geometric dimensions can include the eyeball center position and the eyeball radius. Accordingly, the eyeball center position in the geometric eye model corresponds to the eyeball center position in the user's eyes. The eyeball radius in the geometric eye model corresponds to the eyeball radius in the user's eyes. Of course, this is not the only limitation. Geometric dimensional features can also include the distance between the center of the cornea and the center of the eyeball in the user's eye. The distance between the center of the cornea and the center of the eyeball in the geometric eyeball model is equivalent to the distance between the center of the cornea and the center of the eyeball in the user's eye, and is not restricted here. It is understandable that the geometric dimensional features of different users' eyes may be different.

[0041] Since the pupil center P can be the center point where the pupil surface is tangent to the spherical surface of the eyeball, the normal vector corresponding to the pupil center P in the geometric eyeball model can be used by near-eye display devices to estimate the user's gaze and thus determine the user's eye movement information. Based on this, the user's corresponding geometric eyeball model can be used by near-eye display devices to perform eye movement tracking on the user's eyes.

[0042] In some implementations, the geometric eye model can be pre-constructed. For example, the near-eye display device can acquire multiple preset images of the user's eye captured by a camera module. These preset images can be captured during the user's eye movement, allowing the near-eye display device to pre-construct a corresponding geometric eye model for the user. For instance, the near-eye display device can fit the geometry of the user's eye based on the multiple preset images to obtain the geometric eye model. The geometric eye model can be used to indicate the geometric dimensions of the user's eye, such as the position of the eyeball center, the eyeball radius, and the distance between the corneal center and the eyeball center. Of course, the geometric dimensions of the user's eye are not limited to these, and are not restricted here.

[0043] With a pre-built geometric eye model, near-eye display devices can establish a correlation between the user's corresponding geometric eye model and user identification information. For example, user identification information may include user voiceprint information, fingerprint information, iris information, and other information that can be used to identify the user, without limitation.

[0044] Of course, the geometric eye model can also be constructed in real time. For example, a near-eye display device can construct a geometric eye model corresponding to the user in response to a received preset modeling command. In one exemplary embodiment, when a user first activates the near-eye display device, the device can detect the preset modeling command. Accordingly, the near-eye display device can, in response to the preset modeling command, acquire multiple preset images of the user's eyes captured by the imaging module to construct a geometric eye model corresponding to the user. In another exemplary embodiment, the preset modeling command can also be determined based on preset operations performed by the user on the near-eye display device. These preset operations may include preset touch operations, preset gestures, preset voice commands, etc., and are not limited thereto.

[0045] When a geometric eye model corresponding to a user wearing a near-eye display device is obtained, the geometric eye model can be used by the near-eye display device to perform eye tracking on the user. For example, the near-eye display device can use the geometric eye model to indicate the geometric size characteristics of the user's eyes and determine the user's eye movement information, which helps to improve the convenience and accuracy of eye tracking for the near-eye display device.

[0046] S102. Obtain the target image of the user's eye captured by the shooting module; the target image includes the target light spot formed by the light emitted by the preset light source reflected by the user's eye, and the target pupil area corresponding to the pupil in the user's eye on the target image.

[0047] While a user is wearing a near-eye display device, a preset light source on the device emits light towards the user's eyes. The user's eyes reflect this light. The device's camera module captures an image of the user's eyes. The image captured by the camera module can include the light spot formed by the reflection of the preset light source from the user's eyes. Accordingly, since the image is obtained by the camera module capturing the user's eyes, it can include the pupil area corresponding to the pupil in the image.

[0048] During eye tracking of a user, the near-eye display device can identify the image captured by the camera module of the user's eye as the target image. The target image may include the target light spot formed by the light emitted from a preset light source reflected from the user's eye, and the target pupil area corresponding to the user's pupil in the target image.

[0049] Because the positions of the pupil and cornea change as the user moves their eyes, the position of the pupil's center P within the geometric eye model changes accordingly. Similarly, the position of the cornea's center C within the geometric eye model also changes. Therefore, the positions of the pupil's center P and the cornea's center C within the geometric eye model exhibit a degree of uncertainty. When a target image is acquired, including the target light spot and the target pupil region, the near-eye display device can determine the position of the pupil's center P within the geometric eye model. Since this position corresponds to the target light spot and the target pupil region in the target image, it can also be referred to as the pupil center position corresponding to the target image within the geometric eye model. Accordingly, when the target image is acquired, the target light spot included in the target image can be used by the near-eye display device to determine the position of the cornea in the user's eye at the corneal center C in the geometric eye model. The position of the corneal center C corresponds to the target light spot included in the target image. Therefore, the position of the corneal center C can also be called the corneal center position corresponding to the target image in the geometric eye model, that is, the corneal center position corresponding to the target image.

[0050] For example, a near-eye display device can calculate the position of the pupil center P in a geometric eye model based on the target light spot and the target pupil region included in the target image. The near-eye display device can also estimate the user's gaze by combining the position of the pupil center P in the geometric eye model and the position of the eyeball center E in the geometric eye model, thus obtaining the user's gaze information. Furthermore, the near-eye display device can calculate the pupil radius of the user's eye corresponding to the target image by combining the position of the pupil center P in the geometric eye model, the position of the eyeball center E in the geometric eye model, and the eyeball radius R, thus obtaining the user's pupil radius. Finally, the near-eye display device can obtain the user's eye movement information based on at least one of the user's gaze information and the user's pupil radius.

[0051] Accordingly, the near-eye display device can calculate the position of the corneal center C in the geometric eye model based on the target light spot included in the target image. The near-eye display device can determine the position of the corneal center corresponding to the target image by utilizing the target light spot included in the target image, combined with the corneal reflection geometry corresponding to the imaging module, the target light spot, and the preset light source in the near-eye display device. Since the distance between the corneal center and the eyeball center in the user's eye does not change with eye movement, after determining the position of the corneal center corresponding to the target image, the near-eye display device can update the eyeball center position in the geometric eye model based on the distance between the corneal center and the eyeball center in the geometric eye model. For example, if the near-eye display device experiences systematic drift, such as systematic drift caused by the user's head movements, the near-eye display device can update the eyeball center position in the geometric eye model to compensate for the systematic drift, thereby ensuring the accuracy of subsequent eye tracking of the user by the near-eye display device.

[0052] When a target image of the user's eye is captured by the camera module, the target image can be used by the near-eye display device to perform eye tracking. For example, the near-eye display device can use a geometric eye model to indicate the geometric size characteristics of the user's eye, and combine this with the target light spot and target pupil area included in the target image to determine the user's eye movement information. This helps to improve the convenience and accuracy of eye tracking for the near-eye display device.

[0053] S103. Obtain the user's eye movement information based on geometric size characteristics, target light spot, and target pupil area.

[0054] Before leaving the factory, the near-eye display device is calibrated, allowing it to determine the placement of the preset light source and the camera parameters of the shooting module. These camera parameters may include intrinsic, extrinsic, and distortion parameters, which are not limited here. Correspondingly, the near-eye display device can determine the reference plane corresponding to the shooting module.

[0055] Near-eye display devices can determine a second position of the target light spot on the user's eye based on the target light spot included in the target image.

[0056] For example, a near-eye display device can determine the depth information of a target light spot based on the target light spot included in the target image; and determine the second position of the target light spot on the user's eye based on the depth information of the target light spot.

[0057] In some implementations, where the shooting module may include at least one of a Time of Flight (TOF) camera and a light field camera, the target image may have corresponding depth information, and the near-eye display device may acquire the depth information of the target light spot.

[0058] In other implementations, since the imaging module has been calibrated, the near-eye display device can determine the depth information of the target light spot by combining the target image, the camera parameters of the imaging module, and the reference plane corresponding to the imaging module.

[0059] When light emitted from a preset light source shines on the user's eyes, since the user's eye can be considered a sphere, the depth information at different locations of the user's eye can be different, which is equivalent to different distances of the user's eye from the shooting module. Consequently, the positions of the reflected light from the preset light source to the shooting module also differ at different locations of the user's eye, resulting in different positions of the target light spots in the target image formed by the reflected light from the preset light source to the shooting module. For example... Figure 3 As shown, due to the influence of the geometric positional relationship between the imaging module and the preset light source of the near-eye display device and the user's eye, there is an offset between the second position of the target spot on the user's eye and the position of the target spot on the reference plane corresponding to the imaging module, i.e., parallax exists. The near-eye display device can determine the parallax corresponding to the target spot based on the reference plane corresponding to the imaging module and the target image, and thus determine the depth information of the target spot.

[0060] The near-eye display device can determine the first offset position of the target light spot on the reference plane corresponding to the shooting module based on the first position of the target light spot on the target image and the camera parameters of the shooting module; determine the parallax corresponding to the target light spot based on the first position and the first offset position of the target light spot; and determine the depth information of the target light spot based on the parallax, the distance between the intersection point of the preset light source and the reference plane, the relative horizontal distance between the shooting module and the preset light source, and the focal length of the shooting module. The calculation formula involved in the depth information of the target light spot can be expressed as:

[0061]

[0062] Where, Depth indicates the depth information of the target light spot; D indicates the distance between the intersection point of the preset light source and the reference plane, and the reference plane intersection point indicates the intersection point of the preset light source and the reference plane; d indicates the parallax corresponding to the target light spot; f x Used to indicate the focal length of the shooting module; x cThis is used to indicate the relative horizontal distance between the shooting module and the preset light source. With the shooting module calibrated, the near-eye display device can directly obtain the camera parameters of the shooting module, such as the distance D between the intersection of the preset light source and the reference plane, and the relative horizontal distance x between the shooting module and the preset light source. c And the focal length f of the shooting module x This means that once the shooting module has been calibrated, its camera parameters have been determined, and no restrictions are imposed here. Correspondingly, the parallax d corresponding to the target spot can be determined using the first position of the target spot on the target image and the camera parameters of the shooting module.

[0063] Having determined the depth information of the target light spot, the near-eye display device can combine this depth information with the camera parameters of the imaging module to determine a second position of the target light spot on the user's eye. This second position of the target light spot on the user's eye can then be used by the near-eye display device, in conjunction with a geometric eye model, to determine the user's eye movement information.

[0064] Near-eye display devices can determine the position of the pupil center P in a geometric eye model based on the target pupil region included in the target image. For example... Figure 4 As shown, the near-eye display device can perform perspective projection on the target pupil region to obtain at least two first pupil circles corresponding to the target pupil region. The depth information of the at least two first pupil circles can be a preset depth. The value of the preset depth can vary. Each identical preset depth can correspond to two first pupil circles. When the first pupil circles are determined, the position of the center point of the first pupil circle can be used to determine the pupil center position corresponding to the target image. For example, if the first line connecting the center point of the first pupil circle and the optical center of the camera of the imaging module can cover the pupil center position corresponding to the target image, then the near-eye display device can use the first line connecting the center point of the first pupil circle and the optical center of the camera of the imaging module to determine the pupil center position corresponding to the target image.

[0065] The normal vectors of the two first pupil circles at the same preset depth are different. The near-eye display device can determine the target pupil circle from at least two first pupil circles based on the second position of the target spot on the user's eye. The second position of the target spot is equivalent to its position in the spatial coordinate system corresponding to the shooting module, and can include 3D coordinates. The second position of the target spot on the user's eye can reflect the position of the reflection point when the user's eye reflects light emitted by the preset light source. For example, the near-eye display device can determine the first pupil circle whose normal vector conforms to the corneal reflection geometry corresponding to the light emitted by the preset light source as the target pupil circle based on the geometric positional relationship between the preset light source and the shooting module, combined with the second position of the target spot on the user's eye. For instance, if the normal vector of the first pupil circle conforms to the corneal reflection geometry, the light emitted by the preset light source can enter the shooting range corresponding to the shooting module after reflection by the user's eye, forming the target spot included in the target image. Therefore, the near-eye display device can determine the first pupil circle whose normal vector conforms to the corneal reflection geometry as the target pupil circle. When the normal vector of the first pupil circle does not conform to the geometric relationship of corneal reflection, the normal vector of the first pupil circle may correspond to a user's eye orientation. Under this orientation, the light emitted by the preset light source, after being reflected by the user's eye, cannot enter the shooting range corresponding to the shooting module to form the target spot included in the target image. Alternatively, under this orientation, the light spot formed by the light emitted by the preset light source entering the shooting range corresponding to the shooting module after being reflected by the user's eye differs in position from the target spot included in the target image. Each preset depth can correspond to one target pupil circle. The pupil center position of the target pupil circle can cover the pupil center position corresponding to the target image. For example, the pupil center position of the target pupil circle at a certain preset depth can be used as the pupil center position corresponding to the target image, and the distance between the pupil center position of the target pupil circle and the eyeball center position of the geometric eyeball model is equal to the eyeball radius of the geometric eyeball model.

[0066] For example, the target pupil circle can be used by a near-eye display device to determine the user's eye movement information. The eye movement information may include at least one of the user's gaze information and the pupil radius of the user's eye. Of course, the user's eye movement information is not limited to these, and no limitation is made here.

[0067] For example, geometric dimensions can include the position of the center of the user's eyeball. The normal vector of the target pupil circle can be used to estimate the user's gaze, thus obtaining the user's gaze information. Geometric dimensions can include the position of the center of the user's eyeball. For instance, if the target pupil circles at different preset depths are parallel to each other, then the normal vectors of the target pupil circles at different preset depths are also parallel. If the normal vector of the target pupil circle passes through the center of the eyeball, this normal vector can be used to indicate the direction from the center of the eyeball to the center of the pupil corresponding to the target image. Therefore, the direction indicated by the normal vector of the target pupil circle can be determined as the user's gaze direction, thus obtaining the user's gaze information. The user's gaze information can reflect the user's true gaze information.

[0068] Geometric dimensional features can include the radius of the user's eyeball. For example... Figure 5 As shown, given a defined target pupil circle, the near-eye display device can determine the pupil center position corresponding to the target image by utilizing the eyeball radius and the pupil center positions of the target pupil circles at different preset depths. The distance between the eyeball center and the pupil center position corresponding to the target image is equal to the eyeball radius. Accordingly, the near-eye display device can determine the pupil radius of the user's eye corresponding to the target image based on the radius of the target pupil circle where the pupil center position of the target image is located. The pupil radius of the user's eye corresponding to the target image reflects the user's actual pupil radius under real-time visual information.

[0069] Near-eye display devices can determine a user's eye movement information based on at least one of the user's gaze information and the user's pupil radius.

[0070] When eye tracking is performed on the user based on geometric size features, target light spot, and target pupil area to obtain the user's eye movement information, the near-eye display device can use one or more of the geometric size features, target light spot, and target pupil area to determine at least one of the user's gaze information corresponding to the target image and the user's pupil radius. The user's gaze information can reflect the user's true gaze information, and the user's pupil radius can reflect the user's true pupil radius, which helps to improve the eye tracking accuracy of the near-eye display device.

[0071] For example, geometric features include the location of the center of the user's eyeball and the radius of the eyeball.

[0072] In some implementations, a first position of the target spot on the target image is determined; a second position of the target spot on the user's eye is determined based on the first position of the target spot; perspective projection is performed on the target pupil region to obtain at least two first pupil circles corresponding to the target pupil region; the target pupil circle of the target image is determined from the at least two first pupil circles based on the second position of the target spot; and the user's eye movement information is obtained based on the position of the eyeball center, the eyeball radius, and the target pupil circle.

[0073] For example, when a target image is acquired, the near-eye display device can determine the first position of the target spot on the target image. The near-eye display device can perform image detection processing on the target image to detect the target spot included in the target image and obtain the first position of the target spot on the target image. If the target image is a two-dimensional (2D) image, the first position of the target spot can include 2D coordinates. For example, the first position of the target spot can be represented as (u, v), where u indicates the horizontal pixel position of the target spot on the target image, and v indicates the vertical pixel position of the target spot on the target image.

[0074] A near-eye display device can determine a second position of a target spot on the user's eye based on a first position of the target spot. For example, the near-eye display device can use the camera parameters of the imaging module to transform the first position of the target spot from the image coordinate system corresponding to the target image to the spatial coordinate system corresponding to the imaging module, thus obtaining the second position of the target spot on the user's eye. The second position of the target spot can include three-dimensional (3D) coordinates. For example, the target spot can be represented as a spot Pglint. The second position of the spot Pglint can be represented as (X, Y, Z), where X indicates the horizontal position of the spot Pglint relative to the imaging module, Y indicates the vertical position of the spot Pglint relative to the imaging module, and Z indicates the depth distance of the spot Pglint relative to the imaging module.

[0075] For example, given a target image, a near-eye display device can perform perspective projection on the target pupil region included in the target image. The near-eye display device can perform perspective projection on the target pupil region based on the pinhole imaging principle. For example... Figure 4As shown, during the perspective projection of the target pupil region, a target cone can be constructed with the target pupil region as its base and the camera's optical center / focal point as its vertex. The target pupil region is essentially the intersection line between the target cone and the image plane containing the target image. Based on the constructed target cone, the circular back projection corresponding to the target pupil region can be determined, i.e., at least two first pupil circles corresponding to the target pupil region. Specifically, the circular back projection of the target pupil region at a specified depth consists of two first pupil circles with different normal vectors. These two first pupil circles with different normal vectors are symmetric about the principal axis of an ellipse.

[0076] Taking two first pupil circles with different normal vectors corresponding to the target pupil region at a specified depth as an example, these two first pupil circles can be considered as a double-circle solution of the target pupil region at that specified depth, hereinafter referred to as the double-circle solution. The geometric shape of the target pupil region on the target image is an ellipse. The double-circle solution is due to the symmetry of the target pupil region about its elliptical major axis, while physically, the user's eye can only have one correct eye orientation. Given the second position of the target spot, the near-eye display device can use this second position as the basis for determining the correct solution from the double-circle solution. Since the near-eye display device has been calibrated, the position of the preset light source on the near-eye display device is fixed, and the position of the imaging device on the near-eye display device is fixed. Therefore, the positional distribution of the target spot relative to the user's eye will systematically change due to different user eye orientations. For example, in a double-circle solution, only one solution's normal vector satisfies the corneal reflection geometry corresponding to the light emitted from the preset light source. If the normal vector of the correct solution in the double-circle solution conforms to the corneal reflection geometry, then the light emitted from the preset light source, after being reflected by the user's eye, enters the shooting range of the camera module, forming the target spot included in the target image. If the normal vector of an incorrect solution in the double-circle solution does not conform to the corneal reflection geometry, the normal vector of the incorrect solution may correspond to a user's eye orientation. Under this orientation, the light emitted from the preset light source, after being reflected by the user's eye, cannot enter the shooting range of the camera module. Alternatively, under this orientation, the predicted spot formed by the light emitted from the preset light source entering the shooting range of the camera module after being reflected by the user's eye differs from the target spot included in the target image. For example, the first position of the predicted spot on the target image differs from the first position of the target spot on the target image. Another example is that the second position of the predicted spot on the user's eye differs from the second position of the target spot on the target image. Therefore, by observing the presence and position of the target spot in the target image, the near-eye display device can eliminate incorrect solutions from the double-circle solution and obtain the correct solution from the double-circle solution.

[0077] Accordingly, when perspective projection is performed on the target pupil region to obtain the target cone corresponding to the target pupil region, and at least two first pupil circles corresponding to the target pupil region are determined according to different preset depths, each preset depth can correspond to one target pupil circle. Different target pupil circles are parallel to each other, meaning that the normal vectors of different target pupil circles are the same. Based on this, the number of target pupil circles can include at least one.

[0078] Given a target pupil circle, the near-eye display device can obtain the user's eye movement information based on the geometric dimensions of the user's eye indicated by the geometric eye model and the target pupil circle, such as the position of the eyeball center, the eyeball radius, and the target pupil circle.

[0079] like Figure 5 As shown, when the target pupil circle is determined, the normal vector of the target pupil circle can be used to indicate the direction from the center of the eyeball to the center of the pupil corresponding to the target image. Thus, the near-eye display device can determine the user's gaze information, such as the user's gaze direction, based on the normal vector of the target pupil circle.

[0080] like Figure 5 As shown, given a defined target pupil circle, the near-eye display device can determine a first line connecting the pupil center and the camera optical center based on the pupil center of the target pupil circle and the camera optical center. The near-eye display device can then determine a second line connecting the eyeball center position and the pupil center position corresponding to the target image based on the normal vector of the target pupil circle. The intersection of the first and second lines can be identified as the pupil center position corresponding to the target image. The distance between the eyeball center position and the pupil center position corresponding to the target image is equal to the eyeball radius of the geometric eyeball model. Accordingly, the radius of the target pupil circle containing the pupil center position corresponding to the target image can be determined as the pupil radius of the user's eye corresponding to the target image.

[0081] If at least one of the user's gaze information or the user's pupil radius can be used to determine the user's eye movement information, it will help improve the convenience of eye tracking in near-eye display devices. Correspondingly, when a near-eye display device uses geometric dimensional features indicated by a geometric eye model to determine the user's eye movement information, the user's gaze direction can reflect the user's true gaze information, and the user's pupil radius can reflect the actual pupil radius under the true gaze information, which will help improve the accuracy of eye tracking in near-eye display devices.

[0082] For example, geometrical features include the location of the center of the eyeball at the user's eye and the distance between the center of the cornea and the center of the eyeball.

[0083] In some implementations, the corneal center position corresponding to the target image is determined based on the second position of the target spot on the user's eye; the eye center position is updated based on the corneal center position corresponding to the target image and the distance between the corneal center and the eye center to obtain the updated eye center position.

[0084] When a near-eye display device determines the second position of the target light spot on the user's eye, it can combine the corneal reflection geometry corresponding to the imaging module, the target light spot, and the preset light source in the near-eye display device to determine the corneal center position corresponding to the target image. However, this is not limited to this. When determining the geometric eye model corresponding to the user's eye, the near-eye display device can use the preset light spots included in multiple preset images to fit the corneal curvature radius of the cornea in the geometric eye model. Therefore, the geometric dimensional features of the geometric eye model can also include the corneal curvature radius. Since the corneal curvature radius does not change with the user's eye rotation or head movement, the near-eye display device can combine the second position of the target light spot on the user's eye and the corneal curvature radius of the geometric eye model to determine the corneal center position corresponding to the target image.

[0085] Since the distance between the corneal center and the eyeball center in a user's eye does not change with eye movement or head movement, the near-eye display device, given the corneal center position corresponding to the target image, can calculate the corresponding eyeball center position by combining the distance between the corneal center and the eyeball center in the geometric eye model. The near-eye display device can then determine whether the eyeball center position corresponding to the target image is the same as the eyeball center position in the geometric eye model. If they are different, the near-eye display device can determine that it has experienced a systematic drift and update the eyeball center position corresponding to the target image to the eyeball center position in the geometric eye model. This compensates for the systematic drift in the near-eye display device, allowing it to perform subsequent eye tracking and improving the accuracy of eye tracking.

[0086] The user's eye movement information is obtained based on the updated position of the eyeball center, the radius of the eyeball, and the target pupil circle.

[0087] For example, the process of obtaining the user's eye movement information based on the updated eyeball center position, eyeball radius, and target pupil circle can be referenced from the aforementioned process of obtaining the user's eye movement information based on the eyeball center position, eyeball radius, and target pupil circle, and will not be repeated here. When the updated eyeball center position replaces the previous one to determine the user's eye movement information, the updated eyeball center position can compensate for the systematic drift generated by the near-eye display device during eye tracking, thus avoiding the adverse effects of the near-eye display device's systematic drift on the accuracy of eye tracking.

[0088] Based on this, when a near-eye display device acquires a target image, it can determine the corneal center position corresponding to the target image by combining the second position of the target spot on the user's eye included in the target image. Then, by combining the corneal center position corresponding to the target image with the distance between the corneal center and the eyeball center, the eyeball center position is updated to obtain the updated eyeball center position. This compensates for the systematic drift that occurs during the near-eye display device's eye tracking of the user, avoiding the adverse effects of the systematic drift of the near-eye display device on the accuracy of eye tracking, thus improving the accuracy of the near-eye display device's eye tracking of the user.

[0089] In some implementations, multiple preset images of the user's eyes are captured by the camera module. The preset images include preset light spots formed by light emitted from a preset light source reflected from the user's eyes, and preset pupil areas corresponding to the pupils in the user's eyes on the preset images. The preset pupil areas of each of the multiple preset images are located at different positions on the corresponding preset images. Based on the preset light spots and preset pupil areas of each of the multiple preset images, geometric modeling processing is performed on the user's eyes to obtain a geometric eyeball model corresponding to the user.

[0090] Near-eye display devices can construct a geometric eye model corresponding to the user.

[0091] During the construction of the user's corresponding geometric eye model, the near-eye display device can determine the image of the user's eye captured by the imaging module as a preset image to obtain the preset image. The preset image may include a preset light spot formed by the light emitted by a preset light source reflected from the user's eye, and a preset pupil area corresponding to the pupil in the user's eye on the preset image.

[0092] In constructing the user's corresponding geometric eye model, to improve the accuracy of the model and ensure it fits the user's eye, the near-eye display device can acquire multiple preset images. These preset images can be obtained by capturing images of the user's rotating eye while a preset light source continuously illuminates it. Therefore, the position of the preset pupil region in each preset image will be different. The number of preset images must be at least three.

[0093] When multiple preset images are acquired, the near-eye display device can perform geometric modeling of the user's eye based on the preset light spots and preset pupil regions of each image, thus obtaining a corresponding geometric eyeball model. For example, the near-eye display device can use the preset light spots and preset pupil regions included in each preset image to determine the pupil center position corresponding to each of the multiple preset images. Accordingly, the near-eye display device can use the pupil center positions corresponding to each of the multiple preset images to fit the center position of the eyeball in the geometric eyeball model, and also fit the eyeball radius, thereby determining the geometric eyeball model.

[0094] When near-eye display devices can use multiple preset images to perform geometric modeling of the user's eyes and obtain the corresponding geometric eyeball model, it is beneficial to improve the convenience and accuracy of geometric modeling of the user's eyes.

[0095] In some implementations, a first position of a preset spot included in a preset image is determined on the preset image; a second position of the preset spot on the user's eye is determined based on the first position of the preset spot; a perspective projection is performed on a preset pupil region included in the same preset image to obtain at least two second pupil circles corresponding to the preset pupil region; a preset pupil circle of the preset image is determined from the at least two second pupil circles based on the second position of the preset spot; and geometric modeling is performed on the user's eye based on the preset pupil circles of multiple preset images to obtain a geometric eyeball model corresponding to the user.

[0096] For example, when a preset image is acquired, the preset image can be used to determine a first position of a preset light spot on the preset image. The near-eye display device can determine the first position of the preset light spot included in the preset image on the preset image. The step of determining the first position of the preset light spot included in the preset image on the preset image can refer to the aforementioned step of determining the first position of the target light spot on the target image, and will not be repeated here.

[0097] The near-eye display device can determine a second position of the preset light spot on the user's eye based on the first position of the preset light spot. The steps for determining the second position of the preset light spot on the user's eye can refer to the steps for determining the second position of the target light spot on the user's eye described above, and will not be repeated here.

[0098] For example, when a preset image is acquired, the preset image can be used to determine at least two second pupil circles corresponding to a preset pupil region. The near-eye display device can perform perspective projection on the preset pupil region included in the same preset image to obtain at least two second pupil circles corresponding to the preset pupil region. The at least two second pupil circles can be as follows: Figure 4 As shown. The steps for performing perspective projection on the preset pupil region included in the same preset image can refer to the aforementioned steps for performing perspective projection on the target pupil region, and will not be repeated here.

[0099] Accordingly, since the preset image includes a preset light spot and a preset pupil region, when at least two pupil circles corresponding to the preset pupil region are determined, the preset pupil circle can be determined from at least two second pupil circles based on the second position of the preset light spot. The step of determining the preset pupil circle of the preset image from at least two second pupil circles based on the second position of the preset light spot can refer to the aforementioned step of determining the target pupil circle from at least two first pupil circles based on the second position of the target light spot, and will not be repeated here.

[0100] For the same preset image, there can be a preset pupil circle at each preset depth, which can include preset pupil circles at different preset depths. The pupil center position of the preset pupil circle can cover the pupil center position of the preset image. For example, the pupil center position of the preset pupil circle at a certain preset depth can be used as the pupil center position of the preset image.

[0101] Each preset image can determine at least one preset pupil circle. Since the preset pupil regions of multiple preset images are located at different positions on the corresponding preset images, the normal vectors of the preset pupil circles of multiple preset images are also different accordingly.

[0102] Near-eye display devices can perform geometric modeling of the user's eyes based on the preset pupil circles of multiple preset images to obtain the corresponding geometric eyeball model of the user.

[0103] like Figure 4As shown, given a predetermined pupil circle, since the normal vector of the predetermined pupil circle can be used to indicate the direction from the center of the eyeball in the geometric eyeball model to the center of the pupil corresponding to the predetermined image, the normal vector of the predetermined pupil circle of each predetermined image can pass through the center of the eyeball. Based on this, the near-eye display device can use the predetermined pupil circles of multiple predetermined images to fit the eyeball of the geometric eyeball model to determine the center position of the eyeball and the radius of the eyeball, thereby obtaining the geometric size characteristics of the user's eye indicated by the geometric eyeball model and completing the construction of the geometric eyeball model.

[0104] When a near-eye display device can determine the preset pupil circle of each of multiple preset images and use the preset pupil circles of each of the multiple preset images to perform geometric modeling of the user's eye to obtain the corresponding geometric eyeball model, it is beneficial to improve the convenience of geometric modeling of the user's eye.

[0105] In some implementations, the eyeball center of the user's corresponding geometric eyeball model is fitted based on the normal vectors of the preset pupil circles of multiple preset images to obtain the position of the eyeball center of the geometric eyeball model; the eyeball radius of the geometric eyeball model is determined based on the multiple preset pupil circles and the position of the eyeball center of the geometric eyeball model; and the user's eye is geometrically modeled based on the position of the eyeball center and the eyeball radius to obtain the geometric eyeball model.

[0106] For example, since the normal vector of each preset pupil circle in the preset image can pass through the eyeball center position of the geometric eyeball model, the near-eye display device can perform eyeball center fitting processing on the user's corresponding geometric eyeball model based on the normal vectors of multiple preset pupil circles to obtain the eyeball center position of the geometric eyeball model. The eyeball center position of the geometric eyeball model can be used to indicate the eyeball center position of the user's eye.

[0107] Given multiple preset images, each with its own preset pupil circle, the pupil center position of the preset pupil circle can differ across images. Therefore, a near-eye display device can use the pupil center positions of the preset pupil circles from different images to fit the eyeball radius of a geometric eyeball model. The eyeball radius of the geometric eyeball model can then be used to indicate the radius of the user's eyeball. Based on this, the geometric eyeball model can be used to indicate the geometric dimensional characteristics of the user's eye.

[0108] For example, during the process of fitting the eyeball radius of the geometric eyeball model, the near-eye display device can determine the pupil center position of the preset pupil circle of each of the multiple preset images on the geometric eyeball model; and determine the eyeball radius of the geometric eyeball model based on the distance between the pupil center position of each of the multiple preset images and the eyeball center position of the geometric eyeball model.

[0109] like Figure 6 As shown, the pupil center positions of the preset pupil circles in multiple preset images include, for example, positions P1, P2, and P3. The near-eye display device can calculate the distances between each of positions P1, P2, and P3 and the eyeball center E, and determine the eyeball radius of the geometric eyeball model based on these distances. For example, the near-eye display device can determine the eyeball radius of the geometric eyeball model based on one of the following: the average, median, maximum, minimum, or weighted average of the distances between each of positions P1, P2, and P3 and the eyeball center E.

[0110] Given a defined center and radius of the eyeball in the geometric eyeball model, a near-eye display device can construct a corresponding geometric eyeball model for the user. For example, the near-eye display device can set appropriate parameters for the eyeball within a pre-defined geometric eyeball model based on the center and radius, thereby obtaining the user's corresponding geometric eyeball model.

[0111] When the position of the eyeball center and the eyeball radius of the geometric eyeball model can be determined based on the normal vectors of multiple preset pupil circles, the geometric eyeball model can be used to indicate the geometric size characteristics of the user's eye, which helps to improve the convenience of determining the geometric size characteristics of the user's eye, and thus helps to improve the convenience of determining the geometric eyeball model of the user's eye.

[0112] In some implementations, the normal vector of a preset pupil circle in a preset image is projected onto a preset plane to obtain the normal vector projection line corresponding to the preset pupil circle; the position closest to multiple normal vector projection lines is determined as the eye center position of the geometric eyeball model on the preset plane; the eye center position is back-projected to obtain the eyeball center position of the geometric eyeball model.

[0113] For example, the normal vector of the preset pupil circle of each preset image passes through the center of the eyeball in the geometric eyeball model. Based on this, the near-eye display device can use the normal vectors of the preset pupil circles of multiple preset images to determine the center of the eyeball in the geometric eyeball model.

[0114] The pupil constricts under different environments, meaning the pupil radius varies in real-world scenarios. Consequently, if the pupil radius of each preset pupil circle in different preset images is set to the same preset pupil radius for the near-eye display device to determine the pupil center position of the geometric eye model, the pupil center positions determined by different preset images will each carry different scale errors. For example, when the actual pupil radius increases—that is, the pupil radius of the preset pupil circle is greater than the original preset pupil radius—using the preset pupil radius to calculate the pupil center position of the geometric eye model will result in a calculated pupil center position that is farther from the original preset pupil circle's pupil center position. Conversely, when the actual pupil radius decreases—that is, the pupil radius of the preset pupil circle in the preset image is smaller than the original preset pupil radius—using the preset pupil radius to calculate the pupil center position of the geometric eye model will result in a calculated pupil center position that is closer to the shooting module than the original preset pupil circle's pupil center position. As a result, the preset pupil circle of each preset image will have different degrees of deviation due to the pupil radius deviating from the preset pupil radius, such as uncertainty of "small and near" or "large and far". These deviations cannot be corrected by a uniform global scaling factor.

[0115] To address the aforementioned issues, near-eye display devices can project the normal vectors of the preset pupil circles of all preset images onto a preset plane, obtaining the normal vector projection lines corresponding to the preset pupil circles. These normal vector projection lines can cover the projection corresponding to the center of the preset pupil circles; no restriction is imposed here.

[0116] For at least two second pupil circles corresponding to a preset pupil region on the same preset image, the projection lines of the normal vectors of the second pupil circles are parallel on the preset plane, such as... Figure 7 As shown. The preset pupil circle of the preset image can be represented as (p i ,n i ,r i ), where p i Used to indicate the center position of the preset pupil circle in the i-th preset image, n i The normal vector r used to indicate the preset pupil circle of the i-th preset image. i The pupil radius is used to indicate the preset pupil circle of the i-th preset image, where i indicates the number of preset images. When the normal vector of the preset pupil circle is projected onto a preset plane to obtain the normal vector projection line corresponding to the preset pupil circle, the normal vector projection line can be represented as:

[0117]

[0118] Among them, L i Used to indicate the normal vector projection line corresponding to the i-th preset pupil circle; Used to indicate the starting point of the normal vector projection line corresponding to the i-th preset pupil circle; The direction vector corresponding to the i-th preset pupil circle is used to indicate the extension direction of the projection line of the normal vector corresponding to the i-th preset pupil circle; s is a parameter used to indicate the position of the control point on the projection line of the normal vector corresponding to the i-th preset pupil circle. Used to indicate the set of real numbers.

[0119] By performing the above processing on all preset images, the normal vector projection lines corresponding to each preset image can be obtained. The near-eye display device can calculate the point closest to each normal vector projection line to obtain the eye center position of the geometric eye model on the preset plane. The eye center position is a 2D position, and it corresponds to the eyeball center position of the geometric eye model. The near-eye display device can then back-project the eye center position to obtain the eyeball center position of the geometric eye model.

[0120] With the normal vectors of multiple preset pupil circles determined, the near-eye display device can determine the position of the eyeball center of the geometric eyeball model based on the normal vectors of the multiple preset pupil circles. This helps to improve the convenience of determining the position of the eyeball center of the geometric eyeball model, and thus improves the convenience of geometric modeling of the geometric eyeball model.

[0121] In some implementations, a first line is determined between the preset pupil center of the preset pupil circle and the camera light rays of the shooting module; a second line is determined between the eyeball center position of the geometric eyeball model and the pupil center position corresponding to the preset image, based on the normal vector of the preset pupil circle; the preset pupil center position corresponding to the preset image is determined based on the intersection of the first and second lines; and the eyeball radius of the geometric eyeball model is determined based on the distances between the preset pupil center positions corresponding to each of the multiple preset images and the eyeball center position.

[0122] like Figure 5 As shown, given a predetermined pupil circle, if the first line connecting the predetermined pupil center of the predetermined pupil circle and the optical center of the camera in the imaging module can cover the pupil center position corresponding to the predetermined image, then the near-eye display device can determine the first line connecting the predetermined pupil center of the predetermined pupil circle and the optical center of the camera in the imaging module. The predetermined pupil center of the predetermined pupil circle can be determined based on the pupil center of the second pupil circle. Since the normal vector of the predetermined pupil circle can pass through the eyeball center position of the geometric eyeball model, and the normal vector of the predetermined pupil circle can point from the eyeball center position to the pupil center position corresponding to the predetermined image, the near-eye display device can determine the second line connecting the eyeball center position and the pupil center position corresponding to the predetermined image based on the normal vector of the predetermined pupil circle. Accordingly, the near-eye display device can determine the pupil center position corresponding to the predetermined image based on the intersection of the first and second lines.

[0123] Given multiple preset images, each preset image may have a different pupil center position. The near-eye display device can calculate the distance between the pupil center position and the eyeball center position of each preset image based on the pupil center position and the eyeball center position of each preset image. Then, the near-eye display device can determine the eyeball radius of the geometric eyeball model based on the distances of each preset image.

[0124] like Figure 6 As shown, the pupil center positions of the preset pupil circles in multiple preset images include, for example, positions P1, P2, and P3. The near-eye display device can calculate the distances between each of positions P1, P2, and P3 and the position E of the eyeball center, and determine the eyeball radius of the geometric eyeball model based on the distances between each of positions P1, P2, and P3 and the position E of the eyeball center.

[0125] In one exemplary embodiment, the near-eye display device can be calibrated before leaving the factory, such as calibrating the reference plane corresponding to the shooting module, etc., without limitation. During the process of determining the user's corresponding geometric eye model, the near-eye display device can emit light to the user's eye through a preset light source and capture a preset image of the user's eye through its shooting module. The preset image includes a preset light spot formed by the reflection of the light emitted by the preset light source from the user's eye. The near-eye display device can determine a first position of the preset light spot on the preset image and a second position of the preset light spot on the user's eye. Based on the second position of the preset light spot on the user's eye, the normal vector of the plane where the preset pupil center of the preset image is located is calculated, i.e., the normal vector of the preset pupil circle of the preset image is determined. The near-eye display device can use the normal vector of the preset pupil circle of the preset image to determine the position of the pupil center P corresponding to the preset image. A near-eye display device can capture multiple preset images, which can be taken while the user moves their eyes. The device can then determine the position of the pupil center P corresponding to each of the preset images. Using these positions, it can fit the eyeball center position and radius of a geometric eyeball model, thus completing the construction of the geometric eyeball model. The multiple preset images can include at least three preset images.

[0126] Given that the preset pupil circles of multiple preset images and the center position of the eyeball of the geometric eyeball model are determined, the near-eye display device can use the preset pupil circles and the center position of the eyeball to determine the center position of the pupil corresponding to the preset image, and combine the center positions of the pupils corresponding to multiple preset images to determine the radius of the eyeball of the geometric eyeball model, which helps to improve the convenience of geometric modeling of the geometric eyeball model.

[0127] Given a defined geometric eye model for a user, this model can be used to determine the geometric dimensions of the user's eyes during eye tracking, thereby improving the accuracy of eye tracking in near-eye display devices.

[0128] In some implementations, the corneal center position corresponding to the preset image is determined based on the second position of the preset spot on the user's eye; the distance between the corneal center and the eye center in the geometric eye model is determined based on the corneal center positions corresponding to multiple preset images and the eye center position in the geometric eye model.

[0129] For example, given a predetermined second position of a preset light spot on the user's eye, the near-eye display device can use this second position, combined with the corneal reflection geometry corresponding to the imaging module, the preset light spot, and the preset light source, to determine the corneal center position corresponding to the preset image. Correspondingly, the near-eye display device can use the corneal center positions corresponding to multiple preset images and the eye center position of the geometric eye model to determine the distance between the corneal center and the eye center in the geometric eye model. For instance, the near-eye display device can determine the distance between the corneal center and the eye center in the geometric eye model based on the average of the differences between the corneal center positions corresponding to multiple preset images and the eye center position of the geometric eye model. However, this is not a limitation and is not set forth here.

[0130] In the process of constructing a geometric eye model, near-eye display devices can also use the corneal center positions corresponding to multiple preset images and the eye center position of the geometric eye model to fit the corneal curvature radius of the cornea in the geometric eye model. For example, the corneal curvature radius of the cornea in the geometric eye model can be iteratively optimized to obtain the corneal curvature radius of the cornea in the geometric eye model. No restrictions are imposed here.

[0131] When a near-eye display device uses multiple preset images, each containing a preset light spot, and the position of the eyeball center in a geometric eyeball model to determine the distance between the corneal center and the eyeball center in the geometric eyeball model, the distance between the corneal center and the eyeball center can help the near-eye display device improve the fit between the geometric eyeball model and the user's eye, thereby enabling subsequent eye tracking and improving the accuracy of eye tracking in the near-eye display device.

[0132] The display method for the near-eye display device provided in the above embodiments includes: acquiring a geometric eye model corresponding to a user wearing the near-eye display device; the geometric eye model is used to indicate the geometric size characteristics of the user's eye; acquiring a target image of the user's eye captured by a shooting module; the target image includes a target light spot formed by light emitted from a preset light source reflected from the user's eye, and a target pupil area corresponding to the pupil in the user's eye on the target image; and obtaining the user's eye movement information based on the geometric size characteristics, the target light spot, and the target pupil area.

[0133] Having acquired a geometric eye model corresponding to the user, the near-eye display device can use this model to indicate the geometric dimensions of the user's eyes. By combining these geometric dimensions with the target image's light spot and pupil region, the device can perform eye tracking to obtain the user's eye movement information. Furthermore, if different users have different geometric dimensions, the near-eye display device can utilize the individual geometric eye models of each user to determine their respective eye movement information, thereby improving the accuracy of eye tracking.

[0134] Please see Figure 8 , Figure 8 This is a schematic block diagram of an eye-tracking device for a near-eye display device provided in an embodiment of this application. The eye-tracking device for the near-eye display device can be configured in the near-eye display device or a server to execute the aforementioned eye-tracking method for the near-eye display device. The near-eye display device may include AR glasses, VR glasses, MR glasses, AR helmets, VR helmets, MR helmets, etc., and is not limited thereto. The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms.

[0135] like Figure 8 As shown, the eye-tracking device of the near-eye display device includes a model acquisition module 110, a target image acquisition module 120, and an eye-tracking module 130.

[0136] The model acquisition module is used to acquire a geometric eyeball model corresponding to the user wearing the near-eye display device; the geometric eyeball model is used to indicate the geometric size features of the user's eye.

[0137] The target image acquisition module is used to acquire a target image of the user's eye obtained by the imaging module of the near-eye display device; the target image includes a target light spot formed by the light emitted by the preset light source of the near-eye display device reflected by the user's eye, and a target pupil area corresponding to the pupil in the user's eye on the target image;

[0138] An eye-tracking module is used to obtain the user's eye movement information based on the geometric size features, the target light spot, and the target pupil area.

[0139] For example, the geometric dimensions include the position of the center of the user's eyeball and the radius of the eyeball; the eye tracking module 130 includes a first position determination submodule, a second position determination submodule, a first pupil circle determination submodule, a target pupil circle determination submodule, and an eye movement information determination submodule.

[0140] The first position determination submodule is used to determine the first position of the target spot on the target image;

[0141] The second position determination submodule is used to determine the second position of the target light spot on the user's eye based on the first position of the target light spot;

[0142] The first pupil circle determination submodule is used to perform perspective projection on the target pupil region to obtain at least two first pupil circles corresponding to the target pupil region;

[0143] The target pupil circle determination submodule is used to determine the target pupil circle of the target image from at least two first pupil circles based on the second position of the target spot;

[0144] The eye movement information determination submodule is used to obtain the user's eye movement information based on the position of the eyeball center, the eyeball radius, and the target pupil circle.

[0145] For example, the geometric dimensions include the position of the center of the eyeball of the user's eye and the distance between the center of the cornea and the center of the eyeball; the eye-tracking device includes a first corneal center of position determination submodule and an eyeball center of position update submodule.

[0146] The corneal center position determination submodule is used to determine the corneal center position corresponding to the target image based on the second position of the target spot on the user's eye;

[0147] The eyeball center position update submodule is used to update the eyeball center position based on the corneal center position corresponding to the target image and the distance between the corneal center and the eyeball center, so as to obtain the updated eyeball center position;

[0148] The eye movement information determination submodule includes an eye movement information update submodule.

[0149] The eye movement information update submodule is used to obtain the user's eye movement information based on the updated eyeball center position, the eyeball radius, and the target pupil circle.

[0150] For example, the model acquisition module 110 includes a preset image acquisition submodule and a modeling submodule.

[0151] The preset image acquisition submodule is used to acquire multiple preset images of the user's eyes captured by the shooting module; the preset images include preset light spots formed by the light emitted by the preset light source reflected by the user's eyes, and preset pupil areas corresponding to the pupils in the user's eyes on the preset images; the preset pupil areas of each of the multiple preset images are located at different positions on the corresponding preset images;

[0152] The modeling submodule is used to perform geometric modeling processing on the user's eyes based on the preset light spots and preset pupil regions of each of the multiple preset images, so as to obtain the geometric eyeball model corresponding to the user.

[0153] For example, the modeling submodule includes a third position determination submodule, a fourth position determination submodule, a second pupil circle determination submodule, a preset pupil circle determination submodule, and a geometric modeling submodule.

[0154] The third position determination submodule is used to determine the first position of the preset light spot included in the preset image on the preset image;

[0155] The fourth position determination submodule is used to determine the second position of the preset light spot on the user's eye based on the first position of the preset light spot;

[0156] The second pupil circle determination submodule is used to perform perspective projection on the preset pupil region included in the same preset image to obtain at least two second pupil circles corresponding to the preset pupil region;

[0157] The preset pupil circle determination submodule is used to determine the preset pupil circle of the preset image from at least two second pupil circles based on the second position of the preset light spot;

[0158] The geometric modeling submodule is used to perform geometric modeling processing on the user's eyes based on the preset pupil circles corresponding to each of the multiple preset images, so as to obtain the geometric eyeball model corresponding to the user.

[0159] For example, the geometric modeling submodule includes an eyeball center position determination submodule, an eyeball radius determination submodule, and a modeling submodule.

[0160] The eyeball center position determination submodule is used to perform eyeball center fitting processing on the geometric eyeball model corresponding to the user based on the normal vector of the preset pupil circle of each of the multiple preset images, so as to obtain the eyeball center position of the geometric eyeball model;

[0161] The eyeball radius determination submodule is used to determine the eyeball radius of the geometric eyeball model based on the multiple preset pupil circles and the eyeball center position of the geometric eyeball model.

[0162] The modeling submodule is used to perform geometric modeling of the user's eye based on the position of the eyeball center and the eyeball radius to obtain the geometric eyeball model.

[0163] For example, the eyeball center position determination submodule includes a normal vector projection submodule, an eye center position determination submodule, and a back projection submodule.

[0164] The normal vector projection submodule is used to project the normal vector of the preset pupil circle of the preset image onto a preset plane to obtain the normal vector projection line corresponding to the preset pupil circle.

[0165] The eye center position determination submodule is used to determine the position closest to the projection lines of the multiple normal vectors as the eye center position of the geometric eyeball model on a preset plane;

[0166] The back-projection submodule is used to back-project the center position of the eye to obtain the center position of the eyeball of the geometric eyeball model.

[0167] For example, the eyeball radius determination submodule includes a first connection submodule, a second connection submodule, an intersection point determination submodule, and a distance determination submodule.

[0168] The first connection submodule is used to determine the first connection between the preset pupil center of the preset pupil circle and the camera optical center of the shooting module;

[0169] The second connection submodule is used to determine the second connection between the center position of the eyeball of the geometric eyeball model and the center position of the pupil corresponding to the preset image, based on the normal vector of the preset pupil circle.

[0170] The intersection point determination submodule is used to determine the pupil center position corresponding to the preset image based on the intersection point of the first line and the second line.

[0171] The distance determination submodule is used to determine the eyeball radius of the geometric eyeball model based on the distance between the pupil center position corresponding to each of the multiple preset images and the eyeball center position.

[0172] For example, the eye-tracking model includes a corneal center-of-sphere position fitting submodule and a center-of-sphere distance fitting submodule.

[0173] The corneal center of gravity fitting submodule is used to determine the corneal center of gravity corresponding to the preset image based on the second position of the preset spot on the user's eye.

[0174] The center-of-spot distance fitting submodule is used to determine the distance between the corneal center and the eyeball center in the geometric eyeball model based on the corneal center positions corresponding to each of the multiple preset images and the eyeball center position of the geometric eyeball model.

[0175] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and its modules and units can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0176] The method of this application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0177] For example, the above-described method and apparatus can be implemented as a computer program that can run on a near-eye display device or a server to control the near-eye display device. For example, the near-eye display device may include AR glasses, VR glasses, MR glasses, AR helmets, VR helmets, MR helmets, etc., without limitation. The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms.

[0178] Please see Figure 9 , Figure 9 This is a schematic block diagram of the structure of a near-eye display device provided in an embodiment of this application.

[0179] like Figure 9 As shown, the near-eye display device is equipped with a shooting module and a preset light source; the near-eye display device includes a memory and a processor. The memory and processor can be connected via a system bus, and the memory may include a storage medium and internal memory.

[0180] The storage medium can store the operating system and computer programs. When the computer program is executed, it enables the processor to perform eye-tracking methods for any near-eye display device.

[0181] The processor provides computing and control capabilities to support the operation of the entire near-eye display device.

[0182] Internal memory provides an environment for the execution of computer programs stored in storage media. When executed by a processor, the computer program enables the processor to perform eye-tracking methods for any near-eye display device.

[0183] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the near-eye display device to which the present application is applied. A specific near-eye display device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0184] It should be understood that a processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other convertible logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0185] In one embodiment, the processor is configured to execute a computer program and, when executing the computer program, perform the following steps:

[0186] Obtain a geometric eye model corresponding to the user wearing the near-eye display device; the geometric eye model is used to indicate the geometric size characteristics of the user's eye.

[0187] The camera module captures a target image of the user's eye. The target image includes a target light spot formed by the light emitted from the preset light source reflected from the user's eye, and a target pupil area corresponding to the pupil in the user's eye on the target image.

[0188] The user's eye movement information is obtained based on the geometric size characteristics, the target light spot, and the target pupil region.

[0189] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of eye tracking in the near-eye display device described above can be referred to the corresponding process in the aforementioned embodiments of the eye tracking method for the near-eye display device, and will not be repeated here.

[0190] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method implemented can be referred to in various embodiments of the eye-tracking method of the near-eye display device of this application.

[0191] The computer-readable storage medium can be an internal storage unit of the near-eye display device described in the foregoing embodiments, such as the hard disk or memory of the near-eye display device. Alternatively, the computer-readable storage medium can be an external storage device of the near-eye display device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the near-eye display device.

[0192] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0193] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0194] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above descriptions are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An eye-tracking method for a near-eye display device, characterized in that, The near-eye display device is equipped with a shooting module and a preset light source; The eye-tracking method includes: Obtain a geometric eye model corresponding to the user wearing the near-eye display device; the geometric eye model is used to indicate the geometric size characteristics of the user's eye. The camera module captures a target image of the user's eye. The target image includes a target light spot formed by the light emitted from the preset light source reflected from the user's eye, and a target pupil area corresponding to the pupil in the user's eye on the target image. The user's eye movement information is obtained based on the geometric size characteristics, the target light spot, and the target pupil region.

2. The eye-tracking method according to claim 1, characterized in that, The geometric dimensional features include the position of the center of the user's eyeball and the radius of the eyeball; The step of obtaining the user's eye movement information based on the geometric size features, the target light spot, and the target pupil region includes: Determine the first position of the target spot on the target image; Based on the first position of the target light spot, determine the second position of the target light spot on the user's eye; By performing perspective projection on the target pupil region, at least two first pupil circles corresponding to the target pupil region are obtained; Based on the second position of the target spot, the target pupil circle of the target image is determined from at least two first pupil circles; The user's eye movement information is obtained based on the location of the eyeball center, the eyeball radius, and the target pupil circle.

3. The eye-tracking method according to claim 2, characterized in that, The geometric dimensional features include the position of the center of the eyeball of the user's eye and the distance between the center of the cornea and the center of the eyeball; After determining the second position of the target light spot on the user's eye based on the first position of the target light spot, the eye-tracking method further includes: Based on the second position of the target light spot on the user's eye, the corneal center position corresponding to the target image is determined; Based on the corneal center position corresponding to the target image and the distance between the corneal center and the eyeball center, the eyeball center position is updated to obtain the updated eyeball center position. The step of obtaining the user's eye movement information based on the position of the eyeball center, the eyeball radius, and the target pupil circle includes: The user's eye movement information is obtained based on the updated eyeball center position, the eyeball radius, and the target pupil circle.

4. The eye-tracking method according to any one of claims 1 to 3, characterized in that, The step of obtaining the geometric eye model corresponding to the user wearing the near-eye display device includes: The camera module captures multiple preset images of the user's eyes. Each preset image includes a preset light spot formed by light reflected from the user's eyes by a preset light source, and a preset pupil area corresponding to the user's pupil in the preset image. The preset pupil areas of each of the multiple preset images are located at different positions on the corresponding preset image. Based on the preset light spots and preset pupil regions of each of the multiple preset images, geometric modeling processing is performed on the user's eyes to obtain the corresponding geometric eyeball model.

5. The eye-tracking method according to claim 4, characterized in that, The step of performing geometric modeling processing on the user's eye based on the preset light spots and preset pupil regions of each of the multiple preset images to obtain the corresponding geometric eyeball model includes: Determine the first position of the preset light spot included in the preset image on the preset image; Based on the first position of the preset light spot, determine the second position of the preset light spot on the user's eye; By performing perspective projection on the preset pupil region included in the same preset image, at least two second pupil circles corresponding to the preset pupil region are obtained; Based on the second position of the preset light spot, the preset pupil circle of the preset image is determined from at least two second pupil circles; Based on the preset pupil circles of each of the multiple preset images, geometric modeling is performed on the user's eyes to obtain the corresponding geometric eyeball model.

6. The eye-tracking method according to claim 5, characterized in that, The step of performing geometric modeling processing on the user's eye based on the preset pupil circles of each of the multiple preset images to obtain the corresponding geometric eyeball model includes: Based on the normal vectors of the preset pupil circles of each of the multiple preset images, the eyeball center of the geometric eyeball model corresponding to the user is fitted to obtain the position of the eyeball center of the geometric eyeball model; The radius of the eyeball of the geometric eyeball model is determined based on the multiple preset pupil circles and the position of the eyeball center of the geometric eyeball model. Based on the location of the eyeball center and the eyeball radius, a geometric model of the user's eye is obtained by performing geometric modeling on the eye.

7. The eye-tracking method according to claim 6, characterized in that, The step of performing eyeball center fitting processing on the geometric eyeball model corresponding to the user based on the normal vectors of the preset pupil circles of the multiple preset images to obtain the eyeball center position of the geometric eyeball model includes: The normal vector of the preset pupil circle of the preset image is projected onto the preset plane to obtain the normal vector projection line corresponding to the preset pupil circle; The position closest to the projection lines of the multiple normal vectors is determined as the center position of the eye of the geometric eyeball model on the preset plane; By back-projecting the center position of the eye, the center position of the eyeball in the geometric eyeball model is obtained.

8. The eye-tracking method according to claim 6, characterized in that, The step of determining the eyeball radius of the geometric eyeball model based on the multiple preset pupil circles and the eyeball center position of the geometric eyeball model includes: Determine the first line connecting the preset pupil center of the preset pupil circle and the camera optical center of the shooting module; Based on the normal vector of the preset pupil circle, determine the second line connecting the position of the eyeball center of the geometric eyeball model and the position of the pupil center corresponding to the preset image; The pupil center position corresponding to the preset image is determined based on the intersection of the first line and the second line. The radius of the eyeball in the geometric eyeball model is determined based on the distance between the pupil center position and the eyeball center position corresponding to each of the multiple preset images.

9. The eye-tracking method according to claim 6, characterized in that, After obtaining the position of the eyeball center of the geometric eyeball model, the eye-tracking method further includes: Based on the second position of the preset light spot on the user's eye, determine the corneal center position corresponding to the preset image; Based on the corneal center position corresponding to each of the multiple preset images and the eyeball center position of the geometric eyeball model, the distance between the corneal center and the eyeball center in the geometric eyeball model is determined.

10. An eye-tracking device for a near-eye display device, characterized in that, The eye-tracking device includes: The model acquisition module is used to acquire a geometric eyeball model corresponding to the user wearing the near-eye display device; the geometric eyeball model is used to indicate the geometric size features of the user's eye. The target image acquisition module is used to acquire a target image of the user's eye obtained by the imaging module of the near-eye display device; the target image includes a target light spot formed by the light emitted by the preset light source of the near-eye display device reflected by the user's eye, and a target pupil area corresponding to the pupil in the user's eye on the target image; An eye-tracking module is used to obtain the user's eye movement information based on the geometric size features, the target light spot, and the target pupil area.

11. A near-eye display device, characterized in that, The near-eye display device is equipped with a shooting module and a preset light source; the near-eye display device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the steps of the eye-tracking method for a near-eye display device as described in any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the eye-tracking method for a near-eye display device as described in any one of claims 1 to 9.

Images