Eye tracking device, display device, and storage medium

Abstract

Embodiments of the present application provide an eye movement tracking device, a display device and a storage medium. The eye movement tracking device comprises: an optical display module, the optical display module comprising a display device and an optical module, the optical module comprising a first optical device, the display device emitting first light which is incident on a user's eye after passing through the optical module; and an eye movement tracking module comprising: at least one light source configured to emit second light towards the user's eye, at least part of the second light being reflected after irradiating the eye; and a camera module configured to acquire at least part of the second light reflected after the at least one light source irradiates the eye; wherein the camera module is located between the first optical device and the display device. The above arrangement of the camera module provided by the present application can reduce the inclination angle of the eye movement tracking light, which is conducive to acquiring more reflected light (i.e. second light) (reflected by the user's eye), thereby improving the accuracy of eye movement tracking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of line-of-sight tracking, and particularly relates to an eye movement tracking device, a display device and a storage medium. BACKGROUND

[0002] Currently, some head-mounted display (HMD) devices have an eye movement tracking function, which refers to tracking of eye movement by measuring the position of a fixation point of an eye or the movement of an eyeball relative to a head, or in other words, refers to tracking of the direction of a line of sight of both eyes of a user.

[0003] The implementation of the above eye movement tracking is generally as follows: near-infrared light is emitted to the eyes of a user by a near-infrared light source, and a near-infrared camera installed outside a mirror group (on a reflection path of the near-infrared light reflected by the eyes of the user) is used to take a picture of the eyes of the user, and then the direction of the gaze of the user is inferred by light and back-end analysis.

[0004] Among them, the mirror group structure of some head-mounted display devices is relatively compact, for example, a Pancake mirror group. Under the Pancake architecture, the distance from the eyeballs of a user to the Pancake mirror group is relatively short, and therefore if a near-infrared camera is installed outside the Pancake mirror group, the tilt angle of the movement tracking is too large, and the picture of the eyeballs taken does not meet the requirements of the algorithm. SUMMARY

[0005] The present application provides an eye movement tracking device, a display device and a storage medium, wherein the camera module for obtaining reflected light of the eyes of a user in the eye movement tracking device is arranged between a first optical device and a display device of an optical display module. Compared with arranging a near-infrared camera outside a mirror group, the above arrangement of the camera module provided in the present application can reduce the tilt angle of the light for eye movement tracking, which is conducive to obtaining more reflected light (i.e., second light) (reflected by the eyes of the user), and thus the accuracy of eye movement tracking can be improved.

[0006] In a first aspect, embodiments of the present application provide an eye tracking device, comprising: an optical display module, the optical display module comprising a display device and an optical module, the optical module comprising a first optical device, the first light emitted by the display device being incident on the user's eye after passing through the optical module; and an eye tracking module, comprising: at least one light source configured to emit second light towards the user's eye, at least part of the second light being reflected after illuminating the eye; and a camera module configured to acquire at least part of the second light reflected after the at least one light source illuminates the eye; wherein the camera module is located between the first optical device and the display device. It should be noted that the camera module is located between the first optical device and the display device, and the nearest lens adjacent to the display device in the first optical device has a columnar space with the display device. In an embodiment, the camera module can be located in the columnar space. In another embodiment, the camera module can not be limited to being arranged in the columnar space, but can also be arranged at a point (point B) outside the columnar space along the vertical direction of the center axis of the lens barrel from a point (point A) in the columnar space.

[0007] In some embodiments, at least part of the second light is reflected by the user's glasses and passes through the first optical device, and is acquired by the camera module.

[0008] In some embodiments, the focal length of the optical module is variable.

[0009] In some embodiments, the eye tracking device further comprises a focal length measurement unit configured to trigger focal length detection to determine the current focal length information after the focal length of the optical module is adjusted. In an embodiment, the focal length measurement unit can comprise a Hall sensor, a grating ruler or a sliding rheostat.

[0010] In some embodiments, the distance between the first optical device and the display device is adjustable, and the first optical device is a lens.

[0011] In some embodiments, the first optical device is a variable focal length lens. The variable focal length lens can adjust the focal length by adjusting the distance between the lens and the display device, and the focal length measurement unit can detect the current distance between the lens and the display device after the distance between the lens and the display device is changed. In some cases, the variable focal length lens can be a concave-convex shape variable lens, which changes the focal length by changing the shape of the lens.

[0012] In some embodiments, the optical module further comprises an external lens interface for mounting the myopia correction lens on the lens barrel by magnetic attraction or buckling. In the scenario of myopia correction by an external myopia correction lens, the focal length measurement unit can comprise a storage chip and an identity chip. In one implementation, the storage chip can store the correspondence between the identity information and the optical power information of each myopia correction lens, and the storage chip can be arranged in the optical module. The identity chip stores the identity information (e.g., identification number) of the myopia correction lens, and the identity chip can be arranged on the myopia correction lens. After the myopia correction lens is externally connected to the lens barrel of the optical module, the optical power information (i.e., focal length information) of the myopia correction lens can be matched in the storage chip based on the identity information.

[0013] In some embodiments, when the focal length of the optical module changes, the gaze point direction of the user's eye does not change, and the eye movement tracking device knows that the gaze point of the user's eye does not change.

[0014] In some embodiments, when the focal length of the optical module changes, the light path table suitable for the current focal length information is determined. The specific implementation of determining the light path table suitable for the current focal length information can refer to the light path table correction method provided in the second aspect.

[0015] In some embodiments, the eye movement tracking device further comprises a processor that receives the results obtained by the camera module and knows the gaze point of the user's eye.

[0016] In some embodiments, the optical display module further comprises a second optical device, and at least part of the second light passes through the second optical device before being obtained by the camera module.

[0017] In some embodiments, the second optical device is located between the first optical device and the display device, or the second optical device is located on the side of the first optical device away from the display device. In one implementation, the second optical device can be a lens, and the first optical device and the second optical device can constitute a mirror group, which can be a Pancake mirror group, but is not limited to a Pancake mirror group.

[0018] In a second aspect, the embodiments of the present application further provide an eye movement tracking method for an eye movement tracking device, the eye movement tracking device comprising an optical display module and an eye movement tracking module; the optical display module comprising a display device and an optical module, the optical module comprising a first optical device; the eye movement tracking module comprising at least one light source and a camera module; the method comprising: the first light emitted by the display device passing through the optical module and being incident on the user's eye; the at least one light source being configured to emit the second light to the user's eye, at least part of the second light being reflected after irradiating the eye; and at least part of the second light reflected after the at least one light source irradiating the eye passing through the first optical device being obtained by the camera module.

[0019] In some embodiments, the method further comprises: obtaining focal length information of the optical module; and obtaining the gaze direction of the user's eye. The focal length information can include the focal length of the optical module, etc. In some embodiments, the gaze direction of the user's eye is obtained according to the focal length information of the optical module and the second light obtained by the camera module.

[0020] In some embodiments, the method further comprises: obtaining a set of optical path tables, the set of optical path tables including optical path tables corresponding to at least two myopia correction positions; obtaining a current myopia correction parameter after a myopia correction operation; and

[0021] Performing optical path table interpolation on the set of optical path tables according to the myopia correction parameter to obtain a target optical path table.

[0022] In some embodiments, the method further comprises: obtaining the set of optical path tables from a target memory in which the set of optical path tables is pre-stored.

[0023] In some embodiments, the focal length of the optical module is variable; the eye movement tracking device further comprises a focal length measurement unit configured to trigger focal length detection after the focal length of the optical module is adjusted and obtain current focal length information.

[0024] In some embodiments, when the focal length of the optical module changes, the gaze point direction of the user's eye does not change, and the eye movement tracking device obtains a gaze point of the user's eye that does not change.

[0025] In a third aspect, the embodiments of the present application further provide a method for correcting an optical path table, comprising: obtaining a set of optical path tables, the set of optical path tables including optical path tables corresponding to at least two myopia correction positions; obtaining a current myopia correction parameter after a myopia correction operation; and performing optical path table interpolation on the set of optical path tables according to the myopia correction parameter to obtain a target optical path table.

[0026] In some embodiments, obtaining the set of optical path tables comprises: obtaining the set of optical path tables from a target memory in which the set of optical path tables is pre-stored.

[0027] In some embodiments, obtaining the current myopia correction parameter after the myopia correction operation comprises: obtaining position information of a translated lens in a lens group after the myopia correction operation; and converting the position information to obtain the current myopia correction parameter after the myopia correction operation.

[0028] In some embodiments, the myopia correction operation comprises translating a lens in a lens group to adjust the position of a virtual image distance; and obtaining the position information of the translated lens in the lens group after the myopia correction operation comprises: obtaining current position sensing information provided by a position sensor mounted on the translated lens, wherein the position sensor comprises a Hall sensor, an optical grating ruler, or a sliding rheostat.

[0029] In some embodiments, the current myopia correction parameter after the myopia correction operation is determined according to the position information, including: determining the myopia correction parameter corresponding to the current position sensing information based on the relationship between the pre-calibrated myopia correction parameter and the position sensing information.

[0030] In some embodiments, the myopia correction operation includes adding a myopia lens in front of the lens barrel to adjust the position of the virtual image distance; and the current myopia correction parameter after the myopia correction operation is obtained, including: obtaining the refractive power information of the myopia lens added in front of the lens barrel.

[0031] In some embodiments, the refractive power information of the myopia lens added in front of the lens barrel is obtained, including: obtaining the identity information of the myopia lens added in front of the lens barrel; and matching the refractive power information of the myopia lens added in front of the lens barrel based on the identity information.

[0032] In some embodiments, before the target optical path table is obtained by performing optical path table interpolation on the optical path table set according to the myopia correction parameter, the method further includes: obtaining a position set of a light source (such as a near-infrared LED), the position set of the near-infrared LED including position information of the near-infrared LED under at least two myopia correction parameters; performing interpolation on the position set of the near-infrared LED according to the current myopia correction parameter to obtain a current position of the near-infrared LED under the previous myopia correction parameter; and updating the position information of the near-infrared LED according to the current position of the near-infrared LED under the previous myopia correction parameter.

[0033] In a fourth aspect, the embodiments of the present application further provide a light path table correction method, including: after a myopia correction operation, a target lens in a lens group is moved from a first position to a second position; determining a current myopia correction parameter after the myopia correction operation according to distance information between the first position and the second position; performing optical path table interpolation on a light path table set according to the myopia correction parameter to obtain a target optical path table; wherein the light path table set is pre-stored data, and the light path table set includes optical path tables corresponding to at least two myopia correction positions.

[0034] In a fifth aspect, another embodiment of the present application further provides a light path table correction device, including: a processor and a memory, the memory is used to store at least one instruction, and the instruction is loaded and executed by the processor to realize the light path table correction method provided in the second aspect or the third aspect.

[0035] In a sixth aspect, another embodiment of the present application further provides a head-mounted display device, which includes the light path table correction device provided in the fourth aspect. In an implementation manner, the light path table correction device can be a component element of the head-mounted display device, for example, a chip.

[0036] In a seventh aspect, the embodiments of the present application further provide a head-mounted display device, which includes the eye movement tracking device provided in the first aspect.

[0037] In an eighth aspect, the application further provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the method of the second aspect, the third aspect or the fourth aspect.

[0038] By the above technical solution, the camera module is arranged between the first optical device of the optical display module and the display device, compared with arranging the near-infrared camera outside the mirror group, the above arrangement of the camera module provided by the application can reduce the inclination angle of the eye movement tracking light, which is conducive to obtaining more reflected light (i.e., the second light) (reflected by the user's eyes), and thus the accuracy of eye movement tracking can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor under the premise of the drawings.

[0040] Figure 1 a schematic diagram of a VR system provided by an embodiment of the application;

[0041] Figure 2A a structural schematic diagram of a VR head-mounted display device provided by an embodiment of the application;

[0042] Figure 2B a simplified schematic diagram of a VR head-mounted display device provided by an embodiment of the application;

[0043] Figure 3 a schematic diagram of a head-mounted display device based on a Pancake folded light path in the related art;

[0044] Figure 4 a light polarization schematic diagram of a Pancake mirror group in the related art;

[0045] Figure 5 a structural schematic diagram of a VR head-mounted display device provided by an embodiment of the application;

[0046] Figure 6 an exemplary software structural block diagram of a VR head-mounted display device of an embodiment of the application;

[0047] Figure 7 a schematic diagram of eye movement tracking based on corneal reflection in the related art;

[0048] Figure 8 a schematic diagram of corneal reflection light passing through a Pancake mirror group and a near-infrared camera lens in the related art;

[0049] Figure 9 An optical path table calibration diagram for eye movement tracking in the related art;

[0050] Figure 10a An myopia correction method diagram in the related art;

[0051] Figure 10b Another myopia correction method diagram in the related art;

[0052] Figure 11 A system architecture diagram provided by another embodiment of the present application;

[0053] Figure 12 A flowchart of an optical path table correction method provided by an embodiment of the present application;

[0054] Figure 13 An optical path table calibration diagram provided by another embodiment of the present application;

[0055] Figure 14 A myopia correction parameter acquisition diagram provided by another embodiment of the present application;

[0056] Figure 15 A myopia correction lens installation diagram provided by another embodiment of the present application;

[0057] Figure 16 A myopia correction parameter transmission diagram provided by another embodiment of the present application;

[0058] Figure 17a A myopia correction diagram by rotating a lens barrel provided by another embodiment of the present application;

[0059] Figure 17b A myopia correction diagram by extending a lens barrel provided by another embodiment of the present application;

[0060] Figure 18 A flowchart of an optical path table correction method provided by another embodiment of the present application;

[0061] Figure 19 A structure diagram of an optical path table correction device provided by another embodiment of the present application. DETAILED DESCRIPTION

[0062] To make the purposes, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0063] (1) The at least one involved in the embodiments of the present application includes one or more; wherein the plurality refers to greater than or equal to two. In addition, it should be understood that in the description of the present application, the terms "first", "second", and the like are only used for the purpose of distinguishing the description and cannot be understood as indicating or implying relative importance or indicating or implying order. For example, the first area and the second area do not represent the importance of the two or the order of the two, but only distinguish the areas. In the embodiments of the present application, "and / or" is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0064] (2) Virtual reality (VR) technology is a kind of man-machine interactive means created by means of computer and sensor technology. VR technology integrates computer graphics technology, computer simulation technology, sensor technology, display technology and other scientific technologies, and can create a virtual environment. The virtual environment includes three-dimensional realistic images generated by a computer and played in real time, which brings visual perception to the user; in addition to the visual perception generated by the computer graphics technology, there are auditory, tactile, force perception, movement and other perceptions, even including olfactory and gustatory perceptions, also known as multi-perception; in addition, the head rotation of the user, eyes, gestures, or other human behavior actions can be detected, and the data corresponding to the actions of the user can be processed by the computer, and the actions of the user can be responded in real time, and feedback to the five senses of the user, thereby forming a virtual environment. For example, the user wearing a VR wearable device can see a VR game interface, and can interact with the VR game interface through gestures, handles, etc., as if being in the game.

[0065] (3) Augmented reality (AR) technology refers to superimposing virtual objects generated by a computer on a real-world scene, thereby achieving the augmentation of the real world. That is, in the AR technology, the real-world scene needs to be collected, and then a virtual environment is added to the real world.

[0066] Therefore, the difference between the VR technology and the AR technology is that the VR technology creates a complete virtual environment, and all the objects that the user sees are virtual objects; while the AR technology superimposes virtual objects on the real world, that is, both real objects and virtual objects are included. For example, the user wears transparent glasses, through which the user can see the real environment around, and the glasses can also display virtual objects, so that the user can see both real objects and virtual objects.

[0067] (4) Mixed Reality (MR) is to introduce real scene information (or real scene information) into a virtual environment, to build a bridge of interactive feedback information between the virtual environment, the real world and the user, thereby enhancing the reality of the user experience. Specifically, the real object is virtualized (for example, a camera is used to scan the real object for three-dimensional reconstruction to generate a virtual object), and the virtualized real object is introduced into the virtual environment, so that the user can see the real object in the virtual environment.

[0068] It should be noted that the technical solutions provided in the embodiments of the present application can be applied to the scene where the electronic device for eye tracking is a head-mounted device in a VR scene, an AR scene or an MR scene, and can also be applied to the scene where the electronic device for eye tracking is a non-head-mounted device, for example, the scene of eye tracking using a terminal device (such as a mobile phone, a tablet computer, etc.), a computer monitor, a smart car or a large-screen device such as a television, and the like. For another example, in the driving scene of a smart car, the gaze point of the human eye can also be more accurately determined by the technical solutions provided in the embodiments of the present application, and eye tracking can be more quickly and accurately performed. In short, the technical solutions provided in the embodiments of the present application are applicable to any scene where it is necessary to accurately determine the gaze point of the human eye for eye tracking.

[0069] For the convenience of understanding, the following mainly takes the VR scene as an example for introduction.

[0070] For example, please refer to Figure 1, and an image processing device 200. The VR system can be referred to as a VR split machine. The VR head-mounted display device 100 can be connected with the processing device 200. The connection between the VR head-mounted display device 100 and the processing device 200 includes wired or wireless connection. The wireless connection can be Bluetooth (BT), can be traditional Bluetooth or Bluetooth Low Energy (BLE), wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Zigbee, frequency modulation (FM), near field communication (NFC), infrared (IR), or general 2.4G / 5G frequency band wireless communication connection, etc.

[0071] In some embodiments, the image processing device 200 can perform processing calculation. For example, the image processing device 200 can generate and process images (the processing method will be described later), and then send the processed images to the VR head-mounted display device for display. The image processing device 200 can include a host (such as a VR host) or a server (such as a VR server). The VR host or the VR server can be a device with large computing power. For example, the VR host can be a mobile phone, a tablet computer, a notebook computer, etc., and the VR server can be a cloud server, etc.

[0072] In some embodiments, the VR head-mounted display device 100 can be glasses, a helmet, etc. The VR head-mounted display device 100 is generally provided with two display devices, i.e., a display device 110 and a display device 120. The display device of the VR head-mounted display device 100 can display images to the human eye. In the embodiment shown in the figure, the display device 110 and the display device 120 are wrapped inside the VR glasses, so the display device 110 and the display device 120 are not visible to the outside world. Figure 1 In the embodiment shown in the figure, the display device 110 and the display device 120 are wrapped inside the VR glasses, so the display device 110 and the display device 120 are not visible to the outside world. Figure 1 In the embodiment shown in the figure, the display device 110 and the display device 120 are wrapped inside the VR glasses, so the display device 110 and the display device 120 are not visible to the outside world.

[0073] In some embodiments, the VR head-mounted display device 100 also has the functions of image generation and processing, i.e., the VR head-mounted display device 100 does not need the image processing device 200 in the embodiment. Such a VR head-mounted display device 100 can be referred to as a VR all-in-one machine. Figure 1 In some embodiments, the VR head-mounted display device 100 also has the functions of image generation and processing, i.e., the VR head-mounted display device 100 does not need the image processing device 200 in the embodiment. Such a VR head-mounted display device 100 can be referred to as a VR all-in-one machine.

[0074] Figure 2AThis is a schematic diagram of a VR head-mounted display device 100. Figure 2A As shown in (a), the VR head-mounted display device 100 includes an optical display module 210 and an optical display module 220. The optical display module 210 includes a display device and an optical device 211. The optical display module 220 includes a display screen and an optical device 221. In some embodiments, the optical display modules 210 and 220 further include optical devices. In some embodiments, the optical display modules 210 and 220 can each be two hollow cylindrical lenses, with the optical devices housed within the lenses. The optical devices and display devices are mounted on the VR head-mounted display device 100 via the lenses. When a user wears the VR head-mounted display device 100, the optical display module 210 can be used to display an image to the user's left eye. The optical display module 220 can be used to display an image to the user's right eye. It is understood that... Figure 2A The VR headset 100 shown in (a) may also include other components, such as a support 230 and a bracket 240. The support 230 supports the VR headset 100 on the bridge of the nose, and the bracket 240 supports the VR headset 100 on the ears to ensure stable wearing of the VR headset 100. Figure 2A As shown in (b), at least one eye-tracking module (including M light sources 2501 and at least one camera module 2502) can be provided on the VR head-mounted display device 100 to track the movement of the human eye and thereby determine the gaze point of the human eye. In some embodiments, the end face 100a of the VR head-mounted display device 100 facing the face (or the user's eyes) may be provided with light sources 2501, and the end face 100a of the VR head-mounted display device 100 facing the face (or the user's eyes) may also be provided with camera modules 2502. For example, the light source 2501 is located on the end face 210a of the lens barrel facing the eye, and the end face 210a of the lens barrel facing the eye can be understood as a part of the end face 100a. Taking the eye-tracking module 250 on the optical display module 210 as an example, in some embodiments, the eye-tracking module 250 includes a light source 2501 and a camera module 2502. For example, eight light sources 2501 are arranged on the eye-facing end face 210a of the lens barrel, and eight light sources 2502 can be evenly distributed in a circle. The camera module 2502 can be disposed on the eye-facing end face 210a of the lens barrel. The light sources 2502 and the camera module 2502 can both be arranged around the eye-facing side of the optical device 211.

[0075] For ease of description, please refer to Figure 2B , Figure 2B This can be understood as... Figure 2A A simplified version of the VR head-mounted display device 100, for example,Figure 2B Only optical display modules 210 and 220 are shown; other components are not shown. Figure 2B When a user wears the VR headset 100, the display device 110 is located on the side of the optical device 211 away from the left eye, and the display device 120 is located on the side of the optical device 221 away from the right eye. The optical devices 211 and 221 are symmetrical with respect to the center line of the face or the center line D of the VR headset 100. The center line of the face can be the perpendicular bisector between the right and right eyes; the center line D of the VR headset 100 can be the center line of the end face 100a, or the center line of the support 240, etc. When the display device 110 displays an image, the light emitted by the display device 110 converges to the user's left eye through the optical device 211. When the display device 120 displays an image, the light emitted by the display device 120 converges to the user's right eye through the optical device 221. In some embodiments, the VR head-mounted display device 100 may further include optical elements 212 and 222. Optical elements 212 and 211 form a lens assembly, and optical elements 222 and 221 form a lens assembly (which can be understood as an optical module). The lens assembly may include at least one optical element, and one or more optical elements in the lens assembly can be adjusted to change the optical power of the lens assembly. For example, the position of one or more optical elements in the lens assembly can be moved away from or closer to the display device to change the optical power.

[0076] It should be noted that, Figure 2A or Figure 2B The VR headset display device 100 shown is merely a logical illustration. In specific implementations, the number of optical devices and / or display devices can be flexibly set according to different needs. For example, in some embodiments, display device 110 and display device 120 can be two independent display devices, or two display areas on the same display device. In some embodiments, display device 110 and display device 120 can be displays, such as liquid crystal displays, light-emitting diode (LED) displays, or other types of display devices; this application does not limit the specific type of display device. Figure 2B In the illustrated embodiment, the optical module may include two devices, optical device 222 and optical device 221. In other embodiments, the optical module may include one or more optical devices, or three or more optical devices. The optical device may be one or more of the following: a reflector, a transmissive mirror, or an optical waveguide. It may also improve the field of view; for example, the optical device may be a lens, and multiple lenses may form a lens group. Exemplarily, the optical device may be a Fresnel lens and / or an aspherical lens, etc., but this application does not limit the scope of the embodiments.

[0077] Figure 3 This is a schematic diagram of a head-mounted display device based on a pancake-folded optical path in related technologies. Pancake-folded optical paths are commonly used in head-mounted display devices to redirect the optical path. The following explanation uses either the (left eye) optical display module 210 or the (right eye) optical display module 220 as examples of pancake-folded optical paths. The optical module can include two devices: optical element 222 and optical element 221. Figure 3 As shown, a head-mounted display device based on a pancake folded optical path may include a pancake lens group 10 (i.e., optical module 130 or optical module 140) and a display component 20 (i.e., display device 110 or display device 120). The pancake lens group 10 may be composed of multiple lenses (lenses can be understood as optical devices), and the pancake lens group 10 may contain multiple layers of coating. Specifically, the pancake lens group 10 may include a beam splitter (BS) 101, a first quarter-wave plate (QWP) 102 (hereinafter referred to as QWP1), and a polarization reflector (PR) 103. The display component may include a display screen 201, a polarizer (P) 202, and a second quarter-wave plate (QWP) 203 (hereinafter referred to as QWP2).

[0078] Figure 4 This is a schematic diagram of the light polarization of the Pancake lens group in related technologies, such as... Figure 4As shown, the light from the display screen is folded in the Pancake lens group 10 and finally exits into the user's eye. The multilayer coating in the Pancake lens group 10 can fold the light between the layers of the film. Specifically, the light emitted by the display screen 201 is modulated into linearly polarized light after passing through the polarizer 202, and without loss of generality, the polarization direction can be assumed to be along the y-axis direction. After passing through the QWP2 (203), the light becomes right-handed polarized light, and the fast axis direction of the QWP2 (203) is 45° to the y-axis. Thereafter, the light reaches the half-reflective half-transmissive film 101, a portion of the light is reflected, and another portion of the light is transmitted and passes through the QWP1 (102) to reach the polarized reflective film 103. The fast axis direction of the QWP1 (102) is the same as that of the QWP2 (203), at this time, the light is modulated into linearly polarized light again, and the polarization direction is along the x-axis direction. The polarized reflective film 103 can reflect the polarized light in the x-axis direction and transmit the polarized light in the y-axis direction. Therefore, the light is reflected and passes through the QWP1 (102) to reach the half-reflective half-transmissive film 101, at this time, the light is right-handed polarized light. As before, a portion of the light is transmitted and another portion of the light is reflected. The polarization state of the reflected light becomes left-handed polarized light, and after passing through the QWP1 (102) again, the light is modulated into linearly polarized light again, and the polarization direction is along the y-axis direction. According to the characteristics of the polarized reflective film 103, the light will be transmitted through the polarized reflective film 103 and finally enter the human eye.

[0079] The head-mounted display device has an eye movement tracking function, and the visual axis of the eyeball of the user is obtained through eye movement tracking calibration. The eye movement tracking refers to tracking the movement of the eyeball by measuring the position of the fixation point of the eye or the movement of the eyeball relative to the head, or in other words, the eye movement tracking refers to tracking the direction of the visual line of the eyes of the user.

[0080] It can be understood that more devices can also be included in the VR head-mounted display device 100. For example, as shown in Figure 5 , a structural schematic diagram of a VR head-mounted display device 100 provided by an embodiment of the present application is shown. As Figure 5 shown, the VR head-mounted display device 100 can include a processor 401, a memory 402, a sensor module 403 (which can be used to obtain the posture of the user), a microphone 404, a key 405, an input / output interface 406, a communication module 407, a camera 408, a battery 409, an optical display module 410, and an eye movement tracking module 412, etc.

[0081] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the VR head-mounted display device 100. In other embodiments of the present application, the VR head-mounted display device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0082] The processor 401 is generally configured to control overall operations of the VR head-mounted display device 100, and can include one or more processing units. For example, the processor 401 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a video processing unit (VPU) controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices or integrated in one or more processors. In some embodiments, the processor 401 can also be a microcontroller unit (MCU).

[0083] The processor 401 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 401 is a cache memory. The memory can store instructions or data that have just been used or recycled by the processor 401. If the processor 401 needs to use the instructions or data again, it can directly call them from the memory. This avoids repeated access and reduces the waiting time of the processor 401, thereby improving the efficiency of the system.

[0084] In some embodiments of the present application, the processor 401 can obtain the human eye spot image sent by the camera module in the eye movement tracking module 412, and can also obtain the position of the user's eye, and then calculate the user's gaze point.

[0085] In some embodiments, the processor 401 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, a serial peripheral interface (SPI) interface, etc.

[0086] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 401 can include multiple sets of I2C buses.

[0087] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is usually used to connect the processor 401 and the communication module 407. For example, the processor 401 communicates with the Bluetooth module in the communication module 407 through the UART interface to realize the Bluetooth function.

[0088] The MIPI interface can be used to connect the processor 401 and the display screen in the optical display module 410, the camera 180, and other peripheral devices.

[0089] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 401 and the camera 180, the display screen in the optical display module 410, the communication module 407, the sensor module 403, the microphone 404, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc. In some embodiments, the camera 180 can capture an image including a real object, the processor 401 can fuse the image captured by the camera with a virtual object, and realize the fused image through the optical display module 410.

[0090] The USB interface is an interface conforming to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface can be used to connect a charger to charge the VR head-mounted display device 100, and can also be used to transmit data between the VR head-mounted display device 100 and a peripheral device. It can also be used to connect earphones to play audio through the earphones. The interface can also be used to connect other electronic devices, such as a mobile phone, etc. The USB interface can be USB 3.0, which is used to transmit display port (DP) signals and can transmit high-speed audio and video data.

[0091] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the VR head-mounted display device 100. In some other embodiments of the present application, the VR head-mounted display device 100 can also use different interface connection methods or combinations of multiple interface connection methods in the above embodiments.

[0092] In addition, the VR head-mounted display device 100 can include a wireless communication function, for example, the VR head-mounted display device 100 can receive images from other electronic devices (such as a VR host) for display. The communication module 407 can include a wireless communication module and a mobile communication module. The wireless communication function can be realized through an antenna (not shown), a mobile communication module (not shown), a modem processor (not shown), and a baseband processor (not shown), etc. The antenna is used to transmit and receive electromagnetic wave signals. The VR head-mounted display device 100 can include multiple antennas, each of which can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0093] The mobile communication module can provide a solution for wireless communication including 2th generation (2G) network / 3th generation (3G) network / 4th generation (4G) network / 5th generation (5G) network, etc. applied on the VR head-mounted display device 100. The mobile communication module can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module can receive electromagnetic waves by an antenna, and perform filtering, amplification, etc. on the received electromagnetic waves, and transmit to a modem processor for demodulation. The mobile communication module can also amplify the signal modulated by the modem processor, and radiate as electromagnetic waves by the antenna. In some embodiments, at least part of the function modules of the mobile communication module can be arranged in the processor 401. In some embodiments, at least part of the function modules of the mobile communication module can be arranged in the same device as at least part of the modules of the processor 401.

[0094] The modem processor can include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is configured to demodulate a received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to a loudspeaker, etc.), or displays an image or video through the display screen in the optical display module 410. In some embodiments, the modem processor can be a separate device. In other embodiments, the modem processor can be independent of the processor 401, and arranged in the same device as the mobile communication module or other function modules.

[0095] The wireless communication module can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. applied on the VR head-mounted display device 100. The wireless communication module can be one or more devices integrated with at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 401. The wireless communication module can also receive signals to be sent from the processor 401, frequency modulate them, amplify them, and convert them into electromagnetic wave radiation via an antenna.

[0096] In some embodiments, the antennas and mobile communication module of the VR head-mounted display device 100 are coupled so that the VR head-mounted display device 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0097] The VR head-mounted display device 100 implements a display function through a GPU, an optical display module 410, and an application processor, etc. The GPU is a microprocessor for image processing, connected to the optical display module 410 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 401 can include one or more GPUs that execute program instructions to generate or change display information.

[0098] The memory 402 can be used to store computer-executable program codes including instructions. The processor 401 performs various functional applications and data processing of the VR head-mounted display device 100 by running the instructions stored in the memory 402. The memory 402 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like. The data storage area can store data (such as audio data, a phone book, etc.) created during use of the VR head-mounted display device 100, and the like. In addition, the memory 402 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.

[0099] The VR head-mounted display device 100 can realize an audio function through an audio module, a speaker, a microphone 404, a headset interface, an application processor, and the like. For example, music playing, recording, and the like. The audio module is used to convert digital audio information into an analog audio signal output, and is also used to convert an analog audio input into a digital audio signal. The audio module can also be used to encode and decode an audio signal. In some embodiments, the audio module can be arranged in the processor 401, or part of the function modules of the audio module can be arranged in the processor 401. The speaker, also known as a “loudspeaker”, is used to convert an audio electrical signal into a sound signal. The VR head-mounted display device 100 can listen to music or listen to a hands-free call through the speaker.

[0100] The microphone 404, also known as a “microphone”, “sound receiver”, is used to convert a sound signal into an electrical signal. The VR head-mounted display device 100 can be provided with at least one microphone 404. In other embodiments, the VR head-mounted display device 100 can be provided with two microphones 404, in addition to collecting sound signals, noise reduction functions can also be realized. In other embodiments, the VR head-mounted display device 100 can also be provided with three, four or more microphones 404, to realize the collection of sound signals, noise reduction, and can also identify the source of the sound, realize the directional recording function, and the like.

[0101] The headset interface is used to connect a wired earphone. The headset interface can be a USB interface, or a 3.5 millimeter (mm) open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0102] In some embodiments, the VR head-mounted display device 100 can include one or more buttons 405, which can control the VR head-mounted display device 100 and provide the user with access to functions on the VR head-mounted display device 100. The buttons 405 can be in the form of push buttons, switches, dials, and touch or near touch sensing devices (e.g., touch sensors). Specifically, for example, a user can turn on the optical display module 410 of the VR head-mounted display device 100 by pressing a button. The buttons 405 can include a power button, a volume button, and the like. The buttons 405 can be mechanical buttons. They can also be touch buttons. The VR head-mounted display device 100 can receive button inputs and generate key signal inputs related to user settings and function controls of the VR head-mounted display device 100.

[0103] In some embodiments, the VR head-mounted display device 100 can include an input / output interface 406, which can connect other devices to the VR head-mounted display device 100 through suitable components. The components can include, for example, audio / video jacks, data connectors, and the like.

[0104] The optical display module 410 is configured to present images to the user under the control of the processor 401. The optical display module 410 can convert real pixel image display into virtual image display through one or more optical devices such as mirrors, transmissive lenses, or optical waveguides, to achieve virtual interactive experience or virtual and real combined interactive experience. For example, the optical display module 410 receives image data information sent by the processor 401 and presents corresponding images to the user. In some embodiments, the optical display module 410 can include the optical display module 210 and the optical display module 220.

[0105] In embodiments of the present application, the VR head-mounted display device 100 further includes an eye movement tracking module 412. The eye movement tracking module 412 is configured to track the movement of the human eye and determine the gaze point of the human eye. For example, the pupil position can be located through image processing technology, the pupil center coordinates can be obtained, and the gaze point of the human eye can be calculated. In some embodiments, the eye movement tracking system can determine the gaze point position (or determine the visual line direction) of the user through video eye map method or photodiode response method or pupil corneal reflection method, so as to realize the eye movement tracking of the user.

[0106] It should be noted that in some embodiments of the present specification, a respective eye movement tracking module corresponding to each eye of the user can be provided to synchronously or asynchronously track the eye movement of the two eyes. In some other embodiments of the present specification, an eye movement tracking module can be provided only near a single eye of the user, and the line-of-sight direction of the corresponding eye can be obtained through the eye movement tracking module. According to the relationship between the fixation points of the two eyes (for example, when the user observes an object through the two eyes, the fixation points of the two eyes are generally close or the same), the line-of-sight direction or the fixation point position of the other eye of the user can be determined in combination with the distance between the two eyes of the user.

[0107] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the VR head-mounted display device 100. In some other embodiments of the present application, the VR head-mounted display device 100 can include more or fewer components, or combine some components, or split some components, or different component arrangements, which are not limited in the embodiments of the present application. Figure 2A

[0108] It can be understood that the VR head-mounted display device 100 is an example of an electronic device in the embodiments of the present application, and the electronic device in the embodiments of the present application can also have many other forms, such as an AR wearable device, an MR wearable device, a vehicle-mounted eye movement tracking display device, a smart mobile device, a large-screen display, a smart car, a computer monitor, etc., which are not limited herein.

[0109] Figure 6 FIG. 6 is an example software structure block diagram of the VR head-mounted display device 100 in the embodiments of the present application.

[0110] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, from top to bottom, application program layer 501, application program framework layer 502, runtime 503 and system library 504, and kernel layer 505.

[0111] The application program layer 501 can include a series of application program packages.

[0112] As shown in FIG. 5, the application program package can include camera 501A, calendar 501B, map 501C, WLAN 501D, music 501E, short message 501F, gallery 501G, call 501H, navigation 501I, Bluetooth 501J, video 501K, etc. Application programs (also referred to as applications). Figure 6

[0113] ​​The application framework layer 502 provides an application programming interface (API) and a programming framework for applications of the application layer. The application framework layer includes some pre-defined functions.

[0114] As shown in Figure 6 The application framework layer 502 can include a window manager 5021, a content provider 5022, a telephony manager 5023, a resource manager 5024, a notification manager 5025, a view system 5026, and the like.

[0115] The window manager 5021 is used to manage window programs. The window manager 5021 can acquire a display screen size, determine whether there is a status bar, lock a screen, and the like.

[0116] The content provider 5022 is used to store and acquire data, and make the data accessible to applications. The data can include videos, images, audios, dialed and received calls, browsing history and bookmarks, a phone book, and the like.

[0117] The telephony manager 5023 is used to provide a communication function of the VR head-mounted display device 100. For example, management of a call state (including call connection, call hang-up, and the like).

[0118] The resource manager 5024 provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, and the like.

[0119] The notification manager 5025 enables applications to display notification information in a status bar. The notification manager can be used to convey a message of an informing type, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform of a download completion, a message reminder, and the like. The notification manager can also be a notification that appears in a system top status bar in a form of a graph or a scroll bar text, for example, a notification of an application running in the background, and can also be a notification that appears on a screen in a form of a dialogue interface. For example, a text information is prompted in a status bar, a prompt sound is emitted, an electronic device is vibrated, a light flashes, and the like.

[0120] The view system 5026 includes visual controls, such as a control for displaying text, a control for displaying pictures, and the like. The view system can be used to build an application. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0121] The runtime 503 includes a core library and a virtual machine. The runtime is responsible for scheduling and management of the system.

[0122] The core library includes two parts: one part is a function function called by a programming language (for example, the jave language), and the other part is a core library of a system.

[0123] The application program layer 501 and the application program framework layer 502 run in a virtual machine. The virtual machine executes the programming files (for example, jave files) of the application program layer 501 and the application program framework layer 502 into binary files. The virtual machine is used to perform functions such as management of an object life cycle, stack management, thread management, management of security and exceptions, and garbage collection.

[0124] The system library 504 can include a plurality of function modules. For example: a surface manager 5041, a three-dimensional graphics processing library 5042 (for example, OpenGL ES), a two-dimensional graphics engine 5043 (for example, SGL), media libraries 5044, and the like.

[0125] The surface manager 5041 is used to manage a display subsystem, and provides fusion of two-dimensional (2-Dimensional, 2D) and three-dimensional (3-Dimensional, 3D) layers for a plurality of application programs.

[0126] The media libraries 5044 support playback and recording of a plurality of commonly used audio, video formats, and static image files. The media libraries can support a plurality of audio and video coding formats, for example: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, and the like.

[0127] The three-dimensional graphics processing library 5042 is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, and the like.

[0128] The two-dimensional graphics engine 5043 is a drawing engine for 2D drawing.

[0129] The kernel layer 505 is a layer between hardware and software. The kernel layer 505 at least includes a display driver 5051, a camera driver 5052, an audio driver 5053, and a sensor driver 5054.

[0130] In some embodiments of the present application, the application program framework layer 502 can further include an eye movement tracking function module 5027, which is used to match a light spot in an eye image acquired by the camera driver 5052 with a light source, calculate a line of sight direction of a user, and thus determine a fixation point of the user. In some other embodiments of the present application, the eye movement tracking function module 5027 can also be located in the application program layer 501, the system library 504, or the kernel layer 505, which is not limited herein.

[0131] Figure 7This is a schematic diagram of eye tracking based on corneal reflection in related technologies, such as... Figure 7 As shown, the line connecting the fovea 30a of the user's eyeball 30 and the center of the cornea 30b is the visual axis 30c of the eyeball 30, i.e., the direction of the line of sight. The line connecting the center of the pupil 30d and the center of the cornea 30b is the optical axis 30e of the eyeball 30, i.e., the optical axis of the eyeball 30. There is an angle between the optical axis 30e and the visual axis 30c of the eyeball 30, and some detection results indicate that this angle is 5°. Figure 7 The illustrated eye-tracking architecture based on corneal reflection may include several near-infrared light-emitting diodes (LEDs) D1 and several near-infrared cameras D2. The near-infrared cameras can capture the light spots reflected from the near-infrared LEDs D1 by the cornea 30b of the eyeball, and simultaneously capture the pupil of the eyeball 30, thereby locating the pupil center 30d. According to the law of reflection, the normal determined by the light reflected from the cornea passes through the corneal center 30b. Based on the positions of all the LED light spots 30g, the position of the corneal center 30b can be calculated. Combined with the position of the image 30f at the pupil center, the optical axis 30e of the eyeball 30 can be obtained. In some embodiments, the visual axis 30c of the eyeball 30 is obtained through calibration.

[0132] Under the aforementioned Pancake architecture, the system structure is relatively compact, and the distance between the user's eyeball 30 and the Pancake lens group 10 is relatively short. Therefore, if a near-infrared camera D2 is installed outside the Pancake lens group 10, the tilt angle is too large, and the eyeball image captured cannot meet the algorithm requirements.

[0133] To overcome the above technical problems, an embodiment of the present application provides an eye movement tracking device, which can include an optical display module, the optical display module including a display device and an optical module. The optical module includes a first optical device, and the first light emitted by the display device is incident into the user's eye after passing through the optical module; and an eye movement tracking module, including: at least one light source (for example, a near-infrared LED D1), configured to emit second light to the user's eye, at least part of the second light being reflected after irradiating the eye; a camera module (for example, a near-infrared camera D2), configured to acquire at least part of the second light reflected after the at least one light source irradiates the eye; wherein the camera module is located between the first optical device and the display device. It needs to be explained that the camera module is located between the first optical device and the display device, and the nearest lens in the first optical device to the display device has a columnar space between the display device. In an embodiment, the camera module can be located in the columnar space. In another embodiment, the camera module can not be limited to being arranged in the columnar space, but can also be arranged at a point (point B) extending outward from the columnar space along the vertical direction of the center axis of the lens barrel at a certain distance from any point (point A) in the columnar space.

[0134] In an embodiment, the optical module only includes the first optical device, that is, a single-lens lens barrel is provided. At least part of the second light is reflected by the user's glasses and is acquired by the camera module after passing through the first optical device. In this embodiment, the focal length of the optical module is variable, specifically, the distance between the first optical device and the display device is adjustable, and the first optical device is a lens. In other words, the nearsightedness of the user can be corrected by adjusting the distance between the lens and the display screen 201.

[0135] The eye movement tracking device provided by the embodiment of the present application can further include a focal length measuring unit, which is configured to trigger focal length detection to determine the current focal length information after the focal length of the optical module is adjusted. In an embodiment, the focal length measuring unit can include a Hall sensor, a grating ruler or a sliding rheostat. Further, the focal length measuring unit can detect the current distance between the lens and the display screen 201 after the distance between the lens and the display device is changed.

[0136] In some embodiments, the myopia correction mode can further include an external myopia correction lens, and in some embodiments, the optical module further includes an external lens interface for mounting the myopia correction lens on the lens barrel by magnetic attraction or buckling. In the scenario of myopia correction by an external myopia correction lens, the focal length measurement unit can include a storage chip and an identity chip. In one implementation, the storage chip can store the correspondence between the identity information and the optical power information of each myopia correction lens, and the storage chip can be arranged in the optical module. The identity chip stores the identity information (such as an identification number) of the myopia correction lens, and the identity chip can be arranged on the myopia correction lens. When the myopia correction lens is externally connected to the lens barrel of the optical module, the optical power information (i.e., focal length information) of the myopia correction lens can be matched in the storage chip based on the identity information.

[0137] In the above-mentioned myopia correction scenario of the external myopia correction lens, the trigger condition for triggering the focal length measurement unit to detect the focal length is as follows:

[0138] The external lens interface can include a contact point, which can be used to detect the installation state of the myopia correction lens when the myopia correction lens is externally connected to the lens barrel. When it is determined that the myopia correction lens is installed on the lens barrel, the identity information of the myopia correction lens can be obtained from the myopia correction chip, and the corresponding optical power information can be matched from the storage chip based on the identity information.

[0139] It should be noted that when the focal length of the optical module changes, the gaze point direction of the user's eye does not change, and the gaze point of the user's eye obtained by the eye movement tracking device does not change. In some embodiments, when the focal length of the optical module changes, a light path table suitable for the current focal length information is determined. The eye movement tracking device further includes a processor that receives the results obtained by the camera module and obtains the gaze point of the user's eye.

[0140] In another implementation, the optical display module further includes a second optical device, and at least part of the second light passes through the second optical device before being obtained by the camera module. The second optical device is located between the first optical device and the display device, or the second optical device is located on the side of the first optical device away from the display device. In one implementation, the second optical device can be a lens, and the first optical device and the second optical device can constitute a mirror group, which can be a Pancake mirror group, but is not limited to a Pancake mirror group.

[0141] Figure 8 A schematic diagram of the reflection of corneal light through the Pancake mirror group and the near-infrared camera lens in the related art is shown in FIG. 6. Figure 8As shown, the light reflected by the cornea and the light refracted by the pupil both pass through the Pancake lens group 10 and the near-infrared camera D2 lens after refraction, and then hit the sensor of the near-infrared camera D2. Therefore, in order to calculate the positions of the corneal center of curvature 30b and the pupil center 30d, it is necessary to calibrate the corresponding relationship between the pixel points on the sensor of the near-infrared camera D2 and the chief rays 41 on the eyeball side, that is, to determine the optical path table.

[0142] Figure 9 A schematic diagram of the optical path table calibration of the eye movement tracking in the related art is shown in FIG. 6. Figure 9 As shown in the calibration scheme in the related art, a calibration board 51 is placed in front of the lens barrel, an image of the calibration board 51 is captured by the near-infrared camera D2, and the corresponding relationship between the feature points on the calibration board 51 and the pixels on the sensor of the near-infrared camera D2 is obtained through feature point detection; the calibration board 51 is translated along the optical axis of the lens barrel by a distance, an image of the calibration board 51 is captured by the near-infrared camera D2, and the corresponding relationship between the feature points on the calibration board 51 and the pixels on the sensor of the near-infrared camera D2 is calculated; through interpolation calculation, each pixel on the sensor of the near-infrared camera D2 can find two conjugate points on the calibration board 51 at two different positions, and the line connecting the two conjugate points is the chief ray 41 on the eyeball side, that is, the calibration optical path table of the eye movement tracking system is obtained.

[0143] In the related art, in order to make the head-mounted display device suitable for myopic people, some head-mounted display devices also provide myopia correction functions. Figure 10a A schematic diagram of a myopia correction method in the related art is shown in FIG. 7. Figure 10a As shown, a myopia correction lens can be added in front of the Pancake lens barrel, and the installation method can adopt magnetic attraction or buckle, as shown in FIG. 8. Figure 10a As shown, without the myopia correction lens, the virtual image position is at position 1, and when the myopia correction lens is added, the virtual image is translated from position 1 to position 2 due to the change in optical power, thereby achieving the purpose of correcting the user's myopia. Figure 10b A schematic diagram of another myopia correction method in the related art is shown in FIG. 9. Figure 10b As shown, one or more lenses or display screens 201 in the lens group can be translated in the direction of the optical axis of the lens barrel, and the following figure shows a myopia correction scheme of translating one lens in the Pancake lens group. When the P2 lens is at the solid line position, the virtual image is at position 1, and when the P2 lens is translated to the dashed line position, the virtual image is translated from position 1 to position 2 due to the change in optical power of the Pancake lens group, thereby achieving the purpose of correcting the user's myopia. It should be noted that in the case of a single lens optical module and a fixed user eye position, the distance between the single lens and the display screen 201 is adjusted to translate the virtual image from position 1 to position 2 to achieve the purpose of correcting the user's myopia.

[0144] When different nearsighted users use the same head-mounted display device with myopia correction, the user needs to undergo myopia correction (shifting a lens in the pancake lens group or adding a myopia correction lens of the corresponding power in front of the lens barrel) to suit the current user's myopia situation. However, myopia correction will change the optical path table of the head-mounted display device's eye-tracking system. If calibration is only performed at a certain myopia correction position, the optical path table obtained during eye tracking will have a large error.

[0145] To overcome the aforementioned technical problems, this application provides an optical path table calibration method. This method acquires the current user's myopia correction parameters from a head-mounted display device, then performs optical path table interpolation based on these parameters to obtain a calibrated optical path table. The user's gaze direction is then calculated using the calibrated optical path table, resulting in a more accurate understanding of the current user's gaze direction. By reducing the error caused by myopia correction on the optical path table, the accuracy of eye tracking is improved.

[0146] To implement the above-described optical path table calibration method, this application provides a system architecture in another embodiment, through which corresponding operations are performed by corresponding components of the system architecture to implement the above-described optical path table calibration method. Figure 11 A schematic diagram of a system architecture provided for another embodiment of this application, as shown below. Figure 11 As shown, the system architecture may include a parameter acquisition device 71 and a processor 72. The parameter acquisition device 71 can acquire the myopia correction parameters of the current user of the head-mounted display device. The processor 72 can perform optical path table interpolation based on the acquired myopia correction parameters to obtain a corrected optical path table, and calculate the user's gaze direction using the corrected optical path table, thereby obtaining a more accurate gaze direction for the current user. In one embodiment, the processor 72 can be a microcontroller unit (MCU).

[0147] Figure 12 This is a schematic flowchart of an optical path table calibration method provided in one embodiment of this application, as shown below. Figure 12 As shown, the method may include the following steps:

[0148] Step 801: The processor obtains a set of optical path tables, which includes optical path tables corresponding to at least two myopia correction positions.

[0149] Step 802: The processor obtains the current myopia correction parameters after the myopia correction operation.

[0150] Step 803: The processor interpolates the light path table set according to the myopia correction parameter to obtain a corrected target light path table, and uses the target light path table to solve the line-of-sight direction to obtain the line-of-sight direction of the current user.

[0151] In the implementation of step 801, before the light path table is corrected based on the myopia correction operation, the light path table set can be obtained. Figure 13 The light path table calibration diagram provided for another embodiment of the present application is shown in Figure 13 The light path table set can be obtained by acquiring at least two light path tables corresponding to the light path table calibration at the myopia correction positions.

[0152] The light path table calibration operation can be a pre-completed operation. In an embodiment, all light path tables (light path table set) obtained after pre-calibration can be saved in a corresponding memory, and then the processor 72 can obtain the light path table from the memory. That is, in an embodiment, the "processor obtains the light path table set" in step 801 can be to obtain the light path table set from the above-mentioned memory.

[0153] In the above-mentioned pre-executed light path table calibration process, the operation of calibrating the light path table at a myopia correction position to obtain the corresponding light path table includes: at the current myopia correction position, the near-infrared camera C1 respectively captures the calibration board at two different positions (the calibration board at position P1 and position P2), and finds the conjugate points on the calibration board at the two positions corresponding to each pixel point on the near-infrared camera C1. Specifically, when the calibration board is at position P1, the near-infrared camera C1 captures the calibration board to find the conjugate points on the calibration board corresponding to each pixel point on the near-infrared camera C1. Taking one of the pixel points as an example, as shown in Figure 13 The pixel point (O point) on the near-infrared camera C1 sensor corresponds to the conjugate point A on the calibration board at position P1. In this way, after obtaining the conjugate point on the calibration board at position P1 corresponding to each pixel point, the calibration board is translated to position P2, and the conjugate point on the calibration board at position P2 corresponding to each pixel point on the near-infrared camera C1 is found in the above-mentioned manner. Taking one of the pixel points as an example, after the calibration board is translated to position P2, the conjugate point corresponding to the O point on the calibration board at position P2 is point B. In some embodiments, connecting the conjugate points A and B (vector AB) can obtain the position and direction of the chief ray 41 corresponding to the pixel point (O point). In this way, the chief ray 41 and the direction corresponding to each pixel point are obtained, and then the light path table corresponding to the myopia correction position can be obtained, which can be shown in Table 1.

[0154] Table 1

[0155] u / pixel v / pixel x A / mm]] y A / mm]] z A / mm]] x B / mm]] y B / mm]] z B / mm]] 1 1 1 2 … … … … … … … …

[0156] In an embodiment, the optical path table shown in Table 1 can be an Mx8 matrix, where M represents the number of pixels of the near-infrared camera, the first and second columns (u / pixel column and v / pixel column) in the table are camera pixel coordinates (which can be understood as the serial number of the pixel), and the third to eighth columns (X A column, Y A column, Z A column, X B column, Y B column, Z B column) are the three-dimensional coordinates of the conjugate point A and the conjugate point B corresponding to the pixel, in mm. Among them, X A , Y A , and Z A are the coordinates of point A on the three axes, and X B column, Y B column, and Z B column are the coordinates of point B on the three axes. Through the optical path table, the position and direction of the chief ray 41 corresponding to the camera pixel in the space close to the eyeball can be obtained.

[0157] After one myopia correction of the head-mounted display device, based on the above operation, the optical path table is calibrated again at the current myopia correction position to obtain the optical path table corresponding to the current myopia correction position.

[0158] In an embodiment, the above optical path table calibration operation can be performed based on the number of set myopia correction positions to obtain a corresponding number of optical path tables. For example, the myopia correction range of the myopia correction system of the head-mounted display device is -1D to -7D, if the optical path table calibration is set to be performed at two myopia correction positions, and the two myopia correction positions are -1D correction position and -7D correction position, respectively, then the optical path table can be calibrated at the myopia correction parameter of -1D and -7D, respectively, in the calibration process, to obtain two different optical path tables, which are denoted as optical path table 1D and optical path table 7D, respectively. The obtained optical path table 1D and optical path table 7D are saved in the corresponding memory of the head-mounted display device for the processor 72 to obtain, wherein the memory for storing the optical path table can be the built-in memory of the head-mounted display device, or an external memory or a cloud storage space, which is not limited here.

[0159] In a specific implementation of step 802, in one myopia correction mode, the position of the virtual image distance can be adjusted by manually or electrically translating the lens in the mirror group of the head-mounted display device, so as to achieve the purpose of myopia correction. The mirror group in the head-mounted display device can be various mirror groups with light path folding function, which is not limited herein. In one embodiment, the mirror group in the head-mounted display device can be a Pancake mirror group, and the lens in the Pancake mirror group can be moved to adjust the position of the virtual image distance, so as to achieve the purpose of myopia correction.

[0160] During the above myopia correction operation of the user, the position information of the translated lens can be detected by the parameter acquisition device 71, and the obtained position information is sent to the processor 72.

[0161] The parameter acquisition device 71 can obtain the position information of the translated lens based on the adaptability of the myopia correction mode of the head-mounted display device.

[0162] In one embodiment, in the scenario of adjusting the position of the virtual image distance by translating the lens in the mirror group to achieve myopia correction, the processor 72 obtains the myopia correction parameter corresponding to the current myopia correction operation in step 802 can include the following steps:

[0163] Step 802a: After the myopia correction operation, the parameter acquisition device obtains the position sensing information of the translated lens, and sends the position sensing information of the translated lens to the processor;

[0164] Step 802b: The processor calculates the corresponding myopia correction parameter according to the received position sensing information.

[0165] In a specific implementation of step 802a, the position sensing information of the translated lens can be obtained by a position sensor in the mirror group, which includes but is not limited to a Hall sensor, a grating ruler, a sliding rheostat, etc. In one embodiment, the magnetic flux information of the translated lens can be detected by a Hall sensor. Figure 14 The myopia correction parameter acquisition diagram provided by another embodiment of the present application is as follows: Figure 14As shown, the Hall device 1001 can be mounted on the barrel structure of the head-mounted display device, and the Hall device 1001 does not translate with the P2 lens. In some embodiments, a magnet 1002 can be mounted on the P2 lens, and the mounting position of the magnet 1002 is opposite to the sensing surface of the Hall device 1001. When the P2 lens translates, the distance between the magnet 1002 and the Hall device 1001 changes, and the magnetic flux received by the Hall device 1001 also changes. The position sensing information of the translated lens acquired by the parameter acquisition device can include the magnetic flux received by the Hall device 1001 after the P2 lens translation is completed. In some embodiments, the Hall device 1001 can send the acquired magnetic flux to the processor 72.

[0166] In the implementation of step 802b, after the user adjusts the position of the virtual image by translating the P2 lens, different positions of the virtual image correspond to different myopia correction parameters. After the Hall device 1001 detects and sends the corresponding magnetic flux after the P2 lens translation, the processor 72 can calculate the corresponding myopia correction parameter according to the magnetic flux provided by the Hall device 1001. For example, when the virtual image distance (the distance from the virtual image to the user's eyeball) is 1 m, the myopia correction parameter is -1 D (the user with -1 D myopia can see the far point at a distance of 1 m); when the virtual image distance is 0.2 m, the myopia correction parameter is -5 D. Therefore, the relationship between the magnetic flux received by the Hall device and the myopia correction parameter can be obtained by calibration (this calibration process can be completed before the parameter acquisition device 71 or the head-mounted display device is shipped), that is, the processor 72 can calculate the myopia correction parameter set by the user in the myopia correction operation by detecting the magnetic flux provided by the Hall device 1001. It should be noted that generally, the degree is used to represent the diopter (or optical power) of the glasses, and the degree is equal to the value of the diopter D multiplied by one hundred, for example, -1 D is equal to one hundred degrees of myopia glasses (concave lens).

[0167] In another myopia correction mode, a myopia correction lens can be added in front of the barrel of the head-mounted display device to adjust the position of the virtual image distance, so as to achieve the purpose of myopia correction. In some embodiments, the myopia correction lens can be mounted on the barrel by magnetic attraction or buckling, Figure 15 A myopia correction lens mounting schematic diagram is provided for another embodiment of the present application, as shown in Figure 15 As shown, a magnet 1102 and a storage chip 1103 can be provided on the barrel 1101, and the myopia correction lens is mounted on the barrel 1101 in a magnetic attraction manner, and the optical power information of the myopia correction lens can be stored in the storage chip 1103.

[0168] In one implementation, in the scenario described above where myopia correction is achieved by adjusting the position of the virtual image distance through the addition of myopia correction lenses, the step 802 in which the processor 72 obtains the myopia correction parameters corresponding to the current myopia correction operation may include the following steps:

[0169] Step 802g: Processor 72 obtains the identity information of the myopia correction lens in the current myopia correction operation.

[0170] Step 802h: The processor 72 determines the optical power information of the myopia correction lens based on the obtained identity information of the myopia correction lens.

[0171] In a specific implementation of step 802g, during the myopia correction operation, the currently used myopia correction lens can establish a connection with the processor 72 and inform the processor 72 of the identity information of the currently used myopia correction lens through this connection. Then, the processor 72 can obtain the corresponding lens power information from the storage chip 1103 based on the identity information of the currently used myopia correction lens.

[0172] Figure 16 This is a schematic diagram of myopia correction parameter transmission provided in another embodiment of this application, as shown below. Figure 16 As shown, contacts 1201 can be set at corresponding positions on the lens barrel of the head-mounted display device, and each myopia correction lens 1202 is equipped with an identity chip 1203. The identity chip 1203 stores the identity information of the corresponding myopia correction lens 1202, such as the identity identifier of the myopia correction lens 1202. When the myopia correction lens 1202 is installed on the lens barrel, the myopia correction lens 1202 can establish a connection with the processor 72 through the contacts 1201 on the lens barrel. Through the connection, the identity information stored in the identity chip 1203 of the myopia correction lens 1202 is provided to the processor 72. In other words, the processor 72 can obtain the identity information of the currently installed myopia correction lens 1202 through this connection.

[0173] In other embodiments, the myopia correction lens 1202 equipped with the identity chip 1203 can also establish a connection with the processor 72 in other ways to obtain identity information, and is not limited to... Figure 12 The communication connection method provided in the illustrated embodiment, for example, the identity chip 1203 can be a radio frequency chip, which stores the identity information corresponding to the myopia correction lens 1202. The processor 72 can establish a near-field wireless communication connection with the identity chip 1203 and read the identity information stored in the identity chip 1203 through the near-field wireless communication connection. The identity information may include information such as the focal length or diopter of the myopia correction lens 1202. In some embodiments, the identity chip 1203 and / or the storage chip 1103, contact 1201, etc., can be understood as a focal length measurement unit.

[0174] In the implementation of step 802h, after obtaining the identity information of the currently installed myopia correction lens 1202, the processor 72 can obtain the refractive power information of the myopia correction lens 1202 from the storage chip 1103 based on the identity information of the currently installed myopia correction lens 1202. In an embodiment, obtaining the myopia correction parameter can include obtaining the refractive power information of the myopia correction lens 1202.

[0175] In the implementation of step 803, the processor 72 performs optical path table interpolation on the optical path table set according to the myopia correction parameter to obtain a corrected target optical path table, and uses the target optical path table to calculate the line-of-sight direction to obtain the line-of-sight direction of the current user. The interpolation method includes but is not limited to linear interpolation, polynomial interpolation, spline interpolation, etc. Taking linear interpolation as an example, if the user performs a myopia correction operation, and the processor 72 obtains a myopia correction parameter of -4D, and the optical path table set obtained by the processor includes an optical path table 1D (correction parameter is -1D) and an optical path table 7D (correction parameter is -7D), then after the above myopia correction operation (myopia correction parameter is -4D), the target optical path table obtained by linear interpolation is the average of the optical path table 1D and the optical path table 7D. In some embodiments, the line-of-sight direction of the user can be calculated based on the target optical path table obtained by interpolation.

[0176] The above is the corresponding operation of optical path table correction in the process of myopia correction operation, which is used to emit near-infrared light in the process of eye movement tracking. The position of the near-infrared LED D1 does not affect the eye movement tracking result. Thus, the optical path table suitable for the current myopia correction position is obtained, so that the influence of myopia correction on the optical path table can be reduced, and the accuracy of eye movement tracking is improved.

[0177] If the position of the near-infrared LED D1 used to emit near-infrared light in the process of eye movement tracking affects the eye movement tracking result, the influence of the position of the near-infrared LED D1 needs to be considered in the process of optical path table correction.

[0178] In another myopia correction method, adjusting the position of the virtual image distance by adjusting the lenses in the lens group of the head-mounted display device includes adjusting the distance between the lenses in the lens group by rotating the lens barrel, similar to the focusing method of a single-lens reflex lens. Figure 17a The schematic diagram of myopia correction by rotating the lens barrel provided for another embodiment of the present application is shown in FIG. 8B. Figure 17a As shown in FIG. 8B, the distance between the lenses in the lens group of the head-mounted display device can be adjusted by rotating the lens barrel to adjust the position of the virtual image distance, so as to achieve the purpose of myopia correction.

[0179] However, under the operation of rotating the lens barrel as described above, the position of the near-infrared LED D1 in the xoy plane will change at different near vision degrees. If the lens position adjustment along the optical axis direction also drives the LED, the position of the LED in the z-axis direction will also change. As shown in Figure 17a , the lens barrel rotation drives the LED to move from the solid block (position P3) to the dashed block (position P4).

[0180] In another near vision correction method, adjusting the lens in the lens group of the head-mounted display device to adjust the position of the virtual image distance includes adjusting the distance between the lenses in the lens group by extending or retracting the lens barrel. Figure 17b The schematic diagram of near vision correction by extending or retracting the lens barrel provided for another embodiment of the present application is shown in Figure 17b , when the lens barrel is extended or retracted, the distance between the lenses in the lens group of the head-mounted display device can be adjusted to adjust the position of the virtual image distance, thereby achieving the purpose of near vision correction.

[0181] Under the operation of extending or retracting the lens barrel as described above, the position of the near-infrared LED D1 in the xyz space changes. Specifically, as shown in Figure 17b , when the lens barrel is extended or retracted, the near-infrared LED D1 mounted on the lens barrel translates along the z-axis direction, so that the near-infrared LED D1 moves from the solid block (position P5) to the dashed block (position P6).

[0182] In the above-mentioned near vision correction method (the position of the near-infrared LED D1 changes with the near vision correction operation), since the position of the near-infrared LED D1 changes, when the position of the near-infrared LED D1 changes, the eye movement tracking accuracy will decrease, affecting the user experience.

[0183] To overcome the above technical problems, another embodiment of the present application provides another optical path table calibration method. In order to reduce the LED offset caused by near vision correction, the position of the near-infrared LED D1 after near vision correction can be determined while the optical path table is being calibrated, so that the corresponding user gaze calculation operation can be performed in the case of knowing the position of the near-infrared LED D1, thereby improving the accuracy of user gaze calculation.

[0184] Figure 18 The flowchart of the optical path table calibration method provided for another embodiment of the present application is shown in Figure 18 The method can include the following steps:

[0185] Step 1401: The processor acquires a set of optical path tables and a set of positions of near-infrared LEDs, the set of optical path tables including optical path tables corresponding to at least two near vision correction positions, and the set of positions of near-infrared LEDs including position information of the near-infrared LED under at least two near vision correction parameters.

[0186] Step 1402: The processor acquires the myopia correction parameters corresponding to the current myopia correction operation, and then respectively executes step 1403 and step 1404.

[0187] Step 1403: The processor interpolates the position set of the near-infrared LED according to the myopia correction parameters corresponding to the current myopia correction operation, to obtain the position of the near-infrared LED under the myopia correction parameters corresponding to the current myopia correction operation.

[0188] Step 1404: The processor performs optical path table interpolation on the optical path table set according to the myopia correction parameters corresponding to the current myopia correction operation, obtains the corrected target optical path table, and then executes step 1405.

[0189] Step 1405: According to the position of the near-infrared LED under the myopia correction parameters corresponding to the current myopia correction operation and the target optical path table, the line-of-sight direction is calculated to obtain the line-of-sight direction of the current user.

[0190] In the implementation of step 1401, before the optical path table is corrected based on the myopia correction operation, the optical path table set and the position set of the near-infrared LED can be acquired. The acquisition operation of the optical path table set can be the same as or similar to the acquisition operation of the optical path table set provided in the embodiment shown in the above, and will not be described here. Figure 8 The acquisition operation of the optical path table set provided in the embodiment shown in the above, and will not be described here.

[0191] Since the position of the near-infrared LED in space is a known input of the eye movement tracking algorithm, the myopia correction operation (myopia adjustment) will affect the position of the near-infrared LED in the xyz space, so the position of the near-infrared LED can be re-determined after the myopia correction operation. The acquisition method of the position set of the near-infrared LED can include acquiring the positions of the near-infrared LED under at least two myopia correction parameters.

[0192] The calibration operation of the position of the near-infrared LED can be a pre-completed operation. In an implementation, the position of the near-infrared LED obtained after pre-calibration (the position set of the near-infrared LED) can be saved in a corresponding memory, and then the processor 72 can acquire the position set of the near-infrared LED in the memory.

[0193] In the above pre-executed calibration process of the position of the near-infrared LED, the position of the near-infrared LED can be calibrated at at least two myopia correction positions before the device is shipped, for example, the LED positions under-1D and-7D conditions are calibrated before the device is shipped. Specifically, a fixed point (fixed during myopia correction, for example, the center of the lens barrel) on the lens barrel can be taken as the origin, and the coordinate positions of the near-infrared LED under-1D and-7D conditions can be determined respectively.

[0194] The specific implementation of step 1402 can be the same as or similar to the implementation of step 1401. Figure 12 The myopia correction parameter corresponding to the current myopia correction operation provided by the embodiment shown is acquired in the same or similar manner, and details are not repeated here.

[0195] In the specific implementation of step 1403, after obtaining the myopia correction parameter, the processor 72 can perform interpolation calculation on the set of positions of the near-infrared LED obtained in the previous step (step 1401) according to the myopia correction parameter, to determine the current position of the near-infrared LED after the myopia correction operation (i.e., corresponding to the myopia correction parameter). For example, when the user adjusts the myopia to a -4D position, the LED position at the -4D position is obtained by interpolation. It should be noted that the interpolation of the xy coordinates needs to be first converted to polar coordinates with the center of the lens barrel as the origin, and then the angle of the polar coordinates is interpolated, and finally the rectangular coordinates are converted back. The z coordinate can be directly linearly, polynomially, or spline interpolated. Through the above interpolation calculation, the position of the near-infrared LED after the myopia correction operation is obtained.

[0196] The specific implementation of steps 1404 to 1405 can be the same as or similar to the implementation of steps 1402 to 1403. Figure 12 The specific implementation of step 803 in the embodiment shown is similar, and the difference is that the position of the near-infrared LED can be updated according to the current position of the near-infrared LED after the myopia correction operation calculated in step 1403 before the user's line of sight is calculated, and after the position information of the near-infrared LED is updated, the processor 72 performs ray table interpolation on the set of ray tables according to the myopia correction parameter to obtain a target ray table after correction, and uses the target ray table to calculate the direction of the line of sight to obtain the direction of the user's line of sight.

[0197] Figure 19 The structure diagram of the ray table correction device provided by another embodiment of the present application is shown in Figure 19 The device can include a processor 1501 and a memory 1502, the memory 1502 is used to store at least one instruction, the instruction is loaded and executed by the processor 1501 to realize the ray table correction method provided by any embodiment of the present application. In one embodiment, Figure 19 The ray table correction device shown can be a programmable chip of a head-mounted display device.

[0198] Another embodiment of the present application also provides a computer storage medium, which stores a computer program, and the computer program is executed by a processor to realize the ray table correction method provided by any embodiment of the present application.

[0199] It should be noted that the terminal involved in the embodiments of the present application can include, but is not limited to, a personal computer (PC), a personal digital assistant (PDA), a wireless handheld device, a tablet computer, a mobile phone, an MP3 player, an MP4 player, and the like.

[0200] It can be understood that the application can be an application program (nativeApp) installed on the terminal, or can also be a web program (webApp) of a browser on the terminal, and the embodiments of the present application do not limit this.

[0201] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0202] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0203] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0204] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0205] The integrated unit implemented in the form of the software function unit can be stored in a computer readable storage medium. The software function unit is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0206] The above only describes some embodiments of the present application and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to part or all of the technical features. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An eye tracking device, characterized in that, The device comprises: an optical display module, the optical display module comprising a display device and an optical module, the optical module comprising a first optical device, first light emitted by the display device being incident on a user's eye after passing through the optical module; and an eye movement tracking module, comprising: at least one light source configured to emit second light towards the user's eye, at least part of the second light being reflected after irradiating the eye; a camera module configured to acquire at least part of the second light reflected after the at least one light source irradiates the eye; wherein the camera module is located between the first optical device and the display device; at least part of the second light is reflected by the user's eye and, after passing through the first optical device, is acquired by the camera module; a focal length of the optical module is variable; the eye movement tracking device further comprises a focal length measuring unit configured to trigger focal length detection after the focal length of the optical module is adjusted, and to obtain current focal length information; the eye movement tracking device further comprises a processor configured to: obtain an optical path table applicable to the current focal length information when the focal length of the optical module changes, the optical path table changing with the change of the focal length of the optical module; and obtain a line-of-sight direction according to the optical path table. a distance between the first optical device and the display device is adjustable, and the first optical device is a lens.

2. The eye-tracking apparatus of claim 1, wherein the first optical device is a variable focal length lens.

3. The eye-tracking apparatus of claim 1, wherein, the optical module further comprises an external lens interface for mounting a myopia correction lens on a lens barrel by magnetic attraction or buckling.

4. The eye-tracking apparatus of claim 3, wherein when the focal length of the optical module changes, a gaze point direction of the user's eye does not change, and the eye movement tracking device obtains a gaze point of the user's eye that does not change.

5. The eye-tracking apparatus according to any one of claims 1 to 4, characterized in that the processor receives a result acquired by the camera module and obtains the gaze point of the user's eye.

6. The eye-tracking apparatus according to any one of claims 1 to 4, characterized in that the optical display module further comprises a second optical device, and at least part of the second light passes through the second optical device before being acquired by the camera module.

7. The eye-tracking apparatus according to any one of claims 1 to 4, characterized in that the second optical device is located between the first optical device and the display device, or the second optical device is located on a side of the first optical device away from the display device.

8. The eye-tracking apparatus of claim 7, wherein, the device comprises the eye movement tracking device of any one of claims 1-8.

9. A head-mounted display device, comprising: the eye movement tracking device comprises an optical display module and an eye movement tracking module; 10. An eye tracking method for an eye tracking device, characterized by the optical display module comprises a display device and an optical module, and the optical module comprises a first optical device; the eye movement tracking module comprises at least one light source and a camera module; and the method comprises: first light emitted by the display device is incident on a user's eye after passing through the optical module; the at least one light source is configured to emit second light towards the user's eye, at least part of the second light being reflected after irradiating the eye; and at least part of the second light reflected after the at least one light source irradiates the eye passes through the first optical device and is acquired by the camera module; obtaining focal length information of the optical module; obtaining an optical path table applicable to current focal length information when the focal length of the optical module changes; the optical path table changes with the change of the focal length of the optical module; ​ According to the light path table, a gaze direction of the user's eye is obtained.

11. The method of claim 10, wherein, The method further comprises: obtaining a light path table set, the light path table set comprising light path tables corresponding to at least two myopia correction positions; obtaining a current myopia correction parameter after a myopia correction operation; and performing light path table interpolation on the light path table set according to the myopia correction parameter to obtain a target light path table.

12. The method of claim 11, wherein, The method further comprises: obtaining the light path table set from a target memory in which the light path table set is pre-stored.

13. The method according to any one of claims 10 to 12, characterized in that, The optical module has a variable focal length. The eye movement tracking device further comprises a focal length measurement unit configured to trigger focal length detection after the focal length of the optical module is adjusted to obtain current focal length information.

14. The method of any one of claims 10 to 12, wherein, When the focal length of the optical module changes, the gaze point direction of the user's eye remains unchanged, and the gaze point of the user's eye obtained by the eye movement tracking device remains unchanged.

Images