Optical element and preparation method thereof, eye movement tracking device and near-to-eye display equipment

Abstract

The embodiment of the invention provides an optical element, a preparation method thereof, an eye movement tracking device and near-to-eye display equipment. A reference structure is formed on the optical element, the optical element can transmit visible light, and at least part of infrared light reflected by the eyes is reflected to the camera through the optical element, so that an image collected by the camera comprises an eye pattern, and at least part of the infrared light reflected by the eyes irradiates the reference structure; the image collected by the camera further comprises the reference pattern, so that the sight line direction of the eye can be determined according to the eye pattern and the reference pattern. According to the technical scheme of the embodiment of the invention, the mounting flexibility of the infrared light source and the camera is improved, so that the structural design of the near-to-eye display equipment is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of eye tracking technology, and in particular to an optical element and a preparation method thereof, an eye tracking device, and a near-eye display device. Background Art

[0002] Eye tracking devices usually use a light source to illuminate the human eye, and an image acquisition unit captures the light spot and iris image formed by the human eye, thereby calculating the direction of the human eye's line of sight based on the light spot and iris image. When the eye tracking device is applied to a near-eye display device such as augmented reality (AR) glasses, since the positions of the light source and the image acquisition unit relative to the human eye need to meet certain conditions, the light source and the image acquisition unit are usually set on the frame or lens. However, if the light source and the image acquisition unit are both set on the frame, the size of the frame will increase, affecting the wearing experience and being detrimental to the appearance design of the AR glasses. If the light source or the image acquisition unit is set on the lens, it will block the human eye's line of sight and affect the viewing experience of the AR glasses. Summary of the invention

[0003] Embodiments of the present invention provide an optical element and a method for preparing the same, an eye tracking device, and a near-eye display device, which are intended to improve the installation flexibility of an infrared light source and a camera to facilitate the structural design of a near-eye display device.

[0004] In a first aspect, an embodiment of the present invention provides an optical element, wherein a reference structure is formed on the optical element;

[0005] The optical element is capable of transmitting visible light, and the infrared light reflected by the eye is at least partially reflected by the optical element to the camera, so that the image captured by the camera includes an eye pattern, and the infrared light reflected by the eye is at least partially irradiated to the reference structure, so that the image captured by the camera also includes a reference pattern, thereby being able to determine the line of sight of the eye based on the eye pattern and the reference pattern.

[0006] Optionally, the optical element includes a non-reference structure and the reference structure, and a reflectivity of the non-reference structure is greater than a reflectivity of the reference structure;

[0007] The infrared light reflected by the eye is at least partially reflected to the camera via the non-reference structure, so that the camera can capture the eye pattern.

[0008] Optionally, the optical element comprises a reflective film layer and a functional layer, the functional layer is disposed on one side of the reflective film layer and is provided with a shading area corresponding to the reference pattern, and a portion of the reflective film layer corresponding to the shading area and the shading area form the reference structure; or

[0009] The optical element includes a reflective film, and the reference structure is a hole formed in the reflective film.

[0010] Optionally, the optical element comprises a holographic reflective film, and the holographic reflective film is exposed through a mask to form the reference structure.

[0011] Optionally, the reference pattern includes at least one of the following: a grid, an array pattern, and a QR code.

[0012] In a second aspect, an embodiment of the present invention provides a method for preparing an optical element, comprising the following steps:

[0013] A reflective layer is arranged on one side of the holographic medium layer, and a mask plate is arranged on the other side, wherein the mask plate is provided with a light shielding area corresponding to the reference pattern;

[0014] A laser emitter is arranged in a camera preset area on a side of the mask plate away from the holographic medium layer, and the laser emitter emits an infrared laser toward the holographic medium layer. The infrared laser is reflected by the reflective layer to expose the holographic medium layer, so that the holographic medium layer forms the reference structure, thereby obtaining the holographic reflective film.

[0015] In a third aspect, an embodiment of the present invention further provides a method for preparing an optical element, comprising the following steps:

[0016] A reflective layer is arranged on one side of the holographic medium layer, and a mask is arranged in a preset eye area on the other side of the holographic medium layer, wherein the mask is provided with a reflective area corresponding to the reference pattern;

[0017] A laser emitter is arranged in a camera preset area on the other side of the holographic medium layer, and the laser emitter emits infrared laser toward the holographic medium layer. The infrared laser is reflected to the mask plate through the reflective layer, and part of the infrared laser is reflected to the reflective layer through the reflective area, and is reflected again to the camera preset area through the reflective layer to expose the holographic medium layer, so that the holographic medium layer forms the reference structure, and the holographic reflective film is obtained.

[0018] In a fourth aspect, an embodiment of the present invention provides an eye tracking device, comprising:

[0019] An infrared light source, used for emitting infrared light;

[0020] An optical element, wherein the optical element is the optical element as described in the first aspect, the infrared light is reflected to the optical element through the eye, and the optical element is capable of transmitting visible light;

[0021] a camera, wherein at least a portion of the infrared light reflected by the eye is reflected to the camera via the optical element, so that an image captured by the camera includes an eye pattern, and at least a portion of the infrared light reflected by the eye is irradiated to the reference structure, so that an image captured by the camera also includes a reference pattern; and

[0022] A processing module is electrically connected to the camera, and is used to determine the sight direction of the eye according to the eye pattern and the reference pattern.

[0023] Optionally, the processing module is used to:

[0024] Determine the coordinates of the pupil and / or iris in the eye pattern in the reference pattern according to the image, and use the coordinates as coordinates to be matched;

[0025] The coordinates to be matched are matched with a pre-stored coordinate database, and the sight direction corresponding to the matched pre-stored coordinates is determined as the current sight direction of the eye.

[0026] Optionally, when the processing module matches the coordinates to be matched with a pre-stored coordinate database and determines the sight line direction corresponding to the matched pre-stored coordinates as the current sight line direction of the eye, it is used to:

[0027] When the absolute value of the difference between the coordinate to be matched and one of the pre-stored coordinates is less than or equal to a preset threshold, the coordinate to be matched is matched with the pre-stored coordinate, and the sight direction corresponding to the pre-stored coordinate is determined as the current sight direction of the eye;

[0028] When the absolute value of the difference between the coordinate to be matched and any of the pre-stored coordinates in the pre-stored coordinate database is greater than the preset threshold, the sight direction corresponding to the pre-stored coordinates in the pre-stored coordinate database is interpolated, and the interpolated sight direction matching the coordinate to be matched is determined as the current sight direction of the eye.

[0029] Optionally, the processing module is further used for:

[0030] The coordinates of the pupil and / or iris in the eye pattern in the reference pattern are determined according to the image when the eye is observed along a specified sight line direction, and the coordinates are stored as the pre-stored coordinates in the pre-stored coordinate database.

[0031] Optionally, the eye tracking device includes a frame, a lens provided on the frame, and a wearable component connected to the frame, the optical element is provided on the lens, the camera is provided on the wearable component, and the infrared light source is provided on the frame or the wearable component.

[0032] In a fifth aspect, an embodiment of the present invention provides a near-eye display device, comprising an eye tracking device as described in the fourth aspect above.

[0033] The embodiment of the present invention provides an optical element and a preparation method thereof, an eye tracking device, and a near-eye display device. A reference structure is formed on the optical element, so that when the optical element is applied to the eye tracking device, the infrared light emitted by the infrared light source is transmitted to the eye, and after the infrared light reflected by the eye is transmitted to the optical element, at least part of it can be reflected to the camera through the optical element, so that the image captured by the camera includes an eye pattern, and at least part of it can be irradiated to the reference structure of the optical element, so that the image captured by the camera also includes the reference pattern, so that the line of sight of the eye can be determined according to the eye pattern and the reference pattern, and eye tracking is realized. In the process of realizing eye tracking, the optical element can change the transmission path of the infrared light, so that the camera can capture an image including the eye pattern and the reference pattern, thereby improving the installation position of the infrared light source and the camera more flexibly, and the optical element can transmit visible light, which can prevent the infrared light source, camera and optical element from blocking the user's line of sight, which is beneficial to the structural design of the near-eye display device with eye tracking function. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 A schematic diagram of the structure of an optical element provided by an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of the structure of an eye tracking device provided by an embodiment of the present invention;

[0037] Figure 3 A transmission path diagram of infrared light when an eye tracking device provided by an embodiment of the present invention is working;

[0038] Figure 4 A schematic diagram of the structure of another eye tracking device provided by an embodiment of the present invention;

[0039] Figure 5 A transmission path diagram of infrared light when another eye tracking device provided by an embodiment of the present invention is working;

[0040] Figure 6 A schematic diagram of an image captured by a camera provided in an embodiment of the present invention;

[0041] Figure 7A schematic diagram of an exposure process of a holographic medium layer provided by an embodiment of the present invention;

[0042] Figure 8 A schematic structural diagram of a mask plate provided by an embodiment of the present invention;

[0043] Fig. 9 A schematic diagram of another exposure process of a holographic medium layer provided by an embodiment of the present invention;

[0044] Fig.10 A schematic diagram of a virtual image and an image acquired by a camera provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0047] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0048] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0049] Please refer to Figures 1 to 3 , an embodiment of the present invention provides an optical element, a reference structure 10' is formed on the optical element 10. The optical element 10 can transmit visible light, and the infrared light reflected by the eye 100 is at least partially reflected to the camera 30 via the optical element 10, so that the image captured by the camera 30 includes the eye pattern, and the infrared light reflected by the eye 100 is at least partially irradiated to the reference structure 10', so that the image captured by the camera 30 also includes the reference pattern, so that the line of sight direction of the eye 100 can be determined according to the eye pattern and the reference pattern.

[0050] It can be understood that the eye tracking device includes an infrared light source 20 and a camera 30. When the optical element 10 is applied to the eye tracking device, the infrared light emitted by the infrared light source 20 is reflected to the optical element 10 via the eye 100, and at least a portion of the infrared light reflected to the optical element 10 is reflected to the camera 30 via the optical element 10, so that the image obtained by collecting the infrared light by the camera 30 includes an eye pattern, and at least a portion of the infrared light reflected to the optical element 10 is transmitted to the reference structure 10', and the reference structure 10' can reflect, diffract or absorb the infrared light, so that the image obtained by collecting the infrared light by the camera 30 includes the reference pattern. That is to say, the optical element 10 can change the transmission path of the infrared light reflected by the eye 100, so that the camera 30 can collect an image including the eye pattern and the reference pattern, so that the eye tracking device can determine the line of sight direction of the eye 100 according to the eye pattern and the reference pattern, thereby realizing eye tracking. Since the optical element 10 can change the transmission path of infrared light and can transmit visible light, it is beneficial to make the installation positions of the infrared light source 20 and the camera 30 more flexible while avoiding the optical element 10, the infrared light source 20 and the camera 30 blocking the user's line of sight, so as to facilitate the structural design of the near-eye display device with eye tracking function.

[0051] Please refer to Figures 2 to 5 An embodiment of the present invention provides an eye tracking device, including an infrared light source 20, an optical element 10, a camera 30 and a processing module (not shown). The infrared light source 20 is used to emit infrared light, the optical element 10 is the optical element described above, the infrared light is reflected to the optical element 10 via the eye 100, and the optical element 10 can transmit visible light. The infrared light reflected by the eye 100 is at least partially reflected to the camera 30 via the optical element 10, so that the image captured by the camera 30 includes an eye pattern, and the infrared light reflected by the eye 100 is at least partially irradiated to the reference structure 10', so that the image captured by the camera 30 also includes the reference pattern. The processing module is electrically connected to the camera 30, and the processing module is used to determine the line of sight direction of the eye 100 according to the eye pattern and the reference pattern.

[0052] In some embodiments, the eye tracking device includes a frame 40, a lens 60 disposed on the frame 40, and a wearable component connected to the frame 40, the optical element 10 is disposed on the lens 60, the camera 30 is disposed on the wearable component, and the infrared light source 20 is disposed on the frame 40 or the wearable component. It can be understood that the near-eye display device is usually provided with a frame 40, a lens 60 and a wearable component. When the user wears the near-eye display device, the lens 60 of the near-eye display device is usually located in front of the eye 100, the optical element 10 can reflect infrared light through visible light, the optical element 10 can be stably disposed on the lens 60, will not affect the user's line of sight, and can reflect infrared light reflected by the user's eye 100, and the infrared light source 20 can be disposed on the frame 40 or the wearable component, and the camera 30 can be disposed on the wearable component to avoid the infrared light source 20 and the camera 30 blocking the user's line of sight, and is conducive to controlling the volume of the frame 40, and avoiding the frame 40 from being too large to affect the aesthetics, wearing stability and wearing comfort of the near-eye display device.

[0053] Exemplarily, the infrared light emitted by the infrared light source 20 can be directly transmitted to the user's eye 100 , or the infrared light emitted by the infrared light source 20 can be transmitted to the optical element 10 and then reflected by the optical element 10 to the user's eye 100 .

[0054] For example, taking the application of the eye tracking device to AR glasses / VR glasses as an example, the AR glasses / VR glasses include a frame 40, temples 50, lenses 60, and nose pads 70. The two temples 50 are respectively connected to opposite sides of the frame 40, the two lenses 60 are located between the two temples 50 and are spaced apart on the frame 40, and the nose pads 70 are provided on the frame 40 and between the two lenses 60.

[0055] An exemplary one, Figure 2 FIG. 4 shows a layout of an eye tracking device in AR glasses / VR glasses, such as Figure 2 As shown, the infrared light source 20 can be disposed on the nose pad 70 to face the user's eye 100. The optical element 10 is disposed on the lens 60, for example, on the inner side of the lens 60 close to the user's eye 100, or, on the outer side of the lens 60 away from the user's eye 100, or, inside the lens 60. The camera 30 is disposed on the inner side of the temple 50 and faces the optical element 10.

[0056] Figure 3 Shows Figure 2 The transmission path of infrared light when the eye tracking device in the Figure 3As shown, when the eye tracking device is working, the infrared light source 20 emits infrared light toward the user's eye 100. The infrared light is reflected by the user's eye 100 and then transmitted to the optical element 10. The optical element 10 reflects the infrared light toward the camera 30 again. The camera 30 collects the infrared light and forms an image to obtain an image including an eye pattern and a reference pattern.

[0057] Another exemplary example, Figure 4 Another arrangement of the eye tracking device in AR glasses / VR glasses is shown, Figure 4 As shown, the optical element 10 is arranged on the lens 60, and the infrared light source 20 and the camera 30 are both arranged on the temple 50 and arranged toward the optical element 10. This is beneficial to saving space for the nose pad 70, and at the same time eliminates the need for wiring from the temple 50 to the nose pad 70, making it more convenient to connect the infrared light source 20 and the camera 30.

[0058] Optionally, the infrared light source 20 and the camera 30 may be disposed at intervals on the temple 50, or may be disposed closely on the temple 50. Preferably, the infrared light source 20 and the camera 30 are disposed closely to improve the accuracy of eye tracking and to achieve a more compact appearance.

[0059] Figure 5 Shows Figure 4 The transmission path of infrared light when the eye tracking device in the Figure 5 As shown, when the eye tracking device is working, the infrared light emitted by the infrared light source 20 is transmitted toward the optical element 10, and is transmitted to the user's eye 100 after being reflected by the optical element 10, illuminating the user's eye 100, and then the infrared light is reflected by the user's eye 100 to the optical element 10, and is reflected by the optical element 10 again and transmitted to the camera 30, so that the camera 30 can collect the infrared light and perform imaging, thereby obtaining an image including an eye pattern and a reference pattern.

[0060] Optionally, the infrared light emitted by the infrared light source 20 has a wavelength of 800nm-1000nm, preferably infrared light with a wavelength of 850nm or 940nm.

[0061] Exemplarily, the infrared light source 20 may be a point light source, and the infrared light source 20 occupies a relatively small space.

[0062] Exemplarily, the camera 30 may be an infrared camera 30 that matches the infrared light source 20 , and the camera 30 may only respond to and image infrared light of a specific wavelength emitted by the infrared light source 20 .

[0063] It is worth noting that the shape and size of the optical element 10 can be adjusted according to actual conditions, for example, according to the shape and size of the lens 60. Optionally, the optical element 10 can be square, circular or elliptical, etc., which is not specifically limited here.

[0064] Exemplarily, the central area of ​​the optical element 10 is located on the optical axis of the camera 30 to ensure the integrity of the image captured by the camera 30 .

[0065] It can be understood that the reference pattern is formed by the optical element 10. When the user wears the near-eye display device, the relative positions of the components of the eye tracking device remain unchanged, and the relative positions between the eye tracking device and the user's eye 100 remain unchanged, so that the position of the reference pattern in the image captured by the camera 30 is also relatively fixed. When the user's eye 100 rotates to observe along different sight lines, the relative positions of the pupil and iris of the eye 100 in the eye pattern captured by the camera 30 and the reference pattern are different. Therefore, the user's sight direction can be determined based on the relative position relationship between the pupil, iris and reference pattern in the eye pattern.

[0066] In some embodiments, the reference pattern includes at least one of the following: a grid, an array pattern, a QR code. It is understandable that the reference pattern and the eye pattern are superimposed, and when the reference pattern is a grid, an array pattern or a QR code, it can ensure that the eye pattern can be accurately identified.

[0067] Furthermore, the distribution of the units in the grid, array pattern or QR code in the vertical and horizontal directions is conducive to determining the coordinates of the eye pattern in the reference pattern and ensuring the accuracy of eye tracking.

[0068] For example, the array pattern may be as follows Figure 6 The dot matrix shown.

[0069] For example, Figure 6 As shown in Figure (A), it can be an image captured by the camera 30 when the user's eyes 100 look upward. Figure 6 As shown in Figure (B), it can be the image captured by the camera 30 when the user's eye 100 looks to the lower left. It can be seen that Figure 6 In Figure (A) and Figure (B), the position of the pupil in the eye pattern in the reference pattern is different, and the line of sight direction of the user's eye 100 can be determined based on the position of the pupil in the eye pattern in the reference pattern.

[0070] In some embodiments, the optical element 10 includes a non-reference structure 10" and a reference structure 10', and the reflectivity of the non-reference structure 10" is greater than the reflectivity of the reference structure 10'. The infrared light reflected by the eye 100 is at least partially reflected to the camera 30 via the non-reference structure 10", so that the camera 30 can collect the eye pattern. It can be understood that the reflectivity of the non-reference structure 10" and the reference structure 10' is the reflectivity of infrared light, the reflectivity of the non-reference structure 10" is R1, then 0<R1, the reflectivity of the reference structure 10' is R2, then 0≤R2, and R2<R1. After the infrared light reflected by the eye 100 is transmitted to the optical element 10, part of it is reflected to the camera 30 by the non-reference structure 10", and the other part is irradiated to the reference structure 10' without reflection or less reflection, so that there is a color difference in the image obtained by the camera 30 collecting the infrared light, so that the area of ​​the eye pattern and the reference pattern can be divided according to the color difference, so as to determine the line of sight direction of the eye 100 according to the eye pattern and the reference pattern.

[0071] Preferably, the reflectivity of the non-reference structure 10 ″ and the reference structure 10 ′ satisfies the relationship: R2<0.5R1. When the above relationship is satisfied, it is beneficial for the processing module to more accurately divide the area of ​​the reference pattern after acquiring the image captured by the camera 30, so as to improve the accuracy of determining the line of sight direction of the eye 100 according to the eye pattern and the reference pattern.

[0072] In an optional embodiment, the optical element 10 includes a reflective film layer and a functional layer, the functional layer is disposed on one side of the reflective film layer and is provided with a shading area corresponding to the reference pattern, and the portion of the reflective film layer corresponding to the shading area and the shading area form a reference structure. It can be understood that the functional layer is disposed on one side of the reflective film layer, and the functional layer has a shading area corresponding to the reference pattern, and the shading area is used to block part or all of the infrared light, thereby reducing the reflectivity of the reference structure 10'.

[0073] Exemplarily, the reflective film layer may be a dichroic mirror film layer or a holographic reflective film layer.

[0074] For example, the functional layer may be formed using a crystalline material such as calcium fluoride, a liquid crystal material, or a silicon dioxide film.

[0075] Exemplarily, the shading area may be in a grid shape, an array shape, or a two-dimensional code shape, and the reference pattern in the image captured by the camera 30 may be a corresponding grid, array pattern, or two-dimensional code.

[0076] In another optional implementation, if Figure 1As shown, the optical element 10 includes a reflective film, and the reference structure 10' is a hole formed in the reflective film corresponding to the reference pattern. It can be understood that both visible light and infrared light can pass through the hole, and since infrared light cannot be reflected in the hole, part of the infrared light cannot be reflected to the camera 30, so that the image captured by the camera 30 has the reference pattern corresponding to the hole.

[0077] Exemplarily, the reflective film may be a dichroic mirror film or a holographic reflective film layer.

[0078] Exemplarily, the holes may be in a grid shape, an array shape, or a QR code shape, and the reference pattern in the image captured by the camera 30 may be a corresponding grid, array pattern, or QR code.

[0079] In some embodiments, the optical element 10 includes a holographic reflective film, and the holographic reflective film is exposed through a mask to form a reference structure 10'. It is understandable that the holographic reflective film can be obtained by using a laser in the infrared band to perform mask exposure on the holographic medium layer 11, so that the holographic reflective film records an interference hologram. When the infrared light reflected by the eye 100 is irradiated to the holographic reflective film, the holographic reflective film reflects at least part of the infrared light to the camera 30, so that the image captured by the camera 30 includes the eye pattern, and at least part of the infrared light is irradiated to the reference structure and then diffracted and transmitted to the camera, so that the image captured by the camera 30 includes the reproduced reference pattern. In other words, the structure of the holographic reflective film itself can realize the reflection of infrared light while allowing the infrared light to form a reference pattern in the camera 30, which is beneficial to controlling the volume of the optical element 10, and it is relatively convenient to prepare the holographic reflective film by mask exposure, and the prepared holographic reflective film has little influence on visible light, which is beneficial to ensuring the display effect of the near-eye display device.

[0080] like Figure 6 and Figure 7 As shown, in an optional embodiment, the method for preparing a holographic reflective film includes the following steps: a reflective layer 12 is disposed on one side of a holographic medium layer 11, and a mask plate 13 is disposed on the other side, wherein the mask plate 13 is provided with a shading area 131 corresponding to a reference pattern; a laser emitter (not shown) is disposed in a camera preset area 10a on the side of the mask plate 13 away from the holographic medium layer 11, and the laser emitter emits an infrared laser toward the holographic medium layer 11, and the infrared laser is reflected by the reflective layer 12 to expose the holographic medium layer 11, so that the holographic medium layer 11 forms a reference structure 10', thereby obtaining a holographic reflective film.

[0081] It can be understood that the optical element 10 includes a holographic reflective film, which is obtained by exposing the holographic medium layer 11. The mask plate 13 disposed on one side of the holographic medium layer 11 has a light-shielding area 131 corresponding to the reference pattern, and the infrared laser is blocked and cannot be transmitted to the holographic medium layer 11, while other areas outside the light-shielding area 131 are light-transmitting areas, and the infrared laser can pass through and be transmitted to the holographic medium layer 11. When the laser emitter in the preset area 10a of the camera emits an infrared laser to the holographic medium layer 11, a part of the infrared laser is blocked by the light shielding area 131, so that the position of the holographic medium layer 11 corresponding to the light shielding area 131 cannot record the hologram and does not have a reflection function, and another part of the infrared laser can be transmitted through the light-transmitting area of ​​the mask plate 13 to the holographic medium layer 11 and the reflection layer 12 and can be reflected by the reflection layer 12, and the infrared laser irradiated to the holographic medium layer 11 by the laser emitter and the infrared laser reflected by the reflection layer 12 form interference in the holographic medium layer 11, so that the position of the holographic medium layer 11 corresponding to the position outside the light shielding area 131 records the hologram and has a reflection function, thereby realizing the exposure of the holographic medium layer 11 and obtaining a holographic reflection film. When the holographic reflection film obtained by the exposure of the holographic medium layer 11, the camera 30 and the infrared light source 20 are used for eye tracking, because some positions of the holographic reflection film do not have a reflection function, the infrared light cannot be reflected to the camera 30, so that the image collected by the camera 30 includes the reference pattern.

[0082] The camera preset area 10a is the preset position of the camera 30 relative to the holographic reflective film in the eye tracking device. Specifically, the laser emitter emits infrared laser from the light entrance pupil position of the camera 30 to ensure that more infrared light reflected by the prepared holographic reflective film is collected by the camera 30, thereby ensuring the integrity of the image collected by the camera 30.

[0083] Exemplarily, the infrared laser emitted by the laser transmitter is reflected by the reflective layer 12 and passes through the preset eye area 10b, which is the preset position of the eye 100 relative to the holographic reflective film when the eye tracking device is working.

[0084] For example, the shading area 131 may be in a grid shape, a QR code shape, or a Figure 8 The array shown.

[0085] For example, Figure 7 As shown, the reflective layer 12 is attached to one side of the holographic medium layer 11, and the mask is attached to the other side of the holographic medium layer 11, so as to ensure that the laser can more accurately expose the holographic medium layer 11, improve the accuracy of the hologram that can be formed by the holographic medium layer 11, and thus improve the accuracy of eye tracking of the eye tracking device.

[0086] Optionally, a holographic reflective film may be prepared first and then disposed on the lens 60, or the holographic medium layer 11 may be directly disposed on the lens 60, and then the holographic medium layer 11 may be exposed using the reflective layer 12, the mask plate 13 and the laser emitter to obtain the holographic reflective film.

[0087] like Figure 8 As shown, in another optional embodiment, the preparation method of the holographic reflective film includes the following steps: a reflective layer 12 is set on one side of the holographic medium layer 11, and a mask plate 13 is set in the eye preset area 10b on the other side of the holographic medium layer 11, and the mask plate 13 is provided with a reflective area corresponding to the reference pattern; a laser emitter is set in the camera preset area 10a on the other side of the holographic medium layer 11, and the laser emitter emits an infrared laser toward the holographic medium layer 11, and the infrared laser is reflected to the mask plate 13 through the reflective layer 12, and part of the infrared laser is reflected to the reflective layer 12 through the reflective area, and is reflected again to the camera preset area 10a through the reflective layer 12 to expose the holographic medium layer 11, so that the holographic medium layer 11 forms a reference structure 10', and a holographic reflective film is obtained.

[0088] It can be understood that the optical element 10 includes a holographic reflective film, which is obtained by exposing the holographic medium layer 11. The mask plate 13 arranged in the preset eye area 10b has a reflective area corresponding to the reference pattern, and the reflective area can reflect infrared laser, while the area outside the reflective area reflects less or no infrared laser. The laser emitter in the preset camera area 10a emits infrared laser to the holographic medium layer 11, and the infrared laser is transmitted to the reflective layer 12 after passing through the holographic medium layer 11 and is reflected by the reflective layer 12 to the mask plate 13. Only part of the infrared light is reflected by the reflective area to the holographic medium layer 11, and forms interference with the infrared laser irradiated to the holographic medium layer 11 by the laser emitter and the infrared laser reflected by the reflective layer 12 in the holographic medium layer 11, so that the holographic medium layer 11 records the hologram related to the reference pattern and has a reflection function, so as to realize the exposure of the holographic medium layer 11 and obtain a holographic reflective film. When eye tracking is performed using the holographic reflective film obtained by exposure of the holographic medium layer 11, the camera 30 and the infrared light source 20, the holographic reflective film can reproduce a hologram under infrared light irradiation, so that the pattern collected by the camera 30 includes the reference pattern, and the infrared light reflected by the eye 100 is collected by the camera 30, so that the collected pattern also includes the eye pattern. It is worth noting that when eye tracking is performed using the holographic reflective film obtained by the preparation method, when the camera 30 collects images, the virtual image corresponding to the eye pattern and the virtual image corresponding to the reference pattern can be located at the same or similar object distance relative to the camera 30, and the camera 30 can clearly focus on the eye pattern and the reference pattern at the same time, so that the collected eye pattern and the reference pattern are both clear, so as to determine the sight direction of the eye 100 according to the eye pattern and the reference pattern, thereby improving the accuracy of eye tracking.

[0089] Exemplarily, the reflective area of ​​the mask may be a high-reflective / scattering area that can reflect / scatter laser light in the infrared band, while other areas outside the reflective area are low-reflective / scattering areas that reflect / scatter less or do not reflect / scatter laser light in the infrared band.

[0090] Exemplarily, the reflective area of ​​the mask plate 13 may be in a grid shape, an array shape, or a two-dimensional code shape.

[0091] In some embodiments, the processing module is used to: determine the coordinates of the pupil and / or iris in the eye pattern in the reference pattern based on the image, and use the coordinates as the coordinates to be matched; match the coordinates to be matched with a pre-stored coordinate database, and determine the line of sight direction corresponding to the matched pre-stored coordinates as the line of sight direction of the current eye 100.

[0092] It can be understood that, as described above, when the user's eye 100 observes along different sight directions, the positions of the pupil and / or iris in the eye pattern in the reference pattern are different. The processing module may be provided with a pre-stored coordinate database, which contains multiple pre-stored coordinates. The pre-stored coordinates represent the coordinates of the pupil and / or iris in the eye pattern in the reference pattern, and any pre-stored coordinate has a corresponding sight direction. When the eye tracking device performs eye tracking, the coordinates of the pupil and / or iris in the current eye pattern in the reference pattern can be determined based on the image captured by the camera 30. At this time, the coordinates can be determined as the coordinates to be matched, and the coordinates to be matched can be matched with multiple pre-stored coordinates in the pre-stored coordinate database. If the match is successful, the sight direction corresponding to the matched pre-stored coordinates can be used as the sight direction of the current eye 100 to achieve eye tracking, so that the near-eye display device can be controlled accordingly according to the sight direction of the eye 100.

[0093] Exemplarily, the coordinates of the pupil center in the eye pattern in the reference pattern are determined based on the image to improve the accuracy of eye tracking.

[0094] Exemplarily, the absolute value of the difference between the pre-stored coordinates that match the coordinates to be matched and the coordinates to be matched is smaller than the absolute value of the difference between other pre-stored coordinates and the coordinates to be matched.

[0095] Further, when the processing module matches the coordinates to be matched with the pre-stored coordinate database and determines the sight direction corresponding to the matched pre-stored coordinates as the sight direction of the current eye 100, it is used to: when the absolute value of the difference between the coordinates to be matched and a pre-stored coordinate is less than or equal to a preset threshold, the coordinates to be matched are matched with the pre-stored coordinates, and the sight direction corresponding to the pre-stored coordinates is determined as the sight direction of the current eye 100; when the absolute value of the difference between the coordinates to be matched and any pre-stored coordinates in the pre-stored coordinate database is greater than the preset threshold, the sight direction corresponding to the pre-stored coordinates in the pre-stored coordinate database is interpolated, and the interpolated sight direction matching the coordinates to be matched is determined as the sight direction of the current eye 100. It can be understood that when the absolute value of the difference between the coordinates to be matched and any pre-stored coordinates is greater than the preset threshold, that is, the distance between the coordinates to be matched and any pre-stored coordinates is large, the sight direction corresponding to the pre-stored coordinates is interpolated to obtain the interpolated sight direction corresponding to the coordinates closer to the current coordinates to be matched, and the interpolated sight direction is determined as the sight direction of the current eye 100, so as to improve the accuracy of eye tracking.

[0096] In some embodiments, the processing module is further used to: determine the coordinates of the pupil and / or iris in the eye pattern in the reference pattern according to the image when the eye 100 is observed along the specified line of sight direction, and store the coordinates as pre-stored coordinates in the pre-stored coordinate database. It can be understood that when the eye tracking device is initially used, the user's eye 100 can be guided to observe along the specified line of sight direction, and the camera 30 can be used to capture images. After the processing module obtains the image captured by the camera 30, the coordinates of the pupil and / or iris of the eye pattern in the reference pattern are determined according to the image. At this time, the determined coordinates are pre-stored coordinates, which are stored in the pre-stored coordinate database, and the pre-stored coordinates correspond to the specified line of sight direction, thereby realizing the calibration of the line of sight direction. When the eye tracking device is used for eye tracking in the future, the coordinates of the pupil and / or iris in the eye pattern in the reference pattern can be determined based on the re-acquired image when the user's eye 100 observes along an unknown line of sight direction. The coordinates determined at this time are the coordinates to be matched, and the line of sight direction corresponding to the pre-stored coordinates that match the coordinates to be matched is determined as the line of sight direction of the current eye 100 by matching the coordinates to be matched with the pre-stored coordinates.

[0097] Exemplarily, taking the application of an eye tracking device to a near-eye display device as an example, when a user initially uses the near-eye display device, the near-eye display device can display a virtual image, and guide the user's eye 100 to observe a specified position of the virtual image, so as to guide the user's eye 100 to observe along a specified line of sight direction, thereby calibrating the line of sight direction. For example, the virtual image can be a dot matrix, a square array, a five-pointed star array, etc. Taking the virtual image as a dot matrix as an example, the color / size of a specified point in the dot matrix can be distinguished from the color / size of other points, or the specified point can be made to flash while other points are continuously displayed, so as to distinguish the specified point from other points, thereby guiding the user to look at the specified point, so that the user observes along the specified line of sight direction. Fig. 9 As shown, Fig. 9 , the direction of sight when the eye 100 observes a designated point of the virtual image, and the image including the eye pattern and the reference pattern acquired by the camera are shown. When the eye 100 observes a designated point of the virtual image, the eye 100 observes along the designated direction of sight, and the infrared light source 20 emits infrared light, so that the camera 30 can acquire an image including the eye pattern and the reference pattern, so that the coordinates of the pupil and / or iris of the eye pattern in the image in the reference pattern can be determined according to the image when the user observes along the designated direction of sight, and the coordinates are stored as pre-stored coordinates in the pre-stored coordinate database, and all the pre-stored coordinates are obtained by guiding the user to sequentially gaze at all designated points in the dot matrix to complete the calibration of the sight direction.

[0098] The camera 30 can continuously capture multiple frames of images to obtain the most stable image of the user's eye 100 position, and use the image to calibrate the specified line of sight direction, thereby improving the accuracy of the calibration to optimize the accuracy of eye tracking of the eye tracking device.

[0099] The embodiment of the present invention further provides a near-eye display device, including the eye tracking device as described above. It can be understood that the near-eye display device includes the eye tracking device as described above, and also has all its technical effects, that is, it can avoid blocking the user's line of sight while making the installation position of the infrared light source and the camera more flexible, which is conducive to adjusting the installation position of the infrared light source and the camera according to needs to control the volume of the near-eye display device.

[0100] The embodiment of the present invention also provides an eye tracking method of an eye tracking device, the eye tracking device includes an infrared light source, an optical element and a camera, the infrared light source is used to emit infrared light, the infrared light is reflected by the eye to the optical element, the optical element can transmit visible light, and the optical element is used to reflect the infrared light so that the camera can collect an image including an eye pattern and a reference pattern. The eye tracking method includes steps S10 to S30.

[0101] S10: Acquire the image captured by the camera.

[0102] S20, determining the coordinates of the pupil and / or iris in the eye pattern in the reference pattern according to the image, and using the coordinates as coordinates to be matched.

[0103] S30, matching the coordinates to be matched with a pre-stored coordinate database, and determining the sight direction corresponding to the matched pre-stored coordinates as the sight direction of the current eye.

[0104] The infrared light reflected by the eye is reflected back to the camera through optical elements, so that the camera can capture an image including the eye pattern and the reference pattern, thereby determining the line of sight of the eye based on the coordinates of the pupil and / or iris of the eye pattern in the image in the reference pattern.

[0105] The method of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0106] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The method implemented when the computer program is executed by a processor can refer to the various embodiments of the eye tracking method for a near-eye display device of the present application.

[0107] The computer-readable storage medium may be an internal storage unit of the near-eye display device described in the aforementioned embodiment, such as a hard disk or memory of the near-eye display device. The computer-readable storage medium may also be an external storage device of the near-eye display device, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc., equipped on the near-eye display device.

[0108] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0109] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0110] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An optical element, characterized in that: A reference structure is formed on the optical element; The optical element is capable of transmitting visible light, and the infrared light reflected by the eye is at least partially reflected by the optical element to the camera, so that the image captured by the camera includes an eye pattern, and the infrared light reflected by the eye is at least partially irradiated to the reference structure, so that the image captured by the camera also includes a reference pattern, thereby being able to determine the line of sight of the eye based on the eye pattern and the reference pattern.

2. The optical element according to claim 1, characterized in that The optical element comprises a non-reference structure and the reference structure, and the reflectivity of the non-reference structure is greater than the reflectivity of the reference structure; The infrared light reflected by the eye is at least partially reflected to the camera via the non-reference structure, so that the camera can capture the eye pattern.

3. The optical element according to claim 2, characterized in that The optical element comprises a reflective film layer and a functional layer, the functional layer is arranged on one side of the reflective film layer and is provided with a shading area corresponding to the reference pattern, and a portion of the reflective film layer corresponding to the shading area and the shading area form the reference structure; or The optical element includes a reflective film, and the reference structure is a hole formed in the reflective film.

4. The optical element according to claim 1, characterized in that The optical element includes a holographic reflective film, and the holographic reflective film is exposed through a mask to form the reference structure.

5. The optical element according to any one of claims 1 to 4, characterized in that: The reference pattern includes at least one of the following: a grid, an array pattern, and a two-dimensional code.

6. A method for preparing the holographic reflective film according to claim 4, characterized in that: The following steps are involved: A reflective layer is arranged on one side of the holographic medium layer, and a mask plate is arranged on the other side, wherein the mask plate is provided with a light shielding area corresponding to the reference pattern; A laser emitter is arranged in a camera preset area on a side of the mask plate away from the holographic medium layer, and the laser emitter emits an infrared laser toward the holographic medium layer. The infrared laser is reflected by the reflective layer to expose the holographic medium layer, so that the holographic medium layer forms the reference structure, thereby obtaining the holographic reflective film.

7. A method for preparing the holographic reflective film according to claim 4, characterized in that: The following steps are involved: A reflective layer is arranged on one side of the holographic medium layer, and a mask is arranged in a preset eye area on the other side of the holographic medium layer, wherein the mask is provided with a reflective area corresponding to the reference pattern; A laser emitter is arranged in a camera preset area on the other side of the holographic medium layer, and the laser emitter emits infrared laser toward the holographic medium layer. The infrared laser is reflected to the mask plate through the reflective layer, and part of the infrared laser is reflected to the reflective layer through the reflective area, and is reflected again to the camera preset area through the reflective layer to expose the holographic medium layer, so that the holographic medium layer forms the reference structure, and the holographic reflective film is obtained.

8. An eye tracking device, characterized in that: include: An infrared light source, used for emitting infrared light; An optical element, wherein the optical element is the optical element according to any one of claims 1 to 5, wherein the infrared light is reflected to the optical element through the eye, and the optical element is capable of transmitting visible light; a camera, wherein at least a portion of the infrared light reflected by the eye is reflected to the camera via the optical element, so that the image captured by the camera includes the eye pattern, and at least a portion of the infrared light reflected by the eye is irradiated to the reference structure, so that the image captured by the camera also includes the reference pattern; as well as A processing module is electrically connected to the camera, and is used to determine the sight direction of the eye according to the eye pattern and the reference pattern.

9. The eye tracking device according to claim 8, characterized in that: The processing module is used for: Determine the coordinates of the pupil and / or iris in the eye pattern in the reference pattern according to the image, and use the coordinates as coordinates to be matched; The coordinates to be matched are matched with a pre-stored coordinate database, and the sight direction corresponding to the matched pre-stored coordinates is determined as the current sight direction of the eye.

10. The eye tracking device according to claim 9, characterized in that: When the processing module matches the coordinates to be matched with the pre-stored coordinate database and determines the sight line direction corresponding to the matched pre-stored coordinates as the current sight line direction of the eye, it is used to: When the absolute value of the difference between the coordinate to be matched and one of the pre-stored coordinates is less than or equal to a preset threshold, the coordinate to be matched is matched with the pre-stored coordinate, and the sight direction corresponding to the pre-stored coordinate is determined as the current sight direction of the eye; When the absolute value of the difference between the coordinate to be matched and any of the pre-stored coordinates in the pre-stored coordinate database is greater than the preset threshold, the sight direction corresponding to the pre-stored coordinates in the pre-stored coordinate database is interpolated, and the interpolated sight direction matching the coordinate to be matched is determined as the current sight direction of the eye.

11. The eye tracking device according to claim 9, characterized in that: The processing module is also used for: The coordinates of the pupil and / or iris in the eye pattern in the reference pattern are determined according to the image when the eye is observed along a specified sight line direction, and the coordinates are stored as the pre-stored coordinates in the pre-stored coordinate database.

12. The eye tracking device according to claim 8, characterized in that: The eye tracking device includes a frame, a lens arranged on the frame, and a wearable component connected to the frame, the optical element is arranged on the lens, the camera is arranged on the wearable component, and the infrared light source is arranged on the frame or the wearable component.

13. A near-eye display device, characterized in that: Comprising an eye tracking device as described in any one of claims 8-12.

Images