Optical element, method of manufacturing the same, eye tracking device, near-eye display device

Abstract

Embodiments of the present application provide an optical element, a preparation method thereof, an eye movement tracking device and a near-eye display device. The optical element has a reference structure formed thereon. The optical element can transmit visible light. Infrared light reflected by an eye is at least partially reflected to a camera via the optical element, so that an image captured by the camera includes an eye pattern. The infrared light reflected by the eye is at least partially irradiated to the reference structure, so that the image captured by the camera further includes a reference pattern. The line-of-sight direction of the eye can be determined according to the eye pattern and the reference pattern. The technical solutions of the embodiments of the present application aim to improve the installation flexibility of the infrared light source and the camera, so as to facilitate the structural design of the near-eye display device.

Description

Technical Field

[0001] This invention relates to the field of eye-tracking technology, and in particular to an optical element and its fabrication method, an eye-tracking device, and a near-eye display device. Background Technology

[0002] Eye-tracking devices typically use a light source to illuminate the user's eyes, and an image acquisition unit captures the resulting light spot and iris image. The device then calculates the direction of the user's gaze based on these images. However, when eye-tracking devices are applied to near-eye display devices such as augmented reality (AR) glasses, the relative positions of the light source and image acquisition unit to the user's eyes must meet certain conditions. This often results in the light source and image acquisition unit being mounted on the frame or lenses. However, if both are mounted on the frame, the frame becomes larger, affecting wearability and compromising the AR glasses' aesthetic design. Conversely, if the light source or image acquisition unit is mounted on the lenses, it obstructs the user's line of sight, negatively impacting the viewing experience. Summary of the Invention

[0003] This invention provides an optical element and its fabrication method, an eye-tracking device, and a near-eye display device, aiming to improve the installation flexibility of infrared light sources and cameras to facilitate the structural design of near-eye display devices.

[0004] In a first aspect, embodiments of the present invention provide an optical element on which a reference structure is formed;

[0005] The optical element is capable of transmitting visible light, and at least a portion of the infrared light reflected from the eye is reflected to the camera via the optical element, so that the image captured by the camera includes an eye pattern, and at least a portion of the infrared light reflected from the eye illuminates the reference structure, so that the image captured by the camera also includes a reference pattern, thereby enabling the eye's gaze direction to be determined based on the eye pattern and the reference pattern.

[0006] Optionally, the optical element includes a non-reference structure and the reference structure, wherein the reflectivity of the non-reference structure is greater than the reflectivity of the reference structure;

[0007] The infrared light reflected from the eye is reflected at least partially to the camera via the non-reference structure, so that the camera can capture the eye pattern.

[0008] Optionally, the optical element includes a reflective film layer and a functional layer, the functional layer being disposed on one side of the reflective film layer and having a light-shielding area corresponding to the reference pattern, the portion of the reflective film layer corresponding to the light-shielding area and the light-shielding area forming the reference structure; or

[0009] The optical element includes a reflective film, and the reference structure is a hole formed in the reflective film.

[0010] Optionally, the optical element includes a holographic reflective film, which forms the reference structure by exposure through a mask.

[0011] Optionally, the reference pattern includes at least one of the following: a grid, an array pattern, or a QR code.

[0012] Secondly, embodiments of the present invention provide a method for fabricating an optical element, comprising the following steps:

[0013] A reflective layer is provided on one side of the holographic medium layer, and a mask is provided on the other side. The mask has a light-shielding area corresponding to the reference pattern.

[0014] A laser emitter is placed in a camera preset area on the side of the mask away from the holographic medium layer. The laser emitter emits infrared laser towards the holographic medium layer. The infrared laser is reflected by the reflective layer to expose the holographic medium layer, so that the holographic medium layer forms the reference structure and the holographic reflective film is obtained.

[0015] Thirdly, embodiments of the present invention also provide a method for fabricating an optical element, comprising the following steps:

[0016] A reflective layer is provided on one side of the holographic medium layer, and a mask is provided in the eye preset area on the other side of the holographic medium layer. The mask has a reflective area corresponding to the reference pattern.

[0017] A laser emitter is set in the camera preset area on the other side of the holographic medium layer. The laser emitter emits infrared laser towards the holographic medium layer. The infrared laser is reflected by the reflective layer to the mask. Part of the infrared laser is reflected by the reflective area to the reflective layer, and then reflected again by the reflective layer to the camera preset area to expose the holographic medium layer, so that the holographic medium layer forms the reference structure and the holographic reflective film is obtained.

[0018] Fourthly, embodiments of the present invention provide an eye-tracking device, comprising:

[0019] Infrared light source, used to emit infrared light;

[0020] An optical element, wherein the optical element is as described in the first aspect, the infrared light is reflected by the eye to the optical element, and the optical element is capable of transmitting visible light;

[0021] A camera, wherein at least a portion of the infrared light reflected from the eye is reflected to the camera via the optical element, such that the image captured by the camera includes an eye pattern, and at least a portion of the infrared light reflected from the eye illuminates the reference structure, such that the image captured by the camera also includes a reference pattern; and

[0022] A processing module, electrically connected to the camera, is used to determine the gaze direction of the eye based on the eye pattern and the reference pattern.

[0023] Optionally, the processing module is used for:

[0024] Based on the image, determine the coordinates of the pupil and / or iris in the reference pattern of the eye pattern, and use the coordinates as the coordinates to be matched;

[0025] The coordinates to be matched are matched with the pre-stored coordinate database, and the gaze direction corresponding to the matching pre-stored coordinates is determined as the current gaze direction of the eye.

[0026] Optionally, when the processing module matches the coordinates to be matched with a pre-stored coordinate database, and determines the gaze direction corresponding to the matching pre-stored coordinates as the current gaze direction of the eye, it is used for:

[0027] When the absolute value of the difference between the coordinate to be matched and a pre-stored coordinate is less than or equal to a preset threshold, the coordinate to be matched is matched with the pre-stored coordinate, and the gaze direction corresponding to the pre-stored coordinate is determined as the current gaze direction of the eye.

[0028] When the absolute value of the difference between the coordinate to be matched and any of the pre-stored coordinates in the pre-stored coordinate database is greater than the preset threshold, the gaze direction corresponding to the pre-stored coordinate in the pre-stored coordinate database is interpolated, and the interpolated gaze direction that matches the coordinate to be matched is determined as the current gaze direction of the eye.

[0029] Optionally, the processing module is further configured to:

[0030] Based on the image observed by the eye along a specified line of sight, determine the coordinates of the pupil and / or iris in the reference pattern of the eye pattern, and store the coordinates as the pre-stored coordinates in the pre-stored coordinate database.

[0031] Optionally, the eye-tracking device includes a frame, a lens disposed on the frame, and a wearable component connected to the frame. The optical element is disposed on the lens, the camera is disposed on the wearable component, and the infrared light source is disposed on the frame or the wearable component.

[0032] Fifthly, embodiments of the present invention provide a near-eye display device, including the eye-tracking device as described in the fourth aspect above.

[0033] This invention provides an optical element and its fabrication method, an eye-tracking device, and a near-eye display device. The optical element has a reference structure formed on it. When applied to the eye-tracking device, infrared light emitted from an infrared light source is transmitted to the eye. After being reflected by the eye, the infrared light is transmitted back to the optical element, and at least a portion of it is reflected by the optical element to a camera. This allows the camera to capture an image that includes an eye pattern and at least a portion of the reference structure that illuminates the optical element, further enhancing the image's representation of the reference pattern. The eye's gaze direction can then be determined based on the eye pattern and the reference pattern, thus achieving eye tracking. During eye tracking, the optical element can alter the infrared light transmission path, enabling the camera to capture an image including both the eye pattern and the reference pattern. This allows for greater flexibility in the installation positions of the infrared light source and camera. Furthermore, the optical element's ability to transmit visible light prevents the infrared light source, camera, and optical element from obstructing the user's view, facilitating the structural design of near-eye display devices with eye-tracking functionality. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a schematic diagram of the structure of an optical element provided in an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of the structure of an eye-tracking device provided in an embodiment of the present invention;

[0037] Figure 3 This invention provides a diagram illustrating the transmission path of infrared light during operation of an eye-tracking device.

[0038] Figure 4 This is a schematic diagram of another eye-tracking device provided in an embodiment of the present invention;

[0039] Figure 5 A diagram showing the transmission path of infrared light during operation of another eye-tracking device provided in an embodiment of the present invention;

[0040] Figure 6 A schematic diagram of an image captured by a camera according to an embodiment of the present invention;

[0041] Figure 7This is a schematic diagram of the exposure process of a holographic dielectric layer provided in an embodiment of the present invention;

[0042] Figure 8 This is a schematic diagram of the structure of a mask plate provided in an embodiment of the present invention;

[0043] Figure 9 This is a schematic diagram of the exposure process of another holographic dielectric layer provided in an embodiment of the present invention;

[0044] Figure 10 This is a schematic diagram of a virtual image and an image captured by a camera, provided in an embodiment of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0047] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0048] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0049] Please refer to Figures 1 to 3 This invention provides an optical element 10 on which a reference structure 10' is formed. The optical element 10 is capable of transmitting visible light, and infrared light reflected by the eye 100 is at least partially reflected by the optical element 10 to a camera 30, so that the image captured by the camera 30 includes an eye pattern. The infrared light reflected by the eye 100 is at least partially irradiated by the reference structure 10', so that the image captured by the camera 30 also includes a reference pattern. Thus, the gaze direction of the eye 100 can be determined based on the eye pattern and the reference pattern.

[0050] Understandably, the eye-tracking device includes an infrared light source 20 and a camera 30. When the optical element 10 is applied to the eye-tracking device, the infrared light emitted from the infrared light source 20 is reflected by the eye 100 to the optical element 10. At least a portion of the infrared light reflected to the optical element 10 is reflected by the optical element 10 to the camera 30, so that the image obtained by the camera 30 from the infrared light includes the eye pattern. At least a portion of the infrared light reflected to the optical element 10 is transmitted to the reference structure 10'. The reference structure 10' can reflect, diffract, or absorb the infrared light, so that the image obtained by the camera 30 from the infrared light includes the reference pattern. That is to say, the optical element 10 can change the transmission path of the infrared light reflected by the eye 100, so that the camera 30 can obtain an image including the eye pattern and the reference pattern. Thus, the eye-tracking device can determine the gaze direction of the eye 100 based on the eye pattern and the reference pattern, thereby achieving eye tracking. Since the optical element 10 can change the transmission path of infrared light and transmit visible light, it is advantageous to make the installation position of the infrared light source 20 and the camera 30 more flexible while avoiding obstruction of the user's line of sight by the optical element 10, the infrared light source 20 and the camera 30, so as to facilitate the structural design of near-eye display devices with eye-tracking function.

[0051] Please refer to Figures 2 to 5 This invention provides an eye-tracking device, including an infrared light source 20, an optical element 10, a camera 30, and a processing module (not shown). The infrared light source 20 emits infrared light, and the optical element 10 is the optical element described above. The infrared light is reflected from the eye 100 to the optical element 10, which is capable of transmitting visible light. At least a portion of the infrared light reflected from the eye 100 is reflected by the optical element 10 to the camera 30, so that the image captured by the camera 30 includes an eye pattern. At least a portion of the infrared light reflected from the eye 100 illuminates a reference structure 10', so that the image captured by the camera 30 also includes a reference pattern. The processing module is electrically connected to the camera 30 and is used to determine the gaze direction of the eye 100 based on the eye pattern and the reference pattern.

[0052] In some embodiments, the eye-tracking device includes a frame 40, a lens 60 disposed on the frame 40, and a wearable component connected to the frame 40. An optical element 10 is disposed on the lens 60, a camera 30 is disposed on the wearable component, and an infrared light source 20 is disposed on the frame 40 or the wearable component. It is understood that near-eye display devices typically include a frame 40, a lens 60, and a wearable component. When a user wears the near-eye display device, the lens 60 is typically located in front of the eye 100. The optical element 10 can transmit visible light and reflect infrared light. The optical element 10 can be stably disposed on the lens 60 without affecting the user's vision, while also reflecting infrared light reflected from the user's eye 100. The infrared light source 20 can be disposed on the frame 40 or the wearable component, and the camera 30 can be disposed on the wearable component. This avoids the infrared light source 20 and camera 30 obstructing the user's vision and helps control the size of the frame 40, preventing the frame 40 from becoming too large and affecting the aesthetics, wearing stability, and wearing comfort of the near-eye display device.

[0053] For example, the infrared light emitted by the infrared light source 20 can be directly transmitted to the user's eye 100, or the infrared light emitted by the infrared light source 20 can be transmitted to the optical element 10 and then reflected to the user's eye 100 by the optical element 10.

[0054] For example, taking the application of an eye-tracking device in AR / VR glasses as an example, the AR / VR glasses include a frame 40, temples 50, lenses 60, and nose pads 70. The two temples 50 are respectively connected to opposite sides of the frame 40, the two lenses 60 are located between the two temples 50 and spaced apart on the frame 40, and the nose pads 70 are located on the frame 40 and between the two lenses 60.

[0055] An example, Figure 2 This illustrates one arrangement of eye-tracking devices in AR / VR glasses, such as... Figure 2 As shown, the infrared light source 20 can be disposed on the nose pad 70 and positioned facing the user's eyes 100. The optical element 10 is disposed on the lens 60, for example, on the inner side of the lens 60 near the user's eyes 100, or on the outer side of the lens 60 away from the user's eyes 100, or on the inner side of the lens 60. The camera 30 is disposed on the inner side of the temple 50 and positioned facing the optical element 10.

[0056] Figure 3 It shows Figure 2 The transmission path of infrared light when the eye-tracking device is working, such as... Figure 3As shown, when the eye-tracking device is working, the infrared light source 20 emits infrared light toward the user's eye 100. The infrared light is reflected by the user's eye 100 and transmitted to the optical element 10. The optical element 10 reflects the infrared light back to the camera 30. The camera 30 collects the infrared light and forms an image, obtaining an image including an eye pattern and a reference pattern.

[0057] Another example, Figure 4 This illustrates another arrangement of eye-tracking devices in AR / VR glasses, such as... Figure 4 As shown, the optical element 10 is mounted on the lens 60, and the infrared light source 20 and camera 30 are both mounted on the temple 50 and oriented toward the optical element 10. This helps to save space in the nose pad 70 and eliminates the need for wiring from the temple 50 to the nose pad 70, making it easier to connect the infrared light source 20 and camera 30.

[0058] Optionally, the infrared light source 20 and the camera 30 may be spaced apart on the temple 50, or they may be closely spaced on the temple 50. Preferably, the infrared light source 20 and the camera 30 are closely spaced to improve the accuracy of eye tracking and achieve a more compact appearance.

[0059] Figure 5 It shows Figure 4 The transmission path of infrared light when the eye-tracking device is working, such as... Figure 5 As shown, when the eye-tracking device is working, the infrared light emitted from the infrared light source 20 is transmitted toward the optical element 10, reflected by the optical element 10, and transmitted to the user's eye 100, illuminating the user's eye 100. Then, the infrared light is reflected by the user's eye 100 back to the optical element 10, and reflected again by the optical element 10 and transmitted to the camera 30, so that the camera 30 can collect the infrared light and form an image, obtaining an image including an eye pattern and a reference pattern.

[0060] Optionally, the infrared light emitted by the infrared light source 20 has a wavelength of 800nm-1000nm, preferably infrared light with a wavelength of 850nm or 940nm.

[0061] For example, the infrared light source 20 can be a point light source, and the infrared light source 20 occupies a small space.

[0062] For example, camera 30 may be an infrared camera 30 matched with infrared light source 20, which may respond to and image only infrared light of a specific wavelength emitted by infrared light source 20.

[0063] It is worth noting that the shape and size of the optical element 10 can be adjusted according to the actual situation, for example, it can be adjusted according to the shape and size of the lens 60. Optionally, the optical element 10 can be square, circular or elliptical, etc., and no specific limitation is made here.

[0064] For example, the central region of the optical element 10 is located on the optical axis of the camera 30 to ensure the integrity of the image acquired by the camera 30.

[0065] Understandably, the reference pattern is formed by optical element 10. When the user wears the near-eye display device, the relative positions between the components of the eye-tracking device remain unchanged, as does the relative position between the eye-tracking device and the user's eye 100. This makes the position of the reference pattern in the image captured by the camera 30 relatively fixed. When the user's eye 100 rotates to observe along different viewing directions, the relative positions of the pupil and iris of the eye 100 in the eye pattern captured by the camera 30 are different from the reference pattern. Therefore, the user's viewing direction can be determined based on the relative positional relationship between the pupil, iris, and reference pattern in the eye pattern.

[0066] In some embodiments, the reference pattern includes at least one of the following: a grid, an array pattern, or a QR code. It is understood that the reference pattern and the eye pattern are superimposed, and when the reference pattern is a grid, an array pattern, or a QR code, it ensures that the eye pattern can be accurately identified.

[0067] Furthermore, the distribution of units in the grid, array pattern, or QR code in both the vertical and horizontal directions helps to determine the coordinates of the eye pattern in the reference pattern, ensuring the accuracy of eye tracking.

[0068] For example, the array pattern can be as follows: Figure 6 The dot matrix shown.

[0069] For example, Figure 6 Figure (A) shows an image captured by camera 30 when the user's eye 100 is looking upwards. Figure 6 Figure (B) shows the image captured by camera 30 when the user's eye 100 is looking downwards and to the left. It can be seen that... Figure 6 In Figures (A) and (B), the position of the pupil in the eye pattern is different from that in the reference pattern. The direction of the user's eye 100 can be determined based on the position of the pupil in the eye pattern in the reference pattern.

[0070] In some embodiments, the optical element 10 includes a non-reference structure 10” and a reference structure 10’, where the non-reference structure 10” has a higher reflectivity than the reference structure 10’. Infrared light reflected from the eye 100 is at least partially reflected to the camera 30 via the non-reference structure 10”, allowing the camera 30 to capture the eye pattern. It is understood that the reflectivity of the non-reference structure 10” and the reference structure 10’ is for infrared light; if the reflectivity of the non-reference structure 10” is R1, then 0 < R1; and if the reflectivity of the reference structure 10’ is R2, then 0 ≤ R2, and R2 < R1. After the infrared light reflected from the eye 100 is transmitted to the optical element 10, a portion is reflected to the camera 30 by the non-reference structure 10”, while the other portion illuminates the reference structure 10’ with little or no reflection. This results in color differences in the image obtained by the camera 30 capturing the infrared light, allowing the regions of the eye pattern and the reference pattern to be divided based on these color differences, thus determining the gaze direction of the eye 100.

[0071] Preferably, the reflectivity of the non-reference structure 10” and the reference structure 10’ satisfies the relationship: R2 < 0.5R1. When the above relationship is satisfied, it is beneficial for the processing module to more accurately divide the region of the reference pattern after acquiring the image captured by the camera 30, so as to improve the accuracy of determining the direction of the eye 100’s gaze based on the eye pattern and the reference pattern.

[0072] In one optional embodiment, the optical element 10 includes a reflective film layer and a functional layer. The functional layer is disposed on one side of the reflective film layer and has a light-shielding area corresponding to the reference pattern. The portion of the reflective film layer corresponding to the light-shielding area and the light-shielding area form a reference structure. It is understood that by providing a functional layer on one side of the reflective film layer and having a light-shielding area corresponding to the reference pattern, the light-shielding area blocks part or all of the infrared light, thereby reducing the reflectivity of the reference structure 10'.

[0073] For example, the reflective film layer may be a dichroic mirror film layer or a holographic reflective film layer.

[0074] For example, the functional layer may be formed using a crystalline material such as calcium fluoride, a liquid crystal material, or a thin film such as silicon dioxide.

[0075] For example, the light-blocking area can be in the form of a grid, an array, or a QR code, and the reference pattern in the image captured by the camera 30 can be the corresponding grid, array, or QR code.

[0076] Another alternative implementation, such as Figure 1As shown, the optical element 10 includes a reflective film, and the reference structure 10' is a hole formed in the reflective film corresponding to the reference pattern. It is understood that both visible light and infrared light can pass through the hole. Since infrared light cannot be reflected in the hole, some infrared light cannot be reflected to the camera 30, resulting in the image acquired by the camera 30 containing a reference pattern corresponding to the hole.

[0077] For example, the reflective film may be a dichroic mirror film or a holographic reflective film layer.

[0078] For example, the holes can be in the form of a grid, an array, or a QR code, and the reference pattern in the image captured by the camera 30 can be the corresponding grid, array pattern, or QR code.

[0079] In some embodiments, the optical element 10 includes a holographic reflective film, which forms a reference structure 10' through mask exposure. It is understood that the holographic reflective film can be obtained by mask exposure of the holographic medium layer 11 using an infrared laser, resulting in an interference hologram recorded on the holographic reflective film. When infrared light reflected from the eye 100 illuminates the holographic reflective film, the film reflects at least a portion of the infrared light to the camera 30, so that the image captured by the camera 30 includes the eye pattern. Furthermore, after at least a portion of the light is reflected to the reference structure and diffracted, it is transmitted to the camera, so that the image captured by the camera 30 includes the reconstructed reference pattern. In other words, the holographic reflective film's structure can simultaneously reflect infrared light and form a reference pattern on the camera 30, which is beneficial for controlling the volume of the optical element 10. Moreover, the mask exposure method for preparing the holographic reflective film is relatively convenient, and the resulting holographic reflective film has minimal impact on visible light, which is beneficial for ensuring the display effect of the near-eye display device.

[0080] like Figure 6 and Figure 7 As shown, in an optional embodiment, the method for preparing the holographic reflective film includes the following steps: a reflective layer 12 is provided on one side of the holographic medium layer 11, and a mask 13 is provided on the other side. The mask 13 has a light-shielding area 131 corresponding to the reference pattern. A laser emitter (not shown) is provided in the camera preset area 10a on the side of the mask 13 away from the holographic medium layer 11. The laser emitter emits infrared laser towards the holographic medium layer 11. The infrared laser is reflected by the reflective layer 12 to expose the holographic medium layer 11, so that the holographic medium layer 11 forms a reference structure 10', thereby obtaining the holographic reflective film.

[0081] Understandably, the optical element 10 includes a holographic reflective film, which is obtained by exposing the holographic medium layer 11. The mask 13 disposed on one side of the holographic medium layer 11 has a light-shielding area 131 corresponding to the reference pattern, in which infrared laser is blocked and cannot be transmitted to the holographic medium layer 11, while the other areas outside the light-shielding area 131 are light-transmitting areas, through which infrared laser can be transmitted to the holographic medium layer 11. When the laser emitter in the preset area 10a of the camera emits infrared laser light into the holographic medium layer 11, part of the infrared laser light is blocked by the light-shielding area 131, so that the position of the holographic medium layer 11 corresponding to the light-shielding area 131 cannot record a hologram and does not have a reflection function. The other part of the infrared laser light can be transmitted through the light-transmitting area of ​​the mask plate 13 to the holographic medium layer 11 and the reflective layer 12 and can be reflected by the reflective layer 12. The infrared laser light emitted by the laser emitter to the holographic medium layer 11 and the infrared laser light reflected by the reflective layer 12 form interference in the holographic medium layer 11, so that the position of the holographic medium layer 11 outside the light-shielding area 131 records a hologram and has a reflection function, realizing the exposure of the holographic medium layer 11 and obtaining a holographic reflective film. When eye tracking is performed using the holographic reflective film obtained by exposing the holographic medium layer 11, the camera 30 and the infrared light source 20, because some positions of the holographic reflective film do not have a reflection function, they cannot reflect infrared light to the camera 30, so the image acquired by the camera 30 includes the reference pattern.

[0082] The camera preset area 10a is the pre-set position of the camera 30 relative to the holographic reflective film in the eye-tracking device. Specifically, the laser emitter emits infrared laser light from the entrance pupil position of the camera 30 to ensure that more of the infrared light reflected by the prepared holographic reflective film is collected by the camera 30, thus ensuring the integrity of the image collected by the camera 30.

[0083] For example, the infrared laser emitted by the laser emitter is reflected by the reflective layer 12 and passes through the preset area 10b of the eye, which is the position of the eye 100 relative to the holographic reflective film when the eye-tracking device is working.

[0084] For example, the light-shielding area 131 may be in the form of a grid, a QR code, or something similar. Figure 8 The array shown.

[0085] For example, such as Figure 7 As shown, the reflective layer 12 is attached to one side of the holographic medium layer 11, and the mask is attached to the other side of the holographic medium layer 11 to ensure that the laser can expose the holographic medium layer 11 more accurately, improve the accuracy of the hologram that the holographic medium layer 11 can form, and thus improve the accuracy of eye tracking of the eye tracking device.

[0086] Optionally, a holographic reflective film can be prepared first, and then the holographic reflective film can be placed on the lens 60. Alternatively, the holographic medium layer 11 can be directly placed on the lens 60, and then the holographic medium layer 11 can be exposed using the reflective layer 12, the mask plate 13, and the laser emitter to obtain the holographic reflective film.

[0087] like Figure 8 As shown, in another optional embodiment, the method for preparing the holographic reflective film includes the following steps: a reflective layer 12 is provided on one side of the holographic medium layer 11, and a mask 13 is provided in the eye preset area 10b on the other side of the holographic medium layer 11. The mask 13 has a reflective area corresponding to the reference pattern; a laser emitter is provided in the camera preset area 10a on the other side of the holographic medium layer 11. The laser emitter emits infrared laser towards the holographic medium layer 11. The infrared laser is reflected by the reflective layer 12 to the mask 13. Part of the infrared laser is reflected by the reflective area to the reflective layer 12, and then reflected again by the reflective layer 12 to the camera preset area 10a to expose the holographic medium layer 11, so that the holographic medium layer 11 forms a reference structure 10', and a holographic reflective film is obtained.

[0088] Understandably, the optical element 10 includes a holographic reflective film, which is obtained by exposing the holographic medium layer 11. A mask 13, positioned in the preset eye region 10b, has a reflective area corresponding to the reference pattern. This reflective area reflects infrared laser light, while areas outside the reflective area reflect less or no infrared laser light. The laser emitter in the preset camera region 10a emits infrared laser light into the holographic medium layer 11. After passing through the holographic medium layer 11, the infrared laser light is transmitted to the reflective layer 12 and reflected back to the mask 13. Only a portion of the infrared light is reflected back to the holographic medium layer 11 by the reflective area, and interferes with the infrared laser light emitted from the laser emitter and reflected by the reflective layer 12 within the holographic medium layer 11. This causes the holographic medium layer 11 to record a hologram related to the reference pattern and to possess reflective functionality, thus achieving the exposure of the holographic medium layer 11 and obtaining the holographic reflective film. When eye tracking is performed using the holographic reflective film obtained by exposing the holographic dielectric layer 11, the camera 30, and the infrared light source 20, the holographic reflective film can reproduce the hologram under infrared light illumination, so that the pattern acquired by the camera 30 includes the reference pattern. At the same time, the infrared light reflected by the eye 100 is acquired by the camera 30, so that the acquired pattern also includes the eye pattern. It is worth noting that when eye tracking is performed using the holographic reflective film obtained by this preparation method, when the camera 30 acquires the image, the virtual image corresponding to the eye pattern and the virtual image corresponding to the reference pattern can be located at the same or similar object distance relative to the camera 30. The camera 30 can simultaneously focus clearly on the eye pattern and the reference pattern, so that both the acquired eye pattern and the reference pattern are relatively clear, so as to determine the gaze direction of the eye 100 based on the eye pattern and the reference pattern, thereby improving the accuracy of eye tracking.

[0089] For example, the reflective area of ​​the mask can be a high-reflection / scattering area, capable of reflecting / scattering infrared lasers, while other areas outside the reflective area are low-reflection / scattering areas, reflecting / scattering less or no infrared lasers.

[0090] For example, the reflective area of ​​the mask 13 may be in the form of a grid, an array, or a QR code.

[0091] In some embodiments, the processing module is configured to: determine the coordinates of the pupil and / or iris in the reference pattern based on the image, and use the coordinates as the coordinates to be matched; match the coordinates to be matched with a pre-stored coordinate database, and determine the gaze direction corresponding to the matched pre-stored coordinates as the gaze direction of the current eye 100.

[0092] Understandably, as mentioned above, when the user's eye 100 observes along different viewing directions, the position of the pupil and / or iris in the eye pattern is different in the reference pattern. The processing module can be equipped with a pre-stored coordinate database, which contains multiple pre-stored coordinates. The pre-stored coordinates represent the coordinates of the pupil and / or iris in the eye pattern in the reference pattern, and each pre-stored coordinate has a corresponding viewing direction. When the eye-tracking device performs eye tracking, it can determine the coordinates of the pupil and / or iris in the current eye pattern in the reference pattern based on the image captured by the camera 30. At this time, the coordinates can be determined as the coordinates to be matched, and the coordinates to be matched are matched with multiple pre-stored coordinates in the pre-stored coordinate database. If the match is successful, the viewing direction corresponding to the matched pre-stored coordinates can be used as the current viewing direction of the eye 100 to realize eye tracking, so that the near-eye display device can be controlled accordingly based on the viewing direction of the eye 100.

[0093] For example, the coordinates of the pupil center in the eye pattern within a reference pattern are determined from the image to improve the accuracy of eye tracking.

[0094] For example, the absolute value of the difference between a pre-stored coordinate that matches the coordinate to be matched and the coordinate to be matched is less than the absolute value of the difference between other pre-stored coordinates and the coordinate to be matched.

[0095] Furthermore, the processing module matches the coordinates to be matched with a pre-stored coordinate database. When the gaze direction corresponding to the matched pre-stored coordinate is determined as the gaze direction of the current eye 100, the following steps are taken: when the absolute value of the difference between the coordinates to be matched and a pre-stored coordinate is less than or equal to a preset threshold, the coordinates to be matched are matched with the pre-stored coordinate, and the gaze direction corresponding to the pre-stored coordinate is determined as the gaze direction of the current eye 100; when the absolute value of the difference between the coordinates to be matched and any pre-stored coordinate in the pre-stored coordinate database is greater than the preset threshold, interpolation processing is performed on the gaze direction corresponding to the pre-stored coordinate in the pre-stored coordinate database, and the interpolated gaze direction that matches the coordinates to be matched is determined as the gaze direction of the current eye 100. It can be understood that when the absolute value of the difference between the coordinates to be matched and any pre-stored coordinate is greater than the preset threshold, that is, when the distance between the coordinates to be matched and any pre-stored coordinate is large, interpolation processing is performed on the gaze direction corresponding to the pre-stored coordinate to obtain an interpolated gaze direction corresponding to a coordinate closer to the current coordinates to be matched, and the interpolated gaze direction is determined as the gaze direction of the current eye 100, thereby improving the accuracy of eye tracking.

[0096] In some embodiments, the processing module is further configured to: determine the coordinates of the pupil and / or iris in the reference pattern based on the image when the eye 100 observes along a specified line of sight, and store the coordinates as pre-stored coordinates in a pre-stored coordinate database. It is understood that when the eye-tracking device is initially used, the user's eye 100 can be guided to observe along a specified line of sight, and the camera 30 can capture images. After the processing module acquires the images captured by the camera 30, it determines the coordinates of the pupil and / or iris in the reference pattern based on the images. These determined coordinates are pre-stored coordinates, which are stored in the pre-stored coordinate database, and the pre-stored coordinates correspond to the specified line of sight, thereby achieving the calibration of the line of sight direction. When eye tracking is performed using an eye-tracking device in the future, the coordinates of the pupil and / or iris in the reference pattern can be determined based on the re-acquired image when the user's eye 100 observes along an unknown line of sight. The coordinates determined at this time are the coordinates to be matched. By matching the coordinates to be matched with the pre-stored coordinates, the line of sight corresponding to the pre-stored coordinates that match the coordinates to be matched is determined as the current line of sight of the eye 100.

[0097] For example, taking an eye-tracking device applied to a near-eye display device, when a user initially uses the near-eye display device, the device can display a virtual image. By guiding the user's eye 100 to observe a designated position in the virtual image, the user's eye 100 is guided to observe along a designated line of sight, thereby calibrating the line of sight direction. For example, the virtual image can be a dot matrix, a square array, a pentagram array, etc. Taking a dot matrix virtual image as an example, the color / size of a designated point in the dot matrix can be distinguished from the color / size of other points, or the designated point can be made to flash while other points remain continuously displayed, thus distinguishing the designated point from the other points and guiding the user to focus on the designated point, so that the user observes along the designated line of sight. Figure 9 As shown, Figure 9 The image shows the gaze direction of the eye 100 when observing a designated point in a virtual image, and the image captured by the camera, including the eye pattern and the reference pattern. When the eye 100 observes a designated point in the virtual image, it observes along the designated gaze direction. Simultaneously, the infrared light source 20 emits infrared light, enabling the camera 30 to capture an image including the eye pattern and the reference pattern. Based on the image, the coordinates of the pupil and / or iris of the eye pattern in the image within the reference pattern when the user observes along the designated gaze direction can be determined. These coordinates are then stored as pre-stored coordinates in a pre-stored coordinate database. By guiding the user to sequentially gaze at all designated points in the dot matrix, all pre-stored coordinates are obtained to complete the calibration of the gaze direction.

[0098] The camera 30 can continuously acquire multiple frames of images to obtain the most stable image of the user's eye position 100, and use the image to calibrate the specified gaze direction, thereby improving the accuracy of calibration and optimizing the accuracy of eye tracking of the eye tracking device.

[0099] This invention also provides a near-eye display device, including the eye-tracking device as described above. It is understood that the near-eye display device, including the eye-tracking device as described above, also possesses all its technical advantages, namely, it avoids obstructing the user's view while allowing for more flexible installation positions of the infrared light source and camera, facilitating adjustments to the installation positions of the infrared light source and camera as needed to control the size of the near-eye display device.

[0100] This invention also provides an eye-tracking method for an eye-tracking device. The eye-tracking device includes an infrared light source, an optical element, and a camera. The infrared light source emits infrared light, which is reflected by the eye to the optical element. The optical element transmits visible light and reflects the infrared light so that the camera can capture an image including an eye pattern and a reference pattern. The eye-tracking method includes steps S10 to S30.

[0101] S10. Acquire the image captured by the camera.

[0102] S20. Determine the coordinates of the pupil and / or iris in the reference pattern based on the image, and use the coordinates as the coordinates to be matched.

[0103] S30. Match the coordinates to be matched with the pre-stored coordinate database, and determine the gaze direction corresponding to the matching pre-stored coordinates as the current gaze direction of the eye.

[0104] The infrared light reflected from the eye is reflected back to the camera by optical elements, so that the camera can acquire an image including the eye pattern and a reference pattern, and then determine the direction of the eye's gaze based on the coordinates of the pupil and / or iris in the reference pattern in the image.

[0105] The method of this application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0106] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method implemented can be referred to in various embodiments of the eye-tracking method of the near-eye display device of this application.

[0107] The computer-readable storage medium can be an internal storage unit of the near-eye display device described in the foregoing embodiments, such as the hard disk or memory of the near-eye display device. Alternatively, the computer-readable storage medium can be an external storage device of the near-eye display device, such as a plug-in hard disk, SmartMediaCard (SMC), SecureDigital (SD) card, or FlashCard equipped on the near-eye display device.

[0108] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware embodiments, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0109] It should be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0110] The sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The above descriptions are only specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An optical element, characterized in that, A reference structure is formed on the optical element; The optical element is capable of transmitting visible light, and at least a portion of the infrared light reflected from the eye is reflected to the camera via the optical element, so that the image captured by the camera includes an eye pattern, and at least a portion of the infrared light reflected from the eye illuminates the reference structure, so that the image captured by the camera also includes a reference pattern, thereby enabling the direction of the eye's gaze to be determined based on the coordinates of the pupil and / or iris in the reference pattern.

2. The optical element according to claim 1, characterized in that, The optical element includes a non-reference structure and the reference structure, wherein the reflectivity of the non-reference structure is greater than the reflectivity of the reference structure; The infrared light reflected from the eye is at least partially reflected to the camera via the non-reference structure, so that the camera can capture the eye pattern.

3. The optical element according to claim 2, characterized in that, The optical element includes a reflective film layer and a functional layer. The functional layer is disposed on one side of the reflective film layer and has a light-shielding area corresponding to the reference pattern. The portion of the reflective film layer corresponding to the light-shielding area and the light-shielding area together form the reference structure; or The optical element includes a reflective film, and the reference structure is a hole formed in the reflective film.

4. The optical element according to claim 1, characterized in that, The optical element includes a holographic reflective film, which forms the reference structure by exposure through a mask.

5. The optical element according to any one of claims 1-4, characterized in that, The reference pattern includes at least one of the following: a grid, an array pattern, or a QR code.

6. A method for preparing a holographic reflective film for the optical element according to claim 4, characterized in that, Includes the following steps: A reflective layer is provided on one side of the holographic medium layer, and a mask is provided on the other side. The mask has a light-shielding area corresponding to the reference pattern. A laser emitter is placed in a camera preset area on the side of the mask away from the holographic medium layer. The laser emitter emits infrared laser towards the holographic medium layer. The infrared laser is reflected by the reflective layer to expose the holographic medium layer, so that the holographic medium layer forms the reference structure and the holographic reflective film is obtained.

7. A method for preparing a holographic reflective film for the optical element according to claim 4, characterized in that, Includes the following steps: A reflective layer is provided on one side of the holographic medium layer, and a mask is provided in the eye preset area on the other side of the holographic medium layer. The mask has a reflective area corresponding to the reference pattern. A laser emitter is set in the camera preset area on the other side of the holographic medium layer. The laser emitter emits infrared laser towards the holographic medium layer. The infrared laser is reflected by the reflective layer to the mask. Part of the infrared laser is reflected by the reflective area to the reflective layer, and then reflected again by the reflective layer to the camera preset area to expose the holographic medium layer, so that the holographic medium layer forms the reference structure and the holographic reflective film is obtained.

8. An eye-tracking device, characterized in that, include: Infrared light source, used to emit infrared light; An optical element, wherein the optical element is the optical element as described in any one of claims 1 to 5, wherein the infrared light is reflected by the eye to the optical element, and the optical element is capable of transmitting visible light; A camera, wherein at least a portion of the infrared light reflected from the eye is reflected to the camera via the optical element, such that the image captured by the camera includes an eye pattern, and at least a portion of the infrared light reflected from the eye illuminates the reference structure, such that the image captured by the camera also includes a reference pattern; as well as A processing module, electrically connected to the camera, is used to determine the gaze direction of the eye based on the eye pattern and the reference pattern.

9. The eye-tracking device according to claim 8, characterized in that, The processing module is used for: Based on the image, determine the coordinates of the pupil and / or iris in the reference pattern of the eye pattern, and use the coordinates as the coordinates to be matched; The coordinates to be matched are matched with the pre-stored coordinate database, and the gaze direction corresponding to the matching pre-stored coordinates is determined as the current gaze direction of the eye.

10. The eye-tracking device according to claim 9, characterized in that, The processing module matches the coordinates to be matched with the pre-stored coordinate database. When the gaze direction corresponding to the matching pre-stored coordinates is determined as the current gaze direction of the eye, it is used for: When the absolute value of the difference between the coordinate to be matched and a pre-stored coordinate is less than or equal to a preset threshold, the coordinate to be matched is matched with the pre-stored coordinate, and the gaze direction corresponding to the pre-stored coordinate is determined as the current gaze direction of the eye. When the absolute value of the difference between the coordinate to be matched and any of the pre-stored coordinates in the pre-stored coordinate database is greater than the preset threshold, the gaze direction corresponding to the pre-stored coordinate in the pre-stored coordinate database is interpolated, and the interpolated gaze direction that matches the coordinate to be matched is determined as the current gaze direction of the eye.

11. The eye-tracking device according to claim 9, characterized in that, The processing module is also used for: Based on the image observed by the eye along a specified line of sight, determine the coordinates of the pupil and / or iris in the reference pattern of the eye pattern, and store the coordinates as the pre-stored coordinates in the pre-stored coordinate database.

12. The eye-tracking device according to claim 8, characterized in that, The eye-tracking device includes a frame, a lens disposed on the frame, and a wearable component connected to the frame. The optical element is disposed on the lens, the camera is disposed on the wearable component, and the infrared light source is disposed on the frame or the wearable component.

13. A near-eye display device, characterized in that, Including the eye-tracking device as described in any one of claims 8-12.

Images