Eye tracking display apparatuses, methods and near-eye display devices

Abstract

The present application provides an eye tracking display device and its method and near-eye display equipment, which can solve the problem of not seeing images or image ghosting caused by small eye sockets. The eye tracking display device comprises: a laser scanning projection component for projecting image light with variable image position according to an encoded input image source; a holographic optical element for redirecting and reflecting the image light to image on the retina, and inversely reflecting the image light reflected back via the retina; and an eye tracking component, which comprises a photosensitive element and a light splitting element arranged on the photosensitive side of the photosensitive element. The light splitting element is arranged in the light path between the laser scanning projection component and the holographic optical element, for reflecting the projected image light to propagate to the holographic optical element, and transmitting the inversely reflected image light to propagate to the photosensitive element; and the photosensitive element is used to capture the image light transmitted through the light splitting element to track the eye pupil posture.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of near-eye display, in particular to an eye movement tracking display device, a method thereof and a near-eye display device. BACKGROUND

[0002] In recent years, the related market of near-eye display devices (such as AR or VR glasses) is becoming increasingly active, and the retinal projection imaging scheme has attracted widespread attention because it can achieve near-eye display in the form of near-eye glasses only by a simple projection light path. For example, the existing retinal projection imaging scheme usually first projects an image by cooperating a laser diode (LD) and a micro-electromechanical mirror (MEMS mirror), that is, by laser beam scanning (LBS), and then redirects the image through the function of a holographic optical element (HOE) to make the image re-converge at the pupil of the human eye and directly project onto the retina. However, the existing retinal projection imaging scheme can only see the image at a specific exit pupil position, that is, the eye socket is very small, so it is necessary to develop a practical solution to solve the problem of a small eye socket.

[0003] At present, the mainstream solution to the problem of a small eye socket in the retinal projection imaging scheme is usually divided into two types: one is the pupil replication method, that is, to realize the pupil array through the HOE to realize the expansion of the eye socket; the other is the pupil deflection method, that is, to introduce an eye movement tracking module composed of an additional infrared light source (IR LED) and a position detector (PSD), to obtain real-time eye pupil posture information through a pair of fast mirrors combined with scanning, and to realize pupil deflection through the HOE, which can also solve the problem of a small eye socket.

[0004] However, the above-mentioned pupil replication method requires a high requirement for the production of the HOE due to the need to obtain the pupil array, and when the size of the eye pupil changes due to changes in environmental brightness, the phenomenon of being located between two pupils or crossing two pupils will occur, resulting in the problem of not being able to see the image or image ghosting. The above-mentioned pupil deflection method can display a single point image in real time and does not have the above-mentioned problem, but the eye movement tracking module introduces a scanning mechanism and an additional light source, which increases the complexity of the near-eye display light path, resulting in a large size, heavy weight and high cost of the near-eye display device, which is not conducive to the promotion and popularization of the near-eye display device. SUMMARY

[0005] An advantage of the present application is to provide an eye movement tracking display device, a method thereof and a near-eye display device, which can solve the problem of not being able to see the image or image ghosting caused by a small eye socket, and greatly improve the experience of near-eye display.

[0006] Another advantage of the present application is to provide an eye tracking display device, method thereof and near-eye display device, wherein in one embodiment of the present application, the eye tracking display device is capable of achieving real-time tracking of eye pupil posture while achieving pupil deflection consistent with the direction of eye pupil fixation point.

[0007] Another advantage of the present application is to provide an eye tracking display device, method thereof and near-eye display device, wherein in one embodiment of the present application, the eye tracking display device is capable of using a single-pupil HOE, greatly reducing the difficulty of production and helping to improve energy utilization.

[0008] Another advantage of the present application is to provide an eye tracking display device, method thereof and near-eye display device, wherein in one embodiment of the present application, the eye tracking display device is capable of achieving real-time tracking of eye pupil posture without the need for additional scanning mechanisms and additional infrared light sources, helping to reduce the complexity of the eye tracking optical path and reduce costs.

[0009] Another advantage of the present application is to provide an eye tracking display device, method thereof and near-eye display device, wherein in one embodiment of the present application, the eye tracking display device is capable of eye tracking based on image local feature matching or spot centroid detection, with simple algorithm implementation and high precision, without the need for eye pupil imaging.

[0010] Another advantage of the present application is to provide an eye tracking display device, method thereof and near-eye display device, wherein in order to achieve the above-mentioned purposes, the present application does not need to use expensive materials or complex structures. Therefore, the present application successfully and effectively provides a solution, not only providing a simple eye tracking display device, method thereof and near-eye display device, but also increasing the practicability and reliability of the eye tracking display device, method thereof and near-eye display device.

[0011] In order to achieve the above-mentioned at least one advantage or other advantages and purposes of the present application, the present application provides an eye tracking display device for tracking eye pupil posture for near-eye display, comprising:

[0012] A laser scanning projection assembly for projecting image light with variable image positions according to an encoded input image source;

[0013] A holographic optical element for redirecting and reflecting the image light to image on the retina, and reversely reflecting the image light reflected back via the retina; and

[0014] An eye tracking assembly includes a light sensing element and a light splitting element disposed on a light sensing side of the light sensing element, the light splitting element is disposed in an optical path between the laser scanning projection assembly and the holographic optical element for reflecting image light projected via the laser scanning projection assembly to propagate to the holographic optical element, and for transmitting image light retro-reflected via the holographic optical element back to propagate to the light sensing element; the light sensing element is configured to capture the image light transmitted through the light splitting element to track eye pupil pose.

[0015] According to an embodiment of the present application, the laser scanning projection assembly includes a laser emitter configured to emit laser beams corresponding to image pixels, a micro-electro-mechanical mirror disposed on an emission side of the laser emitter and configured to scan the laser beams emitted via the laser emitter to form image light propagating to a collimating lens, and the collimating lens disposed in an optical path between the micro-electro-mechanical mirror and the light splitting element and configured to collimate the image light from the micro-electro-mechanical mirror to propagate to the light splitting element.

[0016] According to an embodiment of the present application, the light sensing element is a light sensing chip configured to capture the image light retro-reflected through the light splitting element to match image center feature information to determine whether the gaze direction is consistent with the image center exit light ray.

[0017] According to an embodiment of the present application, the light sensing chip is a charge-coupled device or a complementary metal-oxide-semiconductor sensor.

[0018] According to an embodiment of the present application, the light sensing element includes a photodiode and a converging lens disposed in an optical path between the photodiode and the light splitting element, the converging lens is configured to converge the image light retro-reflected through the light splitting element to form a converging light spot on the photodiode; the photodiode is configured to capture the converging light spot to detect a light spot energy centroid to determine whether the gaze direction is consistent with the image center exit light ray.

[0019] According to an embodiment of the present application, the photodiode is selected from one of a PN junction photodiode, a PIN junction photodiode, a photomultiplier tube, and an avalanche photodiode.

[0020] According to an embodiment of the present application, the light splitting element is a beam splitter configured to reflect a portion of the light beams and transmit another portion of the light beams.

[0021] According to an embodiment of the present application, an effective diameter of a reference light of the holographic optical element is greater than an effective diameter of a reconstruction light when the holographic optical element is manufactured.

[0022] According to one embodiment of the present application, the holographic optical element adopts an object light defocus design, and an effective size D of the holographic optical element is equal to 2*F*tan(u / 2), where a converging focal length F of signal light of the holographic optical element is equal to RD+r, an aperture angle u of the signal light of the holographic optical element is equal to 2*(θ+FoV / 2), where RD is an exit pupil distance, r is an eyeball radius, θ is an eyeball rotation angle, and FoV is an aperture angle of the object light.

[0023] According to another aspect of the present application, the present application further provides a near-eye display device, comprising:

[0024] a device body; and

[0025] The eye movement tracking display device according to any one of the above, which is loaded in the device body and used for tracking an eye pupil posture to perform near-eye display.

[0026] According to another aspect of the present application, the present application further provides an eye movement tracking display method, comprising the steps of:

[0027] projecting image light with a variable image position according to an encoded input image source;

[0028] reflecting the projected image light to be redirected and reflected to form an image on a retina;

[0029] reversely reflecting image light reflected by the retina; and

[0030] transmitting the reversely reflected image light to be captured to track an eye pupil posture.

[0031] According to one embodiment of the present application, the step of transmitting the reversely reflected image light to be captured to track an eye pupil posture comprises the steps of:

[0032] capturing the transmitted image light to obtain image information;

[0033] matching the obtained image information with central feature information of the input image source by an image local feature recognition matching manner to obtain a pixel offset of a captured image center position;

[0034] judging whether the pixel offset of the captured image center position and a central pixel offset of the input image source after encoding satisfy a theoretical mapping relationship;

[0035] in response to not satisfying the theoretical mapping relationship, adjusting encoding information of the input image source to repeat the above steps; and

[0036] in response to satisfying the theoretical mapping relationship, completing eye movement tracking display.

[0037] According to one embodiment of the present application, the step of tracking the eye pupil pose by capturing the transmitted retro-reflected image light comprises steps of:

[0038] converging the transmitted image light to form a converging light spot;

[0039] detecting the center of the captured converging light spot by a light spot energy center detection method to obtain the center position of the converging light spot;

[0040] judging whether the center position of the converging light spot and the center pixel offset of the encoded input image source satisfy a theoretical mapping relationship;

[0041] in response to not satisfying the theoretical mapping relationship, adjusting the encoding information of the input image source to repeat the above steps; and

[0042] in response to satisfying the theoretical mapping relationship, completing the eye movement tracking display. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a block diagram of an eye movement tracking display device according to one embodiment of the present application;

[0044] Figure 2 a first example of an eye movement tracking display device according to the above embodiment of the present application is shown;

[0045] Figure 3 an image source encoding schematic diagram of a laser scanning projection component in the eye movement tracking display device according to the above first example of the present application is shown;

[0046] Figure 4 a schematic diagram of a theoretical mapping relationship in the eye movement tracking display device according to the above first example of the present application is shown;

[0047] Figure 5 a process schematic diagram of tracking the eye pupil pose in the eye movement tracking display device according to the above first example of the present application is shown;

[0048] Figure 6 a second example of an eye movement tracking display device according to the above embodiment of the present application is shown;

[0049] Figure 7 a principle schematic diagram of light spot center detection in the eye movement tracking display device according to the above second example of the present application is shown;

[0050] Figure 8A a light path schematic diagram of focusing object light on an eye pupil by a holographic optical element is shown;

[0051] Figure 8BA schematic diagram showing the principle of the holographic optical element using an object light defocus design is shown.

[0052] Figure 8C A schematic diagram showing the optical path of the eye tracking display device according to the above embodiment of the present application when used in defocus is shown.

[0053] Figure 9 A schematic diagram showing the flow of the eye tracking display method according to an embodiment of the present application is shown.

[0054] Figure 10 An example of the eye pupil pose tracking step in the eye tracking display method according to the above embodiment of the present application is shown.

[0055] Figure 11 Another example of the eye pupil pose tracking step in the eye tracking display method according to the above embodiment of the present application is shown.

[0056] Main element symbol explanation: 1, eye tracking display device; 10, laser scanning projection assembly; 11, laser emitter; 12, micro-electromechanical mirror; 13, collimating lens; 20, holographic optical element; 30, eye tracking assembly; 31, light sensing element; 311, light sensing chip; 312, photodiode; 313, converging lens; 32, light splitting element; 320, beam splitter; 2, device main body.

[0057] The above main element symbol explanation further details the present application in combination with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0058] The following description is provided to enable those skilled in the art to implement the present application. The preferred embodiments in the following description are only examples of the present application, and other obvious modifications can be made by those skilled in the art. The basic principles of the present application defined in the following description can be applied to other embodiments, modifications, improvements, equivalents and other technical solutions without departing from the spirit and scope of the present application.

[0059] Those skilled in the art should understand that in the disclosure of the present application, the terms "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the above terms cannot be understood as a limitation of the present application.

[0060] In the present application, the term "a" should be understood as "one or more" in the claims and the specification, i.e. in one embodiment the number of elements can be one, and in other embodiments the number of elements can be more than one. The term "a" should not be understood as meaning only one or singular unless it is explicitly stated that the number of elements is only one or singular in the context of the description of the application. The term "a" should not be understood as limiting the number of elements.

[0061] In the description of the present application, it should be understood that "first", "second" and the like are only used for the purpose of description and should not be understood as indicating or implying relative importance. In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, "connected", "connected" should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through a medium. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0062] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0063] Considering that the existing pupil replication method puts forward higher requirements for the manufacture of HOE, and there is a problem of not being able to see the image or image ghosting; while the existing pupil deflection method needs to introduce a scanning mechanism and an additional light source, which increases the complexity of the near-eye display light path, resulting in a large size, heavy weight, low performance of the near-eye display device, which is not conducive to the promotion and popularization of the near-eye display device. Therefore, the present application creatively proposes an eye movement tracking display device and a method thereof and a near-eye display device, which can solve the problem of not being able to see the image or image ghosting caused by the small eye socket, greatly improving the experience of near-eye display.

[0064] Specifically, referring to the drawings of the specification of the present application Figures 1 to 7 According to an embodiment of the present application, an eye movement tracking display device 1 is provided for tracking eye pupil posture for near-eye display.

[0065] More specifically, as Figure 1 andFigure 2 As shown, the eye tracking display device 1 can include a laser scanning projection component 10, a holographic optical element 20, and an eye tracking component 30. The laser scanning projection component 10 is configured to project image light with variable image positions according to an encoded image source. The holographic optical element 20 is configured to redirect the image light to be imaged on the retina and retro-reflect the image light reflected back via the retina. The eye tracking component 30 includes a photosensitive element 31 and a light splitting element 32 disposed on the photosensitive side of the photosensitive element 31. The light splitting element 32 is disposed in the optical path between the laser scanning projection component 10 and the holographic optical element 20 and is configured to reflect the image light projected via the laser scanning projection component 10 to propagate to the holographic optical element 20 and to transmit the image light retro-reflected back via the holographic optical element 20 to propagate to the photosensitive element 31. The photosensitive element 31 is configured to capture the image light transmitted through the light splitting element 32 to track the eye pupil pose.

[0066] Notably, when the encoding information of the image source input to the laser scanning projection component 10 is changed, the position of the image projected by the laser scanning projection component 10 is also changed accordingly, and the image light redirected by the holographic optical element 20 will propagate to different positions of the human eye. At this time, if the gaze direction of the human eye is consistent with the principal ray angle of the image center, i.e., the image light redirected by the holographic optical element 20 just passes through the eye pupil to be imaged on the retina to achieve better near-eye display, the image light reflected by the retina will be retro-reflected by the holographic optical element 20 to return to the light splitting element 32 along the original path, so as to be captured by the photosensitive element 31 through the light splitting element 32, so that the captured image position obtained by the photosensitive element 31 and the encoded image position input to the laser scanning projection component 10 satisfy a predetermined mapping relationship. If the gaze direction of the human eye is not consistent with the principal ray angle of the image center, i.e., the image light redirected by the holographic optical element 20 cannot pass through the eye pupil or partially pass through the eye pupil to be imaged on the retina, the image light reflected by the retina will not return to the light splitting element 32 along the original path, resulting in that the captured image position obtained by the photosensitive element 31 and the encoded image position input to the laser scanning projection component 10 cannot satisfy the theoretical mapping relationship.

[0067] Therefore, the eye movement tracking display device 1 of the present application can determine whether the gaze direction of the human eye is consistent with the principal ray angle of the image center by judging whether the captured image position obtained via the photosensitive element 31 and the encoded image position input to the laser scanning projection assembly 10 satisfy the predetermined mapping relationship. In particular, when the captured image position obtained via the photosensitive element 31 and the encoded image position input to the laser scanning projection assembly 10 cannot satisfy the predetermined mapping relationship, the eye movement tracking display device 1 of the present application can change the encoded information of the image source input to the laser scanning projection assembly 10 to change the position of the image scanned and projected by the laser scanning projection assembly 10, so that the captured image position obtained via the photosensitive element 31 and the encoded image position input to the laser scanning projection assembly 10 satisfy the theoretical mapping relationship, thereby realizing tracking of the eye pupil posture and ensuring that the eye movement tracking display device 1 can adapt to near-eye display under eye rotation, completely solving the problem of not being able to see the image or image ghosting caused by the small eye socket, and greatly improving the experience of near-eye display.

[0068] According to the above embodiments of the present application, as shown in Figure 1 , Figure 2 and Figure 3 , the laser scanning projection assembly 10 can include a laser emitter 11, a micro-electro-mechanical scanning mirror 12 disposed on the emission side of the laser emitter 11, and a collimating lens 13 disposed on the reflection side of the micro-electro-mechanical scanning mirror 12, the laser emitter 11 being configured to emit a laser beam corresponding to an image pixel; the micro-electro-mechanical scanning mirror 12 being configured to scan the laser beam emitted via the laser emitter 11 to form image light propagating to the collimating lens 13; and the collimating lens 13 being disposed in the optical path between the micro-electro-mechanical scanning mirror 12 and the light splitting element 32 of the eye movement tracking assembly 30, and being configured to collimate the image light from the micro-electro-mechanical scanning mirror 12 to propagate to the light splitting element 32. It can be understood that the laser emitter 11 can be implemented as a laser diode, but is not limited thereto.

[0069] Notably, as shown in Figure 3 and Figure 5 , the laser scanning projection assembly 10 of the present application can realize a change in the image center position by inputting the encoding of the image source during the process of scanning to generate an image; and the center projection ray angle corresponding to the image with a changed center position after scanning by the micro-electro-mechanical scanning mirror 12 will also change, and satisfy a certain pixel offset and projection angle mapping relationship, denoted as (δ pixel , θ1), where δ pixelThe center pixel offset of the image after encoding is θ1, which is the projection angle of the image center projection light after scanning by the MEMS mirror 12. After collimation by the collimating lens 13, the image center offset δ1 = f*tan θ1, where f is the focal length of the collimating lens 13. At this time, the image center offset and the projection angle satisfy a corresponding mapping relationship, denoted as (θ1, δ1).

[0070] For example, in the first example of the present application, as shown in Figure 2 The light sensing element 31 of the eye tracking assembly 30 can be implemented as a light sensing chip 311, which is used to capture the image light returned in reverse through the light splitting element 32 to obtain image information, and then perform image center feature matching to determine whether the gaze direction of the human eye is consistent with the image center exit principal ray. Optionally, the light sensing chip 311 of the present application can be implemented as a charge-coupled device (CDD) or a complementary metal-oxide-semiconductor sensor (CMOS), but is not limited thereto.

[0071] At this time, as shown in Figure 2 and Figure 5 With the change of the projection angle of the image center projection light after scanning by the MEMS mirror 12, the image center position projected on the holographic optical element 20 by the light splitting element 32 also changes, and there is a mapping relationship between the image center position offset δ2 on the holographic optical element 20 and the image center offset δ1 after collimation by the collimating lens 13, denoted as (δ1, δ2). Further, the exit angle of the principal ray redirected by the holographic optical element 20 also changes, and there is a mapping relationship between the exit angle θ2 of the principal ray redirected by the holographic optical element 20 and the image center position offset δ2 on the holographic optical element 20, denoted as (δ2, θ2). Finally, the exit principal ray of the image center combined with the eye pupil rotation, when the gaze direction of the human eye is consistent with the exit principal ray of the image center, the image redirected by the holographic optical element 20 will be received by the human eye to be projected onto the retina. At the same time, the image light received on the retina will also be reflected by the retina to return in reverse along the same path, and then pass through the light splitting element 32 to be captured by the light sensing chip 311. There is a mapping relationship between the image center position offset δ3 captured by the light sensing chip 311 and the exit angle θ2 of the principal ray redirected by the holographic optical element 20, denoted as (θ2, δ3).

[0072] In summary, as shown in Figure 4As shown, when the exit angle of the chief ray of the image center is consistent with the gaze direction of the human eye, the angle of the eye pupil rotation is equal to the exit angle θ2 of the chief ray of the reflected light redirected via the holographic optical element 20, and at this time the offset δ3 of the image center position captured via the photosensitive chip 311 and the offset of the center pixel of the encoded image input into the laser scanning projection assembly 10 pixel satisfies the theoretical mapping relationship, denoted as (δ pixel , δ3), so that the eye movement tracking display device 1 of the present application can determine the pose position of the eye pupil based on the theoretical mapping relationship (δ pixel , δ3), and adjust the encoded information of the input image source according to the pose position of the eye pupil to realize eye movement tracking display.

[0073] It is worth noting that in the above-mentioned first example of the present application, the eye movement tracking display device 1 adopts an image local feature recognition matching method to track eye movement, that is, by matching the center feature information of the image captured via the photosensitive chip 311 with the input image source before encoding, the pixel offset δ3 of the image center position captured via the photosensitive chip 311 can be obtained. In other examples of the present application, the eye movement tracking display device 1 of the present application can also use a light spot energy centroid detection method to track eye movement.

[0074] For example, in the second example of the present application, as Figure 6 shown, the photosensitive element 31 of the eye movement tracking assembly 30 can include a photodiode 312 and a converging lens 313, the converging lens 313 being arranged in the light path between the photodiode 312 and the light splitting element 32, the converging lens 313 being used to converge the image light returned in reverse through the light splitting element 32 to form a converging light spot on the photodiode 312; the photodiode 312 is used to capture the converging light spot to detect the centroid of the light spot to determine whether the gaze direction of the human eye is consistent with the chief ray of the image center exit; in other words, the photodiode 312 is used to detect the centroid of the converging light spot to obtain the centroid position of the converging light spot, and then determine whether the gaze direction of the human eye is consistent with the chief ray of the image center exit. Optionally, the photodiode 312 can be implemented as a PN junction photodiode (abbreviated as PD), a PIN junction photodiode (abbreviated as PIN), a photomultiplier tube (abbreviated as PMT), or an avalanche photodiode (abbreviated as APD), etc., but is not limited thereto.

[0075] It is worth noting that the parameter mapping relationship (δ pixelThe parameter mapping relationship of δ1, θ1, δ2, θ2, δ3) is consistent with that of the first example above, except that in this second example of the application, δ3 is the offset of the principal ray of the image reflected by the retina and returning in the opposite direction along the same path as it passes through the beam splitter 32 to propagate to the converging lens 313; furthermore, as Figure 6 and Figure 7 As shown, the image light returning in reverse is converged by the converging lens 313 to form a converged spot, which is received by the photodiode 312. The photodiode 312 defocuses and detects the centroid position δ4 of the converged spot, where δ4 = (δ f / f)*δ3, where: f is the focal length of the converging lens 313; δ f The distance between the focal point of the photodiode 312 and the converging lens 313 is denoted as (δ3, δ4). In other words, there is a mapping relationship between the centroid position δ4 of the converging spot detected by the photodiode 312 and the offset δ3 of the principal ray of the image as it passes through the beam splitter 32 to the converging lens 313.

[0076] In summary, when the exit angle of the principal ray of the image is consistent with the gaze direction of the human eye, the angle of pupil rotation is equal to the exit angle θ2 of the principal ray reflected by the holographic optical element 20. At this time, the centroid position δ4 of the converging spot detected by the photodiode 312 is δ4 different from the center pixel offset of the coded image input to the laser scanning projection component 10. pixel Satisfying the theoretical mapping relationship, denoted as (δ pixel ,δ4), enabling the eye-tracking display device 1 of this application to be based on the theoretical mapping relationship (δ pixel The eye pose position is determined by δ4), and the encoding information of the input image source is changed according to the eye pose to achieve eye-tracking display.

[0077] According to the above embodiments of this application, as Figure 2 and Figure 6 As shown, the beam-splitting element 32 of the eye-tracking component 30 in the eye-tracking display device 1 can, but is not limited to, be implemented as a beam splitter 320, used to reflect a portion of the light beam and transmit another portion of the light beam. This allows a portion of the image light projected by the laser scanning projection component 10 to be reflected by the beam splitter 320 and propagated to the holographic optical element 20 for near-eye display. Simultaneously, a portion of the image light reflected back by the holographic optical element 20 is transmitted by the beam splitter 320 and propagated to the photosensitive element 31 for eye tracking. It is understood that the beam splitter 320 can be implemented as a half-reflective half-transparent mirror, used to reflect half of the light and transmit half of the light.

[0078] It is worth noting that after the eye movement tracking based on image local feature recognition matching or spot energy centroid detection to obtain the real-time pose information of the eye pupil, a corresponding HOE design is needed to better adapt to the retinal projection display under different eye pupil poses. Specifically, the effective diameter of the reference light of the holographic optical element 20 during manufacturing is larger than the effective diameter of the reconstruction light. In this way, when the projection position of the collimated light on the surface of the holographic optical element 20 changes correspondingly due to the change of the center position of the input image source, the eye movement tracking display device 1 can realize image display that adapts to eye movement. It can be understood that the effective diameter of the reconstruction light of the holographic optical optical element 20 is determined by the scanning angle of the micro-electro-mechanical mirror 12 and the focal length of the collimating lens 13.

[0079] In addition, as Figure 8A shown, if the eye movement tracking display device 1 is used in a way that the object light is focused on the eye pupil, when the eyeball rotates, the eye pupil often translates, so that after the eyeball deviates from the optical axis, there will be a problem of image fading or not being able to see the image. The holographic optical element 20 in the eye movement tracking display device 1 of the present application can be designed with object light defocus, so as to solve this problem by using object light defocus.

[0080] Exemplarily, as Figure 8B shown, the eye movement tracking display device 1 does not affect the clarity of image display when used in defocus; at this time, the aperture angle u of the signal light of the holographic optical element 20 is equal to 2*(θ+FoV / 2), where θ is the rotation angle of the eyeball and FoV is the aperture angle of the object light; the converging focal length F of the signal light of the holographic optical element 20 is equal to RD+r, where RD is the exit pupil distance and r is the radius of the eyeball. In summary, the effective size D of the holographic optical element 20 is equal to 2*F*tan(u / 2), and then the effective diameter of the reference light can be calculated according to the incidence angle of the reference light of the holographic optical element 20. In this way, as Figure 8C shown, when the reconstruction light is incident to different positions on the surface of the holographic optical element 20, the exit angle of the reconstructed object light changes, which is convenient for adapting to image display under different angle eyeball rotation.

[0081] It is worth mentioning that according to another aspect of the present application, the present application further provides a near-eye display device, which can include a device body (not shown in the figure) and the above-mentioned eye movement tracking display device 1, the eye movement tracking display device 1 being loaded in the device body for tracking the pose of the eye pupil to perform near-eye display. It can be understood that the device body of the present application can be but is not limited to being implemented as a glasses frame or a head-mounted device such as a helmet, so as to wear the near-eye display device to perform near-eye display.

[0082] It is worth mentioning that according to another aspect of the present application, asFigure 9 As shown, one embodiment of the present application further provides an eye tracking display method, which can comprise steps of:

[0083] S100: projecting image light with variable image position according to an encoded input image source;

[0084] S200: reflecting the projected image light to be redirected and reflected to image on the retina;

[0085] S300: inversely reflecting the image light reflected back via the retina; and

[0086] S400: transmitting the inversely reflected image light to be captured to track the eye pupil pose.

[0087] Notably, in one example of the present application, as Figure 10 shown, the step S400 of the eye tracking display method can comprise steps of:

[0088] S410: capturing the transmitted image light to obtain image information;

[0089] S420: matching the obtained image information with the central feature information of the input image source by image local feature recognition matching to obtain pixel offset of the captured image center position;

[0090] S430: judging whether the pixel offset of the captured image center position and the encoded central pixel offset of the input image source satisfy a theoretical mapping relationship;

[0091] S440: in response to not satisfying the theoretical mapping relationship, adjusting the encoding information of the input image source to repeat the above steps; and

[0092] S450: in response to satisfying the theoretical mapping relationship, completing the eye tracking display.

[0093] It can be understood that the adjustment direction of the encoding information in the step S440 of the present application is towards the direction of reducing the mapping difference between the pixel offset of the captured image center position and the encoded central pixel offset of the input image source, so as to make the pixel offset of the captured image center position and the encoded central pixel offset of the input image source satisfy the theoretical mapping relationship.

[0094] Notably, in another example of the present application, as Figure 11 shown, the step S400 of the eye tracking display method can also comprise steps of:

[0095] S410': converging the transmitted image light to form a converging light spot to be captured;

[0096] S420': detecting the centroid position of the captured convergent light spot by the light spot centroid detection method;

[0097] S430': judging whether the centroid position of the convergent light spot and the center pixel offset of the input image source after encoding satisfy the theoretical mapping relationship;

[0098] S440': in response to not satisfying the theoretical mapping relationship, adjusting the encoding information of the input image source to repeat the above steps;

[0099] and

[0100] S450': in response to satisfying the theoretical mapping relationship, completing the eye movement tracking display.

[0101] It can be understood that the adjustment direction of the encoding information in the step S440' of the present application is towards the direction of reducing the mapping difference between the centroid position of the convergent light spot and the center pixel offset of the input image source after encoding, and the purpose is to make the centroid position of the convergent light spot and the center pixel offset of the input image source after encoding satisfy the theoretical mapping relationship.

[0102] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0103] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as the limitation of the patent scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which belong to the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. Eye-tracking display apparatus for tracking eye pupil pose for near-eye display, characterized in that, The application comprises: a laser scanning projection assembly for projecting image light with variable image position according to an encoded input image source; a holographic optical element for redirecting the image light to be reflected by the retina and inversely reflecting the image light reflected by the retina; and an eye tracking assembly comprising a light sensing element and a light splitting element disposed on the light sensing side of the light sensing element, the light splitting element being disposed in the optical path between the laser scanning projection assembly and the holographic optical element for reflecting the image light projected by the laser scanning projection assembly to propagate to the holographic optical element and transmitting the image light inversely reflected by the holographic optical element to propagate to the light sensing element; the light sensing element being configured to capture the image light transmitted through the light splitting element to track the eye pupil pose.

2. The eye-tracking display apparatus of claim 1, wherein, The laser scanning projection assembly comprises a laser emitter configured to emit laser beams corresponding to image pixels, a micro-electro-mechanical mirror disposed on the emission side of the laser emitter and configured to scan the laser beams emitted by the laser emitter to form the image light propagating to a collimating lens, and the collimating lens disposed in the optical path between the micro-electro-mechanical mirror and the light splitting element and configured to collimate the image light from the micro-electro-mechanical mirror to propagate to the light splitting element.

3. The eye-tracking display apparatus of claim 1, wherein, The light sensing element is a light sensing chip configured to capture the inversely returned image light transmitted through the light splitting element to determine whether the gaze direction is consistent with the image center exit light ray by matching the image center feature information.

4. The eye-tracking display apparatus of claim 3, wherein, The light sensing chip is a charge-coupled device or a complementary metal-oxide-semiconductor sensor.

5. The eye-tracking display apparatus of claim 1, wherein, The light sensing element comprises a photodiode and a converging lens disposed in the optical path between the photodiode and the light splitting element, the converging lens being configured to converge the inversely returned image light transmitted through the light splitting element to form a converging light spot on the photodiode; the photodiode being configured to capture the converging light spot to determine whether the gaze direction is consistent with the image center exit light ray by detecting the energy centroid of the light spot.

6. The eye-tracking display apparatus of claim 5, wherein, The photodiode is selected from one of a PN junction photodiode, a PIN junction photodiode, a photomultiplier tube, and an avalanche photodiode.

7. The eye-tracking display apparatus of any one of claims 1-6, wherein, The light splitting element is a beam splitter configured to reflect a portion of the light beam and transmit another portion of the light beam.

8. The eye-tracking display apparatus of any one of claims 1-6, wherein, The holographic optical element has an effective diameter of reference light greater than an effective diameter of reconstruction light when manufactured.

9. The eye-tracking display apparatus of claim 8, wherein, The holographic optical element adopts an object light defocusing design, and an effective size D of the holographic optical element is equal to 2xFxtan(u / 2), where a converging focal length F of signal light of the holographic optical element is equal to RD+r, an aperture angle u of the signal light of the holographic optical element is equal to 2x(θ+FoV / 2), where RD is an exit pupil distance, r is an eyeball radius, θ is an eyeball rotation angle, and FoV is an aperture angle of object light.

10. A near-eye display device, characterized by, The application comprises: a device body; and The eye movement tracking display device as claimed in any one of claims 1 to 9, is loaded in the device body for tracking eye pupil pose for near-eye display.

11. An eye-tracking display method, characterized by, The eye movement tracking display device as claimed in any one of claims 1 to 9, comprises the steps of: projecting image light with variable image position according to the encoded input image source; reflecting the projected image light to be redirected and reflected to be imaged on the retina; reversely reflecting the image light reflected back via the retina; and transmitting the reversely reflected image light to be captured for tracking eye pupil pose.

12. The eye-tracking display method of claim 11, wherein, The step of transmitting the reversely reflected image light to be captured for tracking eye pupil pose, comprises the steps of: capturing the transmitted image light to obtain image information; matching the obtained image information with the input image source for central feature information by image local feature recognition matching manner to obtain pixel offset of captured image central position; judging whether the pixel offset of captured image central position and the encoded central pixel offset of the input image source satisfy theoretical mapping relationship or not; in response to not satisfying the theoretical mapping relationship, adjusting the encoding information of the input image source to repeat the above steps; and in response to satisfying the theoretical mapping relationship, completing eye movement tracking display.

13. The eye-tracking display method of claim 11, wherein, The step of transmitting the reversely reflected image light to be captured for tracking eye pupil pose, comprises the steps of: converging the transmitted image light to form converging light spot to be captured; detecting the captured converging light spot for center of mass to obtain center of mass position of the converging light spot by light spot energy center of mass detection manner; judging whether the center of mass position of the converging light spot and the encoded central pixel offset of the input image source satisfy theoretical mapping relationship or not; in response to not satisfying the theoretical mapping relationship, adjusting the encoding information of the input image source to repeat the above steps; and in response to satisfying the theoretical mapping relationship, completing eye movement tracking display.

Images