Image projection system and method of operating the same

Abstract

This invention relates to an image projection system, comprising: an image projection module configured to provide an image beam (ILB) that projects an image frame onto an eye image plane; a sensor assembly configured to provide data indicating lateral and angular deviations between the ILB and the orientation of the eye; an optical assembly configured to guide the ILB to propagate along a propagation path toward the eye, wherein the optical assembly is configured to adjust the angular orientation and lateral position of the ILB in response to an actuation signal; and a controller operatively connected to the sensor assembly and the optical assembly, the controller being configured to process data received from the sensor assembly to generate an actuation signal and send to the optical assembly a first actuation signal configured to compensate for the lateral deviation between the ILB and the lateral position of the pupil of the eye and a second actuation signal configured to compensate for the angular deviation between the orientation of the ILB and the gaze direction of the eye, wherein the controller is configured to send the first actuation signal multiple times during each frame and to send the second actuation signal at the beginning of each frame, and wherein each second actuation signal is generated based on a prediction of the angular position of the eye at the beginning of the corresponding frame.

Description

Technical Field

[0001] The currently disclosed topics relate to image projection technology, and more specifically, to methods and systems for projecting pure / enhanced virtual reality images onto a user's eyes. Background Technology

[0002] Various systems that project virtual reality to one or more user eyes are becoming increasingly popular. Some systems are designed to provide pure virtual reality image projection to the user's eyes, where light from the external scene is blocked from reaching one or more eyes. Other systems are designed to provide enhanced virtual reality perception, where light from the external scene is allowed to reach the eyes while also being enhanced / overlaid by image / video frames projected onto the eyes by the image projection system.

[0003] Typically, conventional projection systems project images toward the user's eyes such that the image is perceived by the user as being located / focused on an intermediate image plane located at a certain distance in front of the eyes (e.g., typically about 4 to several meters away). This intermediate image plane can be either a real image plane in front of the eyes (i.e., the plane on which the projection beam forming the image is actually focused) or a virtual image plane (i.e., the plane on which the projection beam forming the image is perceived by the user's eyes to be focused). In either case, in such conventional image projection systems, the intermediate image plane must be optically relayed to the user's eyes. In other words, because the intermediate image plane (whether virtual or real) is typically placed at a certain finite distance in front of the eyes, it is only focused onto the retina when the eye's lens focuses at that specific distance.

[0004] Alternative projection techniques offer the option of projecting images directly onto the retina of the eye (without an intermediate actual / virtual optical image projection plane outside the eye).

[0005] The problems of projecting images onto the retina of the eye have been recognized in conventional technologies, and various technologies have been developed to provide solutions, such as: International patent application WO15 / 132775 discloses an eye projection system comprising an image generator and an eye projection optical module. The image generator is adapted to acquire data indicating an image, generate multiple beam portions corresponding to pixels of the image, adjust the intensity of each beam portion according to the value of its corresponding image pixel, and guide the beam portion along the total optical propagation path toward the eye projection optical module at a projection angle relative to the total optical propagation path. The projection angle is determined based on the position of the corresponding pixel in the image. The eye-projection optics module includes a gaze-tracking deflector that responds to an input signal indicating the gaze direction β of the user's eye and is configured and operable to deflect the total optical propagation path of the beam portion toward the position of the user's pupil associated with the gaze direction β. The total optical propagation path is deflected such that the portion of the beam incident on the pupil has a pupillary angle of incidence relative to the line of sight through the pupil. Among them, the pupil angle of incidence With projection angle correspond.

[0006] U.S. Patent Application 2018 / 246336 discloses an eye projection device comprising an eye projection system configured to project a light beam to propagate along a propagation path to an eye. The eye projection system includes: an optical assembly defining a total optical path of light; the optical assembly including a deflector arrangement comprising one or more adjustable optical deflectors arranged along the total optical path and configured to define an adjustable propagation path of the light beam from the optical assembly to the eye; wherein the one or more optical deflectors are configured with at least three adjustable deflection parameters that affect the deflection of the propagation path to provide at least three degrees of freedom when adjusting the propagation path of the light beam toward the eye; wherein two of the at least three degrees of freedom are associated with two angular orientations of the propagation path to the eye to compensate for angular changes in the gaze direction of the eye, and at least one of the at least three degrees of freedom is associated with a lateral deflection of the propagation path to compensate for changes in the relative lateral position of the projection system relative to the eye.

[0007] U.S. Patent Application 2020 / 186761 discloses an eye projection system with focus management. The system includes an image projection system for generating a beam modulated to encode image data indicating an image to be projected toward a subject's eye along a beam propagation path; and an optical component located in the path for guiding the beam between the image projection system and the retina of the eye; the component includes a beam divergence component for controllably changing the focusing characteristics of the component and adjusting the divergence of the beam, thereby affecting one or more focusing parameters of one or more portions of the image on the retina.

[0008] U.S. Patent Application 2020 / 174263 discloses a system for projecting images onto the retina. The system includes an image projection module configured and operable to project at least first and second image portions corresponding to first and second portions of an image to be projected onto the fovea and peripheral regions of the retina, respectively; and an optical multiplexing module optically coupled to the image projection module and configured and operable to combine / multiplex the projections of the at least first and second image portions spatially and / or temporally to generate a combined image projection output propagating along a generally common optical path directed toward the retina. The optical multiplexing module is configured and operable to multiplex the projections of the first and second image portions such that the lateral extent of the combined projection is similar to the lateral extent of the second image portion in the combined projection intended to fall on / cover the parafoveal region of the retina, and the lateral extent of the first image portion in the combined projection is smaller and within the lateral extent of the combined projection. The first and second image portions can be projected at different qualities (different resolutions and / or different color depths) and / or different frame rates according to the corresponding foveal and parafoveal regions of the retina they target, thereby optimizing image quality / frame rate and associated processing and projection capabilities based on the anatomy of the retina.

[0009] International patent application WO20 / 157746 discloses an eye-tracking method and apparatus. The method includes receiving image data of at least two images indicating a user's eyes, identifying a region associated with the limbus of the eye in each image; determining a geometric representation of the limbus structure; and determining the three-dimensional position and gaze direction of the user's eyes (i.e., the complete six degrees of freedom of the eye) by triangulation of the geometric representation of the limbus structure in the at least two images.

[0010] The aforementioned references are assigned to the assignee of this application. The references teach background information applicable to the subject matter currently disclosed. Therefore, the entire contents of these publications are incorporated herein by reference in order to teach additional or alternative details, features, and / or technical background where appropriate. Summary of the Invention

[0011] According to one aspect of the currently disclosed subject matter, an image projection system is provided, comprising: an image projection module configured to provide an image beam (ILB) that projects an image frame onto an eye image plane; a sensor assembly configured to provide data indicating lateral and angular deviations between the ILB and the orientation of the eye; an optical assembly configured to guide the ILB to propagate along a propagation path toward the eye, wherein the optical assembly is configured to adjust the angular orientation and lateral position of the ILB in response to an actuation signal; and a controller operatively connected to the sensor assembly and the optical assembly, the controller being configured to process data received from the sensor assembly to generate an actuation signal and send to the optical assembly a first actuation signal configured to compensate for the lateral deviation between the ILB and the lateral position of the pupil of the eye and a second actuation signal configured to compensate for the angular deviation between the orientation of the ILB and the gaze direction of the eye, wherein the controller is configured to send the first actuation signal multiple times during each frame and to send the second actuation signal at the beginning of each frame, and wherein each second actuation signal is generated based on a prediction of the angular position of the eye at the beginning of the corresponding frame.

[0012] According to another aspect, and optionally in combination with other aspects of the currently disclosed subject matter, the optical assembly may include at least one optical deflector having at least two degrees of freedom and configured to adjust the lateral position of the ILB in response to a first actuation signal, and at least one optical deflector having at least two degrees of freedom and configured to adjust the angular orientation of the ILB in response to a second actuation signal. Alternatively, the optical assembly may include at least two optical deflectors, each having at least two degrees of freedom, each deflector configured to adjust the lateral position of the ILB in response to the first actuation signal and to adjust the angular orientation of the ILB in response to the second actuation signal.

[0013] According to another aspect, and optionally in combination with other aspects of the currently disclosed subject matter, the sensor assembly may include an optical port configured to provide a tracking beam (TLB) configured to propagate along a propagation path toward the eye, at least one first optical sensor, and at least one second optical sensor. The at least one first optical sensor may be optically coupled to the overall optical path of the optical assembly and configured to sense the TLB reflected from the eye, thereby obtaining data indicating lateral and angular deviations between the TLB and the eye, and lateral and angular deviations between the ILB and the eye, respectively. The at least one second optical sensor may be optically decoupled from the optical assembly and configured to detect the spatial and angular coordinates of the eye's five degrees of freedom, thereby obtaining data indicating lateral and angular deviations of the ILB. As a non-limiting example, the at least one first optical sensor may be a position and light sensor, and the at least one second optical sensor may be an imaging system. In other embodiments, the sensor assembly may include an optical port and at least two first optical sensors optically coupled to the overall optical path of the optical assembly and configured to sense the TLB reflected from the eye, thereby obtaining data indicating lateral and angular deviations between the TLB and the eye, and lateral and angular deviations between the ILB and the eye, respectively.

[0014] According to another aspect, and optionally in combination with other aspects of the currently disclosed subject matter, the controller can be configured to send a first actuation signal at a near real-time rate based on the maximum rate of the lateral response of the optical components and / or the rendering capability of the image projection module. The first actuation signal can be sent in a closed control loop to reduce lateral deviation. The controller can also be configured to calculate a predicted eye angle position at the beginning of each next frame and generate a second actuation signal based on the calculated predicted eye angle position, the calculation being provided using a predefined eye movement model and data received from the sensor components.

[0015] According to another aspect of the currently disclosed subject matter, a method is provided for operating a system that projects an image frame onto an eye image plane, the system including optical components configured to guide a tracking beam (TLB) and an image beam (ILB) toward the eye along substantially the same propagation path. The method includes: acquiring data indicating lateral and angular deviations between the ILB and the orientation of the eye; processing the acquired data to generate actuation signals and sending to the optical components a first actuation signal configured to compensate for lateral deviations between the ILB and the lateral position of the pupil of the eye, and a second actuation signal configured to compensate for angular deviations between the orientation of the ILB and the gaze direction of the eye, wherein the first actuation signal is sent multiple times during each frame and the second actuation signal is sent at the beginning of each frame, and wherein each second actuation signal is generated based on a prediction of the angular position of the eye at the beginning of the corresponding frame; and adjusting the angular orientation and lateral position of the ILB by the optical components in response to the actuation signals.

[0016] The first sync signal can be sent at near real-time based on the maximum rate of the lateral response of the optical components and / or the rendering capability of the image projection module. The first sync signal can be sent in a closed control loop to reduce lateral deviation.

[0017] The method may further include calculating the predicted eye angle position at the beginning of each next frame, the calculation being provided using a predefined eye movement model and data on the lateral and angular deviations between the indicator ILB and the eye's orientation, and generating a second actuation signal based on the calculated predicted eye angle position.

[0018] According to another aspect, and optionally in combination with other aspects of the currently disclosed subject matter, the optical component may include at least first and second optical deflectors, each optical deflector having at least two degrees of freedom. The method may further include applying a first actuation signal to the first optical deflector to adjust the lateral position of the ILB, and applying a second actuation signal to the second optical deflector to adjust the angular position of the ILB. Alternatively, the method may further include adjusting the lateral position of the ILB by applying the first actuation signal to the first and second optical deflectors, and adjusting the angular position of the ILB by applying the second actuation signal to the first and second optical deflectors.

[0019] According to another aspect, and optionally in combination with other aspects of the currently disclosed subject matter, obtaining data indicating the lateral and angular deviations between the ILB and the eye's orientation may include: providing a tracking beam (TLB) configured to propagate toward the eye along a propagation path; sensing the TLB reflected from the eye by means of at least one first optical sensor optically coupled to the total optical path of the optical components, thereby obtaining data indicating the lateral and angular deviations between the TLB and the eye, and the lateral and angular deviations between the ILB and the eye, respectively; and detecting the spatial and angular coordinates of the eye by means of at least one second optical sensor optically decoupled from the optical components, thereby obtaining data indicating the lateral and angular deviations of the ILB. Attached Figure Description

[0020] To understand the invention and how it can be practiced, embodiments will be described by way of non-limiting examples with reference to the accompanying drawings, in which: Figure 1 The illustration shows a general block diagram of an image projection system according to certain embodiments of the currently disclosed subject matter; Figure 2 A schematic example of the time series of scanning operations of an image projection module known in the art is illustrated. Figure 3 The diagram illustrates a general flowchart of an operational image projection system according to certain embodiments of the currently disclosed subject matter; Figure 4 The diagram illustrates a general block diagram of an eye-tracking controller configured according to certain embodiments of the currently disclosed subject matter; and Figure 5a and Figure 5b The illustration shows a schematic representation of eye coordinate measurement. Detailed Implementation

[0021] Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the invention. However, those skilled in the art will understand that the subject matter disclosed herein can be practiced without these specific details. In other instances, well-known methods, processes, components, and circuits have not been described in detail so as not to obscure the subject matter currently disclosed.

[0022] Unless otherwise specified, it will be appreciated, as is apparent from the following discussion, that throughout this specification, the use of terms such as “processing,” “computing,” “generating,” “obtaining,” “predicting,” or similar terms refers to one or more actions and / or processes by which a computer manipulates data and / or transforms data into other data, said data being represented as physical (such as electronic) quantities and / or said data representing physical objects. The term “computer” should be broadly interpreted to cover any kind of hardware-based electronic device with data processing capabilities, including, as a non-limiting example, the processing and memory circuitry (PMC) in the image projection system disclosed in this application.

[0023] It should be understood that the term "signal" as used herein excludes transient propagation signals, but includes any other signals suitable for the subject matter currently disclosed.

[0024] The operations taught in this article can be performed by a computer constructed specifically for the desired purpose, or by a general-purpose computer configured specifically for the desired purpose via a computer program stored in a non-transitory computer-readable storage medium.

[0025] With this in mind, note Figure 1 The illustration shows a general block diagram of an image projection system configured to project images directly onto the retina of the eye, according to certain embodiments of the currently disclosed subject matter.

[0026] The term “direct projection” or similar terms used in this patent specification should be broadly interpreted as covering an image plane (also referred to below as the “eye image plane”) that focuses a propagating light field (e.g., an image) onto the user’s retina, such focusing being achieved by any kind of arrangement.

[0027] The illustrated image projection system 100 includes an image projection module 101 optically coupled to an imaging module 102. The image projection module 101 is configured to encode at least one light beam based on an image to be formed on the retina and output an encoded image beam (ILB). The imaging module 102 is configured to receive the ILB and guide it in a manner that allows an image to be projected directly onto one or more of the user's eyes. The imaging module 102 includes an inlet optical port (ENP) optically coupled to the image projection module 101 and configured to receive the ILB, and an outlet optical port (EXP) configured to output the ILB toward the eye, thereby enabling an optical path through the imaging module to reach the eye.

[0028] Note that the term "optical port" does not necessarily specify a physical element of the system and may be used herein to specify certain planes / apertures (e.g., virtual planes and / or optical surfaces) that intersect the optical path through which light enters and exits the imaging module.

[0029] Image projection module 101 may include an optical module (not shown) and an image light modulator (not shown) configured to modulate the light beam according to an image to be formed on the retina. The image light modulator may also include an image scanning deflector (e.g., a raster scanning mirror module and / or a two-dimensional angle-adjustable / rotatable deflector capable of performing image scanning, etc.). The image scanning deflector is configured to change its deflection angle over time, such that at different times, it deflects the light beam LB to different orientations corresponding to the positions of different pixels of the image to be projected onto the retina. Therefore, the image scanning deflector enables spatial distribution of the ILB on a two-dimensional image plane (display area) on the user's retina. Non-limiting examples of image projection modules are assigned to the assignee of this application and incorporated herein by reference in U.S. Patent Application 2018 / 246336. Figure 2 A and Figure 2 B is described in detail.

[0030] As a non-limiting example, the image projection module 101 can be configured to enable vertical raster scanning of the display area, such as... Figure 2 As shown in the diagram. During the "scan" phase (201), the ILB scans along the vertical dimension to guide light from the light source to the display area, and during the "retrace" phase (202), the ILB returns to its starting horizontal position for the next scan phase in response to the V-sync signal. No new information is shifted during the retrace phase. In the case of 60Hz scanning, each frame consists of 13.4 milliseconds of projection and 3.3 milliseconds of retrace (idle) time. Pixels are rendered 10–20 milliseconds before being projected.

[0031] Note that the term "frame" is used in this patent specification. N "The beginning" should be broadly interpreted as covering frame ( N -1) and frames N Any point in time during the retrace period.

[0032] Go back to reference Figure 1 The imaging module 102 also includes a controllable and adjustable optical component 103, which is optically located between the ENP and the EXP and is configured to guide the ILB from the ENP through the EXP along the image propagation path to the eye.

[0033] Imaging module 102 also includes eye tracker 104. Eye tracker 104 includes sensor assembly 105 optically coupled to optics assembly 103 and configured to provide data indicating lateral and angular deviations between the ILB and the orientation of the eye, and eye tracking controller 106 operatively connected to sensor assembly 104 and optics assembly 103. Eye tracking controller (ETC) 106 is configured to process data received from sensor assembly to generate actuation signals and send the actuation signals to optics assembly 103.

[0034] Sensor assembly 105 can be optically connected to a tracking light source that emits a tracking beam (TLB). The TLB source emits an IR beam invisible to the human eye and guided to propagate along the optical path of the system's optical components, while interacting with one or more tunable optical deflectors. Sensor assembly 105 includes a plurality of sensors configured to provide an optical port (TP) for the TLB and configured to acquire data indicating lateral and angular deviations between the tracking beam and the orientation of the eye. At least a portion of the sensors is configured to sense the tracking beam reflected from the eye.

[0035] Optical component 103 is aligned with the output and tracking light port of image projection module 101 to receive both the tracking (infrared) beam and the image beam and controllably direct the beam through the propagation path toward the retina guide (this path is also referred to below as the total optical path of the optical component, GOP), so that the TLB can propagate to the eye along the image propagation path.

[0036] Unless otherwise specified, in the following description, the image propagation path to the eye and the tracking propagation path to the eye are referred to together as the "propagation path" because the TLB and ILB share the same propagation path to the eye.

[0037] Optical assembly 103 may include a plurality of deflectors positioned in any suitable arrangement and any one or more desired locations according to embodiments. In some embodiments, the optical assembly may include at least one optical deflector having at least two degrees of freedom and configured to adjust its lateral position toward the propagation path toward the eye in response to an actuation signal, and at least one optical deflector having at least two degrees of freedom and configured to adjust its angular orientation toward the propagation path toward the eye in response to an actuation signal. Alternatively, optical assembly 103 may include at least two optical deflectors, each having at least two degrees of freedom, each deflector configured to adjust both its lateral and angular position toward the propagation path toward the eye in response to an actuation signal.

[0038] Non-limiting examples of controllable optical components are described in U.S. Patent Application 2018 / 246336. Figure 1 A-1F and Figure 3 A is described in detail.

[0039] Sensor assembly 105 may include one or more first optical sensors optically coupled to the overall optical path of the optical assembly and configured to sense a light beam reflected from the eye. As a non-limiting example, the first optical sensor may be a position-sensing photodetector (PSD), a multi-diode array (e.g., a quadrant IR light sensor), a time-delay integrated charge-coupled device (TDI-CCD), a complementary metal-oxide-semiconductor (CMOS) with region of interest (ROI) selection capability, etc. The measured eye positioning data may indicate the deviation between the eye's gaze direction (LOS) and the azimuth of the propagation path of the light beam toward the eye, and / or the deviation between the lateral (X, Y) position of the eye's pupil and the propagation path. Therefore, the first optical sensor may acquire data indicating the angular deviation between the azimuth of the light beam and the eye's gaze direction (hereinafter referred to as "angular deviation") and data indicating the lateral deviation between the lateral position of the eye's pupil and the propagation path (hereinafter referred to as "lateral deviation").

[0040] The sensor assembly 105 may also include one or more second optical sensors, which are optically decoupled from the overall optical path of the optical assembly (and correspondingly from the first optical sensor). The second optical sensor (e.g., an imaging device, including one or more digital or video cameras, a stereo camera with diffuse IR illumination, etc.) may be configured to detect five degrees of freedom of the eye's spatial and angular coordinates. Optionally, the second sensor may include two imaging devices, each capturing an image of the user's eye at its own angle.

[0041] Therefore, in some embodiments, the sensor assembly may include an optical port configured to provide a tracking beam (TLB), at least one first optical sensor optically coupled to the total optical path of the optical assembly 103, and at least one second optical sensor decoupled from the optical assembly, the first and second sensors providing data indicating lateral and angular deviations of the ILB.

[0042] In some other embodiments, the sensor assembly may include at least two first optical sensors optically coupled to the total optical path of the optical assembly 103 at at least two different locations. The sensors may be configured to have at least four degrees of freedom and to measure one or more characteristics of the propagation of the reflected TLB at two or more different locations, and to provide data respectively indicating lateral and angular deviations between the ILB and the eye.

[0043] As a non-limiting example, data on the indicator angle and lateral deviation can be obtained by sensing the beam reflected from the cornea or retina of the eye in response to guiding the tracking beam toward the eye. Data on the indicator angle and lateral deviation can also be obtained through image processing and defining the lateral and angular position of the beam relative to the optical axis of the eye (e.g., through the center of the pupil or limbus).

[0044] Non-limiting examples of sensors in sensor assembly 105 are disclosed in U.S. Patent Application 2018 / 246336, International Patent Application WO20 / 157746, and International Patent Application PCT / IL2021 / 050022, which have been assigned to the assignee of this application and are incorporated herein by reference. A non-limiting example of obtaining data on the indicator angle and lateral deviation by sensing the reflected light beam is disclosed in Israeli Patent Application No. 276354, filed July 28, 2020, which has been assigned to the assignee of this application and is incorporated herein by reference.

[0045] The eye-tracking controller (ETC) 106 is configured to process data received from the sensor assembly to generate an actuation signal and send the actuation signal to the optical assembly 103, as referenced. Figure 3 –5. Further detailed description. The optical component 103 is configured to adjust the angular and lateral positions of the TLB and ILB relative to the eye, respectively, in response to an actuation signal.

[0046] Therefore, according to some embodiments, the image projection system 100 is capable of monitoring both the position of the eye and the gaze / gaze of the eye / pupil, and is capable of directing one or more projected images onto the retina of one or more of the user's eyes respectively. Because changes in eye position / positioning are compensated for, the image can be projected at a fixed location on the retina, unaffected by changes in gaze and / or changes in the relative position between the eye and the projection system.

[0047] It is worth noting that the technology of the currently disclosed subject addresses the compensation for large-scale eye movements (e.g., those associated with changes in gaze direction and / or changes in the position of glasses on the face), while small-scale eye movements, such as saccadic movements (tremors), can be left uncompensated, while still allowing the image to appear completely stable at a fixed position on the retina.

[0048] It should also be noted that the teachings on the currently publicly available topics are not intended for reference. Figure 1 The limitations of the described image projection system. Equivalent and / or modified functionality can be integrated or divided in another way.

[0049] refer to Figure 3 The diagram illustrates a general flowchart of an operational image projection system according to certain embodiments of the currently disclosed subject matter.

[0050] As described in detail above, an image projection system (301) with optical components configured to guide a tracking beam (TLB) and an image beam (ILB) toward the eye along substantially the same propagation path is provided. A sensor component senses the tracking beam reflected from the eye and thereby obtains (302) data indicating lateral and angular deviations. Highly accurate data indicating lateral and angular deviations (302) are also obtained by processing one or more images captured by an imaging device.

[0051] The eye-tracking controller (ETC) processes the data received from the sensor assembly to generate (303) a first actuation signal and sends (306) a first actuation signal configured to compensate for lateral deviation between the ILB / TLB and the eye to the optical assembly. The ETC sends the first actuation signal multiple times per frame (preferably, based on the maximum rate of lateral response of the optical assembly and / or the rendering capability of the image projection module). In response to the first actuation signal, the optical assembly adjusts (308) the lateral orientation of the ILB / TLB.

[0052] In some embodiments, the ETC controller may be configured to generate a first actuation signal based solely on data received from one or more first optical sensors.

[0053] The ETC further processes the data received from the sensor components to calculate (304) the predicted eye corner position at the beginning of each next frame, and generates (305) a second actuation signal based on the calculated predicted position. (See reference...) Figure 4 To elaborate further, ETC uses data from the first and second optical sensors to generate a second actuation signal.

[0054] At the beginning of each frame, the ETC sends (307) a second actuation signal to the optical components, configured to compensate for the predicted angular deviation between the azimuth of the TLB (and, correspondingly, the ILB) and the predicted position of the eye. In response to the second actuation signal, the optical components adjust (309) the angular azimuth of the ILB / TLB. This angular azimuth adjustment can be provided during the retrace phase (e.g., in response to or related to a V-sync signal). During frame projection (scanning phase), the angular position of the beam can remain constant.

[0055] Generate actuation signal reference Figure 4 Further detailed description.

[0056] It should be noted that the teaching of the currently publicly available topics is not subject to... Figure 3Due to limitations of the flowchart shown, the operations may not occur in the order shown. For example, operations 303, 306, 308 and 304, 305, 307, 309 shown in the basic concurrency diagram may be performed at least partially sequentially. It should also be noted that although the flowchart is described with reference to specific elements of an image projection system, this is by no means limiting, and the operations may be performed by elements other than those described herein.

[0057] Figure 4 The illustration shows an overall block diagram of an eye-tracking controller (ETC) 106 configured according to certain embodiments of the currently disclosed subject matter.

[0058] ETC 106 includes a processing and memory circuitry (PMC) 403 configured to receive data acquired by sensor assembly 105. Note that the data acquired by sensor assembly 105 may include data measured by the sensor, images captured by the sensor, and / or biases resulting from initial processing by the sensor assembly. In some embodiments, PMC 403 may be configured to receive data 401 from one or more first sensors at a higher rate than data 402 from one or more second sensors. Optionally, ETC 106 may have an input circuitry (not shown) configured to transform and / or preprocess the data received from sensor assembly 105.

[0059] The PMC 403 includes a processor and memory (not shown separately within the PMC). See reference... Figure 4 In further detail, PMC 403 can be configured to execute several functional modules according to computer-readable instructions implemented on a non-transitory computer-readable storage medium. Such functional modules are referred to below as included in the PMC. PMC 403 includes a lateral control module 404 and an angle prediction and control module 406 operatively connected. Lateral control module 404 is configured to process data 401 received from one or more first sensors to generate a TLB lateral reference 405 providing lateral deviation information. Angle prediction and control module 406 is configured to process data 402 received from a second sensor and the TLB lateral reference to calculate a predicted angular position of the eye and generate a TLB angle reference 407 based on the predicted position.

[0060] According to some embodiments, the angle prediction and control module 406 (hereinafter also referred to as the "prediction module") is configured to calculate the predicted eye angle position at the beginning of each frame. The prediction module 406 provides this calculation using a predefined eye angle movement model and data indicating the lateral and angular deviations between the beam and the eye's orientation in the corresponding current frame, which are received from one or more second sensors and the lateral control module 404.

[0061] The prediction of eye corner position can be achieved using a linear quadratic estimate (e.g., a Kalman filter) applied to the kinematic model described below. Those skilled in the art will readily recognize that the teachings of the subject matter disclosed herein are equally applicable to other eye corner movement models and other prediction techniques (e.g., nonlinear quadratic estimates such as extended Kalman filters and unscented Kalman filters, machine learning models, etc.).

[0062] PMC 403 is also operatively connected to actuator 408, which is configured to generate a first actuation signal based on TLB lateral reference 405 and a second actuation signal based on TLB angular reference 407. (See above reference...) Figure 3 In detail, in response to a first actuation signal, the optical component adjusts the lateral orientation of the ILB / TLB, and in response to a second actuation signal, the optical component adjusts the angular orientation of the ILB / TLB. Actuator 408 is configured to transmit the first actuation signal multiple times during each frame and to transmit the second actuation signal at the beginning of each frame.

[0063] Figure 5a The diagram illustrates the coordinates representing the 5-DOF eye position in the eye coordinate system.

[0064] Eye coordinates: ,in and These are pitch and yaw coordinates.

[0065] In the case of the first optical sensor (based on TLB reflection), the closed lateral control loop based on the TLB reflection from the eye can provide the following function of the eye coordinates:

[0066] For example, for corneal reflexes with lateral control loops ( Figure 5b As shown), this function can be described as:

[0067] In the case of a second optical sensor (imaging device), the number of equations equals the number of coordinates, thus enabling the realization of complete information on lateral and angular deviations:

[0068] For illustrative purposes only, the following description is provided for use in the kinematic model described below.

[0069]

[0070] Where x is the state vector of the eye's kinematic model. r It is the coordinate vector of the eye as defined earlier. and These are the coordinate velocity and acceleration, respectively.

[0071] The following equations describe a general kinematic model of the eye.

[0072]

[0073] f It can be any function that approximates eye movement, and w is the noise processed to model the uncertainty of the deterministic eye movement model. It has a zero-mean normal distribution and a given covariance matrix. A random white noise sequence.

[0074] The above function f linear approximation

[0075] Therefore, the measurement equation model used for the first optical sensor is: This is the noise measured by the first sensor. It is a normal distribution with zero mean and a given covariance matrix. A random white noise sequence.

[0076] It is a linear approximation of the measurement equation of the first sensor.

[0077] The measurement equation used for the second optical sensor is: This is the noise measured by the second sensor. It has a zero-mean normal distribution and a given covariance matrix. A random white noise sequence.

[0078] It is a linear approximation of the measurement equation of the second sensor.

[0079] Accordingly, in Kalman-filter estimation, the time and measurement updates can be expressed as follows: Time update (predictor):

[0080] Measurement update (calibrator):

[0081] According to the table below, the parameters of the calibrator depend on the sensor.

[0082]

[0083] The above non-limiting example demonstrates the use of a model-based predictor / corrector scheme to estimate eye position. The correction described above is based on data received from one or more second sensors and the lateral control module 404.

[0084] The prediction module 406 uses the above (or other suitable) model to predict the required ILB angular position at the beginning of each next frame and the actuator 408 generates a second actuation signal accordingly.

[0085] It should be noted that the teachings on the currently published topics are not for reference. Figure 4 The limitations of ETC are described. Equivalent and / or modified functionality may be integrated or partitioned in another way and may be implemented by any appropriate combination of software and firmware and / or hardware and executed on one or more suitable devices.

[0086] It should be understood that the invention, in its application, is not limited to the details set forth in the description contained herein or illustrated in the accompanying drawings. The invention is capable of other embodiments and can be practiced and performed in various ways. Therefore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered limiting. Consequently, those skilled in the art will recognize that the concepts on which this disclosure is based can readily be used as the basis for designing other structures, methods, and systems for performing several purposes of the subject matter currently disclosed.

[0087] It will also be understood that the system according to the invention can be implemented, at least in part, on a suitably programmed computer. Similarly, the invention contemplates computer programs that can be read by a computer to perform the methods of the invention. The invention also contemplates a non-transitory computer-readable storage device tangibly implementing a program of instructions for performing the methods of the invention, executable by a computer.

[0088] Those skilled in the art will readily recognize that various modifications and alterations can be applied to the embodiments of the invention described above without departing from the scope defined in and by the appended claims.

Claims

1. An image projection system, comprising: An image projection module configured to provide an image beam (ILB) that projects an image frame onto the eye image plane. A sensor assembly configured to provide data indicating lateral and angular deviations between the orientation of the ILB and the eye; An optical component configured to guide the ILB to propagate along a propagation path toward the eye, wherein the optical component is configured to adjust the angular orientation and lateral position of the ILB in response to an actuation signal; as well as A controller, operably connected to a sensor assembly and an optical assembly, is configured to process data received from the sensor assembly to generate actuation signals and send to the optical assembly a first actuation signal configured to compensate for a lateral deviation between the ILB and the lateral position of the pupil of the eye, and a second actuation signal configured to compensate for an angular deviation between the orientation of the ILB and the gaze direction of the eye, wherein the controller is configured to send the first actuation signal multiple times during each frame and to send the second actuation signal at the beginning of each frame, and wherein each second actuation signal is generated based on a prediction of the angular position of the eye at the beginning of the corresponding frame.

2. The image projection system of claim 1, wherein the optical components include at least one optical deflector having at least two degrees of freedom and configured to adjust the lateral position of the ILB in response to a first actuation signal, and at least one optical deflector having at least two degrees of freedom and configured to adjust the angular orientation of the ILB in response to a second actuation signal.

3. The image projection system of claim 1, wherein the optical components include at least two optical deflectors, each optical deflector having at least two degrees of freedom, each deflector being configured to adjust the lateral position of the ILB in response to a first actuation signal and to adjust the angular orientation of the ILB in response to a second actuation signal.

4. The image projection system of any one of claims 1-3, wherein the sensor assembly includes an optical port configured to provide a tracking beam (TLB) configured to propagate toward the eye along a propagation path, at least one first optical sensor, and at least one second optical sensor, and wherein... The at least one first optical sensor is optically coupled to the overall optical path of the optical assembly and configured to sense the TLB reflected from the eye, thereby obtaining data indicating the lateral and angular deviations between the TLB and the eye, and the lateral and angular deviations between the ILB and the eye, respectively; and The at least one second optical sensor is optically decoupled from the optical components and configured to detect the spatial and angular coordinates of the eye's five degrees of freedom, thereby obtaining data indicating the lateral and angular deviations of the ILB.

5. The image projection system of claim 4, wherein the at least one first optical sensor is a position and light sensor, and the at least one second optical sensor is an imaging system.

6. The image projection system of any one of claims 1-3, wherein the sensor assembly includes an optical port configured to provide a tracking beam (TLB) configured to propagate toward the eye along a propagation path and at least two first optical sensors, the at least two first optical sensors being optically coupled to the total optical path of the optical assembly and configured to sense the TLB reflected from the eye, thereby obtaining data indicating lateral and angular deviations between the TLB and the eye, and between the ILB and the eye, respectively.

7. The image projection system according to any one of claims 1-6, wherein the data indicating lateral deviation indicates the lateral position of the light beam relative to the optical axis of the eye passing through the center of the pupil of the eye.

8. The image projection system of any one of claims 1-7, wherein the data indicating angular deviation is obtained by sensing a light beam reflected back from the cornea or retina of the eye in response to directing the TLB toward the eye.

9. The image projection system of any one of claims 1-8, wherein the controller is configured to send a first actuation signal at a near real-time rate based on the maximum rate of the lateral response of the optical components and / or the rendering capability of the image projection module.

10. The image projection system of any one of claims 1-9, wherein the controller is configured to send a first actuation signal in a closed control loop to reduce lateral deviation.

11. The image projection system of any one of claims 1-10, wherein the controller is configured to calculate a predicted eye corner position at the beginning of each next frame and generate a second actuation signal based on the calculated predicted eye corner position, the calculation being provided using a predefined eye movement model and data received from sensor components.

12. A method of operating a system for projecting an image frame onto an eye image plane, the system comprising optical components configured to guide a tracking beam (TLB) and an image beam (ILB) toward the eye along substantially the same propagation path, the method comprising: Obtain data on the lateral and angular deviations between the ILB and the eye's orientation; The acquired data is processed to generate actuation signals and a first actuation signal configured to compensate for the lateral deviation between the ILB and the lateral position of the pupil of the eye and a second actuation signal configured to compensate for the angular deviation between the orientation of the ILB and the gaze direction of the eye are sent to the optical components, wherein the first actuation signal is sent multiple times during each frame and the second actuation signal is sent at the beginning of each frame, and wherein each second actuation signal is generated based on a prediction of the angular position of the eye at the beginning of the corresponding frame; as well as The angular and lateral positions of the ILB are adjusted by optical components in response to the actuation signal.

13. The method of claim 12, wherein the optical components include at least a first optical deflector and a second optical deflector, each optical deflector having at least two degrees of freedom, the method further comprising applying a first actuation signal to the first optical deflector to adjust the lateral position of the ILB, and applying a second actuation signal to the second optical deflector to adjust the angular orientation of the ILB.

14. The method of claim 12, wherein the optical components include at least a first optical deflector and a second optical deflector, each optical deflector having at least two degrees of freedom, the method further comprising adjusting the lateral position of the ILB by applying a first actuation signal to the first optical deflector and the second optical deflector, and adjusting the angular position of the ILB by applying a second actuation signal to the first optical deflector and the second optical deflector.

15. The method of any one of claims 12-14, wherein obtaining data indicating the lateral and angular deviations between the ILB and the eye's orientation comprises: Provides a tracking beam (TLB) configured to propagate toward the eye along the propagation path; By means of at least one first optical sensor optically coupled to the total optical path of the optical components, the TLB reflected from the eye is sensed, thereby obtaining data indicating the lateral and angular deviations between the TLB and the eye, and the lateral and angular deviations between the ILB and the eye, respectively; and By means of at least one second optical sensor that is optically decoupled from the optical components, the spatial and angular coordinates of the eye are detected, thereby obtaining data indicating the lateral and angular deviations of the ILB.

16. The method of claim 15, wherein the at least one first optical sensor is a position and light sensor, and the at least one second optical sensor is an imaging system.

17. The method of any one of claims 12-16, wherein the first actuation signal is transmitted at a near real-time rate based on the maximum rate of the lateral response of the optical component and / or the rendering capability of the image projection module.

18. The method of any one of claims 12-17, wherein the first actuation signal is sent in a closed control loop to reduce lateral deviation.

19. The method of any one of claims 12-18, further comprising calculating a predicted eye angle position at the beginning of each next frame and generating a second actuation signal based on the calculated predicted eye angle position, said calculation being provided using a predefined eye movement model and data obtained indicating the lateral and angular deviations between the ILB and the eye's orientation.

Images