Eye-tracking for retinal projection display systems

Abstract

The retinal projection display system includes at least one visible light source for projecting a visible light image, an infrared light source for projecting infrared light, a scanning mirror having a field of view larger than the visible light image, a reflective surface onto which the visible light image is projected and onto which the infrared light is at least partially reflected toward a user's eye, the reflective surface being larger than the visible light image, at least one infrared light detector for receiving reflected infrared light reflected from the user's eye, and a hardware computation module having a processor and a memory, the hardware computation module configured to determine a user's gaze direction based at least in part on the reflected infrared light.

Description

Related Applications

[0001]

[0001] This application claims the priority and benefit of the co-pending U.S. Provisional Patent Application No. 63 / 239,915, filed on September 1, 2021, with Attorney Docket No. IVS-1017-PR, and assigned to the assignee of this application, entitled "ADAPTIVE EYE-BOX WITH IR LASER IN AR SMART GLASSES" by Heshmati et al., the entire disclosure of which is incorporated herein by reference.

[0002]

[0002] This application is related to U.S. Patent Application No. 17 / 822,619, filed on August 26, 2022, with Attorney Docket No. IVS-1017, and assigned to the assignee of this application, entitled "GAZE TRACKING FOR A RETINAL PROJECTION DISPLAY SYSTEM" by Heshmati et al., the entire disclosure of which is incorporated herein by reference.

Background Art

[0003]

[0003] A retinal projection display (RPD), also referred to as a virtual retinal display (VRD), is used to project an image directly onto the retina through the pupil of the eye. Image rendering is performed at a speed sufficient for the human eye to perceive a continuous video stream of images. Due to the small area, also referred to as the "eye-box," through which the image is projected onto the retina through the pupil, it is essential that an accurate alignment between the RPD and the eye be performed to ensure that the image enters the eye. Further, the direction of the user's line of sight may change during the use of the RPD, thereby changing the position of the eye-box, so it is necessary to consider the change in the line of sight direction during the use of the RPD.

[0004]

[0004] The accompanying drawings, incorporated into and forming part of the description of the embodiments, illustrate various non-exclusive and non-exclusive embodiments of the subject matter and, together with the description of the embodiments, are useful in illustrating the principles of the subject matter discussed below. Unless otherwise specified, drawings referenced in this brief description should be understood as not being drawn to a fixed scale, and similar reference numbers refer to similar parts throughout the various drawings unless otherwise specified. [Brief explanation of the drawing]

[0005] [Figure 1A] This figure shows an exemplary retinal projection display system according to several embodiments. [Figure 1B] This figure shows an exemplary eye-tracking system, including an infrared sensor placed on an eyeglass frame, according to several embodiments. [Figure 1C] This figure shows an exemplary light source module, including an infrared light source, for use in eye-tracking, according to an embodiment. [Figure 1D] This figure shows an exemplary light source module, including an infrared light source and an internal infrared sensor, for use in eye-tracking according to an embodiment. [Figure 2A] This figure shows a functional block diagram of an exemplary retinal projection display system according to several embodiments. [Figure 2B] This figure shows a functional block diagram of an exemplary eye-tracking system according to several embodiments. [Figure 3] This figure shows an exemplary interpupillary distance alignment operation by projecting an alignment image onto a reflective surface, according to several embodiments. [Figure 4A] This figure shows the procedure in an exemplary eye-tracking operation according to several embodiments. [Figure 4B] This figure shows the procedure in an exemplary eye-tracking operation according to several embodiments. [Figure 4C] This figure shows the procedure in an exemplary eye-tracking operation according to several embodiments. [Figure 4D] This figure shows the procedure in an exemplary eye-tracking operation according to several embodiments. [Figure 5A] This figure shows an exemplary scanning pattern across the entire scanning range of a scanning mirror according to one embodiment. [Figure 5B] This figure shows an example scanning pattern when the scanning range of the scanning mirror is the size of the projected image. [Figure 6] This figure shows an exemplary image rendering operation according to one embodiment, where the scanning range of the scanning mirror is larger than the size of the projected image. [Figure 7] This is a block diagram of an exemplary electronic device in which embodiments described herein may be implemented. [Figure 8A] This figure shows an exemplary process of retinal projection according to several embodiments. [Figure 8B] This figure shows an exemplary eye-tracking process according to several embodiments. [Figure 9] This figure shows an exemplary process for determining interpupillary distance alignment according to several embodiments. [Modes for carrying out the invention]

[0006]

[0020] The following description of embodiments is provided merely as an example and is not intended to limit it. Furthermore, it is not intended to be bound by any explicit or implicit theory presented in the above-mentioned background art or in the following description of embodiments.

[0007]

[0021] Various embodiments of the subject matter are referenced in detail here, examples of which are shown in the accompanying drawings. While various embodiments are discussed herein, it will be understood that they are not intended to be limiting. On the contrary, the presented embodiments are intended to encompass alternatives, modifications, and equivalents that may fall within the spirit and scope of the various embodiments defined by the accompanying claims. Furthermore, this description of embodiments includes numerous specific details to provide a complete understanding of the embodiments of the subject matter. However, embodiments may be carried out without these specific details. In other examples, well-known methods, procedures, components, and circuits are not described in detail so as not to unnecessarily obscure the aspects of the embodiments described.

[0008] Notation and Nomenclature

[0022] Some parts of the following detailed description are presented with respect to procedures, logical blocks, processes, and other symbolic representations of operations on data within electrical devices. These descriptions and representations are means used by those skilled in the field of data processing to communicate the content of their work to others skilled in the field in the most effective way. In this application, procedures, logical blocks, processes, etc., are considered to be one or more consistent procedures or instructions that produce a desired result. Procedures are those that require the physical manipulation of physical quantities. Usually, but not always, these quantities take the form of acoustic (e.g., ultrasonic) signals that can be transmitted and received by electronic devices, and / or electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated within electrical devices.

[0009]

[0023] However, it should be noted that all of these terms and similar terms should be associated with appropriate physical quantities and are merely convenient labels applied to those quantities. As will be evident from the following description, unless otherwise specified, throughout the description of the embodiments, any use of terms such as "performing," "determining," "detecting," "directing," "calculating," "correcting," "providing," "receiving," "analyzing," "confirming," "displaying," "presenting," "using," "completing," "instructing," "comparing," "executing," "tracking," "moving," "retrieving," "projecting," "calibrating," "coordinating," "generating," "aligning," "measuring," and "mapping" is understood to refer to the operation and processes of electronic devices such as electrical devices.

[0010]

[0024] The embodiments described herein may be described in the general context of processor-executable instructions residing on some form of non-transient processor-readable medium, such as program modules, executed by one or more computers or other devices. Generally, a program module includes routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. The functions of the program module may be combined or distributed as desired in various embodiments.

[0011]

[0025] In the figures, a single block may be described as performing one or more functions, but in fact, one or more functions performed by that block may be performed in a single component or across multiple components, and / or may be performed using hardware, using software, or using a combination of hardware and software. To clearly show this interchangeability of hardware and software, various exemplary components, blocks, modules, logics, circuits, and steps have been generally described with respect to their functions. Whether such functions are implemented as hardware or as software depends on the particular application example and the design constraints imposed on the overall system. Those skilled in the art may implement the described functions in various ways for each particular application example, but such implementation decisions should not be construed as causing a departure from the scope of the present disclosure. Also, the exemplary ultrasonic sensing system and / or mobile electronic device described herein may include components other than those shown, including well-known components.

[0012]

[0026] Unless otherwise specifically described as implemented in a particular way, the various techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. Any features described as a module or component may also be implemented together in an integrated logic device or separately but interoperably as discrete logic devices. When implemented in software, the techniques may be at least partially realized by a non-transitory processor-readable storage medium comprising instructions that, when executed, perform one or more of the methods described herein. The non-transitory processor-readable data storage medium may form part of a computer program product that may include packaging material.

[0013]

[0027] Non-temporary processor-readable storage media may include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, and other known storage media. The technique may be additionally or alternatively implemented by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures, and which can be accessed, read, and / or executed by a computer or other processor.

[0014]

[0028] The various embodiments described herein may be implemented by one or more motion processing units (MPUs), sensor processing units (SPUs), host processors or their cores, digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), field-programmable gate arrays (FPGAs), programmable logic controllers (PLCs), complex programmable logic devices (CPLDs), discrete gates or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein, or by other equivalent integrated circuits or discrete logic circuits. As used herein, the term "processor" may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. As used herein, the term "processor" can refer to substantially any computing processing unit or device, including, but not limited to, a single-core processor, a single processor with software multithreading capabilities, a multi-core processor, a multi-core processor with software multithreading capabilities, a multi-core processor with hardware multithreading technology, a parallel platform, and a parallel platform with distributed shared memory. Additionally, the processor may utilize nanoscale architectures such as, but not limited to, transistors, switches, and gates based on molecules and quantum dots to optimize space usage or improve the performance of the user equipment. The processor may also be implemented as a combination of computing processing units.

[0015]

[0029] In addition, in some embodiments, the functions described herein may be provided within a dedicated software module or hardware module configured as described herein. Furthermore, the techniques may be fully implemented in one or more circuit or logic elements. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, e.g., a combination of an SPU / MPU and a microprocessor, multiple microprocessors, SPU cores, MPU cores, or one or more microprocessors combined with any other such configuration.

[0016] Summary of the explanation

[0030] The discussion will begin with a description of an exemplary retinal projection display system. The discussion will then continue with a description of an eye-tracking system for the retinal projection display system. Next, an exemplary operation of the retinal projection display system and the eye-tracking system will be described.

[0017]

[0031] Embodiments described herein provide a retinal projection display system comprising: at least one visible light source for projecting a visible light image; an infrared light source for projecting infrared light; a scanning mirror having a field of view larger than that of the visible light image; a reflective surface on which the visible light image is projected and which at least partially reflects the infrared light toward the user's eye, the reflective surface being larger than that of the visible light image; at least one infrared light detector for receiving reflected infrared light reflected from the user's eye; and a hardware computing module comprising a processor and memory, configured to determine the user's line of sight direction at least partially based on the reflected infrared light.

[0018]

[0032] In some embodiments, the reflective surface is at least partially transparent. In some embodiments, the retinal projection system further includes an eyeglass frame configured to be worn by a user, and at least one lens mounted within the eyeglass frame, wherein the reflective surface is located on at least a portion of the at least one lens. In some embodiments, at least one infrared photodetector is located on the eyeglass frame. In some embodiments, at least one infrared photodetector is located inside a module comprising at least one visible light source and an infrared light source.

[0019]

[0033] In some embodiments, the hardware computing module is further configured to scan infrared light across the field of view of a reflective surface. Reflected infrared light reflected from the user's eye is received by at least one infrared light detector. The amount of reflected infrared light across the field of view of the scanning mirror on the reflective surface is measured. The amount of reflected infrared light across the field of view of the scanning mirror on the reflective surface is mapped to generate an infrared reflectance map of the scanning mirror's field of view, which identifies the line of sight direction.

[0020]

[0034] In some embodiments, the hardware computing module is further configured to adjust the operation of a scanning mirror and at least one visible light source for projecting a visible light image onto a reflective surface based on the line of sight, so that the visible light image is projected onto the user's retina. In some embodiments, at least one visible light source and an infrared light source are aligned, and the hardware computing module is further configured to control the scanning mirror to project a visible light image onto the reflective surface in the line of sight direction. In other embodiments, at least one visible light source and an infrared light source are not aligned, and the hardware computing module is further configured to determine the line of sight direction by compensating for the misalignment between at least one visible light source and an infrared light source, and to control the scanning mirror to project a visible light image onto the reflective surface in the line of sight direction. In some embodiments, an interpupillary distance alignment is determined during a calibration operation for the user, and the interpupillary distance alignment identifies the visible region of the reflective surface for the user's known line of sight direction. In some embodiments, the misalignment between at least one visible light source and an infrared light source is at least partially based on the line of sight direction and the interpupillary distance alignment. In some embodiments, the difference between at least one visible light source and an infrared light source is determined during a manufacturing calibration operation and stored in memory.

[0021]

[0035] In some embodiments, at least one visible light source comprises multiple visible light sources, and the misalignment of the visible light sources between the multiple visible light sources is determined during a manufacturing calibration operation and stored in memory. In some embodiments, a hardware computing module is configured to align the multiple visible light sources based at least partially on the misalignment of the visible light sources.

[0022]

[0036] Other embodiments described herein provide a method of retinal projection. A visible light image is projected from at least one visible light source onto a reflective surface using a scanning mirror having a field of view larger than that of the visible light image, the reflective surface being larger than that of the visible light image. Infrared light from an infrared light source is projected onto the reflective surface using a scanning mirror, the infrared light is projected across the field of view of the scanning mirror and at least partially reflected from the reflective surface toward the user's eye. The reflected infrared light reflected toward the user's eye is received by at least one infrared light detector. The user's line of sight is determined at least partially on the reflected infrared light.

[0023]

[0037] In some embodiments, the operation of the scanning mirror and at least one visible light source is adjusted to project a visible light image onto a reflective surface based on the line of sight, so that the visible light image is projected onto the user's retina. In some embodiments, if at least one visible light source and an infrared light source are aligned, the scanning mirror is controlled to project a visible light image onto the reflective surface in the line of sight direction. In other embodiments, if at least one visible light source and an infrared light source are not aligned, a misalignment between the at least one visible light source and an infrared light source is determined. The scanning mirror is controlled to compensate for the misalignment between the at least one visible light source and an infrared light source to determine the line of sight direction and to project a visible light image onto the reflective surface in the line of sight direction.

[0024]

[0038] In some embodiments, determining the misalignment between at least one visible light source and an infrared light source involves obtaining an interpupillary distance alignment for the user, where the interpupillary distance alignment identifies the visible area of ​​the reflective surface relative to the user's known line of sight direction, and the misalignment between at least one visible light source and an infrared light source is at least partially based on the line of sight direction and the interpupillary distance alignment. In other embodiments, determining the misalignment between at least one visible light source and an infrared light source involves obtaining a misalignment between at least one visible light source and an infrared light source, where the misalignment between at least one visible light source and an infrared light source is determined during a manufacturing calibration operation and stored in a memory unit.

[0025]

[0039] In some embodiments, at least one visible light source comprises multiple visible light sources, and the visible light source misalignment between the multiple visible light sources is determined during a manufacturing calibration operation and stored in a memory unit. In some embodiments, the multiple visible light sources are aligned at least partially based on the visible light source misalignment.

[0026]

[0040] In some embodiments, determining the user's line of sight direction based at least partially on reflected infrared light involves measuring the amount of reflected infrared light across the field of view of a scanning mirror on a reflective surface. The amount of reflected infrared light across the field of view of the scanning mirror on the reflective surface is mapped to generate an infrared reflectance map of the scanning mirror's field of view, and the infrared reflectance map identifies the line of sight direction.

[0027] Exemplary retinal projection display system

[0041] Figure 1A shows an exemplary retinal projection display system 100 according to several embodiments. The retinal projection display system 100 includes a light source 110, a scanning mirror 120, and a reflective surface 130. In the illustrated embodiments, the components of the retinal projection display system 100 are contained within an eyeglass frame 160 configured to be worn by a user. The eyeglass frame 160 includes at least one lens 162 mounted therein, and the reflective surface 130 is positioned on or in front of at least a portion of the lens 162 such that the reflective surface 130 is within the user's field of view when looking through the lens 162. It should be understood that the reflective surface 130 may be located on the lens 162 (e.g., a film or a bonding layer) or otherwise positioned between the user's eye and the lens 162. In some embodiments, the reflective surface 130 is at least partially transparent, allowing the user to see through the reflective surface 130 and the lens 162. It should be understood that various components of the retinal projection display system 100, such as the light source 110 and the scanning mirror 120, can be arranged on or within the hollow eyeglass frame 160 (for example, within the hollow 115 of the arm / temple) and positioned so that the light beam 112 is projected onto the reflective surface 130.

[0028]

[0042] During operation, the light source 110 (e.g., a laser) projects an image onto a portion of the reflective surface 130 by generating a light beam 112 that is projected onto the scanning mirror 120. In some embodiments, the light source 110 is a single light source capable of projecting a complete image. In some embodiments, the light source 110 includes multiple light sources, such as separate red, green, and blue (RGB) lasers, that work in coordination to project a complete image. It should be understood that many types of light sources can be used according to the embodiments described.

[0029]

[0043] The scanning mirror 120 is configured to move and direct the light beam 112 so that it is scanned over the reflective surface 130 and each point of the image is positioned on the reflective surface 130, and the reflective surface 130 directs the light beam 112 onto the retina 152 through the pupil 154 of the user's eye 150. It should be understood that various scanning patterns may be used, as described below. It should be understood that the image scanning process is performed at a scanning speed fast enough (e.g., above 60 Hz) so that the user perceives the entire image or the image as a continuous video. In some embodiments, the scanning mirror 120 is a micro-electromechanical (MEMS) device.

[0030]

[0044] The scanning mirror 120 has a field of view (FOV) larger than the intended visible image size, and the reflective surface 130 onto which the visible image is projected is also larger than the intended visible image size. The scanning mirror 120 projects an image onto the visible region of the reflective surface 130 so that the image is projected onto the user's retina 152. The larger FOV allows the retinal projection display system 100 to project the image appropriately into the pupil 154 and onto the retina 152, independently of the movement and rotation of the eye 150. According to some embodiments, the retinal projection display system 100 facilitates the projection and alignment of the intended visible image with the pupil 154 by projecting onto the visible region of the reflective surface 130 on the dynamic range window of the scanning mirror.

[0031]

[0045] Interpupillary distance alignment is used to direct the light beam 112 into the pupil 154 of the eye 150, and identifies the visible area of ​​the reflective surface 130 relative to the user's known line of sight. In some embodiments, interpupillary distance alignment is determined during a calibration operation for the user. In some embodiments, images are displayed at multiple locations on the reflective surface 130 during the calibration operation, and interpupillary distance alignment is determined in response to user feedback identifying the visible area of ​​the reflective surface. For example, user feedback can be provided using the user interface of the retinal projection display system 100 and can be received in many ways, such as voice commands, buttons located on the eyeglass frame 160, or applications on connected devices such as smartphones.

[0032]

[0046] In some embodiments, the retinal projection display system 100 also includes an eye-tracker 140 for tracking the user's line of sight. The visible area of ​​the reflective surface 130 corresponds to the user's line of sight. The scanning mirror 120 is configured to dynamically move the image on the reflective surface 130 according to the user's line of sight and the user's interpupillary distance alignment. Since the interpupillary distance alignment for the user identifies the visible area of ​​the reflective surface 130 for the user's known line of sight, the scanning mirror 120 can move the image according to the line of sight to correspond to the visible area of ​​the reflective surface.

[0033]

[0047] According to some embodiments, the eye-tracker 140 of the retinal projection display system 100 utilizes an infrared light source and at least one infrared sensor (e.g., an infrared light detector) to determine the user's gaze direction. To perform the eye-tracking of the embodiments described, infrared light is projected onto the user's eye, and the reflected infrared light is sensed and used to determine the gaze direction. It should be understood that different parts of the human eye have different reflectivity to infrared light. For example, the pupil of the human eye has little reflectivity to infrared light because most of the infrared light is absorbed by the inner eye. The sclera, the white part that covers most of the outside of the human eyeball, is highly reflective compared to the pupil, and most of the infrared light is reflected from the sclera. The iris, the part of the eye that surrounds the pupil and defines the eye color, is more reflective than the pupil and less reflective than the sclera, and has a reflectivity that is partly dependent on the eye color of the iris.

[0034]

[0048] The eye-tracking in the embodiments described utilizes the reflectivity properties of the outer tissue portion of the eye to identify the user's gaze direction. Figure 1B shows an exemplary eye-tracking system 140, which includes infrared sensors 170 located on an eyeglass frame 160, according to several embodiments. As shown in Figure 1B, multiple infrared sensors 170 are positioned at different locations on the eyeglass frame 160, which is arranged around the lens 162. In the illustrated embodiment, the light source 110 includes an infrared light source and uses a scanning mirror 120 to project infrared light onto a reflective surface 130 (as indicated by arrow 175). The infrared light is reflected from the outer portion of the user's eyeball and received by the infrared sensors 170. The infrared light received by the infrared sensors 170 is used to determine the user's gaze direction.

[0035]

[0049] Figure 1C shows an exemplary light source module 125, including an infrared light source 186, for use in eye-tracking according to an embodiment. In some embodiments, the light source module 125 is positioned on a spectacle frame 160 such that light emitted through an aperture 190 is projected onto a reflective surface 130 using a scanning mirror 120. As shown, the light source 110 includes multiple light sources, including a separate red light source 180, a green light source 182, a blue light source 184, and an infrared light source 186. In some embodiments, the red light source 180, the green light source 182, and the blue light source 184 are lasers that work in coordination to project a complete red, green, and blue (RGB) image. The infrared light source 184 is configured to project infrared light onto the reflective surface 130 via the scanning mirror 120 through the aperture 190 for projection onto the user's eye. In some embodiments, the light source module 125 is located within a cavity 115.

[0036]

[0050] Embodiments in Figures 1B and 1C show an exemplary eye-tracking system 140 in which an infrared light sensor 170 is positioned around a reflective surface 130 to directly receive infrared light reflected from the user's eyes. In other embodiments, one or more infrared light sensors may be positioned to indirectly receive infrared light reflected from the user's eyes.

[0037]

[0051] Figure 1D shows an exemplary light source module 135 for use in eye-tracking according to an embodiment, which includes an infrared light source 186 and an internal infrared sensor 192. In some embodiments, the light source module 135 is positioned on a spectacle frame 160 such that light emitted through an aperture 190 is projected onto a reflective surface 130 using a scanning mirror 120. As shown, the light source 110 includes multiple light sources, including separate red light source 180, green light source 182, blue light source 184, and infrared light source 186. In some embodiments, the red light source 180, green light source 182, and blue light source 184 are lasers that work in coordination to project a complete red, green, and blue (RGB) image. The infrared light source 184 is configured to project infrared light onto the reflective surface 130 via the scanning mirror 120 through the aperture 190 for projection onto the user's eye. In some embodiments, the light source module 135 is positioned within a cavity 115.

[0038]

[0052] The light source module 135 also includes an internal infrared sensor 192 for receiving infrared light reflected from a portion of the user's eye and passing through the aperture 194. In some embodiments, the infrared light reflected from a portion of the user's eye is also reflected from the reflective surface 130 and / or scanning mirror 120, directed through the aperture 194, and received by the infrared sensor 192. The infrared light received by the infrared sensor 192 is used to determine the direction of the user's line of sight.

[0039]

[0053] In some embodiments, to avoid visible image jitter, the scanning mirror 120 is configured to dynamically move the image on the reflective surface 130 according to the user's line of sight in response to the line of sight direction meeting a movement threshold. For example, the scanning mirror 120 moves the image on the reflective surface only when sufficient movement in the line of sight direction is detected. In some embodiments, jitter is addressed by providing a rendered image smaller than the visible area of ​​the reflective surface 130, so that the scanning mirror 120 is configured to dynamically move the image on the reflective surface 130 according to the user's line of sight in response to the image moving outside the visible area. This allows the image to be viewed over a wider range of positions on the reflective surface 130, minimizing jitter.

[0040]

[0054] In some embodiments, to avoid image blurring, the scanning mirror 120 is configured to dynamically move the image on the reflective surface 130 according to the user's line of sight direction after a predetermined time delay following a change in the line of sight direction, allowing the eye 150 to settle into the new line of sight direction before the image is moved.

[0041]

[0055] In some embodiments, the scanning range of the scanning mirror 120 is dynamically adjusted to correspond to the size of the image within the visible region. In other embodiments, the scanning range of the scanning mirror 120 corresponds to the size of the display area of ​​the reflective surface 130, and therefore the light source 110 is activated to display the image only when the scanning mirror 120 is projecting the image within the visible region of the reflective surface 130.

[0042]

[0056] Figure 2A shows a functional block diagram of an exemplary retinal projection display system 200 according to several embodiments. The retinal projection display system 200 includes a light source 210, an infrared light source 212, a scanning mirror 220, an eye-tracker 230, and a reflective surface 240. The light source 210 receives image data 205 from a data source for projection. It should be understood that the image data 205 may include any type of data for displaying or rendering an image, including still image data, video data (e.g., a series of images), or other data for user visualization. In some embodiments, the light source 210 is a single light source capable of projecting a complete image. In some embodiments, the light source 210 includes multiple light sources, such as separate red, green, and blue (RGB) lasers, that work in coordination to project a complete image. It should be understood that many types of light sources may be used according to the embodiments described.

[0043]

[0057] The infrared light source 212 is configured to project infrared light 214 onto the reflective surface 240 (for example, via a scanning mirror 220). It should be understood that the infrared light source 212 can project infrared light 214 to different locations using a different scanning pattern than the visible light source 210. In some embodiments, the retinal projection display system 200 is configured to project infrared light 214 over a wider area than the visible region of the reflective surface 240 onto which visible light is projected. For example, in some embodiments, infrared light 214 is projected over the entire surface of the reflective surface 240. In some embodiments, the light source 210 and the infrared light source 212 are contained within a single light source module.

[0044]

[0058] A light source 210 (e.g., a laser) projects the image 215 onto a scanning mirror 220. The image 215 is projected as a scan of its pixels, and the scanning mirror 220 dynamically moves to position each pixel appropriately on the reflective surface 240 for rendering. The scanning mirror 220 is configured to move and orient the pixels of the image 215 so that each pixel of the image is scanned on the reflective surface 240, positioning each point of the image on the reflective surface 240, which orients the image 215 into the user's pupil and retina. Various scanning patterns may be used, as described below. The image scanning process should be performed at a scanning speed fast enough (e.g., above 60 Hz) for the user to perceive the entire image or a continuous video of the image.

[0045]

[0059] The scanning mirror 220 uses the interpupillary distance alignment 225 and line of sight direction 235 relative to the user to control the position of the pixels in the image 215 so that they are directed onto the user's retina. The interpupillary distance alignment identifies the visible area of ​​the reflective surface 240 relative to the user's known line of sight direction.

[0046]

[0060] Figure 3 shows an exemplary interpupillary distance alignment operation 300 by projecting an alignment image onto a reflective surface, according to several embodiments. In some embodiments, the interpupillary distance alignment operation 300 is determined during a calibration operation for the user. In some embodiments, during the interpupillary distance alignment operation 300, the alignment image 310 is displayed at multiple locations on the reflective surface 330, and the interpupillary distance alignment is determined in response to user feedback identifying the visible area 320 of the reflective surface 330. For example, user feedback can be provided using the user interface of the retinal projection display system 100 and can be received in many ways, such as voice commands, buttons located on the eyeglass frame 160, or applications on connected devices such as smartphones.

[0047]

[0061] During the interpupillary distance alignment operation 300, the user is instructed (e.g., via a user interface) to look in a specific direction (e.g., straight ahead). The alignment image 310 is rendered onto the reflective surface 330 and moved across the dynamic range of the scanning mirror to display the alignment image at multiple locations on the reflective surface 330. The user provides feedback (e.g., when prompted) regarding whether the alignment image 310 is fully visible, partially visible, or not visible. If the alignment image 310 is within the visible area 320 and is, for example, partially or fully visible to the user, the user provides feedback indicating that the alignment image 310 is visible. In some embodiments, the alignment image 310 is configured to assist in alignment. For example, the alignment image 310 may include information that identifies parts of the alignment image 310, such as letters, arrows, colors, or other indicators, which the user can use to indicate which parts of the alignment image 310 to look at, so that the retinal projection display system knows how to move the alignment image 310 within the visible area 320.

[0048]

[0062] As shown in the figure, the alignment image 310 is projected onto the reflective surface 330. In the illustrated example, at the first time interval 340, the alignment image 310 is not within the user's visible area 320 of the reflective surface 330 corresponding to the user's known line of sight direction (e.g., straight ahead). The user provides feedback that the alignment image 310 is not visible to the user. At the second time interval 350, the alignment image 310 is moved to a different position on the reflective surface 330, which is still not within the visible area 320. As shown in the figure, the visible area 320 remains substantially stationary during the interpupillary distance alignment operation 300. The user provides feedback that the alignment image 310 is not visible to the user.

[0049]

[0063] At time 360°, the alignment image 310 is moved to a different position on the reflective surface 330, which is partially within the visible area 320. The user is given feedback that the alignment image 310 is partially visible to the user. At time 470°, the alignment image 310 is moved to a different position on the reflective surface 330, which is fully within the visible area 320. The user is given feedback that the alignment image 310 is fully visible to the user. The position of the alignment image 310 at time 470° is stored and used as the interpupillary distance alignment relative to the user's known line of sight direction (e.g., straight ahead). The interpupillary distance alignment is stored (e.g., in the memory of the retinal projection display system).

[0050]

[0064] Referring to Figure 2A, the eye-tracker 230 is for tracking the user's gaze direction 235. The visible area of ​​the reflective surface 240 corresponds to the user's gaze direction, and as the gaze direction 235 moves, the user's visible area on the reflective surface 240 also moves. The scanning mirror 220 is configured to dynamically move the image 215 on the reflective surface 240 according to the user's gaze direction 235 and the user's interpupillary distance alignment 225. Since the interpupillary distance alignment 225 for the user identifies the visible area of ​​the reflective surface 240 for the user's known gaze direction, the scanning mirror 220 can move the image 215 to correspond to the visible area of ​​the reflective surface 240 according to the gaze direction 235.

[0051]

[0065] Figure 2B shows a functional block diagram of an exemplary eye-tracking system (e.g., eye-tracker 230) according to several embodiments. The eye-tracking system 230 includes one or more infrared sensors 250a-n, an infrared measuring module 260, and a gaze direction determiner 270. The infrared sensors 250a-n are configured to receive and sense infrared light. It should be understood that the eye-tracking system 230 may include any number of infrared sensors 250a-n arranged to receive infrared light reflected from the user's eyes. The infrared measuring module 260 is configured to receive infrared light sensed by the infrared sensors 250a-n and to measure the amount of infrared light received by each infrared sensor 250a-n as a sensing time. In some embodiments, the infrared measuring module 260 is configured to measure the amount of infrared light across the field of view of a scanning mirror 220 on a reflective surface 240.

[0052]

[0066] The line of sight direction determiner 270 is configured to determine the line of sight direction 235 based on the amount of infrared light received by each infrared sensor 250a to n. In some embodiments, the line of sight direction determiner 270 includes an infrared mapping module 272. The infrared mapping module 272 is configured to map the amount of reflected infrared light across the field of view of the scanning mirror 220 on the reflective surface 240 to generate an infrared reflectance map of the field of view of the scanning mirror 220. The infrared reflectance map identifies the line of sight direction based on the intensity of the sensed reflected infrared light.

[0053]

[0067] Figures 4A to 4D illustrate the procedure in an exemplary eye-tracking operation 400 according to several embodiments. During the eye-tracking operation 400, the eye-tracker (e.g., eye-tracker 140 in Figure 1A or eye-tracker 230 in Figure 2A) is configured to track the user's gaze direction. In some embodiments, the eye-tracking operation 400 is performed concurrently with the projection of a visible image, allowing the retinal projection display system to adjust the operation of a scanning mirror and at least one visible light source for projecting a visible light image onto a reflective surface based on the gaze direction, so that the visible light image is projected onto the user's retina.

[0054]

[0068] Figure 4A shows an exemplary scanning pattern of infrared light being scanned across the entire scanning range 410 of the scanning mirror, where the scanning mirror can render an image anywhere on the reflective surface, and the reflective surface is larger than the visible area of ​​the scanning surface. As shown in the figure, the scanning mirror is configured to move in the x and y directions, and the scanning position 420 moves across the entire scanning range 410 according to the scanning pattern, and the scanning position 420 indicates the position of the scanning pattern at a given time.

[0055]

[0069] Infrared light projected onto a reflective surface using a scanning mirror is reflected away from the user's eye, and the reflected infrared light is received by at least one infrared sensor to measure the amount of infrared light at a location across the entire scanning range 410.

[0056]

[0070] Figure 4B shows an exemplary mapping of the intensity of reflected infrared light across the scanning range of a scanning mirror, as sensed by at least one infrared sensor. Due to the reflective properties of different parts of the human eye, the mapping of reflected light intensity indicates the user's line of sight. As shown, the mapping of reflected light intensity includes three scanning regions 430, 432, and 434 of the entire scanning range of the scanning mirror, where scanning region 430 shows the highest reflectivity, scanning region 434 shows the lowest reflectivity, and scanning region 432 shows the reflectivity between scanning regions 430 and 434. For example, scanning region 430 is associated with infrared light reflected from the sclera, scanning region 432 is associated with infrared light reflected from the iris, and scanning region 434 is associated with infrared light reflected from the pupil. Thus, it can be determined that scanning region 434 indicates the user's line of sight. Although the rectangular shapes of scanning regions 430, 432, and 434 are shown, please understand that scanning regions 430, 432, and 434 can have any shape or shape factors.

[0057]

[0071] Figure 4C shows an exemplary projection of a visible image 440 onto a scanning area 434 that identifies the user's line of sight, according to the embodiment. Since the scanning area 434 exhibits the lowest reflectivity across the scanning range of the scanning mirror, it is determined that the scanning area 434 identifies the user's line of sight. Therefore, the visible image 440 is projected onto the scanning area 434 for final projection into the user's pupil and onto the user's retina. It should be understood that Figure 4C shows an example where the infrared and visible light sources are perfectly aligned and not misaligned.

[0058]

[0072] Figure 4D shows an exemplary projection of the visible image 440 onto a scanning mirror when the infrared and visible light sources are not aligned and are misaligned relative to each other. As shown in the figure, region 436 is identified as the user's visible region, as described with respect to Figures 4A to 4C. Because the infrared and visible light sources are not aligned, the unadjusted visible image 440 is not fully projected into region 436.

[0059]

[0073] In some embodiments, the retinal projection display system is configured to determine the line of sight direction by compensating for a misalignment 450 between at least one visible light source and an infrared light source, and to project a visible light image onto a region 436. In some embodiments, interpupillary distance alignment is used to compensate for the misalignment 450. Interpupillary distance alignment for the user identifies the visible region of the reflective surface relative to the user's known line of sight direction, and can be used to adjust the operation of the scanning mirror to move the visible image 440 to correspond to the region 436 of the reflective surface, thereby compensating for the misalignment 450.

[0060]

[0074] In some embodiments, to avoid jitter in the visible image, the visible region (e.g., region 436) is moved only when sufficient movement in the line of sight is detected (e.g., a movement threshold is met). In some embodiments, jitter is addressed by providing a rendered image smaller than the visible region so that the visible region moves in response to the user's line of sight moving outside the visible region. This allows the image to be viewed over a wider range of positions and minimizes jitter. In some embodiments, to avoid image blurring, the visible region is moved according to the user's line of sight after a predetermined time delay following the change in line of sight, allowing the user's eyes to settle into the new line of sight before the image is moved.

[0061]

[0075] Referring to Figure 2A, the scanning mirror 220 projects the image 215 onto the visible area of ​​the reflective surface 240 (e.g., pixel by pixel) so that the image 215 is projected onto the user's retina. In some embodiments, the scanning range of the scanning mirror 220 is dynamically adjusted to correspond to the size of the image in the visible area. In other embodiments, the scanning range of the scanning mirror 220 corresponds to the size of the display area of ​​the reflective surface 240, and therefore the light source 210 is activated to display the image only when the scanning mirror 220 is projecting the image into the visible area of ​​the reflective surface 240.

[0062]

[0076] Figures 5A and 5B show an exemplary image rendering operation 500 according to one embodiment, where the scanning range of the scanning mirror is the size of the projected image. Figure 5A shows an exemplary scanning pattern over the entire scanning range 510 of the scanning mirror, and the scanning mirror can render an image at any point on the reflective surface, where the reflective surface is larger than the visible area of ​​the scanning surface. As shown, the scanning mirror is configured to move in the x and y directions, and the center position 520 is the center position of the user's identified line of sight and visible area. Eye tracking is used to determine the center position of the visible area.

[0063]

[0077] Figure 5B shows an exemplary scanning pattern when the scanning range of the scanning mirror is the size of the projected image. As shown in Figure 5B, the center position 530 is identified using eye-tracking, and the center position 530 is moved relative to the center position 520. The pan angle of the scanning mirror is controlled so that the x and y scanning ranges 550 cover the visible area 540 identified by the center position B. In the illustrated embodiment, the scanning range 550 of the scanning mirror is reduced to the visible area 540 covered by the reflective surface, rather than the entire reflective surface. In some embodiments, infrared reflectivity is measured by an infrared sensor over this reduced scanning range to ensure proper alignment with the user's pupil, which has low reflectivity. If a reflectivity higher than a certain threshold is measured, the scanning pattern is expanded, and eye-tracking operation is restarted to track the new line of sight, as shown in Figures 4A and 4B.

[0064]

[0078] Figure 6 shows an exemplary image rendering operation 600 in one embodiment where the scanning range 610 of the scanning mirror is larger than the size of the projected image. As shown, the scanning mirror is configured to move in the x and y directions, and the center position 620 is the center position of the user's identified line of sight and visible area. Eye tracking is used to determine the center position of the visible area.

[0065]

[0079] As shown in Figure 6, the center position 630 is identified using eye-tracking, and the center position 630 is moved relative to the center position 620. Depending on the line of sight direction by the center position 630, the light source is activated only when the scanning mirror is within the visible area 640. The pan angle of the scanning mirror is controlled so that the x and y scanning ranges 610 cover the entire reflective surface area, but the light source is activated only when the scanning mirror is within the visible area 640.

[0066]

[0080] Figure 7 is a block diagram of an exemplary electronic device 700 that can implement embodiments of the present invention. Figure 7 shows an example of the type of electronic device 700 (e.g., a computer system) that may be used according to or to implement the various embodiments discussed herein. It should be understood that the embodiments of the retinal projection display system described may be implemented using the exemplary electronic device 700.

[0067]

[0081] The electronic device 700 in Figure 7 is merely an example, and it should be understood that the embodiments described herein, without limitation, may operate on or within several different computer systems, including general-purpose networked computer systems, embedded computer systems, mobile electronic devices, smartphones, server devices, client devices, various intermediate devices / nodes, standalone computer systems, media centers, handheld computer systems, multimedia devices, and the like. In some embodiments, the electronic device 700 in Figure 7 is well adapted to have a peripheral tangible computer-readable storage medium 702, which may be combined with, for example, an electronic flash memory data storage device, a floppy disk, a compact disk, a digital multipurpose disk, other disk-based storage devices, a Universal Serial Bus "thumb" drive, a removable memory card, and the like. The tangible computer-readable storage medium is inherently non-temporary.

[0068]

[0082] The electronic device 700 in Figure 7 includes an address / data bus 704 for communicating information and a processor 706A coupled to the bus 704 for processing information and instructions. The bus 704 may be any suitable bus or interface, including, but not limited to, a Peripheral Component Interconnection Express (PCIe) bus, a Universal Serial Bus (USB), a Universal Asynchronous Receiver / Transmitter (UART) serial bus, a suitable Advanced Microcontroller Bus Architecture (AMBA) interface, an Inter-Integrated Circuit (I2C) bus, a Serial Digital Input / Output (SDIO) bus, a Serial Peripheral Interface (SPI), or other equivalents.

[0069]

[0083] As shown in Figure 7, the electronic device 700 is also well suited to a multiprocessor environment in which multiple processors 706A, 706B, and 706C exist. Conversely, the electronic device 700 is also well suited to having a single processor, such as processor 706A. Processors 706A, 706B, and 706C may be any of various types of microprocessors. The electronic device 700 also includes data storage functions such as computer-available volatile memory 708, e.g., random access memory (RAM), coupled to bus 704, for storing information and instructions for processors 706A, 706B, and 706C. The electronic device 700 also includes computer-available non-volatile memory 710, e.g., read-only memory (ROM), coupled to bus 704, for storing static information and instructions for processors 706A, 706B, and 706C. The electronic device 700 also includes a data storage unit 712 (e.g., magnetic or optical disks and disk drives) coupled to the bus 704 for storing information and instructions. The electronic device 700 also includes an alphanumeric input device 714, including alphanumeric keys and function keys, coupled to the bus 704 for communicating information and command selections to the processor 706A or processors 706A, 706B, and 706C. The electronic device 700 also includes a cursor control device 716 coupled to the bus 704 for communicating user input information and command selections to the processor 706A or processors 706A, 706B, and 706C. In one embodiment, the electronic device 700 also includes a display device 718 coupled to the bus 704 for displaying information. Depending on the architecture, different bus configurations may be adopted as needed. For example, additional buses may be used to connect various components of the electronic device 700, such as by using a dedicated bus between the processor 706A and the memory computer-available volatile memory 708 or the computer-available non-volatile memory 710.

[0070]

[0084] Continuing to refer to Figure 7, the display device 718 in Figure 7 may include a light source (e.g., light source 110 in Figure 1A) for projecting image data onto a reflective surface. In other embodiments, the display device 718 may be a liquid crystal display (LCD), a light-emitting diode display (LED) device, a plasma display device, a touchscreen device, or other display device suitable for creating user-recognizable graphic images and alphanumeric characters. The cursor control device 716 enables the computer user to dynamically indicate the movement of a visible symbol (cursor) on the display screen of the display device 718 and to indicate user selection of selectable items displayed on the display device 718. Many implementations of the cursor control device 716 are known in the art, including a trackball, mouse, touchpad, touchscreen, joystick, or special keys on an alphanumeric input device 714 capable of signaling a method of movement or displacement in a given direction. Alternatively, it should be understood that the cursor may be directed and / or actuated via input from the alphanumeric input device 714 using special keys and key sequence commands. The electronic device 700 is also suitable for directing the cursor by other means, such as voice commands. In various embodiments, the alphanumeric input device 714, the cursor control device 716, and the display device 718, or any combination thereof (e.g., a user interface selection device), may operate collectively under the direction of a processor (e.g., processor 706A or processors 706A, 706B, and 706C) to provide a graphical user interface (GUI) 730. The GUI 730 allows the user to interact with the electronic device 700 through a graphical representation presented on the display device 718 by interacting with the alphanumeric input device 714 and / or the cursor control device 716.

[0071]

[0085] The electronic device 700 also includes an I / O device 720 for connecting the electronic device 700 to an external entity. For example, in one embodiment, the I / O device 720 is a modem for enabling wired or wireless communication between the electronic device 700 and an external network, such as the Internet, but not limited to the Internet. In one embodiment, the I / O device 720 includes a transmitter. The electronic device 700 may communicate with the network by transmitting data through the I / O device 720.

[0072]

[0086] Continuing to refer to Figure 7, various other components for the electronic device 700 are shown. Specifically, if present, the operating system 722, applications 724, modules 726, and data 728 are shown to reside typically in one or a combination of computer-readable volatile memory 708 (e.g., RAM), computer-readable non-volatile memory 710 (e.g., ROM), and data storage units 712. In some embodiments, all or some of the various embodiments described herein are stored, for example, as memory locations in RAM 708, computer-readable storage media in data storage units 712, peripheral computer-readable storage media 702, and / or applications 724 and / or modules 726 in other tangible computer-readable storage media.

[0073] Exemplary operation for operating a retinal projection display system

[0087] Figure 8A shows an exemplary process of retinal projection according to several embodiments, Figure 8B shows an exemplary process of eye-tracking according to several embodiments, and Figure 9 shows an exemplary process for determining interpupillary distance alignment according to several embodiments. The steps of these methods are described with reference to elements and / or components of various figures described herein. It will be understood that in some embodiments, the steps may be performed in a different order than described, some of the described steps may not be performed, and / or one or more additional steps may be performed in addition to those described. The flow charts include several steps performed by one or more processors (e.g., a host processor or a sensor processor) under the control of computer-readable and computer-executable instructions stored in a non-temporary computer-readable storage medium, in various embodiments. It will be further understood that one or more steps described in the flow charts may be implemented in hardware, or in combination of hardware and firmware and / or software.

[0074]

[0088] Referring to Figure 8A, Flowchart 800 illustrates an exemplary process of retinal projection according to several embodiments. In step 810 of Flowchart 800, an image from a light source is projected onto a reflective surface larger than the image using a scanning mirror with a field of view larger than the image. In some embodiments, the light source includes multiple visible light sources, and the visible light source misalignments between the multiple visible light sources are determined during a manufacturing calibration operation and stored in a memory unit. In some embodiments, the multiple visible light sources are aligned at least partially based on the visible light source misalignments. In some embodiments, the scanning range of the scanning mirror is dynamically adjusted to correspond to the size of the image in the visible region. In some embodiments, the scanning range of the scanning mirror corresponds to the size of the display area of ​​the reflective surface, and therefore the light source is operated to display the image only when the scanning mirror is projecting the image in the visible region. In some embodiments, the reflective surface is at least partially transparent.

[0075]

[0089] In step 820, the visible area of ​​the reflective surface for the user is determined. In some embodiments, as shown in step 822, the interpupillary distance alignment for the user is obtained (e.g., from memory), and the interpupillary distance alignment identifies the visible area of ​​the reflective surface for the user's known line of sight direction. In some embodiments, as shown in step 824, the user's line of sight direction is tracked, and the visible area corresponds to the line of sight direction.

[0076]

[0090] In some embodiments, step 824 is performed according to flowchart 824 in Figure 8B. Figure 8B shows flowchart 824 of an exemplary eye-tracking process according to some embodiments. In step 830 of flowchart 824, infrared light from an infrared light source is projected onto a reflective surface using a scanning mirror, and the infrared light is projected across the field of view of the scanning mirror and reflected from the reflective surface at least partially toward the user's eye. In step 840, the reflected infrared light reflected from the user's eye is received by at least one infrared light detector.

[0077]

[0091] In step 850, the user's line of sight is determined at least partially based on reflected infrared light. In one embodiment, as shown in step 852, the amount of reflected infrared light across the field of view of the scanning mirror on the reflective surface is measured. In step 854, the amount of reflected infrared light across the field of view of the scanning mirror on the reflective surface is mapped to generate an infrared reflectance map of the field of view of the scanning mirror, and the infrared reflectance map identifies the line of sight.

[0078]

[0092] In step 860, the operation of the scanning mirror and at least one visible light source for projecting a visible light image onto the reflective surface is adjusted based on the line of sight so that the visible light image is projected onto the user's retina. In one embodiment, in step 870, it is determined whether at least one visible light source and an infrared light source are aligned. If at least one visible light source and an infrared light source are aligned, the scanning mirror is controlled to project a visible light image onto the reflective surface in the line of sight, as shown in step 880.

[0079]

[0093] If at least one visible light source and an infrared light source are not aligned, a misalignment between the at least one visible light source and an infrared light source is determined, as shown in step 890, and the misalignment is compensated so that a visible light image is projected onto the reflective surface in the line of sight direction. In some embodiments, the misalignment is determined by obtaining an interpupillary distance alignment for the user, which identifies the visible area of ​​the reflective surface in the user's known line of sight direction, and the misalignment between the at least one visible light source and an infrared light source is at least partially based on the line of sight direction and the interpupillary distance alignment. In other embodiments, the misalignment is determined by obtaining a misalignment between the at least one visible light source and an infrared light source, which is determined during a manufacturing calibration operation and stored in a memory unit.

[0080]

[0094] Returning to Figure 8A, in step 830, the image is oriented over the visible area of ​​the reflective surface so that the image is projected onto the user's retina. In some embodiments, as shown in step 832, the image is dynamically moved over the reflective surface using a scanning mirror according to the user's line of sight and interpupillary distance alignment.

[0081]

[0095] In some embodiments, as shown in step 834, the amount of movement in the line of sight is determined based on tracking the user's line of sight, and in response to the amount of movement in the line of sight satisfying a movement threshold, the image is moved on the reflective surface according to the user's line of sight and the user's interpupillary distance alignment. In some embodiments, as shown in step 836, the image is moved on the reflective surface according to the user's line of sight and the user's interpupillary distance alignment after a predetermined time delay. In some embodiments, as shown in step 838, in response to the determination that the image is outside the visible region, the image is moved on the reflective surface according to the user's line of sight and the user's interpupillary distance alignment.

[0082]

[0096] Figure 9 shows an exemplary flowchart 900 for determining interpupillary distance alignment, for example, during a calibration operation, according to several embodiments. In step 910 of flowchart 900, an alignment image is projected onto a reflective surface. In step 920, user feedback is received regarding the visibility of the alignment image while the user gazes at a known line of sight. In step 930, it is determined, according to the user feedback, whether the alignment image is within the visible area. If the alignment image is not within the visible area, flowchart 900 proceeds to step 940. In step 940, the position of the alignment image is moved to a different position on the reflective surface. If the alignment image is within the visible area, flowchart 900 proceeds to step 950.

[0083]

[0097] In step 950, it is determined, based on user feedback, whether the alignment image is entirely within the visible area. If the alignment image is not entirely within the visible area, flowchart 900 proceeds to step 940. In step 940, the position of the alignment image is moved to a different position on the reflective surface. If the alignment image is entirely within the visible area, flowchart 900 proceeds to step 960. In step 960, an interpupillary distance alignment is determined that identifies the visible area of ​​the reflective surface relative to the user's known line of sight. In some embodiments, the interpupillary distance alignment is stored (e.g., in memory) for retrieval during retinal projection display operation.

[0084] conclusion

[0098] The examples described herein are presented to best illustrate, describe specific uses, and thereby enable those skilled in the art to construct and use embodiments of the described examples. However, those skilled in the art will recognize that the foregoing description and examples are presented for illustrative purposes only. Many aspects of the different exemplary embodiments described above can be combined to form new embodiments. The description is not intended to be exhaustive or to limit embodiments to the exact forms disclosed. Rather, the particular features and actions described above are disclosed as exemplary forms that implement the claims.

[0085]

[0099] Throughout this specification, any reference to “one embodiment,” “a particular embodiment,” “a certain embodiment,” “various embodiments,” “several embodiments,” or similar terms means that a particular feature, structure, or characteristic described in relation to an embodiment is included in at least one embodiment. Therefore, occurrences of such phrases in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, any particular feature, structure, or characteristic of any embodiment can be combined with one or more other features, structures, or characteristics of one or more other embodiments in any suitable manner, without limitation.

[0086]

[0100] In general, this book discloses at least the following:

[0087]

[0101] The retinal projection display system includes at least one visible light source for projecting a visible light image, an infrared light source for projecting infrared light, a scanning mirror having a field of view larger than the visible light image, a reflective surface on which the visible light image is projected and which reflects infrared light at least partially toward the user's eye, the reflective surface being larger than the visible light image, at least one infrared light detector for receiving reflected infrared light reflected from the user's eye, and a hardware computing module comprising a processor and memory, configured to determine the user's line of sight direction at least partially based on the reflected infrared light.

[0088]

[0102] This book further discloses at least the following implementation forms.

[0089]

[0103] The first implementation of the technology described herein is a retinal projection display system,

[0104] A visible light source for projecting a visible light image,

[0105] An infrared light source for projecting infrared light,

[0106] A scanning mirror having a field of view larger than that of a visible light image,

[0107] A reflective surface on which a visible light image is projected and which reflects infrared light at least partially toward the user's eyes, the reflective surface being larger than the visible light image,

[0108] At least one infrared light detector for receiving reflected infrared light reflected from the user's eye,

[0109] The retinal projection display system comprises a hardware computing module having a processor and memory, configured to determine the user's line of sight direction at least partially based on reflected infrared light.

[0090]

[0110] In any further implementation of the aforementioned or subsequent implementations of the retinal projection display system, the hardware computing module is further configured to coordinate the operation of a scanning mirror and at least one visible light source for projecting a visible light image onto a reflective surface based on the line of sight, so that the visible light image is projected onto the user's retina.

[0091]

[0111] In any further implementation of the retinal projection display system described above or below, at least one visible light source and an infrared light source are aligned, and a hardware computing module is further configured to control a scanning mirror to project a visible light image onto a reflective surface in the line of sight direction.

[0092]

[0112] In any further implementation of the retinal projection display system described above or below, at least one visible light source and an infrared light source are not aligned, and the hardware computing module is further configured to compensate for the misalignment between at least one visible light source and an infrared light source to determine the line of sight direction and to control a scanning mirror to project a visible light image onto a reflective surface in the line of sight direction.

[0093]

[0113] In any further implementation of the retinal projection display system described above or below, the interpupillary distance alignment is determined during a calibration operation for the user, and the interpupillary distance alignment identifies the visible area of ​​the reflective surface relative to the user's known line of sight.

[0094]

[0114] In any further implementation of the aforementioned or subsequent implementations of the retinal projection display system, the misalignment between at least one visible light source and an infrared light source is at least partially based on the line of sight direction and interpupillary distance alignment.

[0095]

[0115] In any further implementation of the aforementioned or subsequent implementations of the retinal projection display system, the discrepancy between at least one visible light source and an infrared light source is determined during a manufacturing calibration operation and stored in memory.

[0096]

[0116] In any further implementation of the aforementioned or subsequent implementations of the retinal projection display system, at least one visible light source comprises multiple visible light sources, and the shift between the multiple visible light sources is determined during a manufacturing calibration operation and stored in memory.

[0097]

[0117] In any further implementation of the aforementioned or subsequent implementations of the retinal projection display system, the hardware computing module is configured to align multiple visible light sources at least partially based on the displacement of the visible light sources.

[0098]

[0118] In any further implementation of the aforementioned or subsequent implementations of the retinal projection display system, the hardware computing module is:

[0119] Scan infrared light across the field of view of the reflective surface,

[0120] At least one infrared light detector receives reflected infrared light reflected from the user's eye,

[0121] The amount of reflected infrared light across the field of view of the scanning mirror on the reflective surface is measured.

[0122] The system is further configured to map the amount of reflected infrared light across the field of view of the scanning mirror on a reflective surface to generate an infrared reflectance map of the scanning mirror's field of view, and the infrared reflectance map identifies the line of sight direction.

[0099]

[0123] In any further implementation of the aforementioned or subsequent implementations of the retinal projection display system, the reflective surface is at least partially transparent.

[0100]

[0124] A further implementation of either of the aforementioned or later implementation forms of the retinal projection display system is:

[0125] Eyeglass frames configured to be worn by the user,

[0126] The eyeglass frame further comprises at least one lens mounted within the frame, and the reflective surface is positioned on at least a portion of the at least one lens.

[0101]

[0127] In any further implementation of the retinal projection display system described above or below, at least one infrared light detector is positioned on the eyeglass frame.

[0102]

[0128] In any further implementation of the retinal projection display system described above or below, at least one infrared light detector is located inside a module that includes at least one visible light source and an infrared light source.

[0103]

[0129] Further implementations of the technology described herein include methods for retinal projection displays, which include,

[0130] Projecting a visible light image from at least one visible light source onto a reflective surface larger than the visible light image using a scanning mirror with a field of view larger than the visible light image,

[0131] The process involves projecting infrared light from an infrared light source onto a reflective surface using a scanning mirror, wherein the infrared light is projected across the scanning mirror's field of view and at least partially reflected from the reflective surface toward the user's eye.

[0132] In at least one infrared light detector, reflected infrared light reflected from the user's eye is received,

[0133] This includes determining the user's line of sight direction based at least partially on reflected infrared light.

[0104]

[0134] A further implementation of either of the implementation forms described above or below is:

[0135] The method further includes adjusting the operation of a scanning mirror and at least one visible light source for projecting a visible light image onto a reflective surface based on the line of sight, so that the visible light image is projected onto the user's retina.

[0105]

[0136] In any of the implementation forms described above or below, at least one visible light source and an infrared light source are aligned.

[0137] The method further includes controlling a scanning mirror to project a visible light image onto a reflective surface in the direction of the line of sight.

[0106]

[0138] In any of the implementation forms described above or below, at least one visible light source and an infrared light source are not aligned.

[0139] To determine the shift between at least one visible light source and an infrared light source,

[0140] A method further comprising determining the line of sight direction by compensating for the difference between at least one visible light source and an infrared light source, and controlling a scanning mirror to project a visible light image onto a reflective surface in the line of sight direction.

[0107]

[0141] A further implementation of either of the aforementioned or subsequent implementation forms determines the shift between at least one visible light source and an infrared light source.

[0142] The method further includes obtaining interpupillary distance alignment for a user, the interpupillary distance alignment identifying the visible area of ​​a reflective surface relative to the user's known line of sight direction, and the deviation between at least one visible light source and an infrared light source being at least partially based on the line of sight direction and interpupillary distance alignment.

[0108]

[0143] A further implementation of either of the aforementioned or subsequent implementation forms determines the shift between at least one visible light source and an infrared light source.

[0144] The method further includes obtaining a deviation between at least one visible light source and an infrared light source, wherein the deviation between at least one visible light source and an infrared light source is determined during a manufacturing calibration operation and stored in a memory unit.

[0109]

[0145] A further implementation of either of the implementations described above or below further includes a method in which at least one visible light source comprises multiple visible light sources, and the deviation of the visible light sources between the multiple visible light sources is determined during a manufacturing calibration operation and stored in a memory unit.

[0110]

[0146] A further implementation of either of the above or below implementations further includes a method that further includes aligning a plurality of visible light sources based at least partially on the displacement of a visible light source.

[0111]

[0147] A further implementation of either of the aforementioned or subsequent implementations involves determining the user's line of sight direction at least partially based on reflected infrared light.

[0148] Measuring the amount of reflected infrared light across the field of view of a scanning mirror on a reflective surface,

[0149] The method further includes mapping the amount of reflected infrared light across the field of view of a scanning mirror on a reflective surface to generate an infrared reflectance map of the field of view of the scanning mirror, wherein the infrared reflectance map identifies the line of sight direction.

Claims

1. A visible light source for projecting a visible light image, An infrared light source for projecting infrared light, A scanning mirror having a field of view larger than the aforementioned visible light image, A reflective surface on which the visible light image is projected and on which the infrared light is reflected at least partially toward the user's eyes, the reflective surface being larger than the visible light image, At least one infrared light detector for receiving reflected infrared light reflected from the user's eye, A hardware computing module comprising a processor and memory, wherein the hardware computing module is configured to determine the user's line of sight direction based at least partially on reflected infrared light, Equipped with, The hardware computing module adjusts the operation of a scanning mirror and at least one visible light source for projecting the visible light image onto the reflective surface based on the line of sight, so that the visible light image is projected onto the user's retina. A retinal projection display system in which, if the at least one visible light source and the infrared light source are not aligned, the hardware computing module compensates for the misalignment between the at least one visible light source and the infrared light source by controlling the scanning mirror to project the visible light image onto the region, with the misalignment between the visible light image and the region included in the scanning region indicating the user's line of sight, determined based on the reflected infrared light, within the scanning range of the scanning mirror on the reflective surface, as the misalignment between the at least one visible light source and the infrared light source.

2. The retinal projection display system according to claim 1, wherein interpupillary distance alignment, which is information identifying the visible area of ​​the reflective surface with respect to the user's known line of sight direction, is determined during a calibration operation for the user.

3. The retinal projection display system according to claim 2, wherein the hardware computing module compensates for the misalignment between the at least one visible light source and the infrared light source by moving the visible light image to the visible region identified by the interpupillary distance alignment corresponding to the line of sight.

4. The aforementioned hardware computing module is The infrared light is scanned across the field of view of the reflective surface, The at least one infrared light detector receives reflected infrared light reflected from the user's eye, The amount of reflected infrared light across the field of view of the scanning mirror on the reflective surface is measured. The system is further configured to map the amount of reflected infrared light across the field of view of the scanning mirror on the reflective surface to generate an infrared reflectance map of the field of view of the scanning mirror, The retinal projection display system according to claim 1, wherein the infrared reflectance map identifies the line of sight direction.

5. The retinal projection display system according to claim 1, wherein the reflective surface is at least partially transparent.

6. An eyeglass frame configured to be worn by the aforementioned user, The eyeglass frame comprises at least one lens, Furthermore, The retinal projection display system according to claim 1, wherein the reflective surface is positioned on at least a portion of the at least one lens.

7. The retinal projection display system according to claim 6, wherein the at least one infrared light detector is positioned on the eyeglass frame.

8. The retinal projection display system according to claim 6, wherein the at least one infrared light detector is located inside a module comprising the at least one visible light source and the infrared light source.

9. A method for a retinal projection display, A visible light image from at least one visible light source is projected onto a reflective surface larger than the visible light image using a scanning mirror having a field of view larger than the visible light image. Projecting infrared light from an infrared light source onto the reflective surface using the scanning mirror, wherein the infrared light is projected across the field of view of the scanning mirror and at least partially reflected from the reflective surface toward the user's eye, In at least one infrared light detector, reflected infrared light reflected from the user's eye is received, The user's line of sight direction is determined at least partially based on the reflected infrared light, The operation of the scanning mirror and the at least one visible light source is adjusted based on the line of sight direction so that the visible light image is projected onto the user's retina. If the at least one visible light source and the infrared light source are not aligned, the difference between the region included in the scanning area indicating the user's line of sight direction, determined based on the reflected infrared light, within the scanning range of the scanning mirror on the reflective surface, and the visible light image is determined as the difference between the at least one visible light source and the infrared light source. The scanning mirror is controlled to project the visible light image onto the region, thereby compensating for the misalignment between the at least one visible light source and the infrared light source. Methods that include...

10. The calibration operation for the user further includes determining an interpupillary distance alignment, which is information that identifies the visible area of ​​the reflective surface with respect to the user's known line of sight direction, The method according to claim 9, wherein the misalignment between the at least one visible light source and the infrared light source is compensated by moving the visible light image to the visible region identified by the interpupillary distance alignment corresponding to the line of sight.

11. Determining the user's line of sight direction based at least partially on the reflected infrared light is, To measure the amount of reflected infrared light across the field of view of the scanning mirror on the reflective surface, This includes mapping the amount of reflected infrared light across the field of view of the scanning mirror on the reflective surface to generate an infrared reflectance map of the field of view of the scanning mirror, The method according to claim 9, wherein the infrared reflectance map identifies the line of sight.

Images