Retinal projection display system

Abstract

A retinal projection display system includes a light source for projecting an image, a scanning mirror having a field of view larger than the image, and a reflective surface onto which the image is projected, the reflective surface being larger than the image. The scanning mirror projects the image onto a visible area of ​​the reflective surface such that the image is projected onto a user's retina.

Description

Related Applications

[0001]

[0001] This application claims the priority and benefit of U.S. Provisional Patent Application No. 63 / 260,485, entitled "ADAPTIVE EYE-BOX IN AR SMART GLASSES," filed on August 20, 2021, with Attorney Docket No. IVS-1016-PR, and assigned to the assignee of this application, which is hereby incorporated by reference herein.

[0002]

[0002] This application is related to U.S. Patent Application No. 17 / 820,876, entitled "A RETINAL PROJECTION DISPLAY SYSTEM," filed on August 18, 2022, with Attorney Docket No. IVS-1016, and assigned to the assignee of this application, which is hereby incorporated by reference herein.

Background Art

[0003]

[0003] A retinal projection display (RPD), also referred to as a virtual retinal display (VRD), is used to project an image directly onto the retina through the pupil of the eye. Image rendering is performed at a speed sufficient for the human eye to perceive a continuous video stream of images. Due to the small area, also referred to as the "eye-box," through which the image is projected onto the retina through the pupil, it is essential to perform an accurate alignment between the RPD and the eye to ensure that the image enters the eye. Furthermore, the user's line of sight may change during the use of the RPD, thereby changing the position of the eye-box, so it is necessary to consider the change in the line of sight direction during the use of the RPD.

[0004]

[0004] The accompanying drawings, incorporated into and forming part of the description of the embodiments, illustrate various non-exclusive and non-exclusive embodiments of the subject matter and, together with the description of the embodiments, are useful in illustrating the principles of the subject matter discussed below. Unless otherwise specified, drawings referenced in this brief description should be understood as not being drawn to a fixed scale, and similar reference numbers refer to similar parts throughout the various drawings unless otherwise specified. [Brief explanation of the drawing]

[0005] [Figure 1] This figure shows an exemplary retinal projection display system according to several embodiments. [Figure 2] This figure shows a functional block diagram of an exemplary retinal projection display system according to several embodiments. [Figure 3] This figure shows an exemplary interpupillary distance alignment operation by projecting an alignment image onto a reflective surface, according to several embodiments. [Figure 4] This figure shows an exemplary line-of-sight tracking operation used during the projection of an image onto a reflective surface, according to several embodiments. [Figure 5A] This figure shows an exemplary scanning pattern across the entire scanning range of a scanning mirror according to one embodiment. [Figure 5B] This figure shows an example scanning pattern when the scanning range of the scanning mirror is the size of the projected image. [Figure 6] This figure shows an exemplary image rendering operation according to one embodiment, where the scanning range of the scanning mirror is larger than the size of the projected image. [Figure 7] This is a block diagram of an exemplary electronic device in which embodiments described herein may be implemented. [Figure 8] This figure shows an exemplary process of retinal projection according to several embodiments. [Figure 9] This figure shows an exemplary process for determining interpupillary distance alignment according to several embodiments. [Modes for carrying out the invention]

[0006]

[0015] The following description of embodiments is provided merely as an example and is not intended to limit it. Furthermore, it is not intended to be bound by any explicit or implicit theory presented in the above-mentioned background art or in the following description of embodiments.

[0007]

[0016] Various embodiments of the subject matter are referenced in detail here, examples of which are shown in the accompanying drawings. While various embodiments are discussed herein, it will be understood that they are not intended to be limiting. On the contrary, the presented embodiments are intended to encompass alternatives, modifications, and equivalents that may fall within the spirit and scope of the various embodiments defined by the accompanying claims. Furthermore, this description of embodiments includes numerous specific details to provide a complete understanding of the embodiments of the subject matter. However, embodiments may be carried out without these specific details. In other examples, well-known methods, procedures, components, and circuits are not described in detail so as not to unnecessarily obscure the aspects of the embodiments described.

[0008] Notation and Nomenclature

[0017] Some parts of the following detailed description are presented with respect to procedures, logical blocks, processes, and other symbolic representations of operations on data within electrical devices. These descriptions and representations are means used by those skilled in the field of data processing to communicate the content of their work to others skilled in the field in the most effective way. In this application, procedures, logical blocks, processes, etc., are considered to be one or more consistent procedures or instructions that produce a desired result. Procedures are those that require the physical manipulation of physical quantities. Usually, but not always, these quantities take the form of acoustic (e.g., ultrasonic) signals that can be transmitted and received by electronic devices, and / or electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated within electrical devices.

[0009]

[0018] However, it should be noted that all of these terms and similar terms should be associated with appropriate physical quantities and are merely convenient labels applied to those quantities. As will be evident from the following descriptions, unless otherwise specified, descriptions throughout the descriptions of embodiments that use terms such as “performing,” “determining,” “detecting,” “directing,” “calculating,” “correcting,” “providing,” “receiving,” “analyzing,” “confirming,” “displaying,” “presenting,” “using,” “completing,” “instructing,” “comparing,” “executing,” “tracking,” “moving,” “retrieving,” “projecting,” and “calibrating” are understood to refer to the operation and processes of electronic devices such as electrical devices.

[0010]

[0019] The embodiments described herein may be described in the general context of processor-executable instructions residing on some form of non-transient processor-readable medium, such as program modules, executed by one or more computers or other devices. Generally, a program module includes routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. The functions of the program module may be combined or distributed as desired in various embodiments.

[0011]

[0020] In the diagrams, a single block may be described as performing one or more functions; however, in practice, the one or more functions performed by that block may be performed in a single component or across multiple components, and / or using hardware, software, or a combination of hardware and software. To clearly demonstrate this hardware and software compatibility, various exemplary components, blocks, modules, logic, circuits, and steps have been described in general terms with respect to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. A person skilled in the art may implement the described functions in various ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this disclosure. Furthermore, the exemplary ultrasonic sensing systems and / or mobile electronic devices described herein may include components other than those shown, including well-known components.

[0012]

[0021] The various techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a particular manner. Any feature described as a module or component may also be implemented together in an integrated logic device, or separately as separate but interoperable logic devices. When implemented in software, the technique may be at least partially implemented by a non-temporary processor-readable storage medium having instructions that, when executed, perform one or more of the methods described herein. The non-temporary processor-readable data storage medium may form part of a computer program product, which may include packaging materials.

[0013]

[0022] Non-temporary processor-readable storage media may include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, and other known storage media. The technique may be additionally or alternatively implemented by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures, and which can be accessed, read, and / or executed by a computer or other processor.

[0014]

[0023] The various embodiments described herein may be implemented by one or more processors, such as motion processing units (MPUs), sensor processing units (SPUs), host processors or their cores, digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), field-programmable gate arrays (FPGAs), programmable logic controllers (PLCs), complex-programmable logic devices (CPLDs), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein, or by other equivalent integrated circuits or discrete logic circuits. As used herein, the term “processor” may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. As used herein, the term “processor” may refer to substantially any computing processing unit or device, including, but not limited to, single-core processors, single-processors with software multithreading capabilities, multi-core processors, multi-core processors with software multithreading capabilities, multi-core processors with hardware multithreading technology, parallel platforms, and parallel platforms with distributed shared memory. Furthermore, to optimize space utilization or improve the performance of user devices, the processor may utilize nanoscale architectures such as transistors, switches, and gates based on molecules and quantum dots, but is not limited to these. The processor may also be implemented as a combination of computing processing units.

[0015]

[0024] In addition, in some embodiments, the functions described herein may be provided within a dedicated software module or hardware module configured as described herein. Also, the techniques may be implemented entirely in one or more circuits or logic elements. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, e.g., a combination of an SPU / MPU and a microprocessor, a plurality of microprocessors, SPU cores, MPU cores, or one or more microprocessors in combination with any other such configuration.

[0016] Summary of the Description

[0025] The discussion begins with an explanation of an exemplary retinal projection display system. Next, an exemplary operation of the retinal projection display system is described.

[0017]

[0026] Embodiments herein provide a retinal projection display system that includes a light source for projecting an image, a scanning mirror having a field of view larger than the image, and a reflective surface onto which the image is projected, the reflective surface being larger than the image. The scanning mirror projects the image onto a visible region of the reflective surface such that the image is projected onto the user's retina. In some embodiments, the reflective surface is at least partially transparent. In some embodiments, the retinal projection display further includes an eyeglass frame configured to be worn by the user and at least one lens mounted within the eyeglass frame, and the reflective surface is disposed on at least a portion of at least one lens.

[0018]

[0027] In some embodiments, the interpupillary distance alignment is determined during a calibration operation for the user, and the interpupillary distance alignment identifies the visible region of the reflective surface relative to the known line-of-sight direction of the user. In some embodiments, the image is displayed at multiple locations on the reflective surface during the calibration operation, and the interpupillary distance alignment is determined in response to feedback from the user identifying the visible region of the reflective surface.

[0019]

[0028] In some embodiments, the retinal projection display system also includes an eye tracker for tracking the user's line-of-sight direction, and the visible region corresponds to the line-of-sight direction. The scanning mirror is configured to dynamically move an image on the reflective surface according to the user's line-of-sight direction and the user's interpupillary distance alignment. In some embodiments, the scanning mirror is configured to dynamically move an image on the reflective surface according to the user's line-of-sight direction in response to the line-of-sight direction satisfying a movement threshold. In some embodiments, the scanning mirror is configured to dynamically move an image on the reflective surface according to the user's line-of-sight direction after a predetermined time delay. In some embodiments, the image is smaller than the visible region, and the scanning mirror is configured to dynamically move an image on the reflective surface according to the user's line-of-sight direction in response to the image moving outside the visible region.

[0020]

[0029] In some embodiments, the scanning range of the scanning mirror is dynamically adjusted to correspond to the size of the image within the visible region. In some embodiments, the scanning range of the scanning mirror corresponds to the size of the display area of the reflective surface, and thus the light source is activated to display the image only when the scanning mirror is projecting the image within the visible region.

[0021]

[0030] Other embodiments described herein provide methods for retinal projection. An image from a light source is projected onto a reflective surface using a scanning mirror having a field of view larger than the image, and the reflective surface is larger than the image. In some embodiments, the scanning range of the scanning mirror is dynamically adjusted to correspond to the size of the image in the visible region. In some embodiments, the scanning range of the scanning mirror corresponds to the size of the display area of ​​the reflective surface, and therefore the light source is operated to display the image only when the scanning mirror is projecting the image in the visible region. In some embodiments, the reflective surface is at least partially transparent.

[0022]

[0031] The visible area of ​​the reflective surface for the user is determined. In some embodiments, the interpupillary distance alignment for the user is obtained (e.g., from memory), and the interpupillary distance alignment identifies the visible area of ​​the reflective surface for the user's known line of sight direction. In some embodiments, the user's line of sight direction is tracked, and the visible area corresponds to the line of sight direction.

[0023]

[0032] The image is oriented over the visible area of ​​the reflective surface so that the image is projected onto the user's retina. In some embodiments, the image is dynamically moved over the reflective surface using a scanning mirror according to the user's line of sight and interpupillary distance alignment.

[0024]

[0033] In some embodiments, the amount of movement in the line of sight is determined based on tracking the user's line of sight, and in response to the amount of movement in the line of sight satisfying a movement threshold, the image is moved on the reflective surface according to the user's line of sight and the user's interpupillary distance alignment. In some embodiments, the image is moved on the reflective surface according to the user's line of sight and the user's interpupillary distance alignment after a predetermined time delay. In some embodiments, in response to the determination that the image is outside the visible region, the image is moved on the reflective surface according to the user's line of sight and the user's interpupillary distance alignment.

[0025] Exemplary retinal projection display system

[0034] Figure 1 shows an exemplary retinal projection display system 100 according to several embodiments. The retinal projection display system 100 includes a light source 110, a scanning mirror 120, and a reflective surface 130. In the illustrated embodiments, the components of the retinal projection display system 100 are contained within an eyeglass frame 160 configured to be worn by a user. The eyeglass frame 160 includes at least one lens 162 mounted therein, and the reflective surface 130 is positioned on at least a portion of the lens 162 such that the reflective surface 130 is within the user's field of view when looking through the lens 162. In some embodiments, the reflective surface 130 is at least partially transparent, allowing the user to see through the reflective surface 130 and the lens 162. It should be understood that the various components of the retinal projection display system 100, such as the light source 110 and the scanning mirror 120, may be positioned on or within the hollow eyeglass frame 160 (e.g., within the hollows 115 of the arms / temples) and positioned so that a light beam 112 is projected onto the reflective surface 130.

[0026]

[0035] During operation, the light source 110 (e.g., a laser) projects an image onto a portion of the reflective surface 130 by generating a light beam 112 that is projected onto the scanning mirror 120. In some embodiments, the light source 110 is a single light source capable of projecting a complete image. In some embodiments, the light source 110 includes multiple light sources, such as separate red, green, and blue (RGB) lasers, that work in coordination to project a complete image. It should be understood that many types of light sources can be used according to the embodiments described.

[0027]

[0036] The scanning mirror 120 is configured to move and direct the light beam 112 so that it is scanned over the reflective surface 130 and each point of the image is positioned on the reflective surface 130, and the reflective surface 130 directs the light beam 112 onto the retina 152 through the pupil 154 of the user's eye 150. It should be understood that various scanning patterns may be used, as described below. It should be understood that the image scanning process is performed at a scanning speed fast enough (e.g., above 60 Hz) so that the user perceives the entire image or the image as a continuous video. In some embodiments, the scanning mirror 120 is a micro-electromechanical (MEMS) device.

[0028]

[0037] The scanning mirror 120 has a field of view (FOV) larger than the intended visible image size, and the reflective surface 130 onto which the visible image is projected is also larger than the intended visible image size. The scanning mirror 120 projects an image onto the visible region of the reflective surface 130 so that the image is projected onto the user's retina 152. The larger FOV allows the retinal projection display system 100 to project the image appropriately into the pupil 154 and onto the retina 152, independently of the movement and rotation of the eye 150. According to some embodiments, the retinal projection display system 100 facilitates the projection and alignment of the intended visible image with the pupil 154 by projecting onto the visible region of the reflective surface 130 on the dynamic range window of the scanning mirror.

[0029]

[0038] Interpupillary distance alignment is used to direct the light beam 112 into the pupil 154 of the eye 150, and identifies the visible area of ​​the reflective surface 130 relative to the user's known line of sight. In some embodiments, interpupillary distance alignment is determined during a calibration operation for the user. In some embodiments, images are displayed at multiple locations on the reflective surface 130 during the calibration operation, and interpupillary distance alignment is determined in response to user feedback identifying the visible area of ​​the reflective surface. For example, user feedback can be provided using the user interface of the retinal projection display system 100 and can be received in many ways, such as voice commands, buttons located on the eyeglass frame 160, or applications on connected devices such as smartphones.

[0030]

[0039] In some embodiments, the retinal projection display system 100 also includes an eye-tracker 140 for tracking the user's line of sight. The visible area of ​​the reflective surface 130 corresponds to the user's line of sight. The scanning mirror 120 is configured to dynamically move the image on the reflective surface 130 according to the user's line of sight and the user's interpupillary distance alignment. Since the interpupillary distance alignment for the user identifies the visible area of ​​the reflective surface 130 for the user's known line of sight, the scanning mirror 120 can move the image according to the line of sight to correspond to the visible area of ​​the reflective surface.

[0031]

[0040] In some embodiments, to avoid visible image jitter, the scanning mirror 120 is configured to dynamically move the image on the reflective surface 130 according to the user's line of sight in response to the line of sight direction meeting a movement threshold. For example, the scanning mirror 120 moves the image on the reflective surface only when sufficient movement in the line of sight direction is detected. In some embodiments, jitter is addressed by providing a rendered image smaller than the visible area of ​​the reflective surface 130, so that the scanning mirror 120 is configured to dynamically move the image on the reflective surface 130 according to the user's line of sight in response to the image moving outside the visible area. This allows the image to be viewed over a wider range of positions on the reflective surface 130, minimizing jitter.

[0032]

[0041] In some embodiments, to avoid image blurring, the scanning mirror 120 is configured to dynamically move the image on the reflective surface 130 according to the user's line of sight direction after a predetermined time delay following a change in the line of sight direction, allowing the eye 150 to settle into the new line of sight direction before the image is moved.

[0033]

[0042] In some embodiments, the scanning range of the scanning mirror 120 is dynamically adjusted to correspond to the size of the image within the visible region. In other embodiments, the scanning range of the scanning mirror 120 corresponds to the size of the display area of ​​the reflective surface 130, and therefore the light source 110 is activated to display the image only when the scanning mirror 120 is projecting the image within the visible region of the reflective surface 130.

[0034]

[0043] Figure 2 shows a functional block diagram of an exemplary retinal projection display system 200 according to several embodiments. The retinal projection display system 200 includes a light source 210, a scanning mirror 220, an eye-tracker 230, and a reflective surface. The light source 210 receives image data 205 from a data source for projection. It should be understood that the image data 205 may include any type of data for displaying or rendering an image, including still image data, video data (e.g., a series of images), or other data for user visualization. In some embodiments, the light source 210 is a single light source capable of projecting a complete image. In some embodiments, the light source 210 includes multiple light sources, such as separate red, green, and blue (RGB) lasers, that work in coordination to project a complete image. It should be understood that many types of light sources may be used according to the embodiments described.

[0035]

[0044] A light source 210 (e.g., a laser) projects the image 215 onto a scanning mirror 220. The image 215 is projected as a scan of its pixels, and the scanning mirror 220 dynamically moves to position each pixel appropriately on the reflective surface 240 for rendering. The scanning mirror 220 is configured to move and orient the pixels of the image 215 so that each pixel of the image is scanned on the reflective surface 240, positioning each point of the image on the reflective surface 240, which orients the image 215 into the user's pupil and retina. Various scanning patterns may be used, as described below. The image scanning process should be performed at a scanning speed fast enough (e.g., above 60 Hz) for the user to perceive the entire image or a continuous video of the image.

[0036]

[0045] The scanning mirror 220 uses the interpupillary distance alignment 225 and line of sight direction 235 relative to the user to control the position of the pixels in the image 215 so that they are directed onto the user's retina. The interpupillary distance alignment identifies the visible area of ​​the reflective surface 240 relative to the user's known line of sight direction.

[0037]

[0046] Figure 3 shows an exemplary interpupillary distance alignment operation 300 by projecting an alignment image onto a reflective surface, according to several embodiments. In some embodiments, the interpupillary distance alignment operation 300 is determined during a calibration operation for the user. In some embodiments, during the interpupillary distance alignment operation 300, the alignment image 310 is displayed at multiple locations on the reflective surface 330, and the interpupillary distance alignment is determined in response to user feedback identifying the visible area 320 of the reflective surface 330. For example, user feedback can be provided using the user interface of the retinal projection display system 100 and can be received in many ways, such as voice commands, buttons located on the eyeglass frame 160, or applications on connected devices such as smartphones.

[0038]

[0047] During the interpupillary distance alignment operation 300, the user is instructed (e.g., via a user interface) to look in a specific direction (e.g., straight ahead). The alignment image 310 is rendered onto the reflective surface 330 and moved across the dynamic range of the scanning mirror to display the alignment image at multiple locations on the reflective surface 330. The user provides feedback (e.g., when prompted) regarding whether the alignment image 310 is fully visible, partially visible, or not visible. If the alignment image 310 is within the visible area 320 and is, for example, partially or fully visible to the user, the user provides feedback indicating that the alignment image 310 is visible. In some embodiments, the alignment image 310 is configured to assist in alignment. For example, the alignment image 310 may include information that identifies parts of the alignment image 310, such as letters, arrows, colors, or other indicators, which the user can use to indicate which parts of the alignment image 310 to look at, so that the retinal projection display system knows how to move the alignment image 310 within the visible area 320.

[0039]

[0048] As shown in the figure, the alignment image 310 is projected onto the reflective surface 330. In the illustrated example, at the first time interval 340, the alignment image 310 is not within the user's visible area 320 of the reflective surface 330 corresponding to the user's known line of sight direction (e.g., straight ahead). The user provides feedback that the alignment image 310 is not visible to the user. At the second time interval 350, the alignment image 310 is moved to a different position on the reflective surface 330, which is still not within the visible area 320. As shown in the figure, the visible area 320 remains substantially stationary during the interpupillary distance alignment operation 300. The user provides feedback that the alignment image 310 is not visible to the user.

[0040]

[0049] At time 360°, the alignment image 310 is moved to a different position on the reflective surface 330, which is partially within the visible area 320. The user is given feedback that the alignment image 310 is partially visible to the user. At time 470°, the alignment image 310 is moved to a different position on the reflective surface 330, which is fully within the visible area 320. The user is given feedback that the alignment image 310 is fully visible to the user. The position of the alignment image 310 at time 470° is stored and used as the interpupillary distance alignment relative to the user's known line of sight direction (e.g., straight ahead). The interpupillary distance alignment is stored (e.g., in the memory of the retinal projection display system).

[0041]

[0050] Referring to Figure 2, the eye-tracker 230 is for tracking the user's gaze direction 235. The visible area of ​​the reflective surface 240 corresponds to the user's gaze direction, and as the gaze direction 235 moves, the user's visible area on the reflective surface 240 also moves. The scanning mirror 220 is configured to dynamically move the image 215 on the reflective surface 240 according to the user's gaze direction 235 and the user's interpupillary distance alignment 225. Since the interpupillary distance alignment 225 for the user identifies the visible area of ​​the reflective surface 240 for the user's known gaze direction, the scanning mirror 220 can move the image 215 to correspond to the visible area of ​​the reflective surface 240 according to the gaze direction 235.

[0042]

[0051] Figure 4 shows an exemplary eye-tracking operation 400 used during the projection of an image onto a reflective surface according to several embodiments. During the eye-tracking operation 400, the eye-tracker (e.g., eye-tracker 140 in Figure 1 or eye-tracker 230 in Figure 2) is configured to track the user's gaze direction. Eye-tracking is understood by those skilled in the art, and it should be understood that any type of eye-tracking technique or operation may be used or performed to perform the eye-tracking operation 400. Since the interpupillary distance alignment that identifies the visible area relative to the user's known gaze direction is known, the gaze direction can be used to move the user's visible area relative to any gaze direction.

[0043]

[0052] As shown in the figure, the image is projected onto the visible area 420 of the reflective surface 410. In the illustrated example, at the first time 430, the image is within the user's visible area 420 of the reflective surface 410, corresponding to the user's gaze direction identified by eye tracking. At the second time 440, when the user's gaze direction has shifted, the visible area 420, like the image, shifts in accordance with the user's gaze direction identified by eye tracking at a later time.

[0044]

[0053] In some embodiments, to avoid jitter in the visible image, the visible region 420 is moved only when sufficient movement in the line of sight is detected (e.g., when a movement threshold is met). In some embodiments, jitter is addressed by providing a rendered image smaller than the visible region 420, such that the visible region 420 is moved in response to the user's line of sight moving outside the visible region 420. This allows the image to be viewed over a wider range of positions and minimizes jitter. In some embodiments, to avoid image blurring, the visible region 420 is moved according to the user's line of sight after a predetermined time delay following the change in line of sight, allowing the user's eyes to settle into the new line of sight before the image is moved.

[0045]

[0054] Referring to Figure 2, the scanning mirror 220 projects the image 215 onto the visible area of ​​the reflective surface 240 (e.g., pixel by pixel) so that the image 215 is projected onto the user's retina. In some embodiments, the scanning range of the scanning mirror 220 is dynamically adjusted to correspond to the size of the image in the visible area. In other embodiments, the scanning range of the scanning mirror 220 corresponds to the size of the display area of ​​the reflective surface 240, and therefore the light source 210 is activated to display the image only when the scanning mirror 220 is projecting the image into the visible area of ​​the reflective surface 240.

[0046]

[0055] Figures 5A and 5B show an exemplary image rendering operation 500 according to one embodiment, where the scanning range of the scanning mirror is the size of the projected image. Figure 5A shows an exemplary scanning pattern over the entire scanning range 510 of the scanning mirror, and the scanning mirror can render an image at any point on the reflective surface, where the reflective surface is larger than the visible area of ​​the scanning surface. As shown, the scanning mirror is configured to move in the x and y directions, and the center position 520 is the center position of the user's identified line of sight and visible area. Eye tracking is used to determine the center position of the visible area.

[0047]

[0056] Figure 5B shows an exemplary scanning pattern when the scanning range of the scanning mirror is the size of the projected image. As shown in Figure 5B, the center position 530 is identified using eye-tracking and moved relative to the center position 520. The pan angle of the scanning mirror is controlled so that the x and y scanning ranges 550 cover the visible area 540 identified by the center position B. In the illustrated embodiment, the scanning range 550 of the scanning mirror is reduced to the visible area 540 covering the reflective surface, rather than the entire reflective surface.

[0048]

[0057] Figure 6 shows an exemplary image rendering operation 600 in one embodiment where the scanning range 610 of the scanning mirror is larger than the size of the projected image. As shown, the scanning mirror is configured to move in the x and y directions, and the center position 620 is the center position of the user's identified line of sight and visible area. Eye tracking is used to determine the center position of the visible area.

[0049]

[0058] As shown in Figure 6, the center position 630 is identified using eye-tracking, and the center position 630 is moved relative to the center position 620. Depending on the line of sight direction by the center position 630, the light source is activated only when the scanning mirror is within the visible area 640. The pan angle of the scanning mirror is controlled so that the x and y scanning ranges 610 cover the entire reflective surface area, but the light source is activated only when the scanning mirror is within the visible area 640.

[0050]

[0059] Figure 7 is a block diagram of an exemplary electronic device 700 that can implement embodiments of the present invention. Figure 7 shows an example of the type of electronic device 700 (e.g., a computer system) that may be used according to or to implement the various embodiments discussed herein. It should be understood that the embodiments of the retinal projection display system described may be implemented using the exemplary electronic device 700.

[0051]

[0060] The electronic device 700 in Figure 7 is merely an example, and it should be understood that the embodiments described herein, without limitation, may operate on or within several different computer systems, including general-purpose networked computer systems, embedded computer systems, mobile electronic devices, smartphones, server devices, client devices, various intermediate devices / nodes, standalone computer systems, media centers, handheld computer systems, multimedia devices, and the like. In some embodiments, the electronic device 700 in Figure 7 is well adapted to have a peripheral tangible computer-readable storage medium 702, which may be combined with, for example, an electronic flash memory data storage device, a floppy disk, a compact disk, a digital multipurpose disk, other disk-based storage devices, a Universal Serial Bus "thumb" drive, a removable memory card, and the like. The tangible computer-readable storage medium is inherently non-temporary.

[0052]

[0061] The electronic device 700 in Figure 7 includes an address / data bus 704 for communicating information and a processor 706A coupled to the bus 704 for processing information and instructions. The bus 704 may be any suitable bus or interface, including, but not limited to, a Peripheral Component Interconnection Express (PCIe) bus, a Universal Serial Bus (USB), a Universal Asynchronous Receiver / Transmitter (UART) serial bus, a suitable Advanced Microcontroller Bus Architecture (AMBA) interface, an Inter-Integrated Circuit (I2C) bus, a Serial Digital Input / Output (SDIO) bus, a Serial Peripheral Interface (SPI), or other equivalents.

[0053]

[0062] As shown in Figure 7, the electronic device 700 is also well suited to a multiprocessor environment in which multiple processors 706A, 706B, and 706C exist. Conversely, the electronic device 700 is also well suited to having a single processor, such as processor 706A. Processors 706A, 706B, and 706C may be any of various types of microprocessors. The electronic device 700 also includes data storage functions such as computer-available volatile memory 708, e.g., random access memory (RAM), coupled to bus 704, for storing information and instructions for processors 706A, 706B, and 706C. The electronic device 700 also includes computer-available non-volatile memory 710, e.g., read-only memory (ROM), coupled to bus 704, for storing static information and instructions for processors 706A, 706B, and 706C. The electronic device 700 also includes a data storage unit 712 (e.g., magnetic or optical disks and disk drives) coupled to the bus 704 for storing information and instructions. The electronic device 700 also includes an alphanumeric input device 714, including alphanumeric keys and function keys, coupled to the bus 704 for communicating information and command selections to the processor 706A or processors 706A, 706B, and 706C. The electronic device 700 also includes a cursor control device 716 coupled to the bus 704 for communicating user input information and command selections to the processor 706A or processors 706A, 706B, and 706C. In one embodiment, the electronic device 700 also includes a display device 718 coupled to the bus 704 for displaying information. Depending on the architecture, different bus configurations may be adopted as needed. For example, additional buses may be used to connect various components of the electronic device 700, such as by using a dedicated bus between the processor 706A and the memory computer-available volatile memory 708 or the computer-available non-volatile memory 710.

[0054]

[0063] Continuing to refer to Figure 7, the display device 718 in Figure 7 may include a light source (e.g., light source 110 in Figure 1) for projecting image data onto a reflective surface. In other embodiments, the display device 718 may be a liquid crystal display (LCD), a light-emitting diode display (LED) device, a plasma display device, a touchscreen device, or other display device suitable for creating user-recognizable graphic images and alphanumeric characters. The cursor control device 716 enables the computer user to dynamically indicate the movement of a visible symbol (cursor) on the display screen of the display device 718 and to indicate user selection of selectable items displayed on the display device 718. Many implementations of the cursor control device 716 are known in the art, including a trackball, mouse, touchpad, touchscreen, joystick, or special keys on an alphanumeric input device 714 capable of signaling a method of movement or displacement in a given direction. Alternatively, it should be understood that the cursor may be directed and / or actuated via input from the alphanumeric input device 714 using special keys and key sequence commands. The electronic device 700 is also suitable for directing the cursor by other means, such as voice commands. In various embodiments, the alphanumeric input device 714, the cursor control device 716, and the display device 718, or any combination thereof (e.g., a user interface selection device), may operate collectively under the direction of a processor (e.g., processor 706A or processors 706A, 706B, and 706C) to provide a graphical user interface (GUI) 730. The GUI 730 allows the user to interact with the electronic device 700 through a graphical representation presented on the display device 718 by interacting with the alphanumeric input device 714 and / or the cursor control device 716.

[0055]

[0064] The electronic device 700 also includes an I / O device 720 for connecting the electronic device 700 to an external entity. For example, in one embodiment, the I / O device 720 is a modem for enabling wired or wireless communication between the electronic device 700 and an external network, such as the Internet, but not limited to the Internet. In one embodiment, the I / O device 720 includes a transmitter. The electronic device 700 may communicate with the network by transmitting data through the I / O device 720.

[0056]

[0065] Continuing to refer to Figure 7, various other components for the electronic device 700 are shown. Specifically, if present, the operating system 722, applications 724, modules 726, and data 728 are shown to reside typically in one or a combination of computer-readable volatile memory 708 (e.g., RAM), computer-readable non-volatile memory 710 (e.g., ROM), and data storage units 712. In some embodiments, all or some of the various embodiments described herein are stored, for example, as memory locations in RAM 708, computer-readable storage media in data storage units 712, peripheral computer-readable storage media 702, and / or applications 724 and / or modules 726 in other tangible computer-readable storage media.

[0057] Exemplary operation for operating a retinal projection display system

[0066] Figure 8 shows an exemplary process of retinal projection according to several embodiments, and Figure 9 shows an exemplary process for determining interpupillary distance alignment according to several embodiments. The steps of these methods are described with reference to elements and / or components of various figures described herein. It will be understood that in some embodiments, the steps may be performed in a different order than described, some of the described steps may be omitted, and / or one or more additional steps may be performed in addition to those described. The flow charts, in various embodiments, include several steps performed by one or more processors (e.g., a host processor or a sensor processor) under the control of computer-readable and computer-executable instructions stored in a non-temporary computer-readable storage medium. It will be further understood that one or more steps described in the flow charts may be implemented in hardware, or in combination of hardware and firmware and / or software.

[0058]

[0067] Referring to Figure 8, flowchart 800 illustrates an exemplary process of retinal projection according to several embodiments. In step 810 of flowchart 800, an image from a light source is projected onto a reflective surface larger than the image using a scanning mirror having a field of view larger than the image. In some embodiments, the scanning range of the scanning mirror is dynamically adjusted to correspond to the size of the image in the visible region. In some embodiments, the scanning range of the scanning mirror corresponds to the size of the display area of ​​the reflective surface, and therefore the light source is activated to display the image only when the scanning mirror is projecting the image within the visible region. In some embodiments, the reflective surface is at least partially transparent.

[0059]

[0068] In step 820, the visible area of ​​the reflective surface for the user is determined. In some embodiments, as shown in step 822, the interpupillary distance alignment for the user is obtained (e.g., from memory), and the interpupillary distance alignment identifies the visible area of ​​the reflective surface for the user's known line of sight direction. In some embodiments, as shown in step 824, the user's line of sight direction is tracked, and the visible area corresponds to the line of sight direction.

[0060]

[0069] In step 830, the image is oriented over the visible area of ​​the reflective surface so that the image is projected onto the user's retina. In some embodiments, as shown in step 832, the image is dynamically moved over the reflective surface using a scanning mirror according to the user's line of sight and interpupillary distance alignment.

[0061]

[0070] In some embodiments, as shown in step 834, the amount of movement in the line of sight is determined based on tracking the user's line of sight, and in response to the amount of movement in the line of sight satisfying a movement threshold, the image is moved on the reflective surface according to the user's line of sight and the user's interpupillary distance alignment. In some embodiments, as shown in step 836, the image is moved on the reflective surface according to the user's line of sight and the user's interpupillary distance alignment after a predetermined time delay. In some embodiments, as shown in step 838, in response to the determination that the image is outside the visible region, the image is moved on the reflective surface according to the user's line of sight and the user's interpupillary distance alignment.

[0062]

[0071] Figure 9 shows an exemplary flowchart 900 for determining interpupillary distance alignment, for example, during a calibration operation, according to several embodiments. In step 910 of flowchart 900, an alignment image is projected onto a reflective surface. In step 920, user feedback is received regarding the visibility of the alignment image while the user gazes at a known line of sight. In step 930, it is determined, according to the user feedback, whether the alignment image is within the visible area. If the alignment image is not within the visible area, flowchart 900 proceeds to step 940. In step 940, the position of the alignment image is moved to a different position on the reflective surface. If the alignment image is within the visible area, flowchart 900 proceeds to step 950.

[0063]

[0072] In step 950, it is determined, based on user feedback, whether the alignment image is entirely within the visible area. If the alignment image is not entirely within the visible area, flowchart 900 proceeds to step 940. In step 940, the position of the alignment image is moved to a different position on the reflective surface. If the alignment image is entirely within the visible area, flowchart 900 proceeds to step 960. In step 960, an interpupillary distance alignment is determined that identifies the visible area of ​​the reflective surface relative to the user's known line of sight. In some embodiments, the interpupillary distance alignment is stored (e.g., in memory) for retrieval during retinal projection display operation.

[0064] conclusion

[0073] The examples described herein are presented to best illustrate, describe specific uses, and thereby enable those skilled in the art to construct and use embodiments of the described examples. However, those skilled in the art will recognize that the foregoing description and examples are presented for illustrative purposes only. Many aspects of the different exemplary embodiments described above can be combined to form new embodiments. The description is not intended to be exhaustive or to limit embodiments to the exact forms disclosed. Rather, the particular features and actions described above are disclosed as exemplary forms that implement the claims.

[0065]

[0074] Throughout this specification, any reference to “one embodiment,” “a particular embodiment,” “a certain embodiment,” “various embodiments,” “several embodiments,” or similar terms means that a particular feature, structure, or characteristic described in relation to an embodiment is included in at least one embodiment. Therefore, occurrences of such phrases in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, any particular feature, structure, or characteristic of any embodiment can be combined with one or more other features, structures, or characteristics of one or more other embodiments in any suitable manner, without limitation.

[0066]

[0075] In general, this book discloses at least the following:

[0067]

[0076] A retinal projection display system includes a light source for projecting an image, a scanning mirror with a field of view larger than the image, and a reflective surface onto which the image is projected, the reflective surface being larger than the image. The scanning mirror projects the image onto the visible area of ​​the reflective surface so that the image is projected onto the user's retina.

[0068]

[0077] This book further discloses at least the following implementation forms.

[0069]

[0078] The first implementation of the technology described herein is a retinal projection display system,

[0079] A light source for projecting an image,

[0080] A scanning mirror with a field of view larger than the image,

[0081] A reflective surface onto which an image is projected, comprising a reflective surface larger than the image,

[0082] The scanning mirror is equipped with a retinal projection display system that projects an image onto the visible region of a reflective surface so that the image is projected onto the user's retina.

[0070]

[0083] A further implementation of either of the aforementioned or subsequent implementations includes a retinal projection display system in which the interpupillary distance alignment is determined during a calibration operation for the user, and the interpupillary distance alignment identifies the visible area of ​​a reflective surface relative to the user's known line of sight.

[0071]

[0084] In any further implementation of the retinal projection display system described above or below, the image is displayed at multiple locations on the reflective surface during calibration, and the interpupillary distance alignment is determined in response to user feedback identifying the visible area of ​​the reflective surface.

[0072]

[0085] A further implementation of either of the aforementioned or later implementation forms of the retinal projection display system,

[0086] An eye-tracking device for tracking the direction of a user's gaze, further comprising an eye-tracking device whose visible area corresponds to the direction of gaze,

[0087] The scanning mirror is configured to dynamically move the image on the reflective surface according to the user's line of sight direction and the user's interpupillary distance alignment.

[0073]

[0088] In any further implementation of the aforementioned or subsequent implementations of the retinal projection display system, the scanning mirror is configured to dynamically move the image on the reflective surface according to the user's line of sight in response to the user's line of sight direction satisfying a movement threshold.

[0074]

[0089] In any further implementation of the aforementioned or subsequent implementations of the retinal projection display system, the scanning mirror is configured to dynamically move the image on the reflective surface according to the user's line of sight after a predetermined time delay.

[0075]

[0090] In any further implementation of the aforementioned or subsequent implementations of the retinal projection display system, the image is smaller than the visible area, and the scanning mirror is configured to dynamically move the image on the reflective surface according to the user's line of sight in response to the image moving outside the visible area.

[0076]

[0091] In any further implementation of the retinal projection display system described above or below, the scanning range of the scanning mirror is dynamically adjusted to correspond to the size of the image within the visible region.

[0077]

[0092] In any further implementation of the retinal projection display system described above or below, the scanning range of the scanning mirror corresponds to the size of the display area of ​​the reflective surface, and therefore the light source is activated to display the image only when the scanning mirror is projecting the image within the visible region.

[0078]

[0093] In any further implementation of the aforementioned or subsequent implementations of the retinal projection display system, the reflective surface is at least partially transparent.

[0079]

[0094] A further implementation of either of the aforementioned or later implementation forms of the retinal projection display system,

[0095] Eyeglass frames configured to be worn by the user,

[0096] The eyeglass frame further comprises at least one lens mounted on the eyeglass frame, wherein a reflective surface is located on at least a portion of the at least one lens.

[0080]

[0097] Further implementations of the technology described herein include methods of retinal projection, which include,

[0098] Using a scanning mirror with a field of view larger than the image, the image from the light source is projected onto a reflective surface larger than the image.

[0099] Determining the visible area of ​​the reflective surface for the user,

[0100] This includes orienting an image onto the visible region of a reflective surface so that the image is projected onto the user's retina.

[0081]

[0101] A further implementation of either of the aforementioned or subsequent implementation forms determines the visible area of ​​the user's reflective surface.

[0102] The method includes obtaining interpupillary distance alignment for a user, the interpupillary distance alignment further includes identifying the visible area of ​​a reflective surface relative to the user's known line of sight direction.

[0082]

[0103] In any further implementation of either of the aforementioned or subsequent implementations, the interpupillary distance alignment is determined during the calibration process for the user.

[0104] During the calibration process,

[0105] Projecting the alignment image onto a reflective surface,

[0106] Moving the alignment image across the display area of ​​the reflective surface,

[0107] The method further includes determining an interpupillary distance alignment that identifies the visible area of ​​a reflective surface relative to the user's known line of sight direction, in response to user feedback.

[0083]

[0108] A further implementation of either of the aforementioned or subsequent implementation forms determines the visible area of ​​the user's reflective surface.

[0109] This further includes a method that tracks the user's gaze direction, with the visible area corresponding to the gaze direction.

[0084]

[0110] A further implementation of either of the aforementioned or subsequent implementation forms involves orienting the image onto the visible region of the reflective surface so that the image is projected onto the user's retina.

[0111] The method further includes using a scanning mirror to dynamically move an image on a reflective surface according to the user's line of sight direction and the user's interpupillary distance alignment.

[0085]

[0112] A further implementation of either of the aforementioned or subsequent implementations involves using a scanning mirror to dynamically move the image on the reflective surface according to the user's line of sight direction and the user's interpupillary distance alignment.

[0113] Based on tracking the user's gaze direction, the amount of movement in the gaze direction is determined,

[0114] The method further includes moving an image on a reflective surface according to the user's gaze direction and interpupillary distance alignment in response to the amount of movement in the gaze direction satisfying a movement threshold.

[0086]

[0115] A further implementation of either of the aforementioned or subsequent implementations involves using a scanning mirror to dynamically move the image on the reflective surface according to the user's line of sight direction and the user's interpupillary distance alignment.

[0116] The method further includes moving an image on a reflective surface after a predetermined time delay, according to the user's line of sight direction and the user's interpupillary distance alignment.

[0087]

[0117] Further implementations of either of the aforementioned or subsequent implementations involve an image smaller than the visible area, and using a scanning mirror to dynamically move the image on a reflective surface according to the user's line of sight direction and interpupillary distance alignment,

[0118] Determining whether the image is outside the visible region,

[0119] The method further includes, in response to determining that the image is outside the visible area, moving the image on a reflective surface according to the user's line of sight direction and the user's interpupillary distance alignment.

[0088]

[0120] A further implementation of either of the above or below implementations further includes a method in which the scanning range of the scanning mirror is dynamically adjusted to correspond to the size of the image within the visible region.

[0089]

[0121] A further implementation of either of the above or below implementations further includes a method in which the scanning range of the scanning mirror corresponds to the size of the display area of ​​the reflective surface, and therefore the light source is operated to display the image only when the scanning mirror is projecting the image within the visible area.

[0090]

[0122] A further implementation of either of the aforementioned or subsequent implementations further includes a method in which the reflective surface is at least partially transparent.

Claims

1. A light source for projecting an image, A reflective surface on which the aforementioned image is projected, wherein the reflective surface is larger than the aforementioned image on the reflective surface, A scanning mirror that projects the image onto the visible region of the reflective surface so that the image is projected onto the user's retina, Equipped with, During the calibration operation for the user, interpupillary distance alignment is determined, which is information that identifies the position of the visible area of ​​the reflective surface relative to the user's known line of sight direction. A retinal projection display system in which, during the calibration operation, alignment images for determining the interpupillary distance alignment are sequentially displayed at multiple locations on the reflective surface, and the interpupillary distance alignment is determined in response to user feedback regarding the visibility of the alignment images displayed at each location.

2. The device further includes an eye-tracking device for tracking the direction of the user's gaze, The retinal projection display system according to claim 1, wherein the scanning mirror moves the image on the reflective surface to follow the tracking line direction, which is the line of sight direction tracked by the eye-tracker, and the interpupillary distance alignment.

3. The retinal projection display system according to claim 2, wherein the scanning mirror moves the image from a first visible region, which is the visible region on which the image is projected, to a second visible region, which is the visible region identified by the interpupillary distance alignment corresponding to the tracking gaze direction, in response to the amount of movement in the tracking gaze direction satisfying a movement threshold.

4. The retinal projection display system according to claim 2, wherein the scanning mirror moves the image from a first visible region, which is the visible region on which the image is projected, to a second visible region, which is the visible region identified by the interpupillary distance alignment corresponding to the tracking gaze direction, after a predetermined time delay after the tracking gaze direction changes.

5. The image on the reflective surface is smaller than the visible region, The retinal projection display system according to claim 2, which, in response to determining that the tracking gaze direction is outside the first visible region, which is the visible region on which the image is projected, moves the image from the first visible region to the second visible region, which is the visible region identified by the interpupillary distance alignment corresponding to the tracking gaze direction.

6. The retinal projection display system according to claim 1, wherein the scanning mirror scans the visible region of the reflective surface.

7. The retinal projection display system according to claim 1, wherein the scanning mirror scans the entire reflective surface, and the light source is activated only when the scanning mirror is scanning the visible region.

8. The retinal projection display system according to claim 1, wherein the reflective surface is at least partially transparent.

9. An eyeglass frame configured to be worn by the aforementioned user, The eyeglass frame comprises at least one lens, The retinal projection display system according to claim 1, further comprising the reflective surface being located on at least a portion of the at least one lens.

10. A reflective surface on which an image is projected, wherein the reflective surface is larger than the image on the reflective surface, and a visible area is determined for the user. Using a scanning mirror, the image is projected from a light source onto the visible region so that the image is projected onto the user's retina. Includes, Determining the visible region means During the calibration operation for the user, alignment images for determining interpupillary distance alignment, which is information identifying the position of the visible area of ​​the reflective surface relative to the user's known line of sight direction, are sequentially displayed at multiple locations on the reflective surface. The interpupillary distance alignment is determined in response to user feedback regarding the visibility of the alignment image displayed at each location. A method of retinal projection, including [specific method / technique].

11. Determining the visible area further includes tracking the user's line of sight, The method according to claim 10, wherein, in projecting the image, the image is projected onto the visible region identified by the interpupillary distance alignment corresponding to the tracked line of sight direction, which is the tracked line of sight direction.

12. Projecting the image onto the visible area is: The method according to claim 11, comprising using the scanning mirror to move the image on the reflective surface to follow the tracking line direction, according to the tracking line direction and the interpupillary distance alignment.

13. Moving the image is Determining the amount of movement in the aforementioned tracking line of sight direction, In response to the amount of movement in the tracking gaze direction satisfying the movement threshold, the image is moved from the first visible region, which is the visible region on which the image is projected, to the second visible region, which is the visible region identified by the interpupillary distance alignment corresponding to the tracking gaze direction. The method according to claim 12, including the method described in claim 12.

14. Moving the image is The method according to claim 12, comprising moving the image from a first visible region, which is the visible region on which the image is projected, to a second visible region, which is the visible region identified by the interpupillary distance alignment corresponding to the tracking gaze direction, after a predetermined time delay following a change in the tracking gaze direction.

15. The image on the reflective surface is smaller than the visible region, Moving the aforementioned image is, To determine whether the tracking line of sight direction is outside the first visible region, which is the visible region on which the aforementioned image is projected, In response to the determination that the tracking gaze direction is outside the first visible region, the image is moved from the first visible region to the second visible region, which is the visible region identified by the interpupillary distance alignment corresponding to the tracking gaze direction. The method according to claim 12, including the method described in claim 12.

16. The method according to claim 10, wherein the scanning mirror scans the visible region of the reflective surface.

17. The method according to claim 10, wherein the scanning mirror scans the entire reflective surface, and the light source is activated only when the scanning mirror is scanning the visible region.

18. The method according to claim 10, wherein the reflective surface is at least partially transparent.

Images