Virtual / augmented reality system with reverse-angle diffraction grating

Abstract

The present invention relates to a virtual / augmented reality system with a reverse-angle diffraction grating. A display subsystem for a virtual image generation system includes a planar waveguide device, an optical fiber, at least one light source configured to emit light from a distal end of the optical fiber, and a mechanical drive assembly to which the optical fiber is mounted as a non-fixed flexible cantilever. The drive assembly is configured to displace the distal end of the optical fiber about a fulcrum according to a scanning pattern so that the emitted light diverges from a longitudinal axis coinciding with the fulcrum. The display subsystem further includes an optical modulation device configured to converge light from the optical fiber toward the longitudinal axis, and an optical waveguide input device configured to direct light from the optical modulation device to the planar waveguide device so that the planar waveguide device displays one or more image frames to an end user.

Description

[0001] This application is a divisional application of the application with application number 201680057391.8, application date October 6, 2016, and invention name “Virtual / augmented reality system with reverse angle diffraction grating”. Technical Field

[0002] The present invention generally relates to systems and methods configured to facilitate an interactive virtual or augmented reality environment for one or more users. Background Art

[0003] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears or is perceived as real. Virtual reality (VR) scenarios generally involve the presentation of digital or virtual image information that is opaque to other actual real-world visual input, while augmented reality (AR) scenarios generally involve the presentation of digital or virtual image information as an enhancement to the visualization of the actual world around the user.

[0004] For example, reference Figure 1 , depicting an augmented reality scene 4 in which a user of AR technology sees a real-world park-like setting 6, featuring people, trees, and buildings in the background, as well as a physical platform 8. In addition to these items, the end-user of AR technology also perceives that they "see" a robotic statue 10 standing on the real-world platform 8, and a flying cartoon-like avatar character 12 that appears to be a bee, even though these elements 10, 12 do not exist in the real world. The human visual perception system has proven to be very complex, making the development of VR or AR technology that promotes comfortable, natural, and rich presentation of virtual image elements within other virtual or real-world image elements extremely challenging.

[0005] VR and AR systems typically employ a head-mounted display (or helmet-mounted display, or smart glasses) that is at least loosely coupled to the user's head and, therefore, moves when the end user's head moves. If the end user's head movement is detected by the display system, the data being displayed can be updated to account for the change in head pose (i.e., the orientation and / or position of the user's head).

[0006] As an example, if a user wearing a head-mounted display views a virtual representation of a three-dimensional (3D) object on the display and walks around the area where the 3D object appears, the 3D object can be re-rendered for each viewpoint, giving the end user the perception that he or she is walking around the object occupying real space. If the head-mounted display is used to present multiple objects within a virtual space (e.g., a rich virtual world), measurements of head pose can be used to re-render the scene to match the end user's dynamically changing head position and orientation and enhance immersion in the virtual space.

[0007] AR-enabled head-mounted displays (i.e., simultaneous viewing of real and virtual elements) can have several different types of configurations. In one such configuration, often referred to as a “video see-through” display, a camera captures elements of the real scene, a computing system overlays virtual elements onto the captured real scene, and a non-transparent display presents the composite image to the eye. Another configuration is often referred to as an “optical see-through” display, in which the end user can see through a transparent (or translucent) element in the display system to directly view light from real objects in the environment. A transparent element, often referred to as a “combiner,” overlays the light from the display on top of the end user’s view of the real world.

[0008] VR and AR systems typically employ a display system having a projection subsystem and a display surface positioned in front of an end user's field of view, with the projection subsystem sequentially projecting image frames onto the display surface. In a true three-dimensional system, the depth of the display surface can be controlled at a frame rate or sub-frame rate. The projection subsystem may include one or more optical fibers, wherein light from one or more light sources emits light of different colors into the optical fibers in a set pattern; and a scanning device that scans the optical fibers in a predetermined pattern to create image frames that are sequentially displayed to the end user.

[0009] In one embodiment, the display system includes one or more planar waveguides that are generally parallel to the user's field of view, and light from an optical fiber is injected into the planar waveguide. One or more linear diffraction gratings are embedded in the waveguide to change the angle of the incident light propagating along the waveguide. By changing the angle of the light beyond the threshold of total internal reflection (TIR), the light escapes from one or more sides of the waveguide. The linear diffraction grating has low diffraction efficiency, so each time the light encounters the linear diffraction grating, only a portion of the light can be guided out of the waveguide. By outcoupling light at multiple locations along the grating, the exit pupil of the display system is effectively increased. The display system may further include one or more collimating elements for collimating the light from the optical fiber, and one or more optical coupling elements for optically coupling the collimated light to the edge of the waveguide or optically coupling the collimated light from the edge of the waveguide.

[0010] In a typical fiber-scanning display system, each optical fiber acts as a vibrating cantilever that sweeps a large deflection from a fulcrum to scan light according to a designed scanning pattern. However, due to the large deflection of the collimated light, the optical coupling element must be relatively large, thereby increasing the size of the display system. This size of the optical coupling element becomes even more problematic in the case of stacked waveguide structures, which require the optical components associated with the waveguides at greater distances from the scanning fiber to be larger to accommodate the greater range of scanning the collimated light.

[0011] For example, reference Figure 2 One embodiment of a display system 20 includes one or more light sources 22 that generate image data encoded in the form of spatially and / or temporally varying light; an optical fiber 24 optically coupled to the light source 22; and a collimating element 26 that collimates the light emitted from the distal end of the optical fiber 24. The display system 20 further includes a piezoelectric element 28 to which the optical fiber 24 is mounted or in which it is mounted as a non-fixed flexible cantilever; and drive electronics 30 that are electrically coupled to the piezoelectric element 22 to activate the electrically excited piezoelectric element 28, thereby causing the distal end of the optical fiber 24 to vibrate in a predetermined scanning pattern, creating a deflection 32 about a fulcrum 34.

[0012] The display system 20 includes a waveguide device 38 comprising a plurality of planar waveguides 40a-40e generally parallel to the end user's field of view, and one or more diffractive optical elements (DOEs) 42a-42e associated with each planar waveguide 40. Light originating from the optical fiber 24 propagates along a selected planar waveguide 40 and intersects a corresponding DOE 42, causing a portion of the light to exit the face of the waveguide device 38 toward the end user's eyes, focused at one or more viewing distances depending on the selected planar waveguide 40.

[0013] Display system 20 further includes optical coupling elements in the form of diffractive optical elements (DOEs) 44a-44e, which are integrated into the ends of each planar waveguide 40a-40e and reflect collimated light into the selected planar waveguide 40. As can be seen, as the distance between each DOE 44 and the end of the optical fiber 24 increases, the length of each DOE 44 must increase to accommodate the increased linear span of the deflection angle of the optical fiber 24. This necessarily increases the size and complexity of waveguide device 38 due to the largest DOE 44 (in this case, DOE 44e).

[0014] As another example, see Figure 3 , another embodiment of the display system 50 is Figure 21 is similar to display system 10, except that display system 50 includes an optical coupling element in the form of a light distributing waveguide 52 having DOEs 54a-54e that reflects collimated light into selected planar waveguides 40. The width of distributed waveguide 52 must be large enough to accommodate the maximum linear span of the deflection angle of optical fiber 24, thereby necessarily increasing the size and complexity of waveguide device 38.

[0015] Therefore, there is a need to reduce the size of optical coupling elements used to couple light from one or more optical fibers into one or more planar waveguides in a virtual reality or augmented reality environment. Summary of the Invention

[0016] Embodiments of the present invention relate to apparatuses, systems, and methods for facilitating virtual reality and / or augmented reality interactions for one or more users.

[0017] According to the present invention, a display subsystem for a virtual image generation system for use by an end user is provided. The virtual image generation system may, for example, include a memory for storing a three-dimensional scene and a control subsystem configured to render multiple composite image frames of the three-dimensional scene. In this case, the display subsystem may be configured to sequentially display the multiple image frames to the end user.

[0018] The display subsystem includes a planar waveguide device, an optical fiber, and at least one light source configured to emit light from a distal end of the optical fiber. In one embodiment, the planar waveguide device is configured to be positioned in front of the end user's eyes. The planar waveguide device may have a partially transparent display surface that is configured to be positioned in a field of view between the end user's eyes and the surrounding environment. In one embodiment, the display subsystem may further include a frame structure configured to be worn by the end user, in which case the frame structure may support the planar waveguide device.

[0019] The display subsystem further includes a mechanical drive assembly to which the optical fiber is mounted as a non-fixed flexible cantilever. The drive assembly is configured to displace the distal end of the optical fiber about a fulcrum according to a scanning pattern such that output / emitted light diverges from a longitudinal axis coinciding with the fulcrum. In one embodiment, the mechanical drive assembly includes a piezoelectric element to which the optical fiber is mounted and drive electronics configured to transmit an electrical signal to the piezoelectric element, thereby causing the optical fiber to vibrate according to the scanning pattern. The display subsystem may optionally further include a collimating element configured to collimate light from the optical fiber.

[0020] The display subsystem further includes a light modulation device configured to converge light from the optical fiber toward the longitudinal axis. In one embodiment, the light modulation device is configured to converge the light at a focal point on the longitudinal axis. The focal point can, for example, be located within the optical waveguide input device, such as at the center of the optical waveguide input device along the longitudinal axis. The light modulation device can, for example, include at least one diffraction grating. Each diffraction grating can have a diffraction pattern that matches the geometry of the scanning pattern. For example, if the scanning pattern is a spiral scanning pattern, the diffraction pattern can be a spiral diffraction pattern. In another embodiment, the light modulation device includes two orthogonal diffraction gratings connected in series, such that one of the diffraction gratings diffracts light along a first axis, and the other diffraction grating diffracts light along a second axis orthogonal to the first axis.

[0021] The display subsystem further includes an optical waveguide input device configured to direct light from the optical modulation device to the planar waveguide device so that the planar waveguide device displays one or more image frames to the end user. In one embodiment, the planar waveguide device includes a plurality of planar waveguides configured to display image frames to the end user at different focal points. In this case, the optical waveguide input device may be configured to direct light to selected ones of the plurality of planar waveguides. In another embodiment, the optical waveguide input device includes a plurality of diffractive optical elements extending parallel to the planar waveguides, and the plurality of diffractive optical elements guide the light from the optical modulation device to the planar waveguides, respectively. In yet another embodiment, the optical waveguide input device includes a distributed waveguide extending perpendicular to the planar waveguide, and the distributed waveguide includes a plurality of diffractive optical elements, and the plurality of diffractive optical elements guide the light from the optical modulation device to the planar waveguides, respectively.

[0022] Additional and other objects, features, and advantages of the present invention are described in the detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings illustrate the design and application of embodiments of the present invention, wherein similar elements are designated by common reference numerals. In order to better understand how the above and other advantages and objects of the present invention are obtained, the invention briefly described above will be described in more detail with reference to specific embodiments thereof illustrated in the accompanying drawings. It should be understood that these drawings depict only typical embodiments of the invention and are not to be considered as limiting its scope. The invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0024] Figure 1 It is a picture of a three-dimensional augmented reality scene that can be displayed to an end user by a conventional augmented reality generating device;

[0025] Figure 2is a plan view of one embodiment of a prior art display system that can be used in an augmented reality generating device;

[0026] Figure 3 is a plan view of another embodiment of a prior art display system that can be used in an augmented reality generating device;

[0027] Figure 4 is a block diagram of a virtual image generation system constructed according to one embodiment;

[0028] Figure 5 is Figure 4 A plan view of a schematic frame generated by a virtual image generation system;

[0029] Figure 6 is a plan view of a scan pattern that can be used to generate a frame;

[0030] Figure 7 is a plan view of another scan pattern that can be used to generate a frame;

[0031] Figure 8 is a plan view of another scan pattern that can be used to generate a frame;

[0032] Figure 9 is a plan view of another scan pattern that can be used to generate a frame;

[0033] Figure 10a Can be used for wearing Figure 4 A plan view of a technology for a virtual image generation system;

[0034] Figure 10b Can be worn Figure 4 A plan view of another technique of a virtual image generation system;

[0035] Figure 10c Can be worn Figure 4 A plan view of another technical aspect of a virtual image generation system;

[0036] Figure 10d Can be used for wear Figure 4 A plan view of another technology of a virtual image generation system;

[0037] Figure 11 is Figure 4 A plan view of an embodiment of a display subsystem used in a virtual image generation system;

[0038] Figure 12 is Figure 11 An embodiment of a main planar waveguide for use in a display subsystem;

[0039] Figure 13a is Figure 4 A perspective view of one embodiment of a display subsystem used in a virtual image generation system;

[0040] Figure 13b Specifically shows the light rays extending from a focus Figure 13a A perspective view of the display subsystem;

[0041] Figure 13c Specifically shows the light rays extending from another focus Figure 13a A perspective view of the display subsystem;

[0042] Figure 14 is Figure 4 A plan view of another embodiment of a display subsystem used in a virtual image generation system;

[0043] Figure 15 is a plan view of one embodiment of a planar waveguide device for use in the display subsystem of FIG. 13 ;

[0044] Figure 16 is a plan view of another embodiment of a planar waveguide device used in the display subsystem of FIG. 13 ;

[0045] Figure 17 yes Figure 16 A side view of a planar waveguide device;

[0046] Figure 18a is a plan view of one embodiment of an optical coupling subsystem and scanning device that may be used in the display subsystem of FIG. 13 , particularly illustrating the convergence of the light beam to a focal point in the center of the optical waveguide input device;

[0047] Figure 18b yes Figure 18a A plan view of the optical coupling subsystem and scanning device, particularly showing the convergence of the light beam to a focal point at the edge of the optical waveguide input device;

[0048] Figure 19 is Figure 13a A plan view of one embodiment of an optical waveguide input device for use in an optical coupling subsystem;

[0049] Figure 20 is Figure 13a A plan view of another embodiment of an optical waveguide input device for use in an optical coupling subsystem;

[0050] Figure 21 It is available in Figure 18a A plan view of a spiral diffraction pattern used in an optical modulation device in an optical coupling subsystem;

[0051] Figure 22a It is available in Figure 18aan embodiment of an optical modulation device for use in an optical coupling subsystem; and

[0052] Figure 22b It is available in Figure 18a Another embodiment of an optical modulation device for use in an optical coupling subsystem. DETAILED DESCRIPTION

[0053] The following description relates to display systems and methods used in virtual reality and / or augmented reality systems. However, it should be understood that while the present invention is well suited for use in virtual or augmented reality systems, the present invention may not be limited thereto.

[0054] refer to Figure 4 , one embodiment of a virtual image generation system 100 constructed in accordance with the invention will now be described. The virtual image generation system 100 can be used as an augmented reality subsystem to provide images of virtual objects mixed with physical objects in the field of view of the end user 50. . There are two basic approaches when operating the virtual image generation system 100. The first approach employs one or more imagers (e.g., cameras) to capture images of the surrounding environment. The virtual image generation system 100 mixes the virtual image into data representing the image of the surrounding environment. The second approach employs one or more at least partially transparent surfaces through which the surrounding environment can be seen, and the virtual image generation system 100 generates images of virtual objects on the surfaces.

[0055] The virtual image generation system 100 and the various techniques taught herein can be used in applications other than augmented reality and virtual reality subsystems. For example, the various techniques can be applied to any projection or display subsystem, or to a micro-projector where movement can be performed by the end user's hands rather than the head. Therefore, although generally described herein in terms of augmented reality or virtual reality subsystems, the present teachings should not be limited to such subsystems for such uses.

[0056] At least for augmented reality applications, it is desirable to position various virtual objects in space relative to corresponding physical objects in the field of view of the end user 50. The virtual objects (also referred to herein as one or more virtual labels or callouts) can take any of a variety of forms, and essentially any type of data, information, concept, or logical construct can be represented as an image. Non-limiting examples of virtual objects may include: a virtual text object, a virtual numeric object, a virtual alphanumeric object, a virtual label object, a virtual field object, a virtual chart object, a virtual map object, a virtual instrument object, or a virtual visual representation of a physical object.

[0057] To this end, the virtual image generation system 100 includes: a frame structure 102 worn by the end user 50; a display subsystem 104 carried by the frame structure 102 so that the display subsystem 104 is located in front of the end user's 50 eyes 52; and a speaker 106 carried by the frame structure 102 so that the speaker 106 is located near the ear canal of the end user 50 (optionally, another speaker (not shown) is located near the other ear canal of the end user 50 to provide stereo / shapeable sound control). The display subsystem 104 is designed to present a photo-based radiation pattern to the end user's 50 eyes 52, which can be comfortably perceived as an augmentation of physical reality, with a high level of image quality and three-dimensional perception, and capable of presenting two-dimensional content. The display subsystem 104 presents a sequence of synthetic image frames at a high frequency, providing the perception of a single coherent scene.

[0058] The display subsystem 104 includes a projection subsystem 108 and a partially transparent display screen 110 onto which the projection subsystem 108 projects images. The display screen 110 is located in the field of view of the end user 50 between the eyes 52 of the end user 50 and the surrounding environment.

[0059] In an exemplary embodiment, projection subsystem 108 takes the form of a fiber-scanning-based projection device, and display screen 110 takes the form of a waveguide-based display, into which scanned light from projection subsystem 108 is injected to produce, for example, an image at a single optical viewing distance closer than infinity (e.g., arm's length), images at multiple discrete optical viewing distances or focal planes, and / or stacked layers of images at multiple viewing distances or focal planes to represent a stereoscopic 3D object. The layers in the light field can be stacked close enough together to appear continuous to the human visual subsystem (i.e., one layer is within the confusion cone of the adjacent layer). Additionally or alternatively, even if the layers are stacked more sparsely (i.e., one layer is outside the confusion cone of the adjacent layer), picture elements can be cross-blended between two or more layers to increase the perceived continuity of transitions between layers in the light field. Display subsystem 104 can be monocular or binocular.

[0060] Thus, the display subsystem 104 generates a series of composite image frames that present pixel information of an image of one or more virtual objects to the user. Figure 5, a composite image frame 118 is schematically shown, wherein cells 120a-120m are divided into horizontal rows or lines 122a-122n. Each cell 120 of the frame 118 may specify a value and / or intensity for each of a plurality of colors for each pixel to which the cell 120 corresponds. For example, the frame 118 may specify one or more values ​​for red 124a, one or more values ​​for green 124b, and one or more values ​​for blue 124c for each pixel. The values ​​124 may be specified as a binary representation of each color, e.g., a 4-bit value corresponding to each color. Additionally, each cell 120 of the frame 118 may include a value 124d specifying an amplitude.

[0061] Frame 118 may include one or more fields, collectively referred to as 126. Frame 118 may include a single field. Alternatively, frame 118 may include two or even more fields 126a-126b. Pixel information for the complete first field 126a of frame 118 may be specified before pixel information for the complete second field 126b, for example, in an array, sequence table, or other data structure (e.g., a record, a linked list) before pixel information for the second field 126b. Assuming the rendering subsystem is configured to process more than two fields 126a-126b, a third or even fourth field may follow second field 126b.

[0062] Now refer to Figure 6 , a frame 118 is generated using a raster scan pattern 128. In the raster scan pattern 128, pixels 130 are presented sequentially (only one is called out). The raster scan pattern 128 typically presents pixels from left to right (indicated by arrows 132a, 132b) and then from top to bottom (indicated by arrow 134). Thus, the presentation may start at the upper right corner and traverse the first line 136a from the left to the end of the line. The raster scan pattern 128 then typically starts from the left of the next line. When returning from the end of one line to the beginning of the next line, the presentation may be temporarily blacked out or left blank. This process is repeated line by line until the bottom line 136n is completed, for example at the pixel in the lower right corner. As the frame 118 is completed, a new frame begins, returning to the right of the top line of the next frame. Again, the presentation may be blank when presenting the next frame from the lower left corner back to the upper right corner.

[0063] Many implementations of raster scanning employ a so-called interlace scanning pattern. In an interlace raster scanning pattern, the lines from the first and second fields 126a, 126b are interlaced. For example, when presenting lines of the first field 126a, pixel information from the first field 126a may be used only for odd-numbered lines, while pixel information from the second field 126b may be used only for even-numbered lines. Thus, the first field 126a ( Figure 5) are typically presented before the lines of the second field 126b. The first field 126a may be presented using the pixel information of the first field 126a to sequentially present line 1, line 3, line 5, and so on. Then, the second field 126b of the frame 118 may be presented after the first field 126a using the pixel information of the second field 126b. Figure 5 ) presents line 2, line 4, line 6, etc. in sequence.

[0064] refer to Figure 7 , a spiral scan pattern 140 may be used to generate frame 118 instead of raster scan pattern 128. Spiral scan pattern 140 may include a single spiral scan line 142, which may include one or more complete angular periods (e.g., 360 degrees) that may be referred to as a circle or ring. Figure 6 Like the illustrated raster scan pattern 128 , the pixel information in the spiral scan pattern 140 is used to specify the color and / or intensity of each sequential pixel as the angle increases.

[0065] refer to Figure 8 Alternatively, a Lissajous scan pattern 150 may be used to generate the frame 118. The Lissajous scan pattern 150 may include a single Lissajous scan line 152, which may include one or more complete angular cycles (e.g., 360 degrees) that may be referred to as loops or rings. Alternatively, the Lissajous scan pattern 150 may include two or more Lissajous scan lines 152, each phase shifted relative to one another to nest the Lissajous scan lines 152. Pixel information is used to specify the color and / or intensity of each sequential pixel as the angle increases. The amplitude or radial value specifies a radial dimension 154 from a starting point 156 of the Lissajous scan line 152.

[0066] refer to Figure 9 , optionally, a multi-field spiral scan pattern 158 may be used to generate the frame 118. The multi-field spiral scan pattern 158 includes two or more different spiral scan lines, collectively referred to as 160, and specifically, four spiral scan lines 160a-160d. The pixel information for each spiral scan line 160 may be specified by the fields of the frame. Advantageously, multiple spiral scan lines 160 may be nested simply by shifting the phase between each successive spiral scan line 160. The phase difference between the spiral scan lines 160 should be a function of the total number of spiral scan lines 160 to be employed. For example, the four spiral scan lines 160a-160d may be separated by a 90 degree phase shift. An exemplary embodiment may operate with 10 different spiral scan lines (i.e., sub-spirals) at a 100 Hz refresh rate. With Figure 7Similar to the embodiment of , one or more amplitude or radial values ​​specify a radial dimension 162 from a starting point 164 of the spiral scan line 160 .

[0067] Return Reference Figure 4 The virtual image generation system 100 further includes one or more sensors (not shown) mounted to the frame structure 102 for detecting the position and movement of the head 54 of the end user 50 and / or the position and inter-eye distance of the eyes of the end user 50. Such sensors may include a capture device (such as a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a radio device, and / or a gyroscope.

[0068] For example, in one embodiment, the virtual image generation system 100 includes a head-mounted transducer subsystem 126 that includes one or more inertial sensors to capture inertial measurements indicative of movement of the end user's 50 head 54. This can be used to sense, measure, or collect information about the movement of the end user's 50 head. For example, this can be used to detect motion, velocity, acceleration, and / or position measurements of the end user's 50 head 54.

[0069] The virtual image generation system 100 further includes one or more forward-facing cameras 128, which can be used to capture information about the environment in which the end user 50 is located. The forward-facing cameras 128 can be used to capture information indicating the distance and direction of the end user 50 relative to the environment and specific objects in the environment. When worn on the head, the forward-facing cameras 128 are particularly suitable for capturing information indicating the distance and direction of the end user's 50 head 54 relative to the environment in which the end user 50 is located and specific objects in the environment. For example, the forward-facing cameras 128 can be used to detect head movement, speed, and / or acceleration of head movement. For example, the forward-facing cameras 128 can be used to detect or infer the end user's 50 center of attention, for example, based at least in part on the orientation of the end user's 50 head 54. The orientation can be detected in any direction (e.g., up / down, left, or right relative to the end user's 50 frame of reference).

[0070] The virtual image generation system 100 further includes a pair of rear-facing cameras 129 to track movement, blinking, and depth of focus of the eyes 52 of the end user 50. For example, such eye tracking information can be identified by projecting light at the end user's eyes and detecting the return or reflection of at least some of the projected light.

[0071] The virtual image generation system 100 further includes a patient orientation detection module 130. The patient orientation module 130 detects the instantaneous position of the end user's 50 head 54 and can predict the position of the end user's 50 head 54 based on the position data received from the sensors. Importantly, detecting the instantaneous position of the end user's 50 head 54 helps determine the specific real-world object the end user 50 is looking at, thereby providing an indication of the specific text message to be generated for the real-world object and further providing an indication of the text message to be streamed in the text area. The patient orientation module 130 also tracks the end user's 50 eyes 52 based on the tracking data received from the sensors.

[0072] The virtual image generation system 100 further includes a control subsystem, which can take any of a variety of forms. The control subsystem includes a plurality of controllers, such as one or more microcontrollers; a microprocessor or central processing unit (CPU); a digital signal processor; a graphics processing unit (GPU); other integrated circuit controllers, such as an application-specific integrated circuit (ASIC), a programmable gate array (PGA) such as a field programmable gate array (FPGA), and / or a programmable logic controller (PLU).

[0073] In the illustrated embodiment, the virtual image generation system 100 includes a central processing unit (CPU) 132, a graphics processing unit (GPU) 134, and one or more frame buffers 136. The CPU 132 controls overall operation, while the GPU 134 renders frames (i.e., converts a 3D scene into a 2D image) from 3D data stored in a remote database 150 and stores these frames in the frame buffer 136. Although not shown, one or more additional integrated circuits may control the reading of frames into and / or out of the frame buffer 136 and the operation of the scanning mechanism of the display subsystem 104. For example, dynamic addressing may be employed for reading into and / or out of the frame buffer 146 in the event that a frame is over-rendered. The virtual image generation system 100 further includes a read-only memory (ROM) 138 and a random access memory (RAM) 140. The virtual image generation system 100 further includes a 3D database 142 from which the GPU 134 can access 3D data for one or more scenes used to render a frame.

[0074] The various processing components of the virtual image generation system 100 may be physically contained in distributed subsystems. Figures 10a-10d As shown, the virtual image generation system 100 includes a local processing and data module 144 operatively coupled to the display subsystem 104 and the sensor, such as by a wired or wireless connection 146. The local processing and data module 144 can be mounted in various configurations, such as fixedly attached to the frame structure 102 ( Figure 10a), fixedly attached to the helmet or hat 56 ( Figure 10b ), embedded in a headset, detachably attached to the torso 58 of the end user 50 ( Figure 10c ), or detachably attached to the hip 60 of the end user 50 in a belt-coupled configuration ( Figure 10d The virtual image generation system 100 further includes a remote processing module 148 and a remote database 150 operatively coupled to the local processing and data module 144 , such as by wired or wireless connections 150 , 152 , such that these remote modules 148 , 150 are operatively coupled to each other and serve as available resources for the local processing and data module 144 .

[0075] The local processing and data module 144 may include a low-power processor or controller and digital memory, such as flash memory, both of which may be used to assist in processing, caching, and storing data captured from sensors and / or retrieved and / or processed using the remote processing module 148 and / or remote database 150, which may be transferred to the display subsystem 104 after such processing or retrieval. The remote processing module 148 may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. The remote database 150 may include a relatively large-scale digital data storage facility that may be accessible via the Internet or other network configuration in a "cloud" resource configuration. In one embodiment, all data is stored and all calculations are performed in the local processing and data module 144, allowing for fully autonomous use from any remote module.

[0076] The couplings 146, 152, 154 between the various components described above may include one or more wired interfaces or ports for providing wired or optical communication, or one or more wireless interfaces or ports for providing wireless communication, such as via RF, microwave, and IR. In some implementations, all communication may be wired, while in other implementations, all communication may be wireless. In further implementations, the choice of wired and wireless communication may differ from one embodiment to another. Figures 10a-10d Therefore, the specific choice of wired or wireless communication should not be considered limiting.

[0077] In the illustrated embodiment, subject orientation module 130 is included in local processing and data module 144, while CPU 132 and GPU 134 are included in remote processing module 148, but in alternative embodiments, CPU 132, GPU 124, or portions thereof may be included in local processing and data module 144. 3D database 142 may be associated with remote database 150.

[0078] Now refer to Figure 11 and Figure 12, the display screen 110 includes a main waveguide device 200. The main waveguide device 200 includes one or more main planar waveguides 202 (in Figure 11 and Figure 12 only one is shown in FIG, and one or more diffractive optical elements (DOEs) 204 (in FIG, 204 ) associated with each of at least some of the main waveguides 202 Figure 11 and Figure 12 Only one is shown). Figure 12 As best shown, each main waveguide 202 has a first end 206a and a second end 206b, with the second end 206b being opposite the first end 206a along the length 208 of the main waveguide 202. Each main waveguide 202 has a first face 210a and a second face 210b, with at least the first and second faces 210a, 210b (collectively 210) forming an at least partially internally reflective light path (indicated by arrow 212a and dashed arrow 212b, collectively 212) along at least a portion of the length 208 of the main waveguide 202. The main waveguides 202 can take a variety of forms, with light striking the faces 210 at less than a defined critical angle to provide substantial total internal reflection (TIR). For example, each main waveguide 202 can take the form of a pane or planar surface of glass, fused silica, acrylic, or polycarbonate.

[0079] DOE 204 (in Figure 11 and Figure 12 The DOE 204 (shown by the dot-dashed double line in FIG. 2 ) can take various forms of interrupting the TIR optical path 212, providing multiple optical paths (shown by arrow 214a and dashed arrow 214b, collectively referred to as 214) extending along at least a portion of the length 206 of the main waveguide 202 between an interior 216 and an exterior 218 of the main waveguide 202. The DOE 204 can allow for the location of a distinct object and its focal plane. This can be accomplished on a frame-by-frame, sub-frame-by-subframe, or even pixel-by-pixel basis.

[0080] like Figure 12As shown, light propagates along the main waveguide 202, with at least some reflection or "bounce" caused by TIR propagation. It should be noted that some embodiments may employ one or more reflectors in the internal optical path to promote reflection, such as thin films, dielectric coatings, metallized coatings, etc. The light propagates along the length 208 of the main waveguide 202 and intersects one or more DOEs 204 at various locations along the length 208. The DOEs 204 may be incorporated into the main waveguide 202 or adjacent to or proximate to one or more facets 210 of the main waveguide 202. The DOEs 204 perform at least two functions. The DOEs 204 change the angle of the light, causing some of the light to escape TIR and exit from the interior 216 to the exterior 218 via one or more facets 210 of the main waveguide 202. The DOEs 204 focus the outcoupled light at one or more viewing distances. Thus, a person looking through the facet 210a of the main waveguide 202 can see a digital image at one or more viewing distances.

[0081] refer to Figures 13a-13c , the display screen 110 includes a distributed waveguide device 222 to distribute the light along the first axis ( Figure 13a The distributed waveguide device 222 may relay light along a first axis (e.g., a vertical line or Y-axis) and expand the effective exit pupil of the light along a first axis (e.g., the Y-axis). For example, the distributed waveguide device 222 may include one or more distributed planar waveguides 224 (only one shown) and a DOE 226 (shown by a double-dashed line) associated with each distributed planar waveguide 224. The distributed planar waveguide 224 may be similar or identical to the main waveguide 202 in at least some aspects, but have a different orientation therefrom. Similarly, the DOE 226 may be similar or identical to the DOE 204 in at least some aspects. For example, the distributed planar waveguide 220 and / or the DOE 226 may comprise the same material as the main waveguide 202 and / or the DOE 204, respectively.

[0082] The relayed and exit pupil expanded light is optically coupled from the distributed waveguide device 222 into the main waveguide 202. The main waveguide 202 is optically coupled along a second axis (e.g., Figure 13a The distributed planar waveguide 224 relays light along a horizontal axis (e.g., the X-axis). It should be noted that the second axis may not be perpendicular to the first axis. The main waveguide 202 expands the effective exit pupil of light along the second axis (e.g., the X-axis). In particular, the distributed planar waveguide 224 can relay and expand light along the vertical axis (e.g., the Y-axis) and transfer this light to the main waveguide 202, which relays and expands the light along the horizontal axis (e.g., the X-axis).

[0083] Display screen 110 may generate an image at a single focal plane that can be positioned closer than optical infinity. Figure 13bAs shown, collimated light propagates vertically along the distributed planar waveguide 224 via total internal reflection, repeatedly intersecting the DOE 226 along the way. The DOE 226 preferably has a low diffraction efficiency (e.g., less than 50%). This results in a portion of the light (e.g., 10%) being diffracted toward the edge of the larger main planar waveguide 202 at each point of intersection with the DOE 226, and a portion of the light continuing down the length of the distributed planar waveguide 224 on its original trajectory via TIR. At each point of intersection with the DOE 226, additional light is diffracted toward the entrance of the main waveguide 202. By dividing the incident light into multiple outcoupling groups, the exit pupil of the light is vertically expanded by the DOE 226 in the distributed planar waveguide 224. This vertically expanded light outcoupled from the distributed planar waveguide 224 enters the edge of the main waveguide 202.

[0084] The light entering the main waveguide 202 propagates horizontally along the main waveguide 202 via TIR (e.g. Figure 13b ). As light propagates horizontally along at least a portion of the length of the main waveguide 202 via TIR, the light intersects the DOE 204 at multiple points. The DOE 204 may advantageously be designed or configured to have a phase profile that is the sum of a linear diffraction grating and a radially symmetric diffraction lens. The DOE 204 may advantageously have a low diffraction efficiency. At each point of intersection between the propagating light and the DOE 204, a portion of the light is diffracted toward an adjacent face of the main waveguide 202, allowing the light to escape the TIR and exit from a face of the main waveguide 202. Additionally, the radially symmetric lens aspect of the DOE 204 imparts a focus level to the diffracted light, both shaping the optical wavefront of a single light beam (e.g., imparting curvature) and steering the beam at an angle that matches the designed focus level. As Figure 13b As shown, four beams 228a-228d geometrically extend to a focal point 228, and each beam is advantageously imparted with a convex wavefront profile having a radius centered at the focal point 228 to produce an image or virtual object 230a at a given focal plane.

[0085] refer to Figure 13c , the display screen 110 can generate a multi-focal stereoscopic display, image, or light field. The first group of four light beams 228a-228d geometrically extends to a focal point 230a, and each light beam 228a-228d is advantageously provided with a convex wavefront profile having a radius center at the focal point 230a to produce another portion of the image or virtual object 232a at the corresponding focal plane. The second group of four light beams 228e-228h geometrically extends to a focal point 230b, and each light beam 228e-228h is advantageously provided with a convex wavefront profile having a radius center at the focal point 230b to produce another portion of the image or virtual object 232b at the corresponding focal plane.

[0086] exist Figure 11-1In the embodiment of the display subsystem 104 shown in FIG3 , a single projection subsystem 108 is used to provide image data to a display screen 110. Figure 14 As shown, Figure 11-1 3, the display subsystem 104 can include multiple projection subsystems 108a-108e (only five are shown, collectively referred to as 108) to provide corresponding image data to the display screen 110. The projection subsystems 108 are generally arranged in an array or arrangement along an edge 234 of the display screen 110. For example, the number of planar waveguides 202 and the number of projection subsystems 108 can have a one-to-one (1:1) ratio or correlation.

[0087] The display subsystem 104 can utilize a single main planar waveguide 202. For example, multiple projection subsystems 108 can be arranged in a linear array along the edge 234 of the main planar waveguide 202 closest to the end user's temple. Each projection subsystem 108 injects modulated light encoding sub-image data into the main planar waveguide 202 from a different, corresponding position, thereby generating a different optical path. These different paths can couple light out of the main planar waveguide 202 via multiple DOEs at different angles, focus levels, and / or producing different fill patterns at the exit pupil. The different fill patterns at the exit pupil can be advantageously used to create a light field display. Each layer in a stack or a group of layers in a stack (e.g., three layers) can be configured to produce a corresponding color (e.g., red, blue, green). Thus, for example, a first group of three adjacent layers can be used to produce red, blue, and green light, respectively, at a first focal depth. A second group of three adjacent layers can be used to produce red, blue, and green light, respectively, at a second focal depth. Multiple groups can be used to produce full 3D or 4D color image fields with various focal depths.

[0088] Now refer to Figure 15 Each planar waveguide 202 may include a plurality of DOEs 204a-204d (four shown, each indicated by a double-dashed line, collectively referred to as 204). The DOEs 204 are stacked, arranged, or disposed along an axis 236 that is generally parallel to the field of view of the display screen 110. Although all of the DOEs 204 are shown as being internally located, in some embodiments, one, more, or even all of the DOEs 204 may be located externally of the main waveguide 202.

[0089] In some embodiments, each DOE 204 can be independently turned on or off. That is, each DOE 204 can be activated so that the corresponding DOE 204 diffracts a significant portion of the light that intersects the corresponding DOE 204, or can be deactivated so that the corresponding DOE 204 either barely diffracts the light that intersects the corresponding DOE 204, or diffracts only an insignificant portion of the light. In this context, "significant" means that the human visual system perceives enough light when coupled out of the main waveguide 202, and "insignificant" means that the human visual system does not perceive enough light, or the level is low enough to be ignored by the viewer.

[0090] The switchable DOEs 204 can be turned on one at a time so that only one DOE 204 in the main planar waveguide 202 actively diffracts light in the main planar waveguide 202, exiting one or more facets 210 in the main planar waveguide 202 in an appreciable amount. Alternatively, two or more DOEs 204 can be turned on simultaneously so that their diffraction effects are combined.

[0091] Each DOE 204 in the set of DOEs can have a different phase diagram. For example, each DOE 204 can have a separate phase diagram such that each DOE 204 directs light to a different location in X, Y, or Z when activated. For example, the DOEs 204 can differ from one another in their linear gratings and / or their radially symmetric diffractive lenses. If the DOEs 204 differ from one another in their diffractive lenses, different DOEs 204 (or combinations of DOEs 204) will produce sub-images at different optical viewing distances (i.e., different focal lengths). If the DOEs 204 differ from one another in their linear gratings, different DOEs 204 will produce sub-images that are laterally shifted relative to one another. Such lateral shifting can be advantageously used to create faveate displays, to direct displayed images having non-uniform resolutions or other non-uniform display parameters (e.g., brightness, peak wavelength, polarization, etc.) to different lateral locations, to increase the size of a scanned image, to produce variations in the characteristics of the exit pupil, and / or to generate lightfield displays. Lateral shifting may be advantageously employed to preform tiling or to achieve a tiling effect in the resulting image.

[0092] For example, when the first DOE 204 in the group is turned on, the image may be viewed at an optical viewing distance of 1 meter (eg, Figure 13c An image is generated at the focal point 230b in the main waveguide 202 for a viewer looking toward the main or emitting face 210a of the main waveguide 202. When the second DOE 204 in the group is turned on, an image can be generated at an optical viewing distance of 1.25 meters (e.g., Figure 13bAn image is generated at a focal point 230a in the main waveguide 202 for a viewer looking toward the main or emitting face 210a of the main waveguide 202. By switching the exemplary DOEs 204 on or off in a rapid temporal sequence (e.g., on a frame-by-frame, sub-frame, line-by-line, sub-line, pixel-by-pixel, or sub-pixel basis) and synchronously modulating the image data injected into the main waveguide 202 by the projection subsystem 108, a composite multi-focal stereoscopic image is formed that appears as a single scene to the viewer. By rendering different objects or portions of objects by different DOEs 204 to the image data relayed to the viewer's eyes (at Figure 13c The virtual object or image can be placed at different optical viewing distances, or the virtual object or image can be represented as a 3D stereoscopic image extending through multiple focus planes.

[0093] Now refer to Figure 16 , the display screen 110 may include a plurality of planar waveguides 202a-202d (four shown, collectively referred to as 202). The main waveguides 202a-202d are stacked, arranged, or disposed along an axis 236 that is generally parallel to the field of view of the display screen 110. Each main waveguide 202 includes at least one DOE 204 (shown by a dot-dashed double line, in FIG. Figure 16 Only one is called out in FIG. 1 ). Although all DOEs 204 are shown as being internal, in some embodiments, one, more, or even all of the DOEs 204 may be located external to the main waveguide 202. Additionally or alternatively, although each planar waveguide 202 is illustrated as a single linear array of DOEs 204, one or more of the main waveguides 202 may include two or more stacked, arranged, or disposed DOEs 204, similar to the arrangement with respect to FIG. Figure 15 The described embodiment.

[0094] exist Figure 15 In the embodiment of FIG, each main waveguide 202 may function similarly to the operation of the DOE 204. That is, the DOE 204 of each planar waveguide 202 may each have its own phase diagram, and the phase diagrams of the various DOEs 204 may be different from each other. Figure 15 The embodiment of the present invention adopts the dynamic switching (eg, on / off) of DOE 204, but in Figure 16 In embodiments where the display system 110 selectively routes light to the main waveguide 202 based on the respective phase diagrams, this can be avoided. Instead of or in addition to dynamic switching, the display system 110 can selectively route light to the main waveguide 202 based on the respective phase diagrams. Thus, rather than turning on a specific DOE 204 with a desired phase diagram, the display system 110 can route light to a specific planar waveguide 202 associated with or having a DOE 204 with a desired phase diagram. Again, this can be done instead of or in addition to dynamic switching of the DOE 204.

[0095] In one example, the projection subsystem can be selectively operated to selectively route light to the main waveguide 202 based on the respective phase maps. In another example, each DOE 204 can be independently turned on and off, similar to the description of FIG. Figure 15 The DOE 204 can be turned on and off to selectively route light to the main waveguide 202 based on the respective phase diagrams.

[0096] like Figure 16 As shown, light is emitted outward from two main waveguides 202a and 202d. For ease of illustration, the first planar waveguide 202a produces a planar or flat wavefront at an infinite focal length (indicated by the flat line 238 surrounding the light ray 240, only one example of each is shown for clarity of the drawing). In contrast, the other main waveguide 202d produces a convex wavefront at a limited focal length (e.g., 1 meter) less than infinity (indicated by the arc 242 surrounding the light ray 244, only one example of each is shown for clarity of the drawing). Figure 17 As shown, the main waveguide 202 can be laterally shifted to present and / or optical viewing distances, ie, different focal lengths of the virtual objects 246a - 246c relative to the exit pupil 248 .

[0097] Return Reference Figure 11-1 3. The projection subsystem 108 includes one or more light sources 250 that generate light (e.g., emit light of different colors in a defined pattern) in response to control signals in a predetermined scanning pattern (such as the above relative to Figure 5-9 1. As described herein, a scanning device 252 for scanning light, and an optical coupling subsystem 254 for coupling light from the scanning device 252 into the display screen 110.

[0098] Light source 250 can take any of a number of forms, such as a set of RGB lasers (e.g., laser diodes capable of outputting red, green, and blue light) operable to generate red, green, and blue coherent collimated light, respectively, according to a defined pixel pattern specified in each frame of pixel information or data. Lasers provide high color saturation and are energy efficient.

[0099] The scanning device 252 includes one or more optical fibers 256 (e.g., single-mode optical fibers), each having a proximal end 256a into which light is received from the light source 250 and a distal end 256b from which light is provided to the partially transparent display screen 110. The scanning device 252 further includes a mechanical drive assembly 258 to which the optical fibers 256 are mounted. The drive assembly 258 is configured to displace the distal ends 256b of the optical fibers 256 about a pivot point 260 according to a scanning pattern such that output light is emitted from a longitudinal axis 262 that is coincident with the pivot point 260. It should be understood that although the display subsystem 104 has been described as being implemented using scanning fiber technology, it should be understood that the display subsystem 104 can be based on any display technology, such as a liquid crystal display (LCD), a digital light processing (DLP) display, etc.

[0100] The drive assembly 208 includes a piezoelectric element 264 to which the optical fiber 256 is mounted, and drive electronics 266 configured to transmit an electrical signal to the piezoelectric element 264, thereby causing the distal end 256b of the optical fiber 256 to vibrate according to the scanning pattern. Thus, the operation of the light source 250 and the drive electronics 266 are coordinated in a manner to generate image data that is encoded in the form of spatially and / or temporally varying light.

[0101] In the illustrated embodiment, the piezoelectric element 264 takes the form of a hollow tube, in which case the distal end 256b of the optical fiber 256 is passed through or received by the piezoelectric tube 264. The distal end 256b of the optical fiber 256 protrudes from the piezoelectric tube 264 as a non-fixed flexible cantilever 268 ( Figure 18a and 18b 264). Piezoelectric tube 264 is associated with four quadrant electrodes (not shown). For example, the electrodes may be plated on the exterior, outer surface, or outer perimeter or diameter of piezoelectric tube 264. A core electrode (not shown) is also located on the core, center, inner perimeter, or inner diameter of tube 264.

[0102] Drive electronics 266 are electrically coupled via wires 270 to drive opposing pairs of electrodes (not shown) to independently bend the piezoelectric tube 264 in two axes. The protruding distal tip of the optical fiber 256 has a mechanical resonant mode. The resonant frequency depends on the diameter, length, and material properties of the optical fiber 256. Figure 18a As shown, by vibrating the piezoelectric tube 264 near the first-order mode of mechanical resonance of the fiber cantilever 268, the fiber cantilever 268 is caused to vibrate and can sweep a large deflection around the fulcrum 260. By exciting resonance on two axes, the tip of the fiber cantilever 268 is scanned biaxially in the area filled with 2D scanning. By modulating the intensity of the light source 250 in synchronization with the scanning of the fiber cantilever 268, the light emitted from the fiber cantilever 268 forms an image.

[0103] As briefly discussed above, the optical coupling subsystem 254 optically couples light from the scanning device 252 to the waveguide device 102. The optical coupling subsystem 254 includes an optical waveguide input device 272, such as one or more reflective surfaces, diffraction gratings, mirrors, dichroic mirrors, or prisms, to couple light to the end of the waveguide device 102. The optical coupling subsystem 254 additionally or alternatively includes a collimating element 274 that collimates light from the optical fiber 256.

[0104] As briefly discussed above, light emitted from the scanning device 252 initially diverges from the longitudinal axis 262 as the distal end 256b of the optical fiber 256 vibrates about the fulcrum 260. At each position of the optical fiber 256, the light initially fans out from the distal end 256b and is collimated into a narrow line by the collimating element 274. Without modification, a relatively large optical waveguide input device 272 would be required to accommodate the relatively large deflections in the distal end 256b of the optical fiber 256.

[0105] To this end, the optical coupling subsystem 254 includes an optical modulation device 276 configured to focus light from the collimating element 274 toward the longitudinal axis 262 and, in the embodiment shown, onto a focal point 278 in the center of the optical waveguide input device 272. Figure 18a As shown. It is worth noting that focusing the light at the center of the optical waveguide input device 272 allows the size of the optical waveguide input device 272 to be minimized. That is, focusing the light at the center of the optical waveguide input device 272 minimizes the worst-case divergence span of the scanning light path at the edge of the optical waveguide input device 272. For example, if the light is focused on the front edge of the optical waveguide input device 272, as shown in FIG. Figure 18b As shown, the optical waveguide input device 272 must be made larger to accommodate the larger divergence span of the scanning light path at the rear edge of the optical waveguide input device 272.

[0106] Now refer to Figure 19 , the interaction between the input light modulation device 276 and one embodiment of the optical waveguide input device 272 will be described. In this case, the optical waveguide input device 272 takes the form of a distributed waveguide device 222 having a plurality of optical components. Figures 13a-13c The associated distributed waveguide 224 and DOEs 226a-226e are depicted. Figure 19 As shown, the light modulation device 276 focuses the light from the collimating element 274 onto a focal point 278 located at the DOE 226c, thereby minimizing the size of the DOE 226. As a result, the overall width of the distribution waveguide 224 is minimized. The light is then selectively transmitted down one or more of the main waveguides 202a-202e.

[0107] Now refer to Figure 20Alternatively, the DOE 226 may be directly incorporated into the main waveguide device 200. In this case, the DOE 226 extends parallel to the main planar waveguides 202a-202e of the waveguide device 200, respectively, so that the DOE 226 guides the light from the light modulation device 276 along the main waveguide 202. Figure 20 As shown, light modulation device 276 focuses light from collimating element 274 onto focal point 278 located at DOE 226 c, thereby minimizing the size of DOE 226. Since focal point 278 is located at the center of input light modulation device 276, the lengths of DOEs 226 a and 226 e can be made equal to minimize the worst-case DOE 226. As a result, the overall size of waveguide device 200 is minimized.

[0108] In one embodiment, light modulation device 276 includes one or more diffraction gratings, each of which can be characterized as an optical component having a periodic structure on the order of the wavelength of light that splits and diffracts light into several beams traveling in different directions. The diffraction grating can be transmissive or reflective and can be composed of, for example, surface nano-ridges, nano-patterns, slits, etc., which can be photolithographically printed on a substrate. In alternative embodiments, light modulation device 276 can include one or more lenses.

[0109] In the embodiment shown, the light modulating device 276 has a diffraction pattern that matches the geometry of the scanning pattern so that the collimation of the light is maintained at the target resolution. For example, if a spiral scanning pattern is used, the diffraction pattern can have a pattern of diffraction elements 280, such as Figure 21 If a single diffraction grating is used, each diffraction element can diffract light 282a and 282b inward into two dimensions (e.g., in Figure 22a In the case of a diffraction grating 276a, 276b, the diffraction elements 278a and 278b of each diffraction grating 276a may be arranged in series to diffract the light rays 280a and 280b along one axis of the xy coordinate system (e.g., at the origin of the xy coordinate system). Alternatively, two orthogonal diffraction gratings 276a and 276b may be used, such that the diffraction elements 278a and 278b of one diffraction grating 276a diffract the corresponding light rays 280a and 280b along one axis of the xy coordinate system (e.g., at the origin of the xy coordinate system). Figure 22b In the case of the x-direction), the diffraction elements of the other diffraction grating 276b diffract the light rays 280a and 280b along the other axis of the xy coordinate system (e.g., in Figure 22b in the case of y-direction).

[0110] Although specific embodiments of the present invention have been shown and described, it should be understood that this is not intended to limit the present invention to these embodiments, and that various changes and modifications may be made as will be apparent to those skilled in the art without departing from the spirit and scope of the present invention. Therefore, the present invention is intended to cover substitutions, modifications, and equivalents that may be included within the spirit and scope of the present invention as defined by the claims.

Claims

1. A display subsystem for a virtual image generation system, comprising: at least one waveguide; optical fiber; at least one light source configured to direct light into the proximal end of the optical fiber; a mechanical drive assembly to which the optical fiber is mounted, the mechanical drive assembly being configured to manipulate the optical fiber so as to displace a distal end of the optical fiber about a fulcrum according to a scan pattern and so as to cause light emitted from the distal end of the optical fiber during the scan pattern to diverge from a longitudinal axis coincident with the fulcrum toward a plurality of off-axis scan positions to produce one or more image frames; a light modulation device configured to converge light emitted from the distal end of the optical fiber at each of the plurality of off-axis scanning positions toward the longitudinal axis; as well as an optical waveguide input device configured to guide the light focused by the light modulation device into the at least one waveguide so that the virtual image generation system displays the one or more image frames, wherein the light modulation device comprises two orthogonal diffraction gratings connected in series, such that one of the diffraction gratings diffracts light along a first axis, and the other of the diffraction gratings diffracts light along a second axis orthogonal to the first axis; wherein the light modulation device focuses the light onto a focal point in the center of the optical waveguide input device.

2. The display subsystem according to claim 1, wherein: The light modulation device is configured to converge the light from the optical fiber to a focus on the longitudinal axis.

3. The display subsystem according to claim 2, wherein: The focal point is located within the optical waveguide input device.

4. The display subsystem of claim 1, further comprising a collimating element configured to collimate light from the optical fiber.

5. The display subsystem according to claim 1, wherein: Each of the at least one diffraction grating has a diffraction pattern that matches a geometry of the scan pattern.

6. The display subsystem according to claim 5, wherein: The scanning pattern is a spiral scanning pattern, and the diffraction pattern is a spiral diffraction pattern.

7. The display subsystem according to claim 1, wherein: The mechanical drive assembly includes a piezoelectric element to which the optical fiber is mounted, and drive electronics configured to transmit an electrical signal to the piezoelectric element, thereby causing the optical fiber to vibrate according to the scanning pattern.

8. The display subsystem according to claim 1, wherein: The at least one waveguide includes: a plurality of planar waveguides configured to display the one or more image frames to an end user at different focal points; and the optical waveguide input device is configured to guide light from the optical modulation device to the plurality of planar waveguides.

9. The display subsystem according to claim 8, wherein: The optical waveguide input device includes a plurality of diffractive optical elements extending in parallel along the planar waveguides, respectively, and the plurality of diffractive optical elements guide the light from the light modulation device to the planar waveguides, respectively.

10. The display subsystem according to claim 8, wherein: The optical waveguide input device includes a distributed waveguide extending perpendicular to the planar waveguide, the distributed waveguide includes a plurality of diffraction optical elements, and the plurality of diffraction gratings guide the light from the optical modulation device to the planar waveguide respectively.

11. The display subsystem according to claim 1, wherein: The at least one waveguide is configured to be located in front of an eye of an end user.

12. The display subsystem according to claim 1, wherein: The at least one waveguide has a partially transparent display surface configured to be located in a field of view between an end user's eye and a surrounding environment.

13. The display subsystem of claim 1, further comprising a frame structure configured to be worn by an end user, the frame structure carrying the at least one waveguide.

14. The display subsystem of claim 1, wherein one of the plurality of off-axis scanning positions is an extreme scanning position of the optical fiber.

15. The display subsystem according to claim 1, wherein: The light modulation device is configured to converge light from all off-axis scan positions of the optical fiber toward the longitudinal axis.

16. A virtual image generation system for use by an end user, comprising: a memory storing a three-dimensional scene; a control subsystem configured to render a plurality of composite image frames of the three-dimensional scene; as well as The display subsystem of claim 1 , wherein the display subsystem is configured to sequentially display a plurality of image frames to the end user.

17. The virtual image generation system according to claim 16, wherein: The control subsystem includes a graphics processing unit (GPU).

18. A display subsystem of a virtual image generation system for use by an end user, comprising: at least one waveguide; optical fiber; at least one light source configured to direct light into the proximal end of the optical fiber; a mechanical drive assembly to which the optical fiber is mounted, the mechanical drive assembly being configured to manipulate the optical fiber so as to displace a distal end of the optical fiber about a fulcrum according to a scanning pattern and so as to cause light emitted from the distal end of the optical fiber during the scanning pattern to diverge from a longitudinal axis coincident with the fulcrum; a light modulation device configured to converge light emitted from the distal end of the optical fiber toward the longitudinal axis; as well as an optical waveguide input device configured to guide the light focused by the light modulation device into the at least one waveguide so that the virtual image generation system displays one or more image frames, wherein the light modulation device comprises two orthogonal diffraction gratings connected in series, such that one of the diffraction gratings diffracts light along a first axis, and the other of the diffraction gratings diffracts light along a second axis orthogonal to the first axis; wherein the light modulation device focuses the light onto a focal point in the center of the optical waveguide input device.

19. A display subsystem for a virtual image generation system, comprising: optical fiber; a mechanical drive assembly to which the optical fiber is mounted, the mechanical drive assembly being configured to manipulate the optical fiber so as to displace a distal end of the optical fiber about a fulcrum according to a scan pattern and to cause light emitted from the distal end of the optical fiber during the scan pattern to diverge from a longitudinal axis coincident with the fulcrum toward a plurality of off-axis scan positions to produce one or more image frames; a light modulation device configured to converge light from the distal end of the optical fiber at each of the plurality of off-axis scanning positions toward the longitudinal axis; as well as Optical waveguide input device, wherein the light modulation device comprises two orthogonal diffraction gratings connected in series, such that one of the diffraction gratings diffracts light along a first axis, and the other of the diffraction gratings diffracts light along a second axis orthogonal to the first axis; wherein the light modulation device focuses the light onto a focal point in the center of the optical waveguide input device.

20. The display subsystem according to claim 19, wherein: The light modulation device is configured to converge the light from the optical fiber to a focus on the longitudinal axis.

21. The display subsystem of claim 20, wherein the focal point is located within the light guide input device.

22. The display subsystem of claim 19, further comprising a collimating element configured to collimate light from the optical fiber.

23. The display subsystem according to claim 19, wherein: Each of the two orthogonal diffraction gratings has a diffraction pattern that matches the geometry of the scan pattern.

24. The display subsystem according to claim 23, wherein: The scanning pattern is a spiral scanning pattern, and the diffraction pattern is a spiral diffraction pattern.

25. The display subsystem according to claim 19, wherein: The mechanical drive assembly includes a piezoelectric element to which the optical fiber is mounted, and drive electronics configured to transmit an electrical signal to the piezoelectric element, thereby causing the optical fiber to vibrate according to the scanning pattern.

26. The display subsystem of claim 19, further comprising at least one waveguide, wherein The at least one waveguide includes: a plurality of planar waveguides configured to display the one or more image frames to an end user at different focal points; and the optical waveguide input device is configured to guide light from the optical modulation device to the plurality of planar waveguides.

27. The display subsystem according to claim 26, wherein: The optical waveguide input device includes a plurality of diffractive optical elements extending in parallel along the planar waveguides, respectively, and the plurality of diffractive optical elements guide the light from the light modulation device to the planar waveguides, respectively.

28. The display subsystem according to claim 26, wherein: The optical waveguide input device includes a distributed waveguide extending perpendicular to the planar waveguide, the distributed waveguide includes a plurality of diffraction optical elements, and the plurality of diffraction gratings guide the light from the optical modulation device to the planar waveguide respectively.

29. The display subsystem of claim 26, wherein: The at least one waveguide is configured to be located in front of an eye of the end user.

30. The display subsystem of claim 26, wherein: The at least one waveguide has a partially transparent display surface configured to be located in a field of view between an eye of the end user and a surrounding environment.

31. The display subsystem of claim 26, further comprising a frame structure configured to be worn by the end user, the frame structure carrying the at least one waveguide.

32. The display subsystem of claim 19, wherein one of the plurality of off-axis scan positions is an extreme scan position of the optical fiber.

33. The display subsystem of claim 19, wherein: The light modulation device is configured to converge light from all off-axis scan positions of the optical fiber toward the longitudinal axis.

34. A virtual image generation system for use by an end user, comprising: a memory storing a three-dimensional scene; a control subsystem configured to render a plurality of composite image frames of the three-dimensional scene; as well as The display subsystem of claim 19, wherein the display subsystem is configured to sequentially display a plurality of image frames to the end user.

35. The virtual image generation system according to claim 34, wherein: The control subsystem includes a graphics processing unit (GPU).

36. A display subsystem of a virtual image generation system for use by an end user, comprising: optical fiber; a mechanical drive assembly to which the optical fiber is mounted, the mechanical drive assembly being configured to manipulate the optical fiber so as to displace a distal end of the optical fiber about a fulcrum according to a scanning pattern and so as to cause light emitted from the distal end of the optical fiber during the scanning pattern to diverge from a longitudinal axis coincident with the fulcrum; a light modulation device configured to focus light from the distal end of the optical fiber toward the longitudinal axis; as well as Optical waveguide input device, wherein the light modulation device comprises two orthogonal diffraction gratings connected in series, such that one of the diffraction gratings diffracts light along a first axis, and the other of the diffraction gratings diffracts light along a second axis orthogonal to the first axis; wherein the light modulation device focuses the light onto a focal point in the center of the optical waveguide input device.

37. The display subsystem of claim 36, wherein: The light modulation device is configured to converge the light from the optical fiber to a focus on the longitudinal axis.

38. The display subsystem of claim 36, wherein: The light modulation device is configured to converge light from all off-axis scan positions of the optical fiber toward the longitudinal axis.

39. A display subsystem of a virtual image generation system for use by an end user, comprising: Planar waveguide devices; optical fiber; at least one light source configured to emit light from a distal end of the optical fiber; a mechanical drive assembly to which the optical fiber is mounted as a non-stationary flexible cantilever, the mechanical drive assembly being configured to displace the distal end of the optical fiber about a fulcrum according to a scanning pattern such that emitted light diverges from a longitudinal axis coincident with the fulcrum; a light modulation device configured to focus the light from the optical fiber toward the longitudinal axis; as well as an optical waveguide input device configured to guide the light from the optical modulation device to the planar waveguide device so that the planar waveguide device displays one or more image frames to the end user, wherein the light modulation device comprises two orthogonal diffraction gratings connected in series, such that one of the diffraction gratings diffracts light along a first axis, and the other of the diffraction gratings diffracts light along a second axis orthogonal to the first axis; wherein the light modulation device focuses the light onto a focal point in the center of the optical waveguide input device.

Images