System and method for virtual, augmented, and composite reality

Abstract

To provide a system and a method for preferable virtual, augmented and composite reality.SOLUTION: A method according to the present invention of determining focal depth of a user of a three-dimensional ("3D") display device has a step of pursuing a first visual line path of the user. The method according to the present invention also has a step of analyzing 3D data for discriminating one or more virtual objects along the first visual line path of the user. The method according to the present invention further has a step of discriminating focal depth of a single virtual object as the focal depth of the user when the single virtual object crosses the first visual line path of the user.SELECTED DRAWING: Figure 16

Description

[Technical field]

[0001] (Copyright Notice) A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to anyone copying the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves any and all copyrights whatsoever.

[0002] FIELD OF THE DISCLOSURE The present disclosure relates to virtual reality, augmented reality, and mixed reality imaging, visualization, and display systems and methods. In particular, the present disclosure relates to virtual reality, augmented reality, and mixed reality imaging, visualization, and display systems and methods for determining a viewer's depth of focus. [Background technology]

[0003] Modern computing and display technologies have facilitated the development of virtual reality (VR), augmented reality (AR), and mixed reality (MR) systems. VR systems create a simulated environment for a user to experience. This can be done by presenting computer-generated images to the user through a head-mounted display. The images create a sensory experience that immerses the user within the simulated environment. VR scenarios typically do not involve the presentation of only computer-generated images, but also include actual real-world images.

[0004] AR systems generally complement the real-world environment with simulated elements. For example, an AR system may provide a user with a view of the surrounding real-world environment via a head-mounted display. However, computer-generated images may also be presented on the display to enhance the real-world environment. The computer-generated images may include elements that are contextually related to the real-world environment. Such elements may include simulated text, images, objects, etc. MR systems also introduce simulated objects into the real-world environment, but these objects typically feature a degree of interactivity that exceeds that in AR systems. The simulated elements can often be interactive in real time.

[0005] 1 depicts an example AR / MR scene 2, in which a user sees a real-world park setting 6 featuring people, trees, buildings in the background, and a concrete platform 20. In addition to these items, computer-generated imagery is also presented to the user. The computer-generated imagery may include, for example, a robotic figure 10 standing on the real-world platform 20 and a flying cartoon-like avatar character 12 that appears to be an anthropomorphic bumblebee, although these elements 12, 10 do not actually exist in the real-world environment.

[0006] Different optical systems generate images at different depths to display VR, AR or MR scenarios.

[0007] VR / AR / MR systems can operate in a variety of display modes, including blended light field and discrete light field modes. In blended light field mode ("blended mode"), the display presents a synthetic light field that includes images displayed simultaneously or nearly simultaneously in two or more discrete depth planes. In discrete light field mode ("discrete mode"), the display presents a synthetic light field that includes images displayed in a single depth plane. The illuminated depth planes are controllable in the discrete modes. Some VR / AR / MR systems include an eye tracking subsystem for determining a viewer's focal position. Such systems can illuminate a depth plane that corresponds to the viewer's focal position.

[0008] The complexity of the human visual perception system makes it difficult to produce VR / AR / MR technologies that facilitate comfortable, natural-like, and rich presentation of virtual image elements among other virtual or real-world image elements. Three-dimensional ("3D") image display systems suffer from the vergence-accommodation collision problem. This problem occurs when two optical depth-related biological processes send conflicting depth signals to the user / viewer's brain. Vergence is related to the tendency of the viewer's eyes to rotate to align the optical axis with an object of the viewer's attention at a certain distance. In a binocular system, the point where the optical axes intersect may be called the "vergence point." The amount of rotation of the viewer's eyes during vergence is interpreted by the viewer's brain as an estimated depth. Accommodation is related to the tendency of the viewer's eye lens to focus on an object of the viewer's attention at a certain distance. The focus of the viewer's eyes during convergence and divergence is interpreted by the viewer's brain as different depth estimates. When the convergence and accommodation signals are interpreted by the viewer's brain as the same or similar depth estimates, the 3D viewing experience is natural and comfortable for the viewer. On the other hand, when the convergence and accommodation signals are interpreted by the viewer's brain as substantially different depth estimates, the 3D viewing experience is suboptimal for the viewer and may result in discomfort (eye strain, headaches, etc.) and fatigue. Such a problem is known as convergence-accommodation conflict.

[0009] Portable VR / AR / MR systems also have other limitations, such as size and portability issues, battery life issues, system heating issues, processing power, memory, bandwidth, data sources, component latency, and other system and optical challenges, which may adversely affect VR / AR / MR system performance. These other limitations must be considered when implementing 3D image rendering techniques for natural vergence and accommodation. To accommodate these hardware-related limitations of VR / AR / MR systems, it is often useful to determine the user / viewer's focal point. Although some systems map the user / viewer's focal point to the user / viewer's vergence point, limitations in eye tracking hardware and high speed user / viewer eye movements reduce the accuracy of determining the user / viewer's focal point using this method. Determining the user / viewer's focal point allows the VR / AR / MR system to effectively allocate limited hardware resources. In particular, determining the depth of a user / viewer's focus allows a VR / AR / MR system to direct limited hardware resources to a specific plane that coincides with the user's / viewer's focus.

[0010] One method of determining the depth of focus of a user / viewer is to use depth sensing to generate a depth map of the entire field of view, and then use eye tracking to use the depth map of the field of view to identify the depth of focus of the user / viewer. Depth sensing is the determination of the distance between a known point in three-dimensional ("3D") space (e.g., a sensor) and a point of interest ("POI") on the surface of an object. Depth sensing is also known as texture sensing, since determining the individual distances of multiple POIs on a surface determines the texture of the surface. Depth or texture sensing is useful for many computer vision systems, including mixed reality systems. Generating a depth map can be a demanding process in terms of hardware (e.g., infrared cameras and infrared light projectors) and processing power. In a mobile VR / AR / MR system, the depth map would need to be continually updated, at some cost in terms of processing power, memory, communication channels, etc.

[0011] Improved systems and techniques for processing image data and displaying images are needed, including, for example, systems and techniques for determining a user / viewer's depth of focus for rendering and displaying 3D images to the viewer / user while minimizing vergence-divergence-accommodation conflicts, and for doing so while minimizing demands on the limited graphical processing capabilities of portable VR / AR / MR systems. Improved systems and techniques are needed to address these issues. The systems and methods described herein are configured to address these and other challenges.

[0012] What is needed is a technique or techniques for improving upon older and / or other contemplated approaches. Some of the approaches described in this Background section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Summary of the Invention [Means for solving the problem]

[0013] In one embodiment, a method for determining a focal depth of a user of a three-dimensional ("3D") display device includes tracking a first line of sight path of the user. The method also includes analyzing the 3D data to identify one or more virtual objects along the first line of sight path of the user. The method further includes identifying a depth of only one virtual object as the focal depth of the user when only one virtual object intersects with the first line of sight path of the user.

[0014] In one or more embodiments, each of the one or more virtual objects is displayed at a distinct depth from the user. Analyzing the plurality of 3D data may include analyzing depth segmented data corresponding to the 3D data. The method may also include performing a vergence-divergence analysis on the user to generate a vergence-divergence focal depth for the user, and using the user's vergence-divergence focal depth to refine the accuracy of the user's focal depth.

[0015] In one or more embodiments, when more than one virtual object intersects with the first line of sight path of the user, the method may also include tracking a second line of sight path of the user. The method may also include analyzing the 3D data to identify one or more virtual objects along the second line of sight path of the user. The method may further include identifying a depth of only one 2D image frame as the user's focal depth when only one virtual object intersects with both the first line of sight path and the second line of sight path of the user.

[0016] In another embodiment, a method for determining a focal depth of a user of a three-dimensional ("3D") display device includes tracking a first line of sight path of the user. The method also includes analyzing a plurality of two-dimensional ("2D") image frames displayed by the display device to identify one or more virtual objects along the user's first line of sight path. The method further includes, when only one of the plurality of 2D image frames includes a virtual object along the user's first line of sight path, identifying a depth of only one 2D image frame as the user's focal depth.

[0017] In one or more embodiments, each of the multiple 2D image frames is displayed at a distinct depth from the user. Analyzing the multiple 2D image frames may include analyzing depth segmented data corresponding to the multiple 2D image frames. The method may also include performing a vergence-divergence analysis on the user to generate a vergence-divergence focal depth for the user, and using the user's vergence-divergence focal depth to refine the accuracy of the user's focal depth.

[0018] In one or more embodiments, when more than one of the plurality of 2D image frames includes a virtual object along the first line of sight path of the user, the method may also include tracking a second line of sight path of the user. The method may also include analyzing the plurality of 2D image frames displayed by the display device to identify one or more virtual objects along the second line of sight path of the user. The method may further include identifying a depth of only one of the plurality of 2D image frames as a focal depth of the user when only one of the plurality of 2D image frames includes a virtual object along both the first line of sight path and the second line of sight path of the user. The present invention provides, for example, the following: (Item 1) 1. A method for determining a depth of focus of a user of a three-dimensional ("3D") display device, comprising: Tracking a first gaze path of the user; analyzing the 3D data to identify one or more virtual objects along the first line of sight path of the user; identifying a depth of the only one virtual object as a focal depth of the user when the only one virtual object intersects with the first line of sight path of the user; A method comprising: (Item 2) 2. The method of claim 1, wherein each of the one or more virtual objects is displayed at a distinct depth from the user. (Item 3) 2. The method of claim 1, wherein analyzing the plurality of 3D data includes analyzing depth segmented data corresponding to the 3D data. (Item 4) when more than one virtual object intersects the first line of sight path of the user; Tracking a second gaze path of the user; and analyzing the 3D data to identify the one or more virtual objects along a second line of sight path of the user; identifying a depth of only one 2D image frame as a focal depth of the user when only one virtual object intersects with both a first line of sight path and a second line of sight path of the user; The method of claim 1, further comprising: (Item 5) performing a convergence analysis on the user to generate a convergence depth of focus for the user; using the user's convergence-divergence focal depth to improve the accuracy of the user's focal depth; The method of claim 1, further comprising: (Item 6) 1. A method for determining a depth of focus of a user of a three-dimensional ("3D") display device, comprising: Tracking a first gaze path of the user; analyzing a plurality of two-dimensional ("2D") image frames displayed by the display device to identify one or more virtual objects along the first line of sight path of the user; identifying a depth of only one of the plurality of 2D image frames as a focal depth of the user when only one of the plurality of 2D image frames includes a virtual object along a first line of sight path of the user; A method comprising: (Item 7) 7. The method of claim 6, wherein each of the plurality of 2D image frames is displayed at a distinct depth from the user. (Item 8) 7. The method of claim 6, wherein analyzing the plurality of 2D image frames includes analyzing depth segmentation data corresponding to the plurality of 2D image frames. (Item 9) when more than one of the plurality of 2D image frames includes a virtual object along a first line of sight path of the user, Tracking a second gaze path of the user; and analyzing the plurality of 2D image frames displayed by the display device to identify one or more virtual objects along a second line of sight path of the user; identifying a depth of only one of the plurality of 2D image frames as a focal depth of the user when only one of the plurality of 2D image frames includes a virtual object along both a first line of sight path and a second line of sight path of the user; The method according to item 6, further comprising: (Item 10) performing a convergence analysis on the user to generate a convergence depth of focus for the user; using the user's convergence-divergence focal depth to improve the accuracy of the user's focal depth; The method according to item 6, further comprising: [Brief description of the drawings]

[0019] This patent or patent application contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0020] The drawings described below are for illustration purposes only. The drawings are not intended to limit the scope of the present disclosure. This patent or application contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the U.S. Patent and Trademark Office upon request and payment of the necessary fee.

[0021] The drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale, and elements of similar structure or function are represented by similar reference numerals throughout the drawings. In order to better understand how to obtain the enumerated and other advantages and objects of the various embodiments of the present disclosure, a further detailed description of the present disclosure will be given by reference to specific embodiments thereof as illustrated in the accompanying drawings. With the understanding that these drawings depict only exemplary embodiments of the present disclosure and therefore should not be considered limiting of its scope, the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0022] [Figure 1] FIG. 1 illustrates a user's view of an AR / MR scene using an exemplary AR system.

[0023] [Diagram 2] 2-5 diagrammatically depict a user using a VR / AR / MR system, according to some embodiments. [Diagram 3] 2-5 diagrammatically depict a user using a VR / AR / MR system, according to some embodiments. [Figure 4]2-5 diagrammatically depict a user using a VR / AR / MR system, according to some embodiments. [Diagram 5] 2-5 diagrammatically depict a user using a VR / AR / MR system, according to some embodiments.

[0024] [Figure 6] FIG. 6 diagrammatically depicts various planes of a multi-plane focusing system, according to some embodiments.

[0025] [Figure 7] FIG. 7 diagrammatically depicts a VR / AR / MR system, according to some embodiments.

[0026] [Figure 8] 8A and 8B diagrammatically depict a 3D image including multiple virtual objects from a side perspective view (FIG. 8A) and a rear perspective view (FIG. 8B) according to some embodiments.

[0027] [Figure 9] 9A and 9B diagrammatically depict a 3D image including multiple virtual objects projected onto a single depth plane for display in a discrete mode, from a side perspective view (FIG. 9A) and a rear perspective view (FIG. 9B).

[0028] [Figure 10A] FIG. 10A diagrammatically depicts near and far accommodation zones and accommodation overlap zones of a dual surface blend mode display system, according to some embodiments.

[0029] [Figure 10B] FIG. 10B graphically depicts the relationship between distance from the viewer and diopters.

[0030] [Figure 11]FIG. 11 diagrammatically depicts a 3D image including multiple virtual objects projected onto a single depth plane for display in a discrete mode, from a side perspective view.

[0031] [Figure 12A] FIG. 12A diagrammatically depicts a 3D image including multiple virtual objects projected onto a single depth plane for display in a discrete mode, from a side perspective view.

[0032] [Figure 12B] FIG. 12B diagrammatically depicts a 3D image including multiple virtual objects with multiple virtual depth planes displayed in blend mode from a side perspective view.

[0033] [Figure 13] 13 and 14 diagrammatically depict systems and methods for determining a user / viewer's focal plane from a rear perspective view (FIG. 13) and a side perspective view (FIG. 14), according to some embodiments. [Figure 14] 13 and 14 diagrammatically depict systems and methods for determining a user / viewer's focal plane from a rear perspective view (FIG. 13) and a side perspective view (FIG. 14), according to some embodiments.

[0034] [Figure 15] FIG. 15 diagrammatically depicts a 3D image including multiple virtual objects blurred and projected onto a single determined depth plane for display in a discrete mode from a rear perspective view, in accordance with some embodiments.

[0035] [Figure 16] FIG. 16 is a flowchart illustrating a method for determining a focal plane of a user / viewer, according to some embodiments.

[0036] [Figure 17]FIG. 17 is a block diagram that diagrammatically depicts an illustrative computing system, in accordance with some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] Detailed Description Various embodiments of the present disclosure are directed to systems, methods, and articles of manufacture for Virtual Reality (VR) / Augmented Reality (AR) / Mixed Reality (MR) in a single embodiment or multiple embodiments. Other objects, features, and advantages of the present disclosure are set forth in the detailed description, drawings, and claims.

[0038] Various embodiments will now be described in detail with reference to the drawings, which are provided as illustrative examples to enable those skilled in the art to practice the present disclosure. It should be noted that the drawings and the following examples are not meant to limit the scope of the present disclosure. Certain elements of the present disclosure may be implemented partially or completely using known components (or methods or processes), and only those parts of such known components (or methods or processes) necessary for understanding the present disclosure will be described, and detailed descriptions of other parts of such known components (or methods or processes) will be omitted so as not to obscure the present disclosure. Furthermore, various embodiments encompass currently known and future known equivalents of the components referenced herein as examples.

[0039] Embodiments according to the present disclosure address the problem of implementing VR / AR / MR systems, which often rely on a combination of off-the-shelf and custom components. In some cases, the off-the-shelf components do not possess all of the features or performance characteristics required to implement certain desired aspects of the VR / AR / MR system to be deployed. Some embodiments are directed to approaches to add capabilities and / or repurpose resources to accommodate desired features or performance characteristics of the VR / AR / MR system to be deployed. The accompanying figures and discussion herein present an example environment for a VR / AR / MR system, the present systems, methods, and computer program products.

[0040] Although the head-mounted audiovisual display system and the 3D image rendering system may be implemented independently of the AR / MR system, some embodiments below are described in relation to an AR / MR system for illustrative purposes only. The 3D image rendering and display systems described herein may also be used in a similar manner with VR systems. Problem Summary and Solutions

[0041] As explained above, VR / AR / MR systems also have limitations such as size and portability, battery life, system heating, processing power, memory, bandwidth, data sources, component latency, and other system and optical challenges that may adversely affect VR / AR / MR system performance. These limitations may result in reduced graphical processing and image display demands, countering challenges to improving 3D image rendering and display. VR / AR / MR systems may deploy limited processing and display resources more effectively by identifying a depth plane that corresponds to the user / viewer's focal point. However, using only vergence-divergence movements to identify the user / viewer's focal plane may not be accurate enough. On the other hand, generating a depth map for use in identifying the user / viewer's focal plane may add processing and hardware demands to the VR / AR / MR system.

[0042] For example, due to potential graphical processing and size / portability issues, a VR / AR / MR system, particularly a head-mounted system, may include only enough components to be able to render and display color images at one depth plane per frame at a minimum frame rate (e.g., 60 Hz) for smooth display of moving virtual objects (i.e., discrete mode). An example of such a VR / AR / MR system operating in discrete mode is shown in Figs. 8A-9B. As shown diagrammatically in Fig. 8A, the 3D image includes three virtual objects (near and far cubes 810, 812 and one cylinder 814) adjacent various depth planes (near cube 810 adjacent near depth plane 0 and far cube 812 and cylinder 814 adjacent far depth plane 1). In some embodiments, near depth plane 0 is at about 1.96 diopters and far depth plane 1 is at about 0.67 diopters. Fig. 8B is a perspective view of a viewer of the 3D image shown in Fig. 8A. In FIG. 8B, eye tracking of a user / viewer 816 shows that the viewer's eyes converge to a vergence-divergence point 818 that coincides with the location of the near cube 810. In discrete mode, only a single depth plane is illuminated (i.e., an image is rendered and displayed) per frame. As shown diagrammatically in FIG. 9A, all content of the 3D image (i.e., near and far cubes 810, 812 and cylinder 814) is projected onto near depth plane 0 because the vergence-divergence point 818 coincides with the location of the near cube 810, which is adjacent to the vicinity of depth plane 0. FIG. 9B is a perspective view of a viewer of the 3D image after its content has been projected onto near depth plane 0. Only near depth plane 0 is illuminated, and the viewer's 816 eyes accommodate to near depth plane 0. As explained above, the equating of the user / viewer's focal point with the user / viewer's vergence-divergence point may result in a less accurate determination of this important data point.

[0043] In some embodiments, the projection of all of the 3D image content onto a single depth plane induces only minimal vergence-accommodation conflicts (e.g., minimal user discomfort, eye strain, headaches). This is because there is a loose coupling of accommodation and vergence such that the human brain will tolerate a maximum of about 0.75 diopters of mismatch between accommodation and vergence. As shown in FIG. 10A, this ±0.75 diopter tolerance translates into a near accommodation zone 1010 and a far accommodation zone 1012. Due to the inverse relationship between distance and diopters, the far accommodation zone 1012 is larger than the closer accommodation zone 1010, as shown in FIG. 10B. 10A , with near depth plane 0 and far depth plane 1, the ±0.75 diopter tolerance also results in accommodation zone overlap 1014, where object depths that fall within the accommodation zone overlap 1014 may be displayed in one or both of near depth plane 0 and far depth plane 1, e.g., at different scales, with different brightness and / or color values, etc. In embodiments where all of the 3D image content is located within either the near accommodation zone 1010 or the far accommodation zone 1012, and the eyes of the viewer 816 converge to that depth plane, the projection of all of the 3D image content onto that depth plane induces only minimal vergence-divergence-accommodation collisions.

[0044] Operating in discrete mode allows a VR / AR / MR system to deliver 3D images while functioning within its hardware limitations (e.g., processing, display, memory, communication channels, etc.). However, discrete mode displays require accurate identification of a single depth plane to be illuminated while minimizing demands on system resources. VR / AR / MR systems operating in blended mode can also reduce demands on system resources by focusing their resources on the depth plane that corresponds to the user / viewer's focal point. Again, the present resource-saving method for blended mode displays requires accurate identification of the user / viewer's focal plane.

[0045] The embodiments described herein include 3D image rendering and display systems and methods for use with various VR / AR / MR systems that identify a depth plane that corresponds to a user / viewer's focal point for use in both discrete and blended mode displays while consuming fewer system resources, thereby addressing many of the problems described above. Illustrative VR, AR, and / or MR Systems

[0046] The following description relates to illustrative VR, AR, and / or MR systems with which various 3D image rendering and display system embodiments may be practiced, however, it should be understood that the embodiments are also suitable for use in other types of display systems (including other types of VR, AR, and / or MR systems), and thus the embodiments are not limited to only the illustrative systems disclosed herein.

[0047] The VR / AR / MR systems disclosed herein can include a display that presents computer-generated images (video / image data) to a user. In some embodiments, the display system is wearable, which may advantageously provide a more immersive VR / AR / MR experience. Various components of a VR, AR, and / or MR virtual image system 100 are depicted in FIGS. 2-5. The virtual image generation system 100 includes a frame structure 102 worn by an end user 50, a display subsystem 110 carried by the frame structure 102 such that the display subsystem 110 is positioned in front of the eye of the end user 50, and a speaker 106 carried by the frame structure 102 such that the speaker 106 is positioned adjacent to the ear canal of the end user 50 (optionally, another speaker (not shown) is positioned adjacent to the other ear canal of the end user 50 to provide stereo / shapeable sound control). The display subsystem 110 is designed to present light patterns to the eyes of the end user 50 that may be comfortably perceived as an augmentation to physical reality, with a high level of image quality and three-dimensional perception, and that may be capable of presenting two-dimensional content. The display subsystem 110 presents a sequence of frames at a high frequency, providing the perception of a single coherent scene.

[0048] In the illustrated embodiment, the display subsystem 110 employs an "optical see-through" display through which a user can view direct light from real objects through a transparent (or semi-transparent) element. The transparent element, often referred to as a "combiner," superimposes the light from the display over the user's field of view in the real world. To this end, the display subsystem 110 includes a partially transparent display. In some embodiments, the transparent display may be electronically controlled. In some embodiments, the transparent display may include segmented dimming, controlling the transparency of one or more portions of the transparent display. In some embodiments, the transparent display may include global dimming, controlling the entire transparency of the transparent display. The display is positioned in the field of view of the end user 50 between the eyes of the end user 50 and the surrounding environment such that direct light from the surrounding environment is transmitted through the display to the eyes of the end user 50.

[0049] In the illustrated embodiment, the image projection assembly provides light to a partially transparent display, whereby light is combined with direct light from the surrounding environment and transmitted from the display to the eyes of the user 50. The projection subsystem may be a fiber optic scanning-based projection device, and the display may be a waveguide-based display into which scanned light from the projection subsystem is injected to generate, for example, an image at a single optical viewing distance closer to infinity (e.g., arm's length), images at multiple discrete optical viewing distances or focal planes, and / or image layers stacked at multiple viewing distances or focal planes to represent a stereoscopic 3D object. These layers in the light field may be stacked close enough together to appear continuous to the human visual system (i.e., one layer is within the noise cone of an adjacent layer). Additionally or alternatively, photographic elements (i.e., sub-images) may be blended across two or more layers to increase the perceived continuity of the transition between layers in the light field even when the layers are more sparsely stacked (i.e., one layer is outside the noise cone of an adjacent layer). The display subsystem 110 may be monocular or binocular. The image projection system may be capable of generating blended modes, but system limitations (e.g., power, thermal, speed, etc.) may limit the image projection system to a discrete mode with a single viewing distance per frame.

[0050] The virtual image generation system 100 may also include one or more sensors (not shown) mounted to the frame structure 102 to detect the position and movement of the head 54 of the end user 50 and / or the position and interocular distance of the eyes of the end user 50. Such sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes). Many of these sensors operate under the assumption that the frame 102 onto which they are affixed is in turn substantially fixed to the user's head, eyes, and ears. The virtual image generation system 100 may also include one or more user-facing cameras to track the user's eyes.

[0051] The virtual image generation system 100 may also include a user orientation detection module. The user orientation module may detect the instantaneous position of the end user's 50 head 54 (e.g., via a sensor coupled to the frame 102) and predict the position of the end user's 50 head 54 based on the position data received from the sensor. Detection of the instantaneous position of the end user's 50 head 54 facilitates the determination of a specific real object the end user 50 is looking at, thereby providing an indication of a specific virtual object to be generated in relation to that real object, and further providing an indication of the location where the virtual object should be displayed. The user orientation module may also track the eyes of the end user 50 based on tracking data received from the sensor.

[0052] The virtual image generation system 100 may also include a control subsystem, which may take any of a wide variety of forms. The control subsystem may include a number of controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphic processing units (GPUs), other integrated circuit controllers, such as application specific integrated circuits (ASICs), display bridge chips, display controllers, programmable gate arrays (PGAs), such as field PGAs (FPGAs), and / or programmable logic controllers (PLUs).

[0053] The control subsystem of the virtual image generation system 100 may include a central processing unit (CPU), a graphics processing unit (GPU), one or more frame buffers, and a 3D database for storing 3D data. The CPU may control the overall operation, while the GPU may render frames (i.e., convert the 3D scene into a 2D image) from the 3D data stored in the 3D database and store these frames in the frame buffer. One or more additional integrated circuits may control the reading of frames into and / or from the frame buffer and the operation of the image projection assembly of the display subsystem 110.

[0054] Various processing components of the virtual image generation system 100 may be physically contained within a distributed subsystem. For example, as illustrated in Figures 2-5, the virtual image generation system 100 may include a local processing and data module 130 operatively coupled to a local display bridge 142, a display subsystem 110, and sensors, such as by wired leads or wireless connectivity 136. The local processing and data module 130 may be mounted in a variety of configurations, such as fixedly attached to the frame structure 102 (Figure 2), fixedly attached to a helmet or hat 56 (Figure 3), removably attached to a torso 58 of the end user 50 (Figure 4), or removably attached to a waist 60 of the end user 50 in a belt-type configuration (Figure 5). The virtual image generation system 100 may also include a remote processing module 132 and a remote data repository 134 operably coupled to the local processing and data module 130 and the local display bridge 142 by wired conductors or wireless connectivity 138, 140, etc., such that these remote modules 132, 134 are operably coupled to each other and available as resources to the local processing and data module 130 and the local display bridge 142.

[0055] The local processing and data module 130 and the local display bridge 142 may each include a power-efficient processor or controller and digital memory, such as flash memory, both of which may be utilized to aid in the processing, caching, and storage of data captured from the sensors and / or acquired and / or processed using the remote processing module 132 and / or the remote data repository 134, possibly for processing or retrieval and subsequent passage to the display subsystem 110. The remote processing module 132 may include one or more relatively powerful processors or controllers configured to analyze and process the data and / or image information. The remote data repository 134 may include a relatively large digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module 130 and the local display bridge 142, allowing for fully autonomous use from any remote module.

[0056] The couplings 136, 138, 140 between the various components mentioned above may include one or more wired interfaces or ports to provide wired or optical communication, or one or more wireless interfaces or ports via RF, microwave, IR, and the like to provide wireless communication. In some implementations, all communication may be wired, while in other implementations, all communication may be wireless. In still further implementations, the wired and wireless communication options may differ from those illustrated in Figures 2-5. Thus, the particular options of wired or wireless communication should not be considered limiting.

[0057] In some embodiments, the user orientation module is contained within the local processing and data module 130 and / or the local display bridge 142, while the CPU and GPU are contained within the remote processing module. In alternative embodiments, the CPU, GPU, or portions thereof may be contained within the local processing and data module 130 and / or the local display bridge 142. The 3D database can be associated with the remote data repository 134 or located locally.

[0058] Some VR, AR, and / or MR systems use multiple volume phase holograms, surface relief holograms, or light directing optical elements that incorporate depth plane information to generate images that appear to originate from distinct depth planes. In other words, a diffraction pattern or diffractive optical element (DOE) may be incorporated into or imprinted / embossed on a light directing optical element (LOE, e.g., a planar waveguide) such that as collimated light (a light beam with a substantially planar wavefront) is substantially totally internally reflected along the LOE, it intersects with the diffraction pattern in multiple locations and exits toward the user's eye. The DOE is configured such that light exiting through the LOE is converging and appears to originate from a particular depth plane. The collimated light may be generated using an optical focusing lens ("concentrator").

[0059] For example, a first LOE may be configured to deliver collimated light to the eye that appears to originate from an optical infinity depth plane (0 diopters). Another LOE may be configured to deliver collimated light that appears to originate from a distance of 2 meters (1 / 2 diopters). Yet another LOE may be configured to deliver collimated light that appears to originate from a distance of 1 meter (1 diopters). It may be understood that by using stacked LOE assemblies, multiple depth planes may be created, with each LOE configured to display an image that appears to originate from a particular depth plane. It is understood that a stack may include any number of LOEs. However, at least N stacked LOEs are required to generate N depth planes. Additionally, N, 2N, or 3N stacked LOEs may be used to generate RGB color images at N depth planes.

[0060] To present 3D virtual content to a user, the VR, AR, and / or MR system projects images of the virtual content into the user's eye in either blended or discrete mode, so that they appear to originate from different depth planes in the Z direction (i.e., orthogonal away from the user's eye). In other words, the virtual content can be considered to vary not only in the X and Y directions (i.e., in a 2D plane orthogonal to the central visual axis of the user's eye), but also in the Z direction, so that the user may perceive objects to be very close or at infinite distance or any distance in between. In other embodiments, the user may perceive multiple objects at different depth planes simultaneously. For example, the user may simultaneously see a virtual bird at a distance of 3 meters away from the user and a virtual coffee cup at an arm's length (about 1 meter) from the user. Alternatively, the user may see a virtual dragon appearing from infinity and running towards the user.

[0061] The multi-plane focusing system, in either blended or discrete mode, creates the perception of variable depth by projecting images onto some or all of multiple depth planes located at separate fixed distances in the Z direction from the user's eyes. Now referring to FIG. 6, it should be understood that the multi-plane focusing system may display frames at fixed depth planes 150 (e.g., the six depth planes 150 shown in FIG. 6). Although an MR system can include any number of depth planes 150, one exemplary multi-plane focusing system has six fixed depth planes 150 in the Z direction. When generating virtual content at one or more of the six depth planes 150, a 3D perception is created such that the user perceives one or more virtual objects at variable distances from the user's eyes. Given that the human eye is more sensitive to objects that are closer in distance than objects that appear to be far away, more depth planes 150 are generated closer to the eyes, as shown in FIG. 6. In other embodiments, the depth planes 150 may be placed equidistant from each other.

[0062] The depth plane positions 150 may be measured in diopters, which are units of refractive power equal to the reciprocal of the focal length measured in meters. For example, in some embodiments, the depth plane DP1 may be 1 / 3 diopters away, the depth plane DP2 may be 0.3 diopters away, the depth plane DP3 may be 0.2 diopters away, the depth plane DP4 may be 0.15 diopters away, the depth plane DP5 may be 0.1 diopters away, and the depth plane DP6 may represent infinity (i.e., 0 diopters away). It should be understood that other embodiments may generate the depth planes 150 at other distances / diopters. Thus, when generating virtual content at the strategically placed depth planes 150, the user may perceive the virtual objects in three dimensions. For example, the user may perceive a first virtual object as being close when displayed at the depth plane DP1 while another virtual object appears at infinity at the depth plane DP6. Alternatively, the virtual object may be first displayed in depth plane DP6, then in depth plane DP5, etc., until the virtual object appears very close to the user. It should be understood that the above example is significantly simplified for illustrative purposes. In another embodiment, all six depth planes may be centered on a particular focal distance away from the user. For example, if the virtual content to be displayed is a coffee cup 0.5 meters away from the user, all six depth planes may be generated at different cross sections of the coffee cup, giving the user a very granular 3D view of the coffee cup.

[0063] In some embodiments, the VR, AR, and / or MR system may act as a multi-plane focal system. In other words, all six LOEs may be illuminated nearly simultaneously such that images appearing to originate from six fixed depth planes are generated in rapid succession, with the light source rapidly transmitting image information to LOE1, then LOE2, then LOE3, etc. For example, a portion of a desired image, including an image of the sky at optical infinity, may be thrown at time 1, and an LOE that retains light collimation (e.g., depth plane DP6 from FIG. 6) may be utilized. An image of a closer tree branch may then be thrown at time 2, and an LOE configured to create an image appearing to originate from a depth plane 10 meters away (e.g., depth plane DP5 from FIG. 6) may be utilized. An image of a pen may then be thrown at time 3, and an LOE configured to create an image appearing to originate from a depth plane 1 meter away may be utilized. This type of paradigm can be repeated in a rapid time-series manner so that the user's eyes and brain (eg, visual cortex) perceive the inputs as all being part of the same image.

[0064] In blended mode, such multi-plane focus systems have system requirements in terms of display and processor speed that may not be achievable in portable VR, AR, and / or MR systems. To overcome these system limitations, blended mode systems may display images adjacent to the user / viewer's focal depth more clearly while displaying other images less clearly, at a lower cost in terms of display and processor speed. In discrete mode, such multi-plane focus systems may project all 3D image content regardless of optical depth to a single depth plane that corresponds to the user / viewer's focal position. However, all discrete mode display systems require identification of the user / viewer's focal depth.

[0065] The VR, AR, and / or MR system may project images (i.e., by diverging or converging light beams) that appear to originate from various locations along the Z axis (i.e., depth planes) and generate images for the 3D experience / scenario. As used herein, light beams include, but are not limited to, directional projections of light energy (including visible and invisible light energy) emitted from a light source. Generating images that appear to originate from various depth planes conforms to the vergence and accommodation of the user's eyes for the images and minimizes or eliminates vergence-accommodation conflicts.

[0066] 7, an exemplary embodiment of an AR or MR system 700 (hereinafter referred to as "system 700") is illustrated. System 700 uses stacked light directing optical elements (hereinafter referred to as "LOEs 790"). System 700 generally includes one or more image generation processors 710, one or more light sources 720, one or more controller / display bridges (DBs) 730, one or more spatial light modulators (SLMs) 740, and one or more sets of stacked LOEs 790 that function as a multi-plane focusing system. System 700 may also include an eye tracking subsystem 750.

[0067] The image generation processor 710 is configured to generate the virtual content to be displayed to the user. The image generation processor 710 may convert images or videos associated with the virtual content into a format that can be projected to the user in 3D. For example, when generating 3D content, the virtual content may need to be formatted so that certain image portions are displayed at certain depth planes, while others are displayed at other depth planes. In one embodiment, the images may all be generated at certain depth planes. In another embodiment, the image generation processor 710 may be programmed to provide slightly different images to the right and left eyes such that when viewed together, the virtual content appears coherent and comfortable to the user's eyes.

[0068] The image generation processor 710 may further include memory 712, a GPU 714, a CPU 716, and other circuitry for image generation and processing. The image generation processor 710 may be programmed with desired virtual content to be presented to a user of the system 700. It should be understood that in some embodiments, the image generation processor 710 may be stored within the system 700. In other embodiments, the image generation processor 710 and other circuitry may be stored within a belt pack that is coupled to the system 700. In some embodiments, the image generation processor 710, or one or more components thereof, may be part of a local processing and data module (e.g., local processing and data module 130). As noted above, the local processing and data module 130 may be mounted in a variety of configurations, such as fixedly attached to the frame structure 102 (FIG. 2), fixedly attached to the helmet or hat 56 (FIG. 3), removably attached to the torso 58 of the end user 50 (FIG. 4), or removably attached to the waist 60 of the end user 50 in a belt-connected configuration (FIG. 5).

[0069] An image generation processor 710 is operatively coupled to a light source 720, which projects light associated with the desired virtual content, and one or more spatial light modulators 740. The light source 720 is compact and has high resolution. The light source 720 is operatively coupled to a controller / DB 730. The light source 720 may include color-specific LEDs and lasers arranged in various geometric configurations. Alternatively, the light source 720 may include similar colored LEDs or lasers, each one linked to a specific area of ​​the display's field of view. In another embodiment, the light source 720 may include a broad area emitter, such as an incandescent or fluorescent lamp, with a mask overlay for segmentation of emission area and location. Although the light source 720 is directly connected to the system 700 in FIG. 2B, the light source 720 may also be connected to the system 700 via an optical fiber (not shown). The system 700 may also include a concentrator (not shown) configured to collimate the light from the light source 720.

[0070] The SLM 740 may be reflective (e.g., LCOS, FLCOS, DLP DMD, or MEMS mirror system), transmissive (e.g., LCD), or emissive (e.g., FSD or OLED) in various exemplary embodiments. The type of SLM 740 (e.g., speed, size, etc.) can be selected to improve the creation of a 3D perception. A DLP DMD operated at a higher refresh rate can be easily incorporated into a stationary system 700, but a wearable system 700 may use a DLP of smaller size and power. The power of the DLP changes the way the 3D depth / focal planes are created. The image generation processor 710 is operably coupled to the SLM 740, which encodes the light from the light source 720 with the desired virtual content. The light from the light source 720 may be encoded with image information as it reflects from, emits from, or passes through the SLM 740.

[0071] Light from the SLM 740 is directed to the LOE 790 such that a light beam encoded by the SLM 740 with image data for one depth plane and / or color is effectively propagated along the single LOE 790 for delivery to the user's eye. Each LOE 790 is configured to project an image or sub-image onto the user's retina that appears to originate from a desired depth plane or FOV angular location. The light sources 720 and LOE 790 can thus selectively project images (synchronously encoded by the SLM 740 under control of the controller / DB 730) that appear to originate from various depth planes or locations in space. By sequentially projecting images using each of the light sources 720 and LOE 790 at a sufficiently high frame rate (e.g., 360 Hz for six depth planes at a virtually full volumetric frame rate of 60 Hz), the system 700 can generate 3D images of virtual objects at various depth planes that appear to exist simultaneously in the 3D image.

[0072] Controller / DB 730 communicates with and is operatively coupled to image generation processor 710, light source 720, and SLM 740, and coordinates the synchronous display of images by instructing SLM 740 to encode the light beam from light source 720 with appropriate image information from image generation processor 710. Although the system includes image generation processor 710, controller / DB 730 may also perform at least some of the image generation processes, in some embodiments including, for example, processes of memory 712, GPU 714, and / or CPU 716. In some embodiments, controller / DB 730 may include one or more of the components shown in image generation processor 710, such as, for example, memory 712, GPU 714, and / or CPU 716.

[0073] The system 700 may also include an eye tracking subsystem 750 configured to track the user's eyes and determine the user's convergence point and the user's per-eye direction. The eye tracking subsystem 750 may also be configured to identify a depth plane corresponding to the user / viewer's focal point to more efficiently generate images and / or display images in a discrete mode (e.g., using the user's convergence point and / or the user's per-eye direction). In one embodiment, the system 700 is configured to illuminate a subset of the LOEs 790 in a discrete mode based on input from the eye tracking subsystem 750 such that images are generated in a desired depth plane that coincides with the user's focal point, as shown in FIGS. 9A and 9B. For example, when the user's focal point is at infinity (e.g., the user's eyes are parallel to one another, and other factors are described below), the system 700 may illuminate the LOEs 790 configured to deliver collimated light to the user's eyes such that images appear to arise from optical infinity. In another example, if the eye tracking subsystem 750 determines that the user's focus is one meter away, then a LOE 790 configured to be approximately focused within that range may be illuminated instead. In embodiments where depth planes approximating the entire depth range are illuminated simultaneously (or nearly simultaneously) in blended mode, the eye tracking subsystem 750 may still be enabled to render a more efficient image by identifying the user's focus. EXEMPLARY USER / VIEWER DEPTH OF FOCUS DETERMINATION SYSTEM AND METHOD

[0074] 12A depicts a discrete mode display in which near and far cubes 810, 812, and cylinder 814 are projected onto near depth plane 0 in response to tracking the focus of a viewer 816 on near cube 810, which is adjacent to near depth plane 0. Projection onto near depth plane 0 causes the viewer to accommodate to near depth plane 0, even with far cube 812 and cylinder 814, which are closer to far depth plane 1 than near depth plane 0. To operate in discrete mode, the system must identify or obtain a depth plane that corresponds to the focus of viewer 816 (the "focal plane").

[0075] 13 and 14 depict a system and FIG. 16 depicts a method for determining a focal plane of a viewer 816. In step 1612 (see FIG. 16), the display system (e.g., its eye tracking subsystem) tracks a gaze path 1350 of the viewer 816 (see FIGS. 13 and 14). The display system may use an eye tracking subsystem (described above) to track the gaze path 1350. In some embodiments, the display system tracks one or more gaze paths. For example, the display system may track a gaze path for only the left eye, only the right eye, or both eyes of the viewer 816. In step 1614, the display system analyzes the 3D image data to identify one or more virtual objects along the gaze path of the viewer 816. Analyzing the 3D image data to identify one or more virtual objects along the gaze path of the viewer 816 may include analyzing depth segmented data corresponding to the 3D data. If there is only one virtual object intersecting the line of sight path of the viewer 816, as determined in step 1616, the display system identifies the depth of the intersecting virtual object at the intersection point as the depth of the focal plane of the viewer 816 in step 1618.

[0076] In particular, Figures 13 and 14 show that the line of sight 1350 of the viewer 816 intersects only the near cube 810. Figure 14 shows that the near cube 810 has a depth of 1410, the far cube 812 has a depth of 1412, and the cylinder 814 has a depth of 1414. Figure 14 also shows that the system is capable of projecting images at one or more depths 1408. Because the system has determined that the near cube 810 is the only virtual object that intersects with the line of sight 1305 of the viewer 816, the system identifies the depth 1410 of the near cube 810 as the depth of the focal plane of the viewer 816. Although not required, the system and method for determining the focal plane of the viewer 816 is more accurate if the virtual objects 810, 812, 814 are already displayed to the viewer 816. In such an embodiment, the line of sight path 1350 of the viewer 816 indicates that the viewer 816 is likely to focus on a particular virtual object (e.g., the near cube 810).

[0077] In embodiments where the viewer's line of sight path intersects more than one virtual object (i.e., "no" at step 1616), the display system may proceed along the optional method at steps 1620-1626. At step 1620, the display system tracks the viewer's second line of sight path. In some embodiments, the first line of sight path is associated with the user's first eye and the second line of sight path is associated with the user's second eye. At step 1622, the display system analyzes the 3D image data to identify one or more virtual objects along the viewer's second line of sight path. If there is only one virtual object that intersects both the viewer's line of sight path and the second line of sight path, as determined at step 1624, the display system at step 1626 identifies the depth of the intersecting virtual object as the depth of the viewer's focal plane.

[0078] In some embodiments, the display system may use an eye tracking subsystem to determine the vergence point of the viewer's eyes. The display system may then use the determined vergence point to confirm its identification of the depth of the viewer's focal plane.

[0079] In a discrete display mode embodiment, the display system may use the viewer's identified focal plane (e.g., its depth) to display a virtual object corresponding to the 3D image data.

[0080] FIG. 12B depicts a blended mode composite light field according to some embodiments. In the blended display mode, the near and far cubes 810, 812, and the cylinder 814 are "virtually projected" onto respective virtual depth planes 1210, 1212, 1214 adjacent to the respective locations of the near and far cubes 810, 812, and the cylinder 814 along the optical axis ("Z-axis") of the system. Only the near and far depth planes 0, 1 are actually illuminated, so only the near and far cubes 810, 812, and the cylinder 814 are virtually projected. As explained above, the virtual projection onto the virtual depth planes 1210, 1212, 1214 is simulated through scalar blending of the sub-images projected onto the near and far depth planes 0, 1. The rendering of the 2D images projected onto each of the virtual depth planes 1210, 1212, 1214 may include a blend of objects therein.

[0081] Although identifying a viewer's focal plane (e.g., its depth) is not necessary to display an object in a blended display mode, identifying the viewer's focal plane allows the blended mode display system to operate within its hardware limitations (processing, display, memory, communication channels, etc.), particularly when the viewer's focal plane allows the blended mode display system to efficiently allocate system resources for display of a virtual object at the viewer's focal plane.

[0082] With respect to a blended display mode system, the method may be simplified in that instead of analyzing 3D data and identifying one or more virtual objects along the line of sight of the viewer 816, the display system may analyze only 2D image frames of the blended mode display to identify one or more virtual objects along the line of sight of the viewer 816. This allows the display system to analyze less data, thereby increasing system efficiency and reducing system resource demands.

[0083] The above embodiments include various different methods for determining a user's depth / viewer's focus. Determining a user's / viewer's focus depth allows the display system to project objects in a discrete display mode at the focus depth, thereby reducing demands on the display system. Such reduced demands include display and processor speed, power, and heat generation. Determining a user's / viewer's focus depth also allows the display system to project objects in a blended display mode while distributing system resources to the image on the user's / viewer's focal plane. Furthermore, the user / viewer focal plane determination steps described above are more efficient than other focal plane determination techniques, improving processor performance in terms of speed, power, and heat generation. System Architecture Overview

[0084] 17 is a block diagram of an exemplary computing system 1700 suitable for implementing embodiments of the present disclosure. The computer system 1700 includes a bus 1706 or other communication mechanism for communicating information that interconnects subsystems and devices such as a processor 1707, a system memory 1708 (e.g., RAM), a static storage device 1709 (e.g., ROM), a disk drive 1710 (e.g., magnetic or optical), a communication interface 1714 (e.g., a modem or Ethernet card), a display 1711 (e.g., CRT or LCD), an input device 1712 (e.g., a keyboard), and cursor control.

[0085] According to one embodiment of the present disclosure, computer system 1700 performs specific operations by processor 1707 executing one or more sequences of one or more instructions contained in system memory 1708. Such instructions may be read into system memory 1708 from another computer-readable / usable medium, such as static storage device 1709 or disk drive 1710. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions to implement the present disclosure. Thus, embodiments of the present disclosure are not limited to any specific combination of hardware circuitry and / or software. In one embodiment, the term "logic" shall mean any combination of software or hardware used to implement all or a portion of the present disclosure.

[0086] The terms "computer-readable medium" or "computer usable medium" as used herein refer to any medium that participates in providing instructions to the processor 1707 for execution. Such media may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as the disk drive 1710. Volatile media include dynamic memory, such as the system memory 1708.

[0087] Common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM (e.g., NAND Flash, NOR Flash), any other memory chip or cartridge, or any other medium from which a computer can read.

[0088] In certain embodiments of the present disclosure, execution of sequences of instructions for practicing the present disclosure is performed by a single computer system 1700. According to other embodiments of the present disclosure, two or more computer systems 1700 coupled by a communications link 1715 (e.g., a LAN, PTSN, or wireless network) may cooperate with each other to perform the sequences of instructions required to practice the present disclosure.

[0089] Computer system 1700 may transmit and receive messages, data, and instructions, including programs, i.e., application code, through communications link 1715 and communications interface 1714. Received program code may be executed by processor 1707 as it is received, and / or stored in disk drive 1710 or other non-volatile storage for later execution. A database 1732 in storage medium 1731 may be used to store data accessible by system 1700 via data interface 1733.

[0090] Although the blend mode embodiments described above include two depth planes (e.g., near and far depth planes), other embodiments may include more than two depth planes. Increasing the number of depth planes increases the fidelity with which the virtual depth planes are simulated. However, this increase in fidelity is offset by increased hardware and processing demands that may exceed the resources available in current portable 3D rendering and display systems.

[0091] Certain aspects, advantages, and features of the present disclosure are described herein. It should be understood that not all such advantages may be achieved according to any particular embodiment of the present disclosure. Thus, the present disclosure may be embodied or carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein, without necessarily achieving other advantages as may be taught or suggested herein.

[0092] The embodiments are described in relation to the accompanying drawings. However, it should be understood that the figures are not drawn to scale. Distances, angles, etc. are merely illustrative and do not necessarily bear precise relationship to the actual dimensions and layout of the devices shown. In addition, the foregoing embodiments are described at a level of detail that enables one skilled in the art to make and use the devices, systems, methods, and equivalents described herein. Various modifications are also possible. Components, elements, and / or steps may be altered, added, removed, or rearranged.

[0093] The devices and methods described herein can advantageously be implemented, at least in part, using, for example, computer software, hardware, firmware, or any combination of software, hardware, and firmware. A software module can include computer executable code stored in a computer's memory to perform the functions described herein. In some embodiments, the computer executable code is executed by one or more general-purpose computers. However, those skilled in the art will understand in light of this disclosure that any module that can be implemented using software to be executed on a general-purpose computer can also be implemented using a different combination of hardware, software, or firmware. For example, such modules can be implemented entirely in hardware using a combination of integrated circuits. Alternatively, or in addition, such modules can be implemented entirely or in part using a specialized computer that is designed to perform the specific functions described herein, rather than by a general-purpose computer. In addition, where a method is described that is or can be performed, at least in part, by computer software, it should be understood that such a method can be provided on a non-transitory computer-readable medium that, when read by a computer or other processing device, causes it to perform the method.

[0094] While certain embodiments are explicitly described, other embodiments will be apparent to those of ordinary skill in the art based on this disclosure.

[0095] The various processors and other electronic components described herein are suitable for use with any optical system for projecting light. The various processors and other electronic components described herein are also suitable for use with any audio system for receiving voice commands.

[0096] Various exemplary embodiments of the present disclosure are described herein. These examples are referred to in a non-limiting sense. They are provided to illustrate the broader applicable aspects of the present disclosure. Various changes may be made to the disclosed embodiments described, and equivalents may be substituted without departing from the true spirit and scope of the present disclosure. In addition, many modifications may be made to adapt a particular situation, material, composition, process, process action, or step to the objective, spirit, or scope of the present disclosure. Moreover, as will be understood by those skilled in the art, each of the individual variations described and illustrated herein has discrete components and features that can be easily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present disclosure. All such modifications are intended to be within the scope of the present disclosure and the associated claims.

[0097] The present disclosure includes methods that may be implemented using the subject devices. The methods may include the act of providing such a suitable device. Such provisioning may be implemented by an end user. In other words, the act of "providing" merely requires the end user to obtain, access, approach, locate, configure, activate, power on, or otherwise act to provide the device required in the subject methods. The methods recited herein may occur in any order of the recited events and recited sequences of events that are logically possible.

[0098] Exemplary aspects of the present disclosure have been described above, together with details regarding material selection and manufacturing. As for other details of the present disclosure, these can be understood in conjunction with the above-referenced patents and publications, and are generally known or can be understood by those skilled in the art. The same can be true with respect to the method-based aspects of the present disclosure in terms of additional actions as generally or logically adopted.

[0099] In addition, although the present disclosure has been described with reference to several embodiments, optionally incorporating various features, the present disclosure is not limited to those described or illustrated as each variation of the disclosure is considered. Various modifications may be made to the present disclosure as described, and equivalents (whether listed herein or not included for purposes of some brevity) may be substituted without departing from the true spirit and scope of the present disclosure. In addition, when a range of values ​​is provided, it is to be understood that all intervening values ​​between the upper and lower limits of the range, and any other stated value or intervening values ​​within the stated range, are encompassed within the present disclosure.

[0100] It is also contemplated that any optional features of the variations described may be described and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that there are multiple identical items. More specifically, as used in this specification and the claims associated therewith, the singular forms "a," "an," "said," and "the" include plural references unless specifically stated otherwise. In other words, the use of articles allows for "at least one" of the items of the present subject matter in the above description and in the claims associated with this disclosure. Furthermore, it is noted that such claims may be drafted to exclude any optional element. Thus, this language is intended to serve as a predicate for the use of exclusive terminology such as "solely," "only," and the like in connection with the recitation of claim elements, or the use of a "negative" limitation.

[0101] Without the use of such exclusive terminology, the term "comprising" in a claim associated with this disclosure shall be deemed to permit the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claim, or the addition of features may be deemed to change the nature of the elements recited in such claim. Except as specifically defined herein, all technical and scientific terms used herein should be given the broadest possible commonly understood meaning while maintaining claim legitimacy.

[0102] The scope of the present disclosure should not be limited to the examples provided and / or this specification, but rather should be limited only by the scope of the claim language associated with this disclosure.

[0103] In the foregoing specification, the present disclosure has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made therein without departing from the broader spirit and scope of the present disclosure. For example, the foregoing process flows are described with reference to a particular order of process actions. However, the order of many of the described process actions may be changed without affecting the scope or operation of the present disclosure. The specification and drawings are therefore to be regarded in an illustrative rather than a restrictive sense.

Claims

[Claim 1] The invention described in this specification.

Images