Virtual, augmented and mixed reality systems and methods

Abstract

A method for determining a focal depth of a user of a three-dimensional ("3D") display device includes tracking a first gaze path of the user. The method also includes analyzing 3D data to identify one or more virtual objects along the first gaze path of the user. The method also includes identifying a depth of only one virtual object as the focal depth of the user when the only virtual object intersects the first gaze path of the user.

Description

[0001] Copyright Notice

[0002] A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. Technical Field

[0003] The present disclosure relates to virtual reality, augmented reality and mixed reality imaging, visualization and display systems and methods. In particular, the present disclosure relates to virtual reality, augmented reality and mixed reality imaging, visualization and display systems and methods for determining the depth of observer focus. Background Art

[0004] Modern computing and display technologies have facilitated the development of virtual reality (VR), augmented reality (AR), and mixed reality (MR) systems. VR systems create simulated environments for users to experience. This can be accomplished by presenting computer-generated images to the user via a head-mounted display. The images create a sensory experience of immersing the user in a simulated environment. VR scenes typically involve presentation of only computer-generated images rather than also including actual real-world images.

[0005] AR systems typically supplement the real-world environment with simulated elements. For example, an AR system can provide a user with a view of the surrounding real-world environment via a head-mounted display. However, computer-generated images can also be presented on the display to enhance the real-world environment. The computer-generated images can include elements that are contextually relevant to the real-world environment. Such elements can include simulated text, images, objects, etc. MR systems also introduce simulated objects into the real-world environment, but these objects typically have a greater degree of interactivity than in AR systems. Simulated elements can often interact in real time.

[0006] Figure 1 An example AR / MR scene 2 is depicted in which a user sees a real-world park setting 6 that features people, trees, buildings in the background, and a concrete platform 20. In addition to these items, a computer-generated image is also presented to the user. The computer-generated image may include, for example, a robotic statue 10 standing on the real-world platform 20, and a flying cartoon-like avatar character 2 that appears to be an avatar of a bumblebee, even though these elements 2, 10 do not actually exist in the real-world environment.

[0007] Various optical systems generate images at different depths for displaying VR, AR, or MR scenes.

[0008] VR / AR / MR systems can operate in various display modes, including hybrid light field mode and discrete light field mode. In hybrid light field mode ("hybrid mode"), the display presents a synthetic light field that includes images displayed simultaneously or nearly simultaneously at two or more discrete depth planes. In discrete light field mode ("discrete mode"), the display presents a synthetic light field that includes images displayed at a single depth plane. The illuminated depth plane is controllable in discrete mode. Some VR / AR / MR systems include an eye tracking subsystem for determining the focus of an observer. Such a system can illuminate the depth plane corresponding to the focus of the observer.

[0009] Because the human visual perception system is complex, it is challenging to produce VR / AR / MR technologies that facilitate comfortable, natural-feeling, rich presentation of virtual image elements among other virtual or real-world image elements. Three-dimensional ("3D") image display systems experience a vergence-accommodation conflict problem. This problem occurs when two optical depth-related biological processes send conflicting depth signals to the user / observer's brain. Convergence is related to the tendency of the observer's eyes to rotate to align the optical axis (axis) with an object of the observer's attention at a distance. In a binocular system, the point where the optical axes intersect can be called the "convergence point." The amount of rotation of the observer's eyes during convergence is interpreted by the observer's brain as an estimated depth. Accommodation is related to the tendency of the lens of the observer's eye to focus so that the object of the observer's attention is at a distance. The focus of the observer's eyes during convergence is interpreted by the observer's brain as another estimated depth. When the vergence and accommodation signals are interpreted by the observer's brain as the same or similar estimated depth, the 3D viewing experience is natural and comfortable for the observer. On the other hand, when the vergence and accommodation signals are interpreted by the observer's brain as substantially different estimated depths, the 3D viewing experience is suboptimal for the observer and may cause discomfort (eye strain, headaches, etc.) and fatigue. Such a problem is called vergence-accommodation conflict.

[0010] Portable VR / AR / MR systems also have other limitations, such as size and portability issues, battery life issues, system overheating issues, processing power, memory, bandwidth, data sources, component delays, and other system and optical challenges that may negatively impact VR / AR / MR system performance. These other limitations must be considered when implementing 3D image rendering techniques for natural convergence and accommodation. In order to accommodate these hardware-related limitations of VR / AR / MR systems, it is often useful to determine the focus of the user / observer. While some systems map the focus of the user / observer to the convergence point of the user / observer, limitations of eye tracking hardware and fast user / observer eye movements reduce the accuracy of determining the focus of the user / observer using this method. Determining the focus of the user / observer allows the VR / AR / MR system to efficiently allocate limited hardware resources. In particular, determining the depth of the user / observer focus allows the VR / AR / MR system to direct limited hardware resources to a specific plane that coincides with the user / observer focus.

[0011] One method of determining the depth of a user / observer's focus is to use depth sensing to generate a depth map of the entire field of view, and then use eye tracking to use the depth map of the field of view to identify the depth of the user / observer's focus. Depth sensing is the determination of the distance between a known point in three-dimensional ("3D") space (e.g., a sensor) and a point of interest ("POI") on the surface of an object. Depth sensing is also called texture sensing because determining the respective distances of multiple POIs on a surface determines the texture of that surface. Depth or texture sensing is useful for many computer vision systems, including mixed reality systems. Generating a depth map can be a demanding process in terms of hardware (e.g., infrared cameras and infrared light projectors) and processing power. For mobile VR / AR / MR systems, the depth map will need to be constantly updated at the expense of processing power, memory, communication channels, etc.

[0012] Improved systems and techniques for processing image data and displaying images are needed, including, for example, systems and techniques for determining a user / viewer's depth of focus for rendering and displaying 3D images to the viewer / user while minimizing convergence-accommodation conflicts, and systems and techniques for minimizing the demand on the limited graphics processing capabilities of portable VR / AR / MR systems while doing so. Improved systems and techniques are needed for addressing these issues. The systems and methods described herein are configured to address these and other challenges.

[0013] What is needed is one or more techniques to improve upon legacy techniques and / or other considered approaches.Some of the approaches described in this background section could be pursued, but are not necessarily approaches previously conceived or pursued. Summary of the invention

[0014] In one embodiment, a method for determining a focal depth of a user of a three-dimensional ("3D") display device includes tracking a first gaze path of a user. The method also includes analyzing 3D data to identify one or more virtual objects along the first gaze path of the user. The method also includes identifying a depth of only one virtual object as the focal depth of the user when only one virtual object intersects the first gaze path of the user.

[0015] In one or more embodiments, each of the one or more virtual objects is displayed at a corresponding depth from the user. Analyzing the plurality of 3D data may include analyzing depth segmentation data corresponding to the 3D data. The method may also include performing a vergence analysis on the user to generate a vergence focus depth for the user, and using the vergence focus depth for the user to improve the accuracy of the focus depth for the user.

[0016] In one or more embodiments, when more than one virtual object intersects the first gaze path of the user, the method may further include tracking a second gaze path of the user. The method may further include analyzing the 3D data to identify one or more virtual objects along the second gaze path of the user. The method may further include identifying the depth of the only one 2D image frame as the focus depth of the user when only one virtual object intersects both the first gaze path and the second gaze path of the user.

[0017] In another embodiment, a method for determining a depth of focus of a user of a three-dimensional ("3D") display device includes tracking a first gaze path of the user. The method also includes analyzing a plurality of two-dimensional ("2D") image frames displayed by the display device to identify one or more virtual objects along the first gaze path of the user. The method also includes, when only one of the plurality of 2D image frames includes a virtual object along the first gaze path of the user, identifying a depth of the only one 2D image frame as the depth of focus of the user.

[0018] In one or more embodiments, each of the plurality of 2D image frames is displayed at a corresponding depth from the user. Analyzing the plurality of 2D image frames may include analyzing depth segmentation data corresponding to the plurality of 2D image frames. The method may also include performing a convergence analysis on the user to generate a convergence focus depth for the user; and using the convergence focus depth for the user to improve the accuracy of the focus depth for the user.

[0019] In one or more embodiments, when more than one 2D image frame of a plurality of 2D image frames includes virtual objects along the first gaze path of the user, the method may further include tracking a second gaze path of the user. The method may further include analyzing a plurality of 2D image frames displayed by the display device to identify one or more virtual objects along the second gaze path of the user. The method may further include identifying the depth of only one 2D image frame as the focus depth of the user when only one 2D image frame of a plurality of 2D image frames includes virtual objects along the first gaze path and the second gaze path of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The patent or application file contains at least one drawing executed in color. Copies of the patent or patent application with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0021] The drawings described below are for illustrative purposes only. The drawings are not intended to limit the scope of the present disclosure. This patent or application file contains at least one drawing executed in color. Copies of this patent or patent application with color drawings will be provided by the U.S. Patent and Trademark Office upon request and payment of the necessary fee.

[0022] The accompanying drawings illustrate the design and practicality of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale and that elements of similar structure or function are represented by the same reference numerals throughout the drawings. In order to better understand how to obtain the records and other advantages and purposes of the various embodiments of the present disclosure, a more detailed description of the present disclosure will be provided with reference to its specific embodiments shown in the accompanying drawings. It is understood that these drawings depict only typical embodiments of the present disclosure and therefore should not be considered as limiting its scope, and the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0023] Figure 1 A user's view of an AR / MR scene using an example AR system is shown.

[0024] Figures 2 to 5 A user using a VR / AR / MR system is schematically depicted according to some embodiments.

[0025] Figure 6 The planes of a multi-plane focusing system are schematically depicted according to some embodiments.

[0026] Figure 7 A VR / AR / MR system according to some embodiments is schematically depicted.

[0027] Fig. 8A and Figure 8B From the side perspective ( Fig. 8A ) and rear perspective ( Figure 8B ) schematically depicts a 3D image including multiple virtual objects according to some embodiments.

[0028] Fig. 9A and Fig. 9B From the side perspective ( Fig. 9A ) and rear perspective ( Fig. 9B ) schematically depicts a 3D image comprising multiple virtual objects projected onto a single depth plane for display in a discrete mode.

[0029] Fig. 10A Schematically depicted are near and far accommodation zones and an accommodation overlap zone of a dual-plane mixed-mode display system in accordance with some embodiments.

[0030] Fig. 10B The relationship between the distance from the observer and the diopter is schematically depicted.

[0031] Fig.11 A 3D image including multiple virtual objects projected onto a single depth plane for display in a discrete mode is schematically depicted from a side perspective.

[0032] Fig. 12A A 3D image including multiple virtual objects projected onto a single depth plane for display in a discrete mode is schematically depicted from a side perspective.

[0033] Fig. 12B A 3D image including multiple virtual objects having multiple virtual depth planes displayed in a hybrid mode is schematically depicted from a side perspective.

[0034] Fig.13 and Fig.14 Schematically depicts a method for viewing a device from a rear perspective ( Fig.13 ) and side perspective ( Fig.14 )Systems and methods for determining the focal plane of a user / observer.

[0035] Fig.15 Schematically depicted is a 3D image including multiple virtual objects blurred and projected onto a single determined depth plane for display in a discrete mode from a rear perspective in accordance with some embodiments.

[0036] Fig.16 is a flow chart illustrating a method for determining a focal plane of a user / observer according to some embodiments.

[0037] Fig.17 is a block diagram that schematically depicts an illustrative computing system in accordance with some embodiments. DETAILED DESCRIPTION

[0038] Various embodiments of the present disclosure are directed to systems, methods, and articles for virtual reality (VR) / augmented reality (AR) / mixed reality (MR) in a single embodiment or in multiple embodiments. Other objects, features, and advantages of the present disclosure are described in the detailed description, drawings, and claims.

[0039] Various embodiments will now be described in detail with reference to the accompanying drawings, which are provided as illustrative examples to enable those skilled in the art to practice the present disclosure. It should be noted that the following figures and embodiments are not intended to limit the scope of the present disclosure. In the case where some elements of the present disclosure can be partially or fully realized using known components (or methods or processes), only such known components (or methods or processes) will be described for those parts necessary for understanding the present disclosure, and the detailed description of other parts of such known components (or methods or processes) will be omitted to avoid confusing the present disclosure. Further, various embodiments encompass the current and future known equivalents of the components mentioned herein by way of illustration.

[0040] Embodiments according to the present disclosure solve the problem that the implementation of VR / AR / MR systems often relies on a combination of off-the-shelf components and custom components. In some cases, off-the-shelf components do not have all the features or performance characteristics required to implement certain desired aspects of the VR / AR / MR system to be deployed. Some embodiments relate to methods for adding capabilities and / or reusing resources to adjust the desired features or performance characteristics of the VR / AR / MR system to be deployed. The figures and discussions herein present example environments, systems, methods, and computer program products for VR / AR / MR systems.

[0041] The head-mounted audio-visual display system and the 3D image rendering system can be implemented independently of the AR / MR system, but some of the following embodiments are described with respect to the AR / MR system for illustrative purposes only. The 3D image rendering and display system described herein can also be used with a VR system in a similar manner.

[0042] Summary of Problems and Solutions

[0043] As described above, VR / AR / MR systems have limitations, such as size and portability, battery life, system overheating, processing power, memory, bandwidth, data sources, component delays, and other system and optical challenges that may negatively impact VR / AR / MR system performance. These limitations may result in reducing graphics processing and image display requirements becoming a counter-challenge to improving 3D image rendering and display. VR / AR / MR systems can more efficiently deploy limited processing and display resources by identifying a depth plane corresponding to the user / observer's focus. However, using only convergence to identify the user / observer's focal plane may not be accurate enough. On the other hand, generating a depth map for use in identifying the user / observer's focal plane may increase processing and hardware requirements for VR / AR / MR systems.

[0044] For example, due to potential graphics processing and size / portability issues, VR / AR / MR systems, particularly head-mounted systems, may only include enough components to be able to render and display color images at one depth plane per frame at a minimum frame rate (e.g., 60 Hz) for smooth display of moving virtual objects (i.e., discrete mode). FIG. 8A to FIG. 9B An example of such a VR / AR / MR system operating in discrete mode is shown in FIG. Fig. 8A Schematically shown, the 3D image includes three virtual objects (near and far cubes 810, 812 and a cylinder 814) adjacent to various depth planes (near cube 810 adjacent to near depth plane 0 and far cube 812 and cylinder 814 adjacent to far depth plane 1). In some embodiments, near depth plane 0 is at about 1.96 diopters, and far depth plane 1 is at about 0.67 diopters. Figure 8B yes Fig. 8A The 3D image is shown from the viewer's perspective. Figure 8B In the example, tracking the user / viewer's eyes 816 indicates that the viewer's eyes tend to converge to a point 818 that coincides with the position of the near cube 810. In discrete mode, only one depth plane is illuminated per frame (i.e., image rendering and display). Fig. 9A As schematically shown, since the convergence point 818 coincides with the position of the near cube 810 adjacent to the near depth plane 0 , all contents of the 3D image (ie, the near and far cubes 810 , 812 and the cylinder 814 ) are projected onto the near depth plane 0 . Fig. 9B is the viewer's perspective view of the 3D image after it has been projected onto near depth plane 0. Only near depth plane 0 is illuminated, and the eyes of viewer 816 are adjusted to near depth plane 0. As described above, equating the user / viewer's focal point with the user / viewer's point of convergence may lead to a less accurate determination of this important data point.

[0045] In some embodiments, projecting all the contents of a 3D image onto a single depth plane triggers only minimal vergence-accommodation conflict (e.g., minimal user discomfort, eye fatigue, headaches). This is because there is a loose coupling between accommodation and vergence, such that the human brain can tolerate a maximum mismatch of about 0.75 diopters between accommodation and vergence. Fig. 10A As shown, this ±0.75 diopter tolerance translates into a near accommodation zone 1010 and a far accommodation zone 1012. Due to the inverse relationship between distance and diopter, as Fig. 10B As shown, the far adjustment area 1012 is larger than the near adjustment area 1010. Fig. 10A 0 and far depth plane 1 in the embodiment depicted in , the ±0.75 diopter tolerance also results in accommodation zone overlap 1014, where object depths falling within accommodation zone overlap 1014 may be displayed at near depth plane 0 and far depth plane 1, e.g., at different scales, with different brightness and / or color values, etc. In embodiments where all content of the 3D image is in the near accommodation zone 1010 or the far accommodation zone 1012 and the eye of the observer 816 is oriented toward that depth plane, projecting all content of the 3D image onto that depth plane triggers only minimal vergence-accommodation conflict.

[0046] Operating in discrete mode allows the VR / AR / MR system to deliver 3D images while operating within its hardware limitations (e.g., processing, display, memory, communication channels, etc.). However, discrete mode display requires accurate identification of a single depth plane to be illuminated while minimizing the demand on system resources. A VR / AR / MR system operating in hybrid mode can also reduce the demand on system resources by focusing those resources on the depth plane corresponding to the user / observer's focal point. Furthermore, this resource conservation approach for hybrid mode display requires accurate identification of the user / observer's focal plane.

[0047] The embodiments described herein include 3D image rendering and display systems and methods for various VR / AR / MR systems. These 3D image rendering and display systems and methods identify a depth plane corresponding to the user / viewer's focus for use in discrete mode and mixed mode displays while reducing consumed system resources, thereby solving many of the problems described above.

[0048] Illustrative VR, AR and / or MR Systems

[0049] The following description relates to illustrative VR, AR, and / or MR systems in which embodiments of various 3D image rendering and display systems may be practiced. However, it should be understood that these embodiments are also applicable to applications in other types of display systems (including other types of VR, AR, and / or MR systems), and therefore, the embodiments are not limited to the illustrative systems disclosed herein.

[0050] The VR / AR / MR systems disclosed herein may include a display that presents computer-generated images (video / image data) to a user. In some embodiments, the display system is wearable, which may advantageously provide a more immersive VR / AR / MR experience. The various components of the VR, AR, and / or MR virtual image system 100 are shown in FIG. Figures 2 to 5 . The virtual image generation system 100 includes a frame structure 102 worn by the end user 50, a display subsystem 110 carried by the frame structure 102 so that the display subsystem 110 is located in front of the eyes of the end user 50, and a speaker 106 carried by the frame structure 102 so that the speaker 106 is positioned adjacent to the ear canal of the end user 50 (optionally, another speaker (not shown) is positioned adjacent to another ear canal of the end user 50 to provide stereo / plastic sound control). The display subsystem 110 is designed to present a light pattern to the eyes of the end user 50 that can be comfortably perceived as an enhancement of physical reality, with a high level of image quality and three-dimensional perception, as well as being able to present two-dimensional content. The display subsystem 110 presents a sequence of frames at a high frequency that provides the perception of a single coherent scene.

[0051] In the illustrated embodiment, the display subsystem 110 employs an "optical see-through" display through which a user can directly view light from real objects via a transparent (or translucent) element. The transparent element, often referred to as a "combiner," superimposes the light from the display on the user's view of the real world. To this end, the display subsystem 110 includes a partially transparent display.

[0052] In some embodiments, the transparent display may be electronically controlled. In some embodiments, the transparent display may include segmented dimming to control the transparency of one or more portions of the transparent display.

[0053] In some embodiments, the transparent display may include global dimming to control the transparency of the entire transparent display. The display is positioned in the field of view of the end user 50 between the eyes of the end user 50 and the surrounding environment so that direct light from the surrounding environment is transmitted through the display to the eyes of the end user 50.

[0054] In the illustrated embodiment, the image projection assembly provides light to a partially transparent display, which is combined with direct light from the surrounding environment and transmitted from the display to the eyes of the user 50. The projection subsystem can be a fiber scanning based projection device, and the display can be a waveguide based display into which the scanned light from the projection subsystem is injected to produce, for example, an image at a single optical viewing distance closer than infinity (e.g., arm's length), images at multiple discrete optical viewing distances or focal planes, and / or image layers stacked at multiple viewing distances or focal planes to represent a volumetric 3D object. The layers in the light field can be stacked closely enough together to appear continuous to the human visual system (i.e., one layer is within the confusion cone of an adjacent layer). Additionally or alternatively, picture elements (i.e., sub-images) can be blended across two or more layers to increase the perceived continuity of transitions between layers in the light field, even if the layers are more sparsely stacked (i.e., one layer is outside the confusion cone of an adjacent layer). The display subsystem 110 can be monocular or binocular. While an image projection system may be able to generate mixed patterns, system limitations (eg, power, heat, speed, etc.) may limit the image projection system to discrete patterns with a single viewing distance per frame.

[0055] The virtual image generation system 100 may also include one or more sensors (not shown) mounted to the frame structure 102 for detecting the position and movement of the end user 50's head 54 and / or the end user 50's eye position and eye distance. Such sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radios, and / or gyroscopes). Many of these sensors operate under the assumption that the frame 102 to which they are fixed is in turn substantially fixed to the user's head, eyes, and ears. The virtual image generation system 100 may also include one or more user-facing cameras to track the user's eyes.

[0056] The virtual image generation system 100 may also include a user orientation detection module. The user orientation module detects the instantaneous position of the head 54 of the end user 50 (e.g., via a sensor coupled to the framework 102) and may predict the position of the head 54 of the end user 50 based on the position data received from the sensor. Detecting the instantaneous position of the head 54 of the end user 50 facilitates determining the specific real object that the end user 50 is looking at, thereby providing an indication of a specific virtual object to be generated in relation to the real object, and further providing an indication of the position of the virtual object to be displayed. The user orientation module may also track the eyes of the end user 50 based on the tracking data received from the sensor.

[0057] The virtual image generation system 100 may also include a control subsystem that may take any of a variety of forms. The control subsystem includes a plurality of controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers such as application specific integrated circuits (ASICs), display bridge chips, display controllers, programmable gate arrays (PGAs), such as field PGAs (FPGAs), and / or programmable logic controllers (PLUs).

[0058] The control subsystem of the virtual image generation system 100 may include a central processing unit (CPU), a graphics processing unit (GPU), one or more frame buffers, and a three-dimensional database for storing three-dimensional scene data. The CPU may control the overall operation, while the GPU may render frames (i.e., convert the three-dimensional scene into a two-dimensional image) from the three-dimensional data stored in the three-dimensional database and store the frames in the frame buffer. One or more additional integrated circuits may control the reading and / or reading out of frames from the frame buffer and the operation of the image projection components of the display subsystem 110.

[0059] The various processing components of the virtual image generation system 100 may be physically contained in distributed subsystems. Figures 2 to 5 As shown, the virtual image generation system 100 may include a local processing and data module 130, which is operably coupled to a local display bridge 142, a display subsystem 110, and a sensor, such as via a wired lead or a wireless connection 136. The local processing and data module 130 can be installed in a variety of configurations, such as fixedly attached to the frame structure 102 ( Figure 2 ), fixedly attached to a helmet or hat 56 ( Figure 3 ), removably attached to the torso 58 of the end user 50 ( Figure 4 ), or removably attached to the hip 60 of the end user 50, in a belt coupling configuration ( Figure 5 ). The virtual image generation system 100 may also include a remote processing module 132 and a remote data repository 134, which are operably coupled to the local processing and data module 130 and the local display bridge 142, such as through wired leads or wireless connections 138, 140, so that these remote modules 132, 134 are operably coupled to each other and available as resources to the local processing and data module 130 and the local display bridge 142.

[0060] The local processing and data module 130 and the local display bridge 142 may each include a power efficient processor or controller, and digital memory, such as flash memory, both of which may be used to assist in processing, high buffering, and storage of data captured from sensors and / or acquired and / or processed using the remote processing module 132 and / or remote data repository 134, possibly for transmission to the display subsystem 110 after such processing or acquisition. The remote processing module 132 may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. The remote data repository 134 may include a relatively large-scale digital data storage facility that may be available via the Internet or other network configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module 130 and the local display bridge 142, which allows for fully autonomous use from any remote module.

[0061] The couplings 136, 138, 140 between the various components described above may include one or more wired interfaces or ports for providing wired or optical communications, or one or more wireless interfaces or ports for providing wireless communications, such as via RF, microwave, and IR. In some embodiments, all communications may be wired, while in other embodiments, all communications may be wireless. In further embodiments, the selection of wired and wireless communications may differ from one embodiment to another. Figures 2 to 5 Therefore, the specific choice of wired or wireless communication should not be considered limiting.

[0062] In some embodiments, the user orientation module is included in the local processing and data module 130 and / or the local display bridge 142, while the CPU and GPU are included in the remote processing module. In alternative embodiments, the CPU, GPU, or portions thereof may be included in the local processing and data module 130 and / or the local display bridge 142. The 3D database may be associated with the remote data repository 134 or may be locally located.

[0063] Some VR, AR, and / or MR systems use multiple volume phase holograms, surface relief holograms, or light-guiding optical elements embedded with depth plane information to generate images that appear to originate from corresponding depth planes. In other words, a diffraction pattern or diffractive optical element (DOE) may be embedded within or imprinted / embossed on a light-guiding optical element (LOE; e.g., a planar waveguide) such that, because collimated light (a light beam having a substantially planar wavefront) is substantially totally internally reflected along the LOE, it intersects the diffraction pattern at multiple locations and emerges toward the user's eyes. The DOE is configured such that the light emerging from the LOE is marginalized so that they appear to originate from a particular depth plane. The collimated light may be generated using an optical focusing lens ("condenser").

[0064] For example, a first LOE can be configured to deliver collimated light to the eye that appears to originate from an optically infinite depth plane (0 diopters). Another LOE can be configured to deliver collimated light that appears to originate from a distance of 2 meters (1 / 2 diopters). Yet another LOE can be configured to deliver collimated light that appears to originate from a distance of 1 meter (1 diopters). By using stacked LOE assemblies, it will be appreciated that multiple depth planes can be created, with each LOE being configured to display an image that appears to originate from a particular depth plane. It will be appreciated that a stack may include any number of LOEs. However, at least N stacked LOEs are required to generate N depth planes. Further, N, 2N, or 3N stacked LOEs can be used to generate RGB color images at N depth planes.

[0065] In order to present 3D virtual content to the user, the VR, AR and / or MR system projects images of the virtual content into the user's eyes so that they appear to originate from various depth planes in the Z direction (i.e., orthogonally away from the user's eyes) in a mixed or discrete mode. In other words, the virtual content can change not only in the X and Y directions (i.e., in a 2D plane orthogonal to the central visual axis of the user's eyes), but also appear to change in the Z direction, so that the user can feel that the object is very close, or at an infinite distance, or at any distance in between. In other embodiments, the user can perceive multiple objects at different depth planes at the same time. For example, the user may simultaneously see a virtual bird at a distance of 3 meters away from the user and a virtual coffee cup at an arm's length (about 1 meter) from the user. Alternatively, the user may see a virtual dragon appear from an infinite distance and run towards the user.

[0066] A multi-plane focusing system creates a perception of variable depth by projecting images in a blended or discrete pattern onto some or all of a plurality of depth planes located at respective fixed distances from the user's eyes in the Z direction. Figure 6 It should be understood that the multi-plane focusing system can be used at a fixed depth plane 150 (e.g., Figure 6 150). Although the MR system may include any number of depth planes 150, an exemplary multi-plane focusing system has six fixed depth planes 150 in the Z direction. When generating virtual content, one or more of the six depth planes 150 create a 3D perception so that the user perceives one or more virtual objects at different distances from the user's eyes. Considering that the human eye is more sensitive to objects that are closer than objects that appear far away, more depth planes 150 are generated closer to the eyes, such as Figure 6 In other embodiments, the depth planes 150 may be placed equidistant from each other.

[0067] The depth plane position 150 may be measured in diopters, which is a unit of refractive power equal to the reciprocal of the focal length measured in meters. For example, in some embodiments, the depth plane DP1 may be outside 1 / 3 diopters, the depth plane DP2 may be outside 0.3 diopters, the depth plane DP3 may be outside 0.2 diopters, the depth plane DP4 may be outside 0.15 diopters, the depth plane DP5 may be outside 0.1 diopters, and the depth plane DP6 may represent infinity (i.e., outside 0 diopters). It should be understood that other embodiments may generate the depth plane 150 at other distances / diopters. Therefore, when virtual content is generated at strategically placed depth planes 150, the user is able to perceive virtual objects in three dimensions. For example, when displayed in the depth plane DP1, the user may perceive the first virtual object as being close to him, while another virtual object appears at infinity at the depth plane DP6. Alternatively, the virtual object may be displayed first at the depth plane DP6, then at the depth plane DP5, and so on, until the virtual object appears very close to the user. It should be understood that the above example is significantly simplified for illustrative purposes. In another embodiment, all six depth planes may be focused at a specific focal distance away from the user. For example, if the virtual content to be displayed is a coffee cup half a meter away from the user, all six depth planes may be generated at different cross-sections of the coffee cup, giving the user a highly granular 3D view of the coffee cup.

[0068] In some embodiments, the VR, AR, and / or MR system can operate as a multi-plane focusing system. In other words, all six LOEs can be illuminated at nearly the same time, so that images that appear to originate from six fixed depth planes are generated in rapid succession, with the light source rapidly delivering image information to LOE 1, then LOE 2, then LOE 3, and so on. For example, a portion of a desired image can be injected at time 1, including an image of the sky at optical infinity, and the LOEs that maintain collimation of the light (e.g., from Figure 6 Then, an image of the closer tree branch can be injected at time 2, and a depth plane 10 (e.g., from 10 meters away) can be used to create a depth plane 10 that appears to originate from 10 meters away. Figure 6 Then, an image of the pen may be injected at time 3, and an LOE configured to create an image that appears to originate from depth plane 1 1 meter away may be used. This type of paradigm may be repeated in a rapid temporal sequential manner so that the user's eyes and brain (e.g., visual cortex) perceive the input as all parts of the same image.

[0069] In hybrid mode, such multi-plane focusing systems have system requirements in terms of display and processor speed that may not be achievable in portable VR, AR, and / or MR systems. To overcome these system limitations, hybrid mode systems can more clearly display images adjacent to the focal depth of the user / observer, while displaying other images less clearly at a lower cost in terms of display and processor speed. In discrete mode, such multi-plane focusing systems can project all 3D image content into a single depth plane corresponding to the user / observer's focal point, regardless of optical depth. However, all discrete mode display systems require identification of the user / observer's focal depth.

[0070] VR, AR, and / or MR systems can project images that appear to originate from various locations along the Z axis (i.e., depth planes) (i.e., by diverging or converging light beams) to generate images for a 3D experience / scene. As used in this application, light beams include, but are not limited to, directional projections of light energy (including visible and invisible light energy) radiated from a light source. Generating images that appear to originate from various depth planes is consistent with the user's eye's convergence and accommodation of the image, and minimizes or eliminates convergence-accommodation conflicts.

[0071] Reference now Figure 7 , an exemplary embodiment of an AR or MR system 700 (hereinafter referred to as "system 700") is shown. System 700 uses a stacked light-guiding optical element (hereinafter referred to as "LOE790"). System 700 generally includes one or more image generation processors 710, one or more light sources 720, one or more controller / display bridges (DB) 730, one or more spatial light modulators (SLM) 740, and one or more sets of stacked LOE 790 used as a multi-plane focusing system. System 700 may also include an eye tracking subsystem 750.

[0072] The image generation processor 710 is configured to generate virtual content to be displayed to the user. The image generation processor 710 can convert images or videos associated with the virtual content into a format that can be projected to the user in 3D. For example, when generating 3D content, it may be necessary to format the virtual content so that parts of a particular image are displayed at a particular depth plane, while other parts are displayed at other depth planes. In one embodiment, all images can be generated at a particular depth plane. In another embodiment, the image generation processor 710 can be programmed to provide slightly different images to the right eye and the left eye so that when viewed together, the virtual content appears coherent and comfortable to the user's eyes.

[0073] The image generation processor 710 may also include a memory 712, a GPU 714, a CPU 716, and other circuits for image generation and processing. The image generation processor 710 may be programmed with desired virtual content to be presented to a user of the system 700. It should be understood that in some embodiments, the image generation processor 710 may be housed in the system 700. In other embodiments, the image generation processor 710 and other circuits may be housed in a belt pack coupled to the system 700. In some embodiments, the image generation processor 710 or one or more components thereof may be part of a local processing and data module (e.g., local processing and data module 130). As mentioned above, the local processing and data module 130 may be installed in a variety of configurations, such as fixedly attached to the frame structure 102 ( Figure 2 ), fixedly attached to a helmet or hat 56 ( Figure 3 ), removably attached to the torso 58 of the end user 50 ( Figure 4 ), or removably attached to the hip 60 of the end user 50, in a belt coupling configuration ( Figure 5 ).

[0074] The image generation processor 710 is operably coupled to a light source 720 and one or more spatial light modulators 740 that project light associated with the desired virtual content. The light source 720 is compact and has a high resolution. The light source 720 is operably coupled to the controller / DB 730. The light source 720 may include LEDs and lasers of specific colors arranged in various geometric configurations. Alternatively, the light source 720 may include LEDs or lasers of similar colors, each connected to a specific area of ​​the field of view of the display. In another embodiment, the light source 720 may include a wide-area emitter, such as an incandescent or fluorescent lamp covered with a mask for segmenting the emission areas and locations. Although the light source 720 is in Figure 2 B is directly connected to the system 700, but the light source 720 can be connected to the system 700 via an optical fiber (not shown). The system 700 can also include a condenser (not shown) configured to collimate the light from the light source 720.

[0075] In various exemplary embodiments, the SLM 740 can be reflective (e.g., LCOS, FLCOS, DLP DMD, or MEMS mirror system), transmissive (e.g., LCD), or emissive (e.g., FSD or OLED). The type of SLM 740 (e.g., speed, size, etc.) can be selected to improve the creation of 3D perception. Although a DLP DMD operating at a higher refresh rate can be easily incorporated into a fixed system 700, a wearable system 700 can use a DLP of smaller size and power. The power of the DLP changes how the 3D depth plane / focal plane is created. The image generation processor 710 is operably coupled to the SLM 740, which encodes the light from the light source 720 with the desired virtual content. The light from the light source 720 can be encoded with image information when it is reflected, emitted, or passed through the SLM 740.

[0076] Light from the SLM 740 is directed to the LOE 790 so that a light beam encoded by the SLM 740 with image data for one depth plane and / or color is efficiently propagated along a single LOE 790 for delivery to the user's eye. Each LOE 790 is configured to project an image or sub-image that appears to originate from a desired depth plane or FOV angular position onto the user's retina. The light source 720 and LOE 790 can thus selectively project images that appear to originate from various depth planes or spatial locations (synchronously encoded by the SLM 740 under the control of the controller / DB 730). By sequentially projecting images using each of the light sources 720 and LOE 790 at a sufficiently high frame rate (e.g., 360 Hz for six depth planes at an effective full volume frame rate of 60 Hz), the system 700 can generate 3D images of virtual objects at various depth planes that appear to exist simultaneously in the 3D image.

[0077] The controller / DB 730 communicates with and is operably coupled to the image generation processor 710, the light source 720, and the SLM 740 to coordinate the synchronous display of images by instructing the SLM 740 to encode the light beams from the light source 720 with the appropriate image information from the image generation processor 710. Although the system includes the image generation processor 710, in some embodiments, the controller / DB 730 may also perform at least some image generation processes, including processes such as the memory 712, the GPU 714, and / or the CPU 716. In some embodiments, the controller / DB 730 may include one or more components shown in the image generation processor 710, such as, for example, the memory 712, the GPU 714, and / or the CPU 716.

[0078] The system 700 may also include an eye tracking subsystem 750 configured to track the user's eyes to determine the user's point of convergence and the direction of each of the user's eyes. The eye tracking subsystem 750 may also be configured to identify a depth plane corresponding to the user's / viewer's focus in order to more efficiently generate images and / or display images in a discrete pattern (e.g., using the user's point of convergence and / or the direction of each of the user's eyes). In one embodiment, the system 700 is configured to illuminate a subset of the LOEs 790 in a discrete pattern based on input from the eye tracking subsystem 750 so that an image is generated at a desired depth plane that coincides with the user's focus, such as Fig. 9A and 9B As shown. For example, if the user's focus is at infinity (e.g., the user's eyes are parallel to each other, as well as other factors described below), the system 700 can illuminate a LOE 790 that is configured to deliver collimated light to the user's eyes so that the image appears to originate from optical infinity. In another example, if the eye tracking subsystem 750 determines that the user's focus is 1 meter away, a LOE 790 that is configured to focus approximately within that range can be illuminated instead. In embodiments where depth planes approximating the entire depth range are illuminated simultaneously (or nearly simultaneously) in a hybrid mode, the eye tracking subsystem 750 can still enable more efficient image rendering by identifying the user's focus.

[0079] Illustrative User / Observer Depth of Focus Determination System and Method

[0080] Fig. 12A A discrete mode display is depicted in which a near cube 810, a far cube 812, and a cylinder 814 are projected onto a near depth plane 0, which is adjacent to the near depth plane 0 in response to tracking the focus of an observer 816 to the near cube 810. Projection to near depth plane 0 causes the observer to accommodate to near depth plane 0, even though far cube 812 and cylinder 814 are closer to far depth plane 1 than to near depth plane 0. In order to operate in discrete mode, the system must identify or obtain the depth plane corresponding to the focus of the observer 816 (the "focal plane").

[0081] Fig.13 and Fig.14 Describes the system, and Fig.16 A method for determining the focal plane of the observer 816 is depicted. In step 1612 (see Fig.16 ), the display system (e.g., its eye tracking subsystem) tracks the gaze path 1350 of the observer 816 (see Fig.13 and Fig.14). The display system may track the gaze path 1350 using the eye tracking subsystem (described above). In some embodiments, the display system tracks one or more gaze paths. For example, the display system may track a gaze path for only the left eye, only the right eye, or both eyes of the observer 816. At step 1614, the display system analyzes the 3D image data to identify one or more virtual objects along the gaze path of the observer 816. Analyzing the 3D image data to identify one or more virtual objects along the gaze path of the observer 816 may include analyzing depth segmentation data corresponding to the 3D data. If there is only one virtual object that intersects the gaze path of the observer 816 as determined at step 1616, then at step 1618, the display system identifies the depth of the intersecting virtual object at the intersection as the depth of the focal plane of the observer 816.

[0082] In particular, Fig.13 and Fig.14 The gaze path 1350 of the observer 816 is shown to intersect only the near cube 810 . Fig.14 Near cube 810 is shown having a depth of 1410 , far cube 812 has a depth of 1412 , and cylinder 814 has a depth of 1414 . Fig.14 It is also shown that the system is able to project images at one or more depths 1408. Because the system has determined that the near cube 810 is the only virtual object that intersects the gaze path 1305 of the observer 816, the system identifies the depth 1410 of the near cube 810 as the depth of the focal plane of the observer 816. Although not necessary, this system and method for determining the focal plane of the observer 816 is more accurate when the virtual objects 810, 812, 814 have been displayed to the observer 816. In such an embodiment, the gaze path 1350 of the observer 816 indicates that the observer 816 is likely focused on a particular virtual object (e.g., near the cube 810).

[0083] In embodiments where the observer's gaze path intersects more than one virtual object (i.e., "No" at step 1616), the display system may proceed along an optional method at steps 1620 to 1626. At step 1620, the display system tracks a second gaze path of the observer. In some embodiments, the first gaze path is associated with a first eye of the user, and the second gaze path is associated with a second eye of the user. At step 1622, the display system analyzes the 3D image data to identify one or more virtual objects along the observer's second gaze path. If there is only one virtual object that intersects the observer's gaze path and the second gaze path as determined at step 1624, then at step 1626, the display system identifies the depth of the intersecting virtual object as the depth of the observer's focal plane.

[0084] In some embodiments, the display system can use the eye tracking subsystem to determine the convergence point of the observer's eyes. The display system can then use the determined convergence point to confirm its identification of the depth of the observer's focal plane.

[0085] In discrete display mode embodiments, the display system may use the identified focal plane of the observer (eg, its depth) to display virtual objects corresponding to the 3D image data.

[0086] Fig. 12B A hybrid mode synthetic light field is depicted in accordance with some embodiments. In hybrid display mode, the near and far cubes 810, 812 and cylinder 814 are "virtually projected" along the optical axis ("Z axis") of the system onto respective virtual depth planes 1210, 1212, 1214 adjacent respective positions of the near and far cubes 810, 812 and cylinder 814. The near and far cubes 810, 812 and cylinder 814 are only virtually projected, as only the near and far depth planes 0, 1 are actually illuminated. As described above, the virtual projection onto the virtual depth planes 1210, 1212, 1214 is simulated by a scaled blend of sub-images projected onto the near and far depth planes 0, 1. The rendering of the 2D image to be projected at each of the virtual depth planes 1210, 1212, 1214 may include blending of objects therein.

[0087] Although displaying objects in a hybrid display mode does not require identification of the observer's focal plane (e.g., its depth), identification of the observer's focal plane does allow the hybrid mode display system to operate within its hardware limitations (processing, display, memory, communication channels, etc.), particularly if the observer's focal plane allows the hybrid mode display system to efficiently allocate system resources for displaying virtual objects in the observer's focal plane.

[0088] For a mixed display mode system, the method may be simplified because instead of analyzing 3D data to identify one or more virtual objects along the gaze path of the observer 816, the display system may only analyze the 2D image frames of the mixed mode display to identify one or more virtual objects along the gaze path of the observer 816. This allows the display system to analyze less data, thereby increasing system efficiency and reducing system resource requirements.

[0089] The above-described embodiments include various different methods for determining the depth of focus of a user / observer. Determining the depth of focus of a user / observer allows a display system to project objects at that focal depth in a discrete display mode, thereby reducing the demands on the display system. Such reduced demands include display and processor speed, power, and heat generation. Determining the depth of focus of a user / observer also allows a display system to project objects in a hybrid display mode while allocating system resources to images on the user / observer focal plane. Further, the user / observer focal plane determination steps described above are more efficient than other focal plane determination techniques and improve system performance in terms of speed, power, and heat generation.

[0090] System Architecture Overview

[0091] Fig.17 17 is a block diagram of an illustrative computing system 1700 suitable for implementing embodiments of the present disclosure. Computer system 1700 includes a bus 1706 or other communication mechanism for communicating information that interconnects subsystems and devices such as a processor 1707, system memory 1708 (e.g., RAM), static storage device 1709 (e.g., ROM), disk drive 1710 (e.g., magnetic or optical), communication interface 1714 (e.g., a modem or Ethernet card), display 1711 (e.g., CRT or LCD), input device 1712 (e.g., keyboard), and cursor control.

[0092] According to one embodiment of the present disclosure, the computer system 1700 performs specific operations by executing one or more sequences of one or more instructions contained in the system memory 1708 by the processor 1707. Such instructions can be read into the system memory 1708 from another computer readable / usable medium (e.g., static storage device 1709 or disk drive 1710). In an alternative embodiment, hard-wired circuits can be used instead of or in combination with software instructions to implement the present disclosure. Therefore, the embodiments of the present disclosure are not limited to any specific combination of hardware circuits and / or software. In one embodiment, the term "logic" should mean any combination of software or hardware used to implement all or part of the present disclosure.

[0093] As used herein, the term "computer-readable medium" or "computer-usable medium" refers to any medium that participates in providing instructions to processor 1707 for execution. Such media may take a variety of forms, including, but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as disk drive 1710. Volatile media include dynamic memory, such as system memory 1708.

[0094] Common forms of computer-readable media include, for example, floppy disks, diskettes, hard disks, magnetic tape, any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM (e.g., NAND flash, NOR flash), any other memory chip or cassette, or any other medium from which a computer can read.

[0095] In an embodiment of the present disclosure, the execution of the instruction sequence for practicing the present disclosure is performed by a single computer system 1700. According to other embodiments of the present disclosure, two or more computer systems 1700 coupled via a communication link 1715 (e.g., a LAN, PTSN, or wireless network) can coordinate with each other to execute the instruction sequence required for practicing the present disclosure.

[0096] Computer system 1700 can send and receive messages, data and instructions, including programs, i.e., application code, via communication link 1715 and communication interface 1714. Received program code can be executed by processor 1707 as it is received, and / or stored in disk drive 1710 or other non-volatile storage device for later execution. Database 1732 in storage medium 1731 can be used to store data accessible by system 1700 via data interface 1733.

[0097] Although the hybrid mode embodiments described above include two depth planes (e.g., near and far depth planes), other embodiments may include more than two depth planes. Increasing the number of depth planes increases the fidelity of the simulated virtual depth planes. However, this increase in fidelity is offset by an increase in hardware and processing requirements, which may exceed the resources available in current portable 3D rendering and display systems.

[0098] Certain aspects, advantages and features of the present disclosure have been described herein. It should be understood that all such advantages need not necessarily be realized according to any particular embodiment of the present disclosure. Therefore, the present disclosure may be implemented or performed in a manner that realizes or optimizes one advantage or group of advantages as taught herein without necessarily realizing other advantages as may be taught or suggested herein.

[0099] Embodiments have been described in conjunction with the accompanying drawings. However, it should be understood that the drawings are not drawn to scale. Distances, angles, etc. are merely illustrative and do not necessarily have an exact relationship to the actual dimensions and layout of the device shown. In addition, the above-described embodiments have been described at a level of detail to allow one of ordinary skill in the art to make and use the devices, systems, methods, etc. described herein. A wide variety of variations are possible. Components, elements, and / or steps may be changed, added, removed, or rearranged.

[0100] The devices and methods described herein can advantageously be implemented at least in part using, for example, computer software, hardware, firmware, or any combination of software, hardware, and firmware. A software module may include a computer executable code stored in a computer memory for performing the functions described herein. In some embodiments, the computer executable code is executed by one or more general-purpose computers. However, it will be appreciated by those skilled in the art that, according to the present disclosure, any module that can be implemented using software to be executed on a general-purpose computer can also be implemented using different combinations of hardware, software, or firmware. For example, such a module can be implemented entirely in hardware using a combination of integrated circuits. Alternatively or additionally, such a module can be implemented entirely or partially using a special-purpose computer designed to perform the specific functions described herein rather than by a general-purpose computer. In addition, in the case of describing a method that is at least partially executed by computer software or can be at least partially executed by computer software, it should be understood that such a method can be provided on a non-transient computer-readable medium that, when read by a computer or other processing device, enables it to execute the method.

[0101] While certain embodiments have been explicitly described, other embodiments will become apparent to one of ordinary skill in the art based on this disclosure.

[0102] The various processors and other electronic components described herein are suitable for use with any optical system for projecting light. The various processors and other electronic components described herein are also suitable for use with any audio system for receiving voice commands.

[0103] Various exemplary embodiments of the present disclosure are described herein. Reference is made to these examples in a non-restrictive sense. They are provided to illustrate the wider applicable aspects of the present disclosure. Various changes can be made to the described present disclosure and equivalents can be substituted without departing from the true spirit and scope of the present disclosure. In addition, many modifications can be made to adapt specific situations, materials, compositions of matter, processes, (one or more) process actions or (one or more) steps to (one or more) purposes, spirit or scope of the present disclosure. Moreover, as will be understood by those skilled in the art, each individual variation described and shown herein has discrete components and features that can be easily separated or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present disclosure. All such modifications are intended to be within the scope of the claims associated with the present disclosure.

[0104] The present disclosure includes methods that can be performed using subject equipment. The method may include the action of providing such suitable equipment. Such provision may be performed by an end user. In other words, the "providing" action only requires the end user to obtain, access, approach, locate, set, activate, power on or other actions to provide the necessary equipment in the method. The methods recorded herein may be performed in any logically possible order of recording events and in the recorded order of events.

[0105] Exemplary aspects of the present disclosure have been described above along with details about material selection and manufacturing. As for other details of the present disclosure, these can be understood in conjunction with the patents and disclosures mentioned above and generally known or understood by those skilled in the art. In terms of additional actions as commonly or logically used, the method-based aspects of the present disclosure may be equally applicable.

[0106] In addition, although the present disclosure has been described with reference to several examples that optionally include various features, the present disclosure will not be limited to the invention described or indicated as expected relative to each variant of the present disclosure. Various changes can be made to the described disclosure and equivalents (whether recorded in this article or not included for some simplicity) can be substituted without departing from the true spirit and scope of the present disclosure. In addition, where a range of values ​​is provided, it should be understood that each intermediate value between the upper and lower limits of the range and any other claimed value or intermediate value in the claimed range are included in the present disclosure.

[0107] Moreover, it is contemplated that any optional features of the described variants may be set forth and claimed independently or in combination with any one or more of the features described herein. References to singular items include the possibility of having multiple identical items. More particularly, as used herein and in the claims associated therewith, unless otherwise specifically stated, the singular forms "one", "an", "said" and "the" include plural indicators. In other words, the use of articles allows "at least one" in the subject items in the above description and claims associated with the present disclosure. It should also be noted that such claims can be written to exclude any optional elements. In this way, the statement is intended to be used as a prior basis for using special terms such as "only", "only" etc. or using "negative" restrictions in conjunction with the recording of claim elements.

[0108] In the absence of such special terms, the term "comprising" in the claims associated with the present disclosure shall allow the inclusion of any additional elements - regardless of whether a given number of elements are recited in such claims, or the addition of features may be considered to transform the nature of the elements recited in such claims. Except as specifically defined herein, all technical and scientific terms used herein shall be given the broadest commonly understood meaning possible while maintaining claim validity.

[0109] The breadth of the present disclosure will not be limited to the examples provided and / or this description, but instead will be defined solely by the scope of the claim language associated with the present disclosure.

[0110] In the foregoing description, the present disclosure has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made thereto within the broader spirit and scope of the present disclosure. For example, the process flow described above is described with reference to a specific order of process actions. However, the order of many of the described process actions may be changed without affecting the scope or operation of the present disclosure. Therefore, the description and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A method for determining a focal depth of a user of a three-dimensional display device, comprising: Tracking a first gaze path of the user, the first gaze path corresponding to a first eye of the user and not corresponding to a second eye of the user; analyzing three-dimensional data to identify one or more virtual objects along the first gaze path of the user based on the first gaze path; as well as When it is determined based on the identified one or more virtual objects and the first gaze path that only one virtual object intersects the first gaze path of the user, analyzing the three-dimensional data to determine the depth of the only one virtual object; and The focus depth of the user is determined as a determined depth of the only one virtual object determined using the three-dimensional data for the only one virtual object and only the first gaze path of the user.

2. The method according to claim 1, wherein: Each of the one or more virtual objects is displayed at a corresponding depth from the user.

3. The method according to claim 1, wherein: Analyzing the plurality of three-dimensional data includes analyzing depth segmentation data corresponding to the three-dimensional data.

4. The method according to claim 1, further comprising: When more than one virtual object intersects the first gaze path of the user: tracking a second gaze path of the user, the second gaze path corresponding to a second eye of the user and not corresponding to the first eye of the user; analyzing the three-dimensional data to identify one or more virtual objects along the second gaze path of the user based on the second gaze path; as well as When it is determined that only one virtual object intersects both the first gaze path and the second gaze path of the user based on the identified one or more virtual objects and the first gaze path and the second gaze path, analyzing the three-dimensional data to determine the depth of the only one virtual object; and The focus depth of the user is determined as the determined depth of the only one virtual object.

5. A method for determining a focal depth of a user of a three-dimensional display device, comprising: Tracking a first gaze path of the user, the first gaze path corresponding to a first eye of the user and not corresponding to a second eye of the user; analyzing a plurality of two-dimensional image frames displayed by the display device to identify one or more virtual objects along the first gaze path of the user based on the first gaze path; as well as When it is determined based on the identified one or more virtual objects and the first gaze path that only one of the plurality of two-dimensional image frames includes a virtual object along the first gaze path of the user, analyzing the plurality of two-dimensional image frames to determine a depth of the only one two-dimensional image frame; as well as The focus depth of the user is determined as a determined depth of the only one two-dimensional image frame determined using the three-dimensional data for the virtual object and only the first gaze path of the user.

6. The method according to claim 5, wherein: Each of the plurality of two-dimensional image frames is displayed at a corresponding depth from the user.

7. The method according to claim 5, wherein: Analyzing the plurality of two-dimensional image frames includes analyzing depth segmentation data corresponding to the plurality of two-dimensional image frames.

8. The method according to claim 5, further comprising: When more than one of the plurality of two-dimensional image frames includes a virtual object along the first gaze path of the user: tracking a second gaze path of the user, the second gaze path corresponding to a second eye of the user and not corresponding to the first eye of the user; analyzing the plurality of two-dimensional image frames displayed by the display device to identify one or more virtual objects along the second gaze path of the user based on the second gaze path; as well as When it is determined that only one of the plurality of two-dimensional image frames includes a virtual object along the first gaze path and the second gaze path of the user based on the identified one or more virtual objects and the first gaze path and the second gaze path, analyzing the three-dimensional data to determine the depth of the only one two-dimensional image frame; and The focus depth of the user is determined as the determined depth of the only one two-dimensional image frame.

Images