Virtual, augmented, and mixed reality system and method

By selectively blurring 3D images based on depth data, VR/AR/MR systems address convergence-divergence motion-accommodation conflicts, enhancing user comfort and reducing resource demands.

JP2024167359AInactive Publication Date: 2024-12-03MAGIC LEAP INC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024150823
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2019-12-06
Filing Date
2024-09-02
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

VR/AR/MR systems face challenges in minimizing convergence-divergence motion-accommodation conflicts, which cause user discomfort such as eye strain and headaches, due to the mismatch between convergence-divergence motion and accommodation signals, and are limited by system resources like processing power and portability.

Method used

Implement methods for rendering and displaying 3D images with selective blurring based on known depth data, either in discrete or blended modes, to align convergence-divergence motion with accommodation cues, reducing system resource demands.

Benefits of technology

The methods provide a more natural 3D viewing experience by minimizing conflicts and reducing system resource consumption, improving user comfort and performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024167359000001_ABST
    Figure 2024167359000001_ABST
Patent Text Reader

Abstract

To provide a virtual, augmented, and mixed reality system and method.SOLUTION: A method for displaying three-dimensional ("3D") images comprises rendering frames of 3D image data. The method also comprises analyzing the frames of 3D image data to generate best known depth data. The method further comprises using the best known depth data to segment the 3D image data into near and far frames of two-dimensional ("2D") image data corresponding to near and far depths, respectively. Additionally, the method comprises displaying to a user near and far 2D image frames corresponding to the near and far frames of the 2D image data at the near and far depths, respectively.SELECTED DRAWING: Figure 16
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to anyone copying the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves any and all copyrights whatsoever.

[0002] FIELD OF THE DISCLOSURE This disclosure relates to virtual reality, augmented reality, and mixed reality imaging, visualization, and display systems and methods. In particular, this disclosure relates to virtual reality, augmented reality, and mixed reality imaging, visualization, and display systems and methods for blurring display content. [Background technology]

[0003] Modern computing and display technologies have facilitated the development of virtual reality (VR), augmented reality (AR), and mixed reality (MR) systems. VR systems create a simulated environment for a user to experience. This can be done by presenting computer-generated images to the user through a head-mounted display. The images create a sensory experience that immerses the user within the simulated environment. VR scenarios typically do not involve the presentation of only computer-generated images, but also include actual real-world images.

[0004] AR systems generally complement the real-world environment with simulated elements. For example, an AR system may provide a user with a view of the surrounding real-world environment via a head-mounted display. However, computer-generated images may also be presented on the display to enhance the real-world environment. The computer-generated images may include elements that are contextually related to the real-world environment. Such elements may include simulated text, images, objects, etc. MR systems also introduce simulated objects into the real-world environment, but these objects typically feature a degree of interactivity that exceeds that in AR systems. The simulated elements can often be interactive in real time.

[0005] 1 depicts an example AR / MR scene 2, in which a user sees a real-world park setting 6 featuring people, trees, buildings in the background, and a concrete platform 20. In addition to these items, computer-generated imagery is also presented to the user. The computer-generated imagery may include, for example, a robotic figure 10 standing on the real-world platform 20 and a flying cartoon-like avatar character 12 that appears to be an anthropomorphic bumblebee, although these elements 12, 10 do not actually exist in the real-world environment.

[0006] Different optical systems generate images at different depths to display VR, AR or MR scenarios.

[0007] VR / AR / MR systems can operate in a variety of display modes, including blended light field and discrete light field modes. In blended light field mode ("blended mode"), the display presents a synthetic light field that includes images displayed simultaneously or nearly simultaneously in two or more discrete depth planes. In discrete light field mode ("discrete mode"), the display presents a synthetic light field that includes images displayed in a single depth plane. The illuminated depth planes are controllable in the discrete modes. Some VR / AR / MR systems include an eye tracking subsystem for determining a viewer's focal position. Such systems can illuminate a depth plane that corresponds to the viewer's focal position.

[0008] The complexity of the human visual perception system makes it difficult to produce VR / AR / MR technologies that facilitate comfortable, natural-like, and rich presentation of virtual image elements among other virtual or real-world image elements. Three-dimensional ("3D") image display systems suffer from the vergence-accommodation collision problem. This problem arises when two optical depth-related biological processes send conflicting depth signals to the viewer / user's brain. Vergence is related to the tendency of the viewer's eyes to rotate to align the optical axis with an object of the viewer's attention at a certain distance. In a binocular system, the point where the optical axes intersect may be called the "vergence point." The amount of rotation of the viewer's eyes during vergence is interpreted by the viewer's brain as an estimated depth. Accommodation is related to the tendency of the viewer's eye lens to focus on an object of the viewer's attention at a certain distance. The focus of the viewer's eyes during convergence and divergence is interpreted by the viewer's brain as different depth estimates. When the convergence and accommodation signals are interpreted by the viewer's brain as the same or similar depth estimates, the 3D viewing experience is natural and comfortable for the viewer. On the other hand, when the convergence and accommodation signals are interpreted by the viewer's brain as substantially different depth estimates, the 3D viewing experience is suboptimal for the viewer and may result in discomfort (eye strain, headaches, etc.) and fatigue. Such a problem is known as convergence-accommodation conflict.

[0009] Blurring is another depth cue associated with accommodation. When a viewer's eyes are focused / accommodated to a particular optical depth, objects located at other optical depths naturally appear out of focus, i.e., blurred. Blended mode displays achieve blurring naturally by presenting images at two or more depth planes. Some discrete mode displays present images without blurring, resulting in some degree of vergence-accommodation conflict and viewer discomfort. Even blended mode displays may benefit from blurring of some 3D image content. For example, 3D image content located at a minimum distance ("minimum clip") where the content is visible (i.e., very close to the viewer's eyes) will be naturally blurred even when the viewer is focused on the content. Similarly, 3D image content located at a maximum distance ("far clip") where the content is visible (i.e., toward an infinite distance away from the viewer's eyes) will be naturally blurred even when the viewer is focused on the content.

[0010] Portable VR / AR / MR systems also have other limitations, such as size and portability issues, battery life issues, system heating issues, processing power, memory, bandwidth, data sources, component latency, and other system and optical challenges that can adversely affect VR / AR / MR system performance. These other limitations must be considered when implementing 3D image rendering techniques for natural convergence and accommodation.

[0011] Improved systems and techniques for processing image data and displaying images are needed, including, for example, systems and techniques for rendering and displaying 3D images to a viewer / user while minimizing vergence-divergence-accommodation conflicts, and for doing so while minimizing demands on the limited graphical processing capabilities of portable VR / AR / MR systems. Improved systems and techniques are needed to address these issues. The systems and methods described herein are configured to address these and other challenges.

[0012] What is needed is a technique or techniques for improving upon older and / or other contemplated approaches. Some of the approaches described in this Background section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Summary of the Invention [Means for solving the problem]

[0013] In one embodiment, a method for displaying a three-dimensional ("3D") image in a discrete mode includes rendering a frame of 3D image data. The method also includes analyzing the frame of 3D image data and generating best known depth data. The method further includes identifying a first set of virtual objects / pixels having individual best known depths within a first depth range. Furthermore, the method includes identifying a second set of virtual objects / pixels having individual best known depths within a second range. In addition, the method includes blurring the second set of virtual objects / pixels based on the individual best known depths to generate a second set of virtual objects / pixels. The method also includes displaying the first set of virtual objects / pixels and the blurred second set of virtual objects / pixels to a user at a first depth corresponding to the first depth range.

[0014] In one or more embodiments, the method also includes blurring the first set of virtual objects / pixels based on the respective best known depths to generate the first set of virtual objects / pixels, and displaying the second set of virtual objects / pixels and the blurred first set of virtual objects / pixels to the user at a second depth corresponding to the second depth range. The method may also include projecting a frame of 3D image data to generate a two-dimensional ("2D") image frame. Blurring the second set of virtual objects / pixels may include blurring portions of the 2D image frame corresponding to the second set of virtual objects / pixels to generate a partially blurred 2D image frame. Displaying the first set of virtual objects / pixels and the blurred second set of virtual objects / pixels to the user may include displaying the partially blurred 2D image frame to the user. The method may also include combining the first set of virtual objects / pixels with the blurred second set of virtual objects / pixels to generate a partially blurred two-dimensional ("2D") image frame. Displaying the first set of virtual objects / pixels and the blurred second set of virtual objects / pixels to the user may include displaying the partially blurred 2D image frame to the user. Blurring the second set of virtual objects / pixels based on the respective best known depths to generate the blurred second set of virtual objects / pixels may include blurring each of the second set of virtual objects / pixels to the same extent.

[0015] In one or more embodiments, blurring each of the second set of virtual objects / pixels based on the respective best known depths to generate the blurred second set of virtual objects / pixels includes variably blurring each of the second set of virtual objects / pixels, where the respective degree of blurring of each of the second set of virtual objects / pixels is proportional to the respective best known depth. The degree of blurring may be directly proportional to the respective best known depth. The second depth range may include a far clip. The far clip may correspond to approximately 1.15 diopters.

[0016] In one or more embodiments, blurring the second set of virtual objects / pixels based on the respective best known depths to generate the blurred second set of virtual objects / pixels includes maximally blurring the object / pixel when the best known depth of the object / pixel is greater than a maximum threshold. Maximally blurring the object / pixel may include removing the object / pixel. The degree of blurring may be inversely proportional to the respective best known depth. The second depth range may include a minimum clip. The minimum clip may correspond to approximately 2.7 diopters.

[0017] In one or more embodiments, blurring the second set of virtual objects / pixels based on the respective best known depths to generate the blurred second set of virtual objects / pixels includes maximally blurring the object / pixel when the best known depth of the object / pixel is less than a minimum threshold. Maximally blurring the object / pixel includes removing the object / pixel. The degree of blurring may be linearly, logarithmically, or asymmetrically proportional to the respective best known depth.

[0018] In another embodiment, a method for displaying a three-dimensional ("3D") image in a blended mode includes rendering a frame of 3D image data. The method also includes analyzing the frame of 3D image data and generating best known depth data. The method further includes identifying a virtual object / pixel having a best known depth outside the sharp display range. Furthermore, the method includes blurring the virtual object / pixel based on the best known depth to generate a blurred virtual object / pixel. In addition, the method includes using the best known depth data to segment the 3D image data into near and far frames of two-dimensional ("2D") image data corresponding to near and far depths, respectively. At least one of the near and far frames of 2D image data includes the blurred virtual object / pixel. The method also includes displaying near and far 2D image frames corresponding to the near and far frames of 2D image data at the near and far depths, respectively, to a user. The present invention provides, for example, the following: (Item 1) 1. A method for displaying a three-dimensional ("3D") image in a discrete mode, comprising: Rendering a frame of 3D image data; analyzing the frames of 3D image data to generate best known depth data; Identifying a first set of virtual objects / pixels having a respective best known depth within a first depth range; identifying a second set of virtual objects / pixels having respective best known depths within a second range; blurring the second set of virtual objects / pixels based on the respective best known depths to generate a blurred second set of virtual objects / pixels; displaying the first set of virtual objects / pixels and the blurred second set of virtual objects / pixels to a user at a first depth corresponding to the first depth range; A method comprising: (Item 2) Blurring the first set of virtual objects / pixels based on the respective best known depths to generate a first set of blurred virtual objects / pixels; displaying the second set of virtual objects / pixels and the blurred first set of virtual objects / pixels to a user at a second depth corresponding to the second depth range; The method of claim 1, further comprising: (Item 3) projecting the frame of 3D image data to generate a two-dimensional ("2D") image frame; Blurring the second set of virtual objects / pixels includes blurring portions of the 2D image frame that correspond to the second set of virtual objects / pixels to generate a partially blurred 2D image frame; displaying the first set of virtual objects / pixels and the blurred second set of virtual objects / pixels to the user includes displaying the partially blurred 2D image frame to the user. The method according to item 1. (Item 4) combining the first set of virtual objects / pixels with the blurred second set of virtual objects / pixels to generate a partially blurred two-dimensional ("2D") image frame; displaying the first set of virtual objects / pixels and the blurred second set of virtual objects / pixels to the user includes displaying the partially blurred 2D image frame to the user. The method according to item 1. (Item 5) 2. The method of claim 1, wherein blurring the second set of virtual objects / pixels based on their respective best known depths to generate blurred second set of virtual objects / pixels includes blurring each of the second set of virtual objects / pixels to the same extent. (Item 6) Blurring the second set of virtual objects / pixels based on the respective best known depths to generate the blurred second set of virtual objects / pixels includes variably blurring each of the second set of virtual objects / pixels; the respective degree of blurring of each of the second set of virtual objects / pixels is proportional to the respective best known depth; The method according to item 1. (Item 7) 7. The method of claim 6, wherein the degree of blur is directly proportional to the respective best known depth. (Item 8) 8. The method of claim 7, wherein the second depth range includes a far clip. (Item 9) 9. The method of claim 8, wherein the far clip corresponds to approximately 1.15 diopters. (Item 10) 8. The method of claim 7, wherein blurring the second set of virtual objects / pixels based on their respective best known depths to generate blurred second set of virtual objects / pixels includes maximally blurring objects / pixels when the best known depths of the objects / pixels are greater than a maximum threshold. (Item 11) Item 11. The method of item 10, wherein maximally blurring the object / pixel comprises deleting the object / pixel. (Item 12) 7. The method of claim 6, wherein the degree of blur is inversely proportional to the respective best known depth. (Item 13) Item 13. The method of item 12, wherein the second depth range includes a minimum clip. (Item 14) Item 14. The method of item 13, wherein the minimum clip corresponds to approximately 2.7 diopters. (Item 15) Item 13. The method of item 12, wherein blurring the second set of virtual objects / pixels based on their respective best known depths to generate blurred second set of virtual objects / pixels includes maximally blurring objects / pixels when the best known depths of the objects / pixels are less than a minimum threshold. (Item 16) Item 16. The method of item 15, wherein maximally blurring the object / pixel comprises deleting the object / pixel. (Item 17) 7. The method of claim 6, wherein the degree of blur is linearly proportional to the respective best known depth. (Item 18) 7. The method of claim 6, wherein the degree of blur is logarithmically proportional to the respective best known depth. (Item 19) 7. The method of claim 6, wherein the degree of blur is asymmetrically proportional to the respective best known depths. (Item 20) 1. A method for displaying a three-dimensional ("3D") image in a blended mode, comprising: Rendering a frame of 3D image data; analyzing the frames of 3D image data to generate best known depth data; Identifying a virtual object / pixel with a best known depth outside the sharp viewing range; Blurring the virtual object / pixel based on the best known depth to generate a blurred virtual object / pixel; using the best known depth data to segment the 3D image data into near and far frames of two-dimensional ("2D") image data corresponding to near and far depths, respectively, where at least one of the near and far frames of 2D image data includes the blurred virtual object / pixel; displaying to a user near and far 2D image frames corresponding to said near and far frames of 2D image data at near and far depths, respectively; A method comprising: [Brief description of the drawings]

[0019] This patent or patent application contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0020] The drawings described below are for illustration purposes only. The drawings are not intended to limit the scope of the present disclosure. This patent or application contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the U.S. Patent and Trademark Office upon request and payment of the necessary fee.

[0021] The drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale, and elements of similar structure or function are represented by similar reference numerals throughout the drawings. In order to better understand how to obtain the enumerated and other advantages and objects of the various embodiments of the present disclosure, a further detailed description of the present disclosure will be given by reference to specific embodiments thereof as illustrated in the accompanying drawings. With the understanding that these drawings depict only exemplary embodiments of the present disclosure and therefore should not be considered limiting of its scope, the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0022] [Figure 1] FIG. 1 illustrates a user's view of an AR / MR scene using an exemplary AR system.

[0023] [Diagram 2] 2-5 diagrammatically depict a user using a VR / AR / MR system, according to some embodiments. [Diagram 3]2-5 diagrammatically depict a user using a VR / AR / MR system, according to some embodiments. [Figure 4] 2-5 diagrammatically depict a user using a VR / AR / MR system, according to some embodiments. [Diagram 5] 2-5 diagrammatically depict a user using a VR / AR / MR system, according to some embodiments.

[0024] [Figure 6] FIG. 6 diagrammatically depicts various planes of a multi-plane focusing system, according to some embodiments.

[0025] [Figure 7] FIG. 7 diagrammatically depicts a VR / AR / MR system, according to some embodiments.

[0026] [Figure 8] 8A and 8B diagrammatically depict a 3D image including multiple virtual objects from a side perspective view (FIG. 8A) and a rear perspective view (FIG. 8B) according to some embodiments.

[0027] [Figure 9] 9A and 9B diagrammatically depict a 3D image including multiple virtual objects projected onto a single depth plane for display in a discrete mode, from a side perspective view (FIG. 9A) and a rear perspective view (FIG. 9B).

[0028] [Figure 10A] FIG. 10A diagrammatically depicts near and far accommodation zones and accommodation overlap zones of a dual surface blend mode display system, according to some embodiments.

[0029] [Figure 10B] FIG. 10B graphically depicts the relationship between distance from the viewer and diopters.

[0030] [Figure 11] FIG. 11 diagrammatically depicts a 3D image including multiple virtual objects projected onto a single depth plane for display in a discrete mode, from a side perspective view.

[0031] [Figure 12A] FIG. 12A diagrammatically depicts a 3D image including multiple virtual objects projected onto a single depth plane for display in a discrete mode, from a side perspective view.

[0032] [Figure 12B] FIG. 12B diagrammatically depicts a 3D image including multiple virtual objects with multiple virtual depth planes displayed in blend mode from a side perspective view.

[0033] [Figure 13] FIG. 13 diagrammatically depicts a 3D image including multiple virtual objects blurred and projected onto a single depth plane for display in a discrete mode from a rear perspective view, according to some embodiments.

[0034] [Figure 14] FIG. 14 diagrammatically depicts a 3D image including multiple virtual objects with multiple virtual depth planes to be displayed in blended mode from a side perspective view according to some embodiments.

[0035] [Figure 15] FIG. 15 diagrammatically depicts a 3D image including multiple virtual objects projected onto a surface for display in blended mode, from a rear perspective view.

[0036] [Figure 16] FIG. 16 diagrammatically depicts a 3D image including multiple virtual objects blurred and projected onto a surface for display in blended mode from a rear perspective view, according to some embodiments.

[0037] [Figure 17] FIG. 17 is a block diagram that diagrammatically depicts an illustrative computing system, in accordance with some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] Detailed Description Various embodiments of the present disclosure are directed to systems, methods, and articles of manufacture for Virtual Reality (VR) / Augmented Reality (AR) / Mixed Reality (MR) in a single embodiment or multiple embodiments. Other objects, features, and advantages of the present disclosure are set forth in the detailed description, drawings, and claims.

[0039] Various embodiments will now be described in detail with reference to the drawings, which are provided as illustrative examples to enable those skilled in the art to practice the present disclosure. It should be noted that the drawings and the following examples are not meant to limit the scope of the present disclosure. Certain elements of the present disclosure may be implemented partially or completely using known components (or methods or processes), and only those parts of such known components (or methods or processes) necessary for understanding the present disclosure will be described, and detailed descriptions of other parts of such known components (or methods or processes) will be omitted so as not to obscure the present disclosure. Furthermore, various embodiments encompass currently known and future known equivalents of the components referenced herein as examples.

[0040] Embodiments according to the present disclosure address the problem of implementing VR / AR / MR systems, which often rely on a combination of off-the-shelf and custom components. In some cases, the off-the-shelf components do not possess all of the features or performance characteristics required to implement certain desired aspects of the VR / AR / MR system to be deployed. Some embodiments are directed to approaches to add capabilities and / or repurpose resources to accommodate desired features or performance characteristics of the VR / AR / MR system to be deployed. The accompanying figures and discussion herein present an example environment for a VR / AR / MR system, the present systems, methods, and computer program products.

[0041] Although the head-mounted audiovisual display system and the 3D image rendering system may be implemented independently of the AR / MR system, some embodiments below are described in relation to an AR / MR system for illustrative purposes only. The 3D image rendering and display systems described herein may also be used in a similar manner with VR systems. (Problem Overview and Solutions)

[0042] As described above, VR / AR / MR systems have 3D image rendering and display limitations that can lead to vergence-accommodation conflicts and associated degradation of the VR / AR / MR experience. In particular, the lack of accommodation-induced blur in discrete modes, i.e., modes in which images are rendered / displayed at one depth plane per frame, can exacerbate vergence-accommodation conflicts and associated viewer experience degradation. Due to accommodation-induced blur in natural vision, real-world content located at a different depth (i.e., diopters) than the depth at which the user is currently focused will be naturally blurred as a function of human perception. VR / AR / MR systems also have limitations such as size and portability, battery life, system heating, processing power, memory, bandwidth, data sources, component latency, and other system and optical challenges that can adversely affect VR / AR / MR system performance. These limitations can result in reduced graphical processing and image display demands, which can be counter-challenges to improve 3D image rendering and display.

[0043] For example, due to potential graphical processing and size / portability issues, a VR / AR / MR system, particularly a head-mounted system, may include only enough components to be able to operate in discrete mode to be able to render and display color images in one depth plane per frame at a minimum frame rate (e.g., 60 Hz) for smooth display of moving virtual objects (i.e., discrete mode). An example of such a VR / AR / MR system operating in discrete mode is shown in FIGS. 8A-9B. As shown diagrammatically in FIG. 8A, the 3D image includes three virtual objects (near and far cubes 810, 812 and one cylinder 814) adjacent various depth planes (near cube 810 adjacent near depth plane 0 and far cube 812 and cylinder 814 adjacent far depth plane 1). In some embodiments, near depth plane 0 is at about 1.96 diopters and far depth plane 1 is at about 0.67 diopters. FIG. 8B is a perspective view of a viewer of the 3D image shown in FIG. 8A. In FIG. 8B, eye tracking of the viewer / user 816 shows that the viewer's eyes converge to a vergence-divergence point 818 that coincides with the location of the near cube 810. In discrete mode, only a single depth plane is illuminated (i.e., the image is rendered and displayed) per frame. As shown diagrammatically in FIG. 9A, all content of the 3D image (i.e., near and far cubes 810, 812 and cylinder 814) is projected onto near depth plane 0 because the vergence-divergence point 818 coincides with the location of the near cube 810, which is adjacent to the vicinity of depth plane 0. FIG. 9B is a perspective view of a viewer of the 3D image after its content has been projected onto near depth plane 0. Only near depth plane 0 is illuminated, and the viewer's 816 eyes accommodate to near depth plane 0.

[0044] In some embodiments, the projection of all of the 3D image content onto a single depth plane induces only minimal vergence-accommodation conflicts (e.g., minimal user discomfort, eye strain, headaches). This is because there is a loose coupling of accommodation and vergence such that the human brain will tolerate a maximum of about 0.75 diopters of mismatch between accommodation and vergence. As shown in FIG. 10A, this ±0.75 diopter tolerance translates into a near accommodation zone 1010 and a far accommodation zone 1012. Due to the inverse relationship between distance and diopters, the far accommodation zone 1012 is larger than the closer accommodation zone 1010, as shown in FIG. 10B. 10A , with near depth plane 0 and far depth plane 1, the ±0.75 diopter tolerance also results in accommodation zone overlap 1014, where object depths that fall within the accommodation zone overlap 1014 may be displayed in one or both of near depth plane 0 and far depth plane 1, e.g., at different scales, with different brightness and / or color values, etc. In embodiments where all of the 3D image content is located within either the near accommodation zone 1010 or the far accommodation zone 1012, and the eyes of the viewer 816 converge to that depth plane, the projection of all of the 3D image content onto that depth plane induces only minimal vergence-divergence-accommodation collisions.

[0045] In some embodiments, the projection of all of the 3D image content onto a single depth plane may induce significant vergence-accommodation conflicts (e.g., user discomfort, eye strain, headaches) enough to affect / limit the VR / AR / MR experience. For example, in the embodiment depicted in FIG. 10A, the far cube 812 and cylinder 814 are located within the far accommodation zone 1012 and outside the near accommodation zone 1010. When the viewer 816's attention is directed to the near cube 810, the viewer 816 may still see the far cube 812 and cylinder 814 in the viewer's 816 field of view, as shown in FIG. 8B. The far cube 812 and cylinder 814 do not behave in an optically identical manner to the real-world content at their respective depths, as shown in FIG. 11. A mismatch between the accommodation estimated depth (i.e., near) and the vergence estimated depth (i.e., far) for the far cube 812 and cylinder 814 can result in vergence-accommodation conflicts and associated degradation of the VR / AR / MR experience (e.g., user discomfort, eye strain, headaches).

[0046] Further, the far cube 812 and cylinder 814 will not have natural accommodation-induced depth blur. To simplify 3D image rendering, the near and far cubes 810, 812 and cylinder 814 are all rendered at full sharpness in depth plane 0. As discussed above, the sharpness / blur of the 3D image content forms another cue that the brain uses to estimate depth. Thus, a mismatch between the sharpness / blur estimated depth (i.e., near) and the vergence-divergence estimated depth (i.e., far) for the far cube 812 and cylinder 814 may exacerbate the vergence-divergence-accommodation conflict and the associated degradation of the VR / AR / MR experience. In the embodiment depicted in Figures 9A and 9B, the more distant objects (i.e., the far cube 812 and cylinder 814) appear at full sharpness when they are projected onto the near depth plane 0. The sharpness of these distant objects on the near depth plane 0 may exacerbate the vergence-accommodation conflict for the viewer. The possibility of selecting an incorrect depth plane due to eye tracking errors and delays in switching depth planes may prolong the vergence-accommodation conflict for the viewer.

[0047] Embodiments described herein include 3D image rendering and display systems and methods for use with various VR / AR / MR systems that render and display 3D images in a discrete mode with blur, minimizing vergence-divergence-accommodation conflicts while consuming fewer system resources, thereby addressing many of the problems described above. (Illustrative VR, AR, and / or MR Systems)

[0048] The following description relates to illustrative VR, AR, and / or MR systems with which various 3D image rendering and display system embodiments may be practiced, however, it should be understood that the embodiments are also suitable for use in other types of display systems (including other types of VR, AR, and / or MR systems), and thus the embodiments are not limited to only the illustrative systems disclosed herein.

[0049] The VR / AR / MR systems disclosed herein can include a display that presents computer-generated images (video / image data) to a user. In some embodiments, the display system is wearable, which may advantageously provide a more immersive VR / AR / MR experience. Various components of a VR, AR, and / or MR virtual image system 100 are depicted in FIGS. 2-5. The virtual image generation system 100 includes a frame structure 102 worn by an end user 50, a display subsystem 110 carried by the frame structure 102 such that the display subsystem 110 is positioned in front of the eye of the end user 50, and a speaker 106 carried by the frame structure 102 such that the speaker 106 is positioned adjacent to the ear canal of the end user 50 (optionally, another speaker (not shown) is positioned adjacent to the other ear canal of the end user 50 to provide stereo / shapeable sound control). The display subsystem 110 is designed to present light patterns to the eyes of the end user 50 that may be comfortably perceived as an augmentation to physical reality, with a high level of image quality and three-dimensional perception, and that may be capable of presenting two-dimensional content. The display subsystem 110 presents a sequence of frames at a high frequency, providing the perception of a single coherent scene.

[0050] In the illustrated embodiment, the display subsystem 110 employs an "optical see-through" display through which a user can view direct light from real objects via a transparent (or semi-transparent) element. The transparent element, often referred to as a "combiner," superimposes the light from the display over the user's field of view in the real world. To this end, the display subsystem 110 includes a partially transparent display. In some embodiments, the transparent display may be electronically controlled. In some embodiments, the transparent display may include segmented dimming, controlling the transparency of one or more portions of the transparent display. In some embodiments, the transparent display may include global dimming, controlling the entire transparency of the transparent display. The display is positioned in the field of view of the end user 50 between the eyes of the end user 50 and the surrounding environment such that direct light from the surrounding environment is transmitted through the display to the eyes of the end user 50.

[0051] In the illustrated embodiment, the image projection assembly provides light to a partially transparent display, whereby light is combined with direct light from the surrounding environment and transmitted from the display to the eyes of the user 50. The projection subsystem may be a fiber optic scanning-based projection device, and the display may be a waveguide-based display into which scanned light from the projection subsystem is injected to generate, for example, an image at a single optical viewing distance closer to infinity (e.g., arm's length), images at multiple discrete optical viewing distances or focal planes, and / or image layers stacked at multiple viewing distances or focal planes to represent a stereoscopic 3D object. These layers in the light field may be stacked close enough together to appear continuous to the human visual system (i.e., one layer is within the noise cone of an adjacent layer). Additionally or alternatively, photographic elements (i.e., sub-images) may be blended across two or more layers to increase the perceived continuity of the transition between layers in the light field even when the layers are more sparsely stacked (i.e., one layer is outside the noise cone of an adjacent layer). The display subsystem 110 may be monocular or binocular. The image projection system may be capable of generating blended modes, but system limitations (e.g., power, thermal, speed, etc.) may limit the image projection system to a discrete mode with a single viewing distance per frame.

[0052] The virtual image generation system 100 may also include one or more sensors (not shown) mounted to the frame structure 102 to detect the position and movement of the head 54 of the end user 50 and / or the position and interocular distance of the eyes of the end user 50. Such sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes). Many of these sensors operate under the assumption that the frame 102 onto which they are affixed is in turn substantially fixed to the user's head, eyes, and ears. The virtual image generation system 100 may also include one or more user-facing cameras to track the user's eyes.

[0053] The virtual image generation system 100 may also include a user orientation detection module. The user orientation module may detect the instantaneous position of the end user's 50 head 54 (e.g., via a sensor coupled to the frame 102) and predict the position of the end user's 50 head 54 based on the position data received from the sensor. Detection of the instantaneous position of the end user's 50 head 54 facilitates the determination of a specific real object the end user 50 is looking at, thereby providing an indication of a specific virtual object to be generated in relation to that real object, and further providing an indication of the location where the virtual object should be displayed. The user orientation module may also track the eyes of the end user 50 based on tracking data received from the sensor.

[0054] The virtual image generation system 100 may also include a control subsystem, which may take any of a wide variety of forms. The control subsystem may include a number of controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphic processing units (GPUs), other integrated circuit controllers, such as application specific integrated circuits (ASICs), display bridge chips, display controllers, programmable gate arrays (PGAs), such as field PGAs (FPGAs), and / or programmable logic controllers (PLUs).

[0055] The control subsystem of the virtual image generation system 100 may include a central processing unit (CPU), a graphics processing unit (GPU), one or more frame buffers, and a 3D database for storing 3D data. The CPU may control the overall operation, while the GPU may render frames (i.e., convert the 3D scene into a 2D image) from the 3D data stored in the 3D database and store these frames in the frame buffer. One or more additional integrated circuits may control the reading of frames into and / or from the frame buffer and the operation of the image projection assembly of the display subsystem 110.

[0056] Various processing components of the virtual image generation system 100 may be physically contained within a distributed subsystem. For example, as illustrated in Figures 2-5, the virtual image generation system 100 may include a local processing and data module 130 operatively coupled to a local display bridge 142, a display subsystem 110, and sensors, such as by wired leads or wireless connectivity 136. The local processing and data module 130 may be mounted in a variety of configurations, such as fixedly attached to the frame structure 102 (Figure 2), fixedly attached to a helmet or hat 56 (Figure 3), removably attached to a torso 58 of the end user 50 (Figure 4), or removably attached to a waist 60 of the end user 50 in a belt-type configuration (Figure 5). The virtual image generation system 100 may also include a remote processing module 132 and a remote data repository 134 operably coupled to the local processing and data module 130 and the local display bridge 142 by wired conductors or wireless connectivity 138, 140, etc., such that these remote modules 132, 134 are operably coupled to each other and available as resources to the local processing and data module 130 and the local display bridge 142.

[0057] The local processing and data module 130 and the local display bridge 142 may each include a power-efficient processor or controller and digital memory, such as flash memory, both of which may be utilized to aid in the processing, caching, and storage of data captured from the sensors and / or acquired and / or processed using the remote processing module 132 and / or the remote data repository 134, possibly for processing or retrieval and subsequent passage to the display subsystem 110. The remote processing module 132 may include one or more relatively powerful processors or controllers configured to analyze and process the data and / or image information. The remote data repository 134 may include a relatively large digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module 130 and the local display bridge 142, allowing for fully autonomous use from any remote module.

[0058] The couplings 136, 138, 140 between the various components mentioned above may include one or more wired interfaces or ports to provide wired or optical communication, or one or more wireless interfaces or ports via RF, microwave, IR, and the like to provide wireless communication. In some implementations, all communication may be wired, while in other implementations, all communication may be wireless. In still further implementations, the wired and wireless communication options may differ from those illustrated in Figures 2-5. Thus, the particular options of wired or wireless communication should not be considered limiting.

[0059] In some embodiments, the user orientation module is contained within the local processing and data module 130 and / or the local display bridge 142, while the CPU and GPU are contained within the remote processing module. In alternative embodiments, the CPU, GPU, or portions thereof may be contained within the local processing and data module 130 and / or the local display bridge 142. The 3D database can be associated with the remote data repository 134 or located locally.

[0060] Some VR, AR, and / or MR systems use multiple volume phase holograms, surface relief holograms, or light directing optical elements that incorporate depth plane information to generate images that appear to originate from distinct depth planes. In other words, a diffraction pattern or diffractive optical element (DOE) may be incorporated into or imprinted / embossed on a light directing optical element (LOE, e.g., a planar waveguide) such that as collimated light (a light beam with a substantially planar wavefront) is substantially totally internally reflected along the LOE, it intersects with the diffraction pattern in multiple locations and exits toward the user's eye. The DOE is configured such that light exiting through the LOE is converging and appears to originate from a particular depth plane. The collimated light may be generated using an optical focusing lens ("concentrator").

[0061] For example, a first LOE may be configured to deliver collimated light to the eye that appears to originate from an optical infinity depth plane (0 diopters). Another LOE may be configured to deliver collimated light that appears to originate from a distance of 2 meters (1 / 2 diopters). Yet another LOE may be configured to deliver collimated light that appears to originate from a distance of 1 meter (1 diopters). It may be understood that by using stacked LOE assemblies, multiple depth planes may be created, with each LOE configured to display an image that appears to originate from a particular depth plane. It is understood that a stack may include any number of LOEs. However, at least N stacked LOEs are required to generate N depth planes. Additionally, N, 2N, or 3N stacked LOEs may be used to generate RGB color images at N depth planes.

[0062] To present 3D virtual content to a user, the VR, AR, and / or MR system projects images of the virtual content into the user's eye in either blended or discrete mode, so that they appear to originate from different depth planes in the Z direction (i.e., orthogonal away from the user's eye). In other words, the virtual content can be considered to vary not only in the X and Y directions (i.e., in a 2D plane orthogonal to the central visual axis of the user's eye), but also in the Z direction, so that the user may perceive objects to be very close or at infinite distance or any distance in between. In other embodiments, the user may perceive multiple objects at different depth planes simultaneously. For example, the user may simultaneously see a virtual bird at a distance of 3 meters away from the user and a virtual coffee cup at an arm's length (about 1 meter) from the user. Alternatively, the user may see a virtual dragon appearing from infinity and running towards the user.

[0063] The multi-plane focusing system, in either blended or discrete mode, creates the perception of variable depth by projecting images onto some or all of multiple depth planes located at separate fixed distances in the Z direction from the user's eyes. Now referring to FIG. 6, it should be understood that the multi-plane focusing system may display frames at fixed depth planes 150 (e.g., the six depth planes 150 shown in FIG. 6). Although an MR system can include any number of depth planes 150, one exemplary multi-plane focusing system has six fixed depth planes 150 in the Z direction. When generating virtual content at one or more of the six depth planes 150, a 3D perception is created such that the user perceives one or more virtual objects at variable distances from the user's eyes. Given that the human eye is more sensitive to objects that are closer in distance than objects that appear to be far away, more depth planes 150 are generated closer to the eyes, as shown in FIG. 6. In other embodiments, the depth planes 150 may be placed equidistant from each other.

[0064] The depth plane positions 150 may be measured in diopters, which are units of refractive power equal to the reciprocal of the focal length measured in meters. For example, in some embodiments, the depth plane DP1 may be 1 / 3 diopters away, the depth plane DP2 may be 0.3 diopters away, the depth plane DP3 may be 0.2 diopters away, the depth plane DP4 may be 0.15 diopters away, the depth plane DP5 may be 0.1 diopters away, and the depth plane DP6 may represent infinity (i.e., 0 diopters away). It should be understood that other embodiments may generate the depth planes 150 at other distances / diopters. Thus, when generating virtual content at the strategically placed depth planes 150, the user may perceive the virtual objects in three dimensions. For example, the user may perceive a first virtual object as being close when displayed at the depth plane DP1 while another virtual object appears at infinity at the depth plane DP6. Alternatively, the virtual object may be first displayed in depth plane DP6, then in depth plane DP5, etc., until the virtual object appears very close to the user. It should be understood that the above example is significantly simplified for illustrative purposes. In another embodiment, all six depth planes may be centered on a particular focal distance away from the user. For example, if the virtual content to be displayed is a coffee cup 0.5 meters away from the user, all six depth planes may be generated at different cross sections of the coffee cup, giving the user a very granular 3D view of the coffee cup.

[0065] In some embodiments, the VR, AR, and / or MR system may act as a multi-plane focal system. In other words, all six LOEs may be illuminated nearly simultaneously such that images appearing to originate from six fixed depth planes are generated in rapid succession, with the light source rapidly transmitting image information to LOE1, then LOE2, then LOE3, etc. For example, a portion of a desired image, including an image of the sky at optical infinity, may be thrown at time 1, and an LOE that retains light collimation (e.g., depth plane DP6 from FIG. 6) may be utilized. An image of a closer tree branch may then be thrown at time 2, and an LOE configured to create an image appearing to originate from a depth plane 10 meters away (e.g., depth plane DP5 from FIG. 6) may be utilized. An image of a pen may then be thrown at time 3, and an LOE configured to create an image appearing to originate from a depth plane 1 meter away may be utilized. This type of paradigm can be repeated in a rapid time-series manner so that the user's eyes and brain (eg, visual cortex) perceive the inputs as all being part of the same image.

[0066] In blended mode, such a multi-plane focal system has system requirements in terms of display and processor speed that may not be achievable in a portable VR, AR, and / or MR system. In discrete mode, such a multi-plane focal system may project all 3D image content, regardless of optical depth, to a single depth plane corresponding to the viewer's focus position to prevent out-of-plane content from flickering due to system speed limitations.

[0067] The VR, AR, and / or MR system may project images (i.e., by diverging or converging light beams) that appear to originate from various locations along the Z axis (i.e., depth planes) and generate images for the 3D experience / scenario. As used herein, light beams include, but are not limited to, directional projections of light energy (including visible and invisible light energy) emitted from a light source. Generating images that appear to originate from various depth planes conforms to the vergence and accommodation of the user's eyes for the images and minimizes or eliminates vergence-accommodation conflicts.

[0068] 7, an exemplary embodiment of an AR or MR system 700 (hereinafter referred to as "system 700") is illustrated. System 700 uses stacked light directing optical elements (hereinafter referred to as "LOEs 790"). System 700 generally includes one or more image generation processors 710, one or more light sources 720, one or more controller / display bridges (DBs) 730, one or more spatial light modulators (SLMs) 740, and one or more sets of stacked LOEs 790 that function as a multi-plane focusing system. System 700 may also include an eye tracking subsystem 750.

[0069] The image generation processor 710 is configured to generate the virtual content to be displayed to the user. The image generation processor 710 may convert images or videos associated with the virtual content into a format that can be projected to the user in 3D. For example, when generating 3D content, the virtual content may need to be formatted so that certain image portions are displayed at certain depth planes, while others are displayed at other depth planes. In one embodiment, the images may all be generated at certain depth planes. In another embodiment, the image generation processor 710 may be programmed to provide slightly different images to the right and left eyes such that when viewed together, the virtual content appears coherent and comfortable to the user's eyes. The image generation processor 710 may also be configured to blur portions of the image or video.

[0070] The image generation processor 710 may further include memory 712, a GPU 714, a CPU 716, and other circuitry for image generation and processing. The image generation processor 710 may be programmed with desired virtual content to be presented to a user of the system 700. It should be understood that in some embodiments, the image generation processor 710 may be stored within the system 700. In other embodiments, the image generation processor 710 and other circuitry may be stored within a belt pack that is coupled to the system 700. In some embodiments, the image generation processor 710, or one or more components thereof, may be part of a local processing and data module (e.g., local processing and data module 130). As noted above, the local processing and data module 130 may be mounted in a variety of configurations, such as fixedly attached to the frame structure 102 (FIG. 2), fixedly attached to the helmet or hat 56 (FIG. 3), removably attached to the torso 58 of the end user 50 (FIG. 4), or removably attached to the waist 60 of the end user 50 in a belt-connected configuration (FIG. 5).

[0071] An image generation processor 710 is operatively coupled to a light source 720, which projects light associated with the desired virtual content, and one or more spatial light modulators 740. The light source 720 is compact and has high resolution. The light source 720 is operatively coupled to a controller / DB 730. The light source 720 may include color-specific LEDs and lasers arranged in various geometric configurations. Alternatively, the light source 720 may include similar colored LEDs or lasers, each one linked to a specific area of ​​the display's field of view. In another embodiment, the light source 720 may include a broad area emitter, such as an incandescent or fluorescent lamp, with a mask overlay for segmentation of emission area and location. Although the light source 720 is directly connected to the system 700 in FIG. 2B, the light source 720 may also be connected to the system 700 via an optical fiber (not shown). The system 700 may also include a concentrator (not shown) configured to collimate the light from the light source 720.

[0072] The SLM 740 may be reflective (e.g., LCOS, FLCOS, DLP DMD, or MEMS mirror system), transmissive (e.g., LCD), or emissive (e.g., FSD or OLED) in various exemplary embodiments. The type of SLM 740 (e.g., speed, size, etc.) can be selected to improve the creation of a 3D perception. A wearable system 700 may use a DLP of smaller size and power, while a LP DMD operated at a higher refresh rate can be easily incorporated into a stationary system 700. The power of the DLP changes the way the 3D depth / focal planes are created. The image generation processor 710 is operably coupled to the SLM 740, which encodes the light from the light source 720 with the desired virtual content. The light from the light source 720 may be encoded with image information as it reflects from, emits from, or passes through the SLM 740.

[0073] Light from the SLM 740 is directed to the LOE 790 such that a light beam encoded by the SLM 740 with image data for one depth plane and / or color is effectively propagated along the single LOE 790 for delivery to the user's eye. Each LOE 790 is configured to project an image or sub-image onto the user's retina that appears to originate from a desired depth plane or FOV angular location. The light sources 720 and LOE 790 can thus selectively project images (synchronously encoded by the SLM 740 under control of the controller / DB 730) that appear to originate from various depth planes or locations in space. By sequentially projecting images using each of the light sources 720 and LOE 790 at a sufficiently high frame rate (e.g., 360 Hz for six depth planes at a virtually full volumetric frame rate of 60 Hz), the system 700 can generate 3D images of virtual objects at various depth planes that appear to exist simultaneously in the 3D image.

[0074] The controller / DB 730 communicates with and is operatively coupled to the image generation processor 710, the light source 720, and the SLM 740, and coordinates the synchronous display of images by instructing the SLM 740 to encode the light beam from the light source 720 with appropriate image information from the image generation processor 710. Although the system includes the image generation processor 710, the controller / DB 730 may also perform at least some of the image generation processes, in some embodiments including, for example, the processes of the memory 712, the GPU 714, and / or the CPU 716. In some embodiments, the controller / DB 730 may include one or more of the components shown in the image generation processor 710, such as, for example, the memory 712, the GPU 714, and / or the CPU 716.

[0075] System 700 may also include an eye tracking subsystem 750 configured to track the user's eyes and determine the user's convergence / divergence point / focus. In one embodiment, system 700 is configured to illuminate a subset of LOEs 790 in a discrete mode based on input from eye tracking subsystem 750 such that an image is generated at a desired depth plane that coincides with the user's convergence / divergence point, as shown in FIGS. 9A and 9B. For example, if the user's eyes are parallel to one another, system 700 may illuminate LOEs 790 configured to deliver collimated light to the user's eyes such that the image appears to originate from optical infinity. In another example, if eye tracking subsystem 750 determines that the user's convergence / divergence point is one meter away, LOEs 790 configured to be approximately focused within that range may instead be illuminated. In embodiments where depth planes approximating the entire depth range are illuminated simultaneously (or nearly simultaneously) in blended mode, the eye tracking subsystem 750 is not required to determine the user's convergence-divergence movement point in order to select one or more depth planes to illuminate. (Illustrative Blend Mode Rendering and Display with Bokeh)

[0076] 12A depicts a discrete mode display in which near and far cubes 810, 812, and cylinder 814 are projected onto near depth plane 0 in response to tracking the convergence-divergence motion point of a viewer 816 to near cube 810, which is adjacent to near depth plane 0. Projection onto near depth plane 0 causes the viewer to accommodate to near depth plane 0, even with far cube 812 and cylinder 814, which are closer to far depth plane 1 than near depth plane 0.

[0077] 9A and 9B, the distance cube 812 and cylinder 814 are rendered with full sharpness even though they are closer to the distance plane 1 than to the near plane 0. As explained above, this sharpness of illuminated out-of-plane objects introduces accommodative cues that can exacerbate vergence-divergence-accommodation conflicts.

[0078] FIG. 13 depicts a discrete mode display with blurring, according to some embodiments. Like the discrete mode display shown in FIG. 9B, the discrete mode display shown in FIG. 13 projects near and far cubes 810, 812, and cylinder 814 (see FIG. 9A) onto near depth plane 0 in response to tracking the vergence-divergence motion point of a viewer 816 to near cube 810, adjacent to near depth plane 0. Unlike the discrete mode display shown in FIG. 9B, the discrete mode display shown in FIG. 13 also blurs far cube 812′ and cylinder 814′, providing a more natural accommodation cue for these illuminated out-of-plane objects.

[0079] The display system can analyze the 3D image data shown in FIGS. 8A and 8B and generate best known depth data for the virtual objects therein. The display system may generate the best known depth data using a disparity map in the 3D image data and / or a data set in a depth buffer. The display system can then analyze the best known depth data and identify one or more objects to be blurred to generate one or more blurred objects. The one or more blurred objects, along with one or more non-blurred objects, are then projected onto a depth plane corresponding to the viewer's vergence-divergence motion point / focal point. The projected image is a partially blurred 2D image that includes illuminated out-of-plane blurred objects and illuminated in-plane sharp objects.

[0080] The display system may also determine the intensity of blur based on the best known depth of the object to be blurred and the depth of the viewer's convergence / divergence point / focus. For example, the intensity of blur of an object may be related to the difference / distance between its best known depth (e.g., the average of the best known depths of all pixels in the object) and the depth of the viewer's convergence / divergence point / focus. The relationship between blur and difference / distance may be linear, logarithmic, asymmetric, etc. Blurring each of the illuminated out-of-plane objects to a different degree based on its distance from the viewer's convergence / divergence point / focus provides even more natural accommodation cues for these objects. In some embodiments, all of the illuminated out-of-plane objects may be blurred at the same time. In these embodiments, the intensity of blur may be based on the average of the best known depths of all pixels in all of the illuminated out-of-plane objects.

[0081] The above embodiments include various different methods for blurring illuminated out-of-plane objects and providing more natural accommodation cues for those objects. The present blurring enables a display system to project objects with more natural accommodation cues in discrete display modes, thereby reducing demands on the display system. Such reduced demands include display and processor speed, power, and heat generation. Furthermore, the blur object identification and blur intensity determination steps described above are more efficient in terms of speed, power, and heat generation than other blurring techniques, improving processor performance. (Illustrative Blend Mode Rendering and Display with Bokeh)

[0082] FIG. 12B depicts a blended mode composite light field according to some embodiments. In the blended display mode, the near and far cubes 810, 812, and the cylinder 814 are "virtually projected" onto respective virtual depth planes 1210, 1212, 1214 adjacent to the respective locations of the near and far cubes 810, 812, and the cylinder 814 along the optical axis ("Z-axis") of the system. Only the near and far depth planes 0, 1 are actually illuminated, so only the near and far cubes 810, 812, and the cylinder 814 are virtually projected. As explained above, the virtual projection onto the virtual depth planes 1210, 1212, 1214 is simulated through scalar blending of the sub-images projected onto the near and far depth planes 0, 1. The rendering of the 2D images projected onto each of the virtual depth planes 1210, 1212, 1214 may include a blend of objects therein.

[0083] FIG. 14 depicts a "blend mode" for rendering and displaying 3D images, according to some embodiments. The blend mode display presents a synthetic light field generated from a combination of at least two discrete depth planes 0,1. By configuring the rendering and display system to position near depth plane 0 and far depth plane 1 within a certain number of diopters of each other (e.g., 1.5 diopters), due to the tolerance between accommodation and vergence-divergence movements (e.g., ±0.75 diopters), these two depth planes 0,1 can cover the entire 3D field of view of the viewer from near the minimum cutoff distance of vision to near infinity, as shown in FIG. 10A. The blend mode synthetic light field simulates the viewer's accommodation for any position between the two depth planes 0,1 as a scalar blend of the sub-images as projected onto the two depth planes 0,1. Through scalar blending of sub-images in the two depth planes 0, 1, the blend mode composite light field can simulate accommodation to one or more virtual depth planes 1410. The blend mode increases the fidelity of the viewer's accommodation, resulting in a more natural depth perception.

[0084] In the embodiment depicted in FIG. 14, the near cube 810 is close to the minimum clipping distance 2 of the display system, i.e., the distance at which objects are no longer visible to a viewer using the display system. In some embodiments, the minimum clip 2 may be about 37 cm or 2.7 dpt. FIG. 15 depicts an image produced by a blended mode display system at near plane 0, according to some embodiments. Even though the near cube 810 is adjacent to the minimum clip 2, the embodiment in FIG. 15 renders the near cube 810 in full sharpness. The sharpness of the near cube 810 contrasts with the blur of real-world objects near the minimum clip 2 of the system, resulting in an erroneous accommodation cue. Thus, this erroneous accommodation cue may exacerbate vergence-divergence accommodation conflicts in a user of the system viewing the image in FIG. 15. The embodiment in Figure 16, on the other hand, blurs the near cube 810', thereby providing a more natural accommodation cue for objects near the minimum clip 2 of the display system. A similar type of blurring can also be applied to objects near the far clip (i.e., infinity). In some embodiments, the far clip may be approximately 87 cm or 1.15 dpt.

[0085] The display system can analyze the 3D image data shown in FIG. 14 and generate therein best known depth data for the virtual object. The display system may generate the best known depth data using a disparity map in the 3D image data and / or a data set in a depth buffer. The display system can then analyze the best known depth data and identify an object or objects adjacent to the minimum or far clip that should be blurred to generate a blurred object. The blurred object is then projected onto a separate depth plane that corresponds to the location of the object in the 3D image displayed in blend mode. The projected image may be a fully blurred 2D image or a partially blurred 2D image that includes the near clipped blurred object and other objects.

[0086] The display system may also determine the intensity of blur based on the best known depth of the object to be blurred and the depth of the minimum or far clip. For example, the intensity of blur of an object may be related to the difference / distance between its best known depth (e.g., the average of the best known depths of all pixels in the object) and the minimum or far clip. The relationship between blur and difference / distance may be linear, logarithmic, asymmetric, etc. Blurring each near clip object to a different degree based on its distance from the minimum or far clip provides an even more natural accommodation cue for these objects. Alternatively, all near clip objects may be blurred to the same degree. Near clip objects may be blurred / faded out until they are part of a clip or a part of a far clip at which point the object may be completely removed. For example, an object located in the far clip may be completely removed unless the viewer is congested by the object. In other embodiments, virtual objects may appear at a short distance beyond the minimum and / or far clip.

[0087] Although techniques for blurring near-clipped objects have been described with respect to blended mode displays, such techniques are equally applicable to discrete mode displays, since discrete mode displays may also project 3D images that include near-clipped objects.

[0088] The above embodiments include various different methods for blurring near-clipped objects and providing more natural accommodation cues for those objects. The present blurring enables a display system to project objects with more natural accommodation cues in blended or discrete display modes, thereby reducing demands on the display system. Such reduced demands include display and processor speed, power, and heat generation. Furthermore, the blur object identification and blur intensity determination steps described above are more efficient in terms of speed, power, and heat generation than other blurring techniques, improving processor performance. (System Architecture Overview)

[0089] 17 is a block diagram of an exemplary computing system 1700 suitable for implementing embodiments of the present disclosure. The computer system 1700 includes a bus 1706 or other communication mechanism for communicating information that interconnects subsystems and devices such as a processor 1707, a system memory 1708 (e.g., RAM), a static storage device 1709 (e.g., ROM), a disk drive 1710 (e.g., magnetic or optical), a communication interface 1714 (e.g., a modem or Ethernet card), a display 1711 (e.g., CRT or LCD), an input device 1712 (e.g., a keyboard), and cursor control.

[0090] According to one embodiment of the present disclosure, computer system 1700 performs specific operations by processor 1707 executing one or more sequences of one or more instructions contained in system memory 1708. Such instructions may be read into system memory 1708 from another computer-readable / usable medium, such as static storage device 1709 or disk drive 1710. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions to implement the present disclosure. Thus, embodiments of the present disclosure are not limited to any specific combination of hardware circuitry and / or software. In one embodiment, the term "logic" shall mean any combination of software or hardware used to implement all or a portion of the present disclosure.

[0091] The terms "computer-readable medium" or "computer usable medium" as used herein refer to any medium that participates in providing instructions to the processor 1707 for execution. Such media may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as the disk drive 1710. Volatile media include dynamic memory, such as the system memory 1708.

[0092] Common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM (e.g., NAND Flash, NOR Flash), any other memory chip or cartridge, or any other medium from which a computer can read.

[0093] In certain embodiments of the present disclosure, execution of sequences of instructions for practicing the present disclosure is performed by a single computer system 1700. According to other embodiments of the present disclosure, two or more computer systems 1700 coupled by a communications link 1715 (e.g., a LAN, PTSN, or wireless network) may cooperate with each other to perform the sequences of instructions required to practice the present disclosure.

[0094] Computer system 1700 may transmit and receive messages, data, and instructions, including programs, i.e., application code, through communications link 1715 and communications interface 1714. Received program code may be executed by processor 1707 as it is received, and / or stored in disk drive 1710 or other non-volatile storage for later execution. A database 1732 in storage medium 1731 may be used to store data accessible by system 1700 via data interface 1733.

[0095] Although the blend mode embodiments described above include two depth planes (e.g., near and far depth planes), other embodiments may include more than two depth planes. Increasing the number of depth planes increases the fidelity with which the virtual depth planes are simulated. However, this increase in fidelity is offset by increased hardware and processing demands that may exceed the resources available in current portable 3D rendering and display systems.

[0096] Certain aspects, advantages, and features of the present disclosure are described herein. It should be understood that not all such advantages may be achieved according to any particular embodiment of the present disclosure. Thus, the present disclosure may be embodied or carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein, without necessarily achieving other advantages as may be taught or suggested herein.

[0097] The embodiments are described in relation to the accompanying drawings. However, it should be understood that the figures are not drawn to scale. Distances, angles, etc. are merely illustrative and do not necessarily bear precise relationship to the actual dimensions and layout of the devices shown. In addition, the foregoing embodiments are described at a level of detail that enables one skilled in the art to make and use the devices, systems, methods, and equivalents described herein. Various modifications are also possible. Components, elements, and / or steps may be altered, added, removed, or rearranged.

[0098] The devices and methods described herein can advantageously be implemented, at least in part, using, for example, computer software, hardware, firmware, or any combination of software, hardware, and firmware. A software module can include computer executable code stored in a computer's memory to perform the functions described herein. In some embodiments, the computer executable code is executed by one or more general-purpose computers. However, those skilled in the art will understand in light of this disclosure that any module that can be implemented using software to be executed on a general-purpose computer can also be implemented using a different combination of hardware, software, or firmware. For example, such modules can be implemented entirely in hardware using a combination of integrated circuits. Alternatively, or in addition, such modules can be implemented entirely or in part using a specialized computer that is designed to perform the specific functions described herein, rather than by a general-purpose computer. In addition, where a method is described that is or can be performed, at least in part, by computer software, it should be understood that such a method can be provided on a non-transitory computer-readable medium that, when read by a computer or other processing device, causes it to perform the method.

[0099] While certain embodiments are explicitly described, other embodiments will be apparent to those of ordinary skill in the art based on this disclosure.

[0100] The various processors and other electronic components described herein are suitable for use with any optical system for projecting light. The various processors and other electronic components described herein are also suitable for use with any audio system for receiving voice commands.

[0101] Various exemplary embodiments of the present disclosure are described herein. These examples are referred to in a non-limiting sense. They are provided to illustrate the broader applicable aspects of the present disclosure. Various changes may be made to the disclosed embodiments described, and equivalents may be substituted without departing from the true spirit and scope of the present disclosure. In addition, many modifications may be made to adapt a particular situation, material, composition, process, process action, or step to the objective, spirit, or scope of the present disclosure. Moreover, as will be understood by those skilled in the art, each of the individual variations described and illustrated herein has discrete components and features that can be easily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present disclosure. All such modifications are intended to be within the scope of the present disclosure and the associated claims.

[0102] The present disclosure includes methods that may be implemented using the subject devices. The methods may include the act of providing such a suitable device. Such provisioning may be implemented by an end user. In other words, the act of "providing" merely requires the end user to obtain, access, approach, locate, configure, activate, power on, or otherwise act to provide the device required in the subject methods. The methods recited herein may occur in any order of the recited events and recited sequences of events that are logically possible.

[0103] Exemplary aspects of the present disclosure have been described above, together with details regarding material selection and manufacturing. As for other details of the present disclosure, these can be understood in conjunction with the above-referenced patents and publications, and are generally known or can be understood by those skilled in the art. The same can be true with respect to the method-based aspects of the present disclosure in terms of additional actions as generally or logically adopted.

[0104] In addition, although the present disclosure has been described with reference to several embodiments, optionally incorporating various features, the present disclosure is not limited to those described or illustrated as each variation of the disclosure is considered. Various modifications may be made to the present disclosure as described, and equivalents (whether listed herein or not included for purposes of some brevity) may be substituted without departing from the true spirit and scope of the present disclosure. In addition, when a range of values ​​is provided, it is to be understood that all intervening values ​​between the upper and lower limits of the range, and any other stated value or intervening values ​​within the stated range, are encompassed within the present disclosure.

[0105] It is also contemplated that any optional features of the variations described may be described and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that there are multiple identical items. More specifically, as used in this specification and the claims associated therewith, the singular forms "a," "an," "said," and "the" include plural references unless specifically stated otherwise. In other words, the use of articles allows for "at least one" of the items of the present subject matter in the above description and in the claims associated with this disclosure. Furthermore, it is noted that such claims may be drafted to exclude any optional element. Thus, this language is intended to serve as a predicate for the use of exclusive terminology such as "solely," "only," and the like in connection with the recitation of claim elements, or the use of a "negative" limitation.

[0106] Without the use of such exclusive terminology, the term "comprising" in a claim associated with this disclosure shall be deemed to permit the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claim, or the addition of features may be deemed to change the nature of the elements recited in such claim. Except as specifically defined herein, all technical and scientific terms used herein should be given the broadest possible commonly understood meaning while maintaining claim legitimacy.

[0107] The scope of the present disclosure should not be limited to the examples provided and / or this specification, but rather should be limited only by the scope of the claim language associated with this disclosure.

[0108] In the foregoing specification, the present disclosure has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made therein without departing from the broader spirit and scope of the present disclosure. For example, the foregoing process flows are described with reference to a particular order of process actions. However, the order of many of the described process actions may be changed without affecting the scope or operation of the present disclosure. The specification and drawings are therefore to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method for displaying a three-dimensional ("3D") image to a user, the method comprising: Rendering a frame of 3D image data; generating depth data by analyzing the frames of 3D image data; identifying a set of virtual objects / pixels having individual depths within a depth range that includes one of a near clip and a far clip; Blurring the set of virtual objects / pixels when a user's eyes are converged / diverged relative to the set of virtual objects / pixels, the blurring of the virtual objects / pixels being based on the determined difference / distance between a depth of the virtual object / pixel and a depth of the one or both of the near clip and the far clip, respectively, thereby generating a blurred virtual object / pixel; Displaying the set of blurred virtual objects / pixels to the user; A method comprising:

2. The method further includes generating a two-dimensional ("2D") image frame by projecting the frame of the 3D image data; Blurring the set of virtual objects / pixels comprises blurring portions of the 2D image frame that correspond to the set of virtual objects / pixels to generate a partially blurred 2D image frame; The method of claim 1 , wherein displaying the set of blurred virtual objects / pixels to the user comprises displaying the partially blurred 2D image frame to the user.

3. The method further includes generating a partially blurred two-dimensional ("2D") image frame by combining a second set of virtual objects / pixels with the set of blurred virtual objects / pixels; The method of claim 1 , wherein displaying the set of blurred virtual objects / pixels to the user comprises displaying the partially blurred 2D image frame to the user.

4. The method of claim 1, further comprising generating the set of blurred virtual objects / pixels by blurring the set of virtual objects / pixels based on their individual depths.

5. The method of claim 4, further comprising blurring each of the sets of virtual objects / pixels to the same extent.

6. The method further comprising variably blurring each of the sets of virtual objects / pixels; The method of claim 4 , wherein the respective degree of blur for each of the sets of virtual objects / pixels is proportional to the respective depth.

7. The method of claim 4, wherein the degree of blurring is directly proportional to the individual depths.

8. The method of claim 4, wherein the degree of blurring is one of: linearly proportional to the individual depths; logarithmically proportional to the individual depths; or asymmetrically proportional to the individual depths.

9. The method of claim 1, wherein the depth range includes the far clip.

10. The method of claim 9, wherein the far clip corresponds to approximately 1.15 diopters.

11. The method of claim 1, wherein the depth range includes the near clip.

12. The method of claim 11, wherein the near clip corresponds to approximately 2.7 diopters.

13. The method of claim 1, further comprising tracking convergence and divergence movement points of the user to the set of virtual objects / pixels.

14. A method for displaying a three-dimensional ("3D") image, the method comprising: generating depth data by analyzing the frames of 3D image data; identifying a virtual object / pixel having depth within the 3D image data; determining a difference / distance between a depth of the virtual object / pixel and a depth of one or both of a near clip and a far clip, respectively; generating a blurred virtual object / pixel by blurring the virtual object / pixel based on the determined difference / distance between a depth of the virtual object / pixel and a depth of the one or both of the near and far clips, respectively; Displaying the blurred virtual object / pixel to a user. A method comprising:

15. The method according to claim 1, using the depth data to segment the 3D image data into near and far frames of two-dimensional ("2D") image data corresponding to near and far depths, respectively, where at least one of the near and far frames of the 2D image data includes the blurred virtual object / pixel; displaying to a user near and far 2D image frames corresponding to the near and far frames of the 2D image data at the near and far depths, respectively; The method of claim 14 further comprising:

16. The method of claim 14, wherein one or both of the near clip and the far clip include the near clip.

17. The method of claim 14, wherein one or both of the near clip and the far clip include the far clip.

18. The method of claim 14, wherein one or both of the near clip and the far clip include the near clip and the far clip.

19. The method of claim 14, wherein the virtual object / pixel is blurred as a linear or logarithmic or asymmetric function of the determined difference / distance between the depth of the virtual object / pixel and the depth of one or both of the near and far clips, respectively.

Citation Information

Patent Citations

  • Stereoscopic viewing system / method and storage medium storing computer program displaying a pair of pictures viewed from two different viewpoints so that they can stereoscopically be viewed

    JP1998083460A

  • Program, information storage medium, and image generation system

    JP2012173821A

  • Information processing device, information processing method and program

    JP2013105183A

  • Information processing program, information processing device, information processing system, and information processing method

    JP2014235657A

  • Image processing device

    WO2017191703A1