Multi-focal-plane-based method for generating stereoscopic viewpoints in DIBR systems (MFP-DIBR)

Abstract

Some embodiments of an exemplary method may include receiving an input image with depth information, mapping the input image to a set of focal plane images, orienting the set of focal plane images using head orientation information to provide stereoscopic disparity between the left and right eyes, and displaying the oriented set of focal plane images. Some embodiments of another exemplary method may include receiving a description of three-dimensional (3D) content, receiving information from a tracker indicative of a viewer's movement relative to a real-world environment, synthesizing motion parallax by varying multiple focal planes of the 3D content in response to receiving the information indicative of the viewer's movement, and rendering an image on a multi-focal plane display using the varied multi-focal plane rendering.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional patent application of and claims the benefit under 35 U.S.C. §119(e) of U.S. Provisional Patent Application No. 62 / 647,568, entitled "Multifocal Plane Based Method to Produce Stereoscopic Viewpoints in a DIBR System (MFP-DIBR)," filed March 23, 2018, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Wearable glasses-type near-eye displays (NEDs) are popular in virtual reality and gaming, and more recently in augmented reality. Glasses that use a mobile phone as the display element are an example of a popular, low-cost device for viewing virtual or camera-captured images. Generally, all of these displays present a sequence of stereoscopic image pairs. Stereoscopic rendering may also be used with external displays, including those for stereoscopic 3D (S3D) TVs.

[0003] Many previous MFP-based or 3D-based viewpoint generation systems may have lower visual quality because the viewpoint can change slightly with user movement. Many near-eye displays that support natural accommodation lack support for viewpoint changes, even small viewpoint changes relative to the display when the eye is turned (slightly changing the viewpoint within the eyebox). Disocclusion is naturally content-dependent, but large viewpoint changes or mismatches are more noticeable. Creating virtual viewpoints for stereoscopic video or motion parallax may be used, for example, in 3DoF+ extensions of 360° video and other 3DoF applications. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Rahul Narain et al., Optimal Presentation of Imagery with Focus Cues on Multi-Plane Displays, 34(4) ACM TRANSACTIONS ON GRAPHICS 59:1–12 (August 2015) (“Narain”) [Non-patent document 2] Pan et al., 3D Video Disparity Scaling for Preference and Prevention of Discomfort, 7863 PROCEEDINGS OF SPIE (2011) (“Pan”) [Non-patent document 3] Linwei Zhu et al., View-Spatial-Temporal Post-Refinement for View Synthesis in 3D Video Systems, 28 Signal Processing: Image Communication 1342~1357 (2013) (“Zhu”) [Non-patent document 4] Sergey Sukhanov, 3D Content Production, ST. PETERSBURG ELECTROTECHNICAL UNIVERSITY (undated presentation, retrieved from www.slideshare.net / mikhailvink / sergey-a-sukhanov-3d-content-production, website indicates publication date March 12, 2011) (last accessed February 26, 2019) [Non-patent document 5] Kristoph Fehn, A 3D-TV Approach Using Depth-Image-Based Rendering (DIBR), Proceedings of VIIP '03 (September 2003) (Benalmadena, Spain) ("Fehn") [Non-patent document 6] Shibata, Takashi et al., Visual Discomfort with Stereo Displays: Effects of Viewing Distance and Direction of Vergence-Accommodation Conflict, PROCEEDINGS OF THE INT'L SOC. FOR OPTICS AND PHOTONICS (SPIE) 7863 (2011) (78630P1~78630P9) ("Shibata") [Non-Patent Document 7] Shiro Suyama et al., Apparent 3-D Image Perceived from Luminance-Modulated Two 2-D Images Displayed at Different Depths, 44 (8) Vision Research 785-793 (April 2004) (“Suyama”) [Non-patent document 8] Simon J. Watt et al., Real-World Stereoscopic Performance in Multiple-Focal-Plane Displays: How Far Apart Should the Image Planes Be?, SPIE: Stereoscopic Displays and Applications XXIII (February 2012) [Non-Patent Document 9] Simon J. Watt et al., Achieving Near-Correct Focus Cues in a 3-D Display Using Multiple Image Planes, 5666 Proceedings of SPIE 393~401 (2005) (``Watt 2005'') [Non-Patent Document 10] Kurt Akeley et al., A Stereo Display Prototype with Multiple Focal Distances, 23(3) ACM TRANSACTIONS ON GRAPHICS (TOG) 804–813 (2004) (“Akeley”) [Non-Patent Document 11] X. Hu & H. Hua, Design and Assessment of a Depth-Fused Multi-Focal-Plane Display Prototype, 10(4) IEEE / OSA Journal of Display Technology 308~316 (2014) (“Hu&Hua”) [Non-Patent Document 12] File:2D plus depth.png, WIKIPEDIA, / / commons.wikimedia.org / w / index.php?curid=5489771 (last accessed February 22, 2019) [Non-Patent Document 13] Xuyuan Xu et al., Depth Map Misalignment Correction and Dilation for DIBR View Synthesis, 28 Signal Processing: Image Communication 1023~1045 (2013) [Non-Patent Document 14] Shibata et al., The Zone of Comfort: Predicting Visual Discomfort with Stereo Displays, 11(8):11 J. OF VISION 1–29 (2011) (“Shibata II”) [Non-Patent Document 15] Paul V. Johnson et al., Dynamic Lens and Monovision 3D Displays to Improve Viewer Comfort, 24(11) Opt. Exp. 11808~11827 (2016) [Non-Patent Document 16] Zannoli et al., Blur and the Perception of Depth at Occlusion, 16(6):17 Journal of Vision 1–25 (2016) [Non-Patent Document 17] J.P. Rolland et al., Multifocal Planes Head-Mounted Displays, 39 Appl. Opt. 3209-3215 (2000) ("Rolland") Summary of the Invention

[0005] An exemplary method according to some embodiments may include receiving an input image with depth information, mapping the input image to a plurality of focal plane images using the depth information, orienting the plurality of focal plane images using head orientation information to provide stereoscopic disparity between the left and right eyes, and displaying the oriented plurality of focal plane images.

[0006] In some embodiments of the exemplary method, at least receiving the input image and displaying the oriented multiple focal plane images is performed by a multi-focal plane (MFP) display.

[0007] In some embodiments of the example method, orienting the plurality of focal plane images may include rotating one or more of the plurality of focal plane images.

[0008] In some embodiments of the example method, rotating one or more of the plurality of focal plane images may include rotating a mismatch vector for one or more of the focal plane images relative to a vector parallel to the horizon.

[0009] In some embodiments of the example method, orienting the plurality of focal plane images may include shifting one or more of the plurality of focal plane images.

[0010] In some embodiments of the example method, shifting one or more of the plurality of focal plane images may include scaling a disparity vector for the respective one or more focal plane images.

[0011] In some embodiments of the example method, shifting one or more of the plurality of focal plane images may include rotating one or more of the plurality of focal plane images relative to a fixed origin.

[0012] In some embodiments of the example method, shifting one or more of the plurality of focal plane images includes moving one or more of the plurality of focal plane images by an offset relative to a fixed origin.

[0013] Some embodiments of the example method may further include processing the multiple focal plane images to address image mismatch caused by shifting the focal plane images.

[0014] In some embodiments of the example method, processing the plurality of focal plane images may include filtering one or more of the plurality of focal plane images.

[0015] In some embodiments of the example method, processing the plurality of focal plane images may include determining depth blending weights for at least one of the plurality of focal plane images and generating at least one of the plurality of focal plane images using the respective depth blending weights.

[0016] In some embodiments of the example method, generating at least one of the plurality of focal plane images includes multiplying at least one of the plurality of textures with a respective depth blending weight.

[0017] In some embodiments of the example method, displaying the oriented multiple focal plane images may include combining the shifted focal plane images to generate a stereoscopic image pair.

[0018] Some embodiments of the example method may further include measuring motion tracking sensor readings of the viewer's position relative to the real-world environment to generate head orientation information.

[0019] Some embodiments of the example method may further include synthesizing motion parallax by altering one or more of the plurality of focal plane images in response to generating the head orientation information.

[0020] In some embodiments of the exemplary method, the synthesized motion parallax may include scaling one or more of the multiple focal plane images relative to each other using head orientation information, which may indicate axial motion.

[0021] Some embodiments of the example method may further include filtering the input image for low frequency content and redistributing the low frequency content into multiple focal plane images.

[0022] Some embodiments of the example method may further include filtering the input image for high frequency content and decomposing the high frequency content into multiple focal plane images.

[0023] An exemplary apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform one of the exemplary methods described above.

[0024] An exemplary method according to some embodiments may include receiving a description of three-dimensional (3D) content, receiving information from a tracker indicative of a viewer's movement relative to a real-world environment, synthesizing motion parallax by varying multi-focal planes of the 3D content in response to receiving the information indicative of the viewer's movement from the tracker, and rendering an image on a multi-focal plane display using the varied multi-focal plane rendering.

[0025] In some embodiments of the exemplary method, the synthesized motion parallax may include scaling the multi-focal planes relative to each other using information indicative of the viewer's motion, which may be axial.

[0026] An exemplary apparatus according to some embodiments may include a processor (e.g., one or more processors) and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform one of the exemplary methods described above.

[0027] An exemplary method according to some embodiments may include determining a plurality of focal plane images to be mapped to an input image, orienting the plurality of focal plane images using head orientation information to provide stereoscopic disparity between the left and right eyes, and displaying the oriented plurality of focal plane images.

[0028] In some embodiments of the example method, determining the plurality of focal plane images includes receiving the plurality of focal plane images.

[0029] In some embodiments of the example method, determining the multiple focal plane images includes mapping the input image to the multiple focal plane images to generate the multiple focal plane images.

[0030] An exemplary apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform one of the exemplary methods described above.

[0031] An exemplary method according to some embodiments may include receiving three-dimensional (3D) video content, receiving a viewer orientation signal, determining image data for one or more focal planes of a display using the 3D video content and the viewer orientation signal, and rendering one or more focal plane images on the display using the image data.

[0032] An exemplary apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform the exemplary methods described above. [Brief explanation of the drawings]

[0033] A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings, in which like reference numerals in the figures indicate like elements, and in which: [Figure 1A] FIG. 1 is a system diagram of an exemplary system illustrating an exemplary communication system, according to some embodiments. [Figure 1B] 1B is a system diagram of an example system illustrating an example wireless transmit / receive unit (WTRU) that may be used within the communication system illustrated in FIG. 1A, in accordance with some embodiments. [Figure 2A] FIG. 1 is a schematic diagram illustrating an exemplary set of focal planes displayed at different depths from a user. [Figure 2B] 1 is an illustration showing an exemplary focal plane image. [Figure 3] 1 is a graph illustrating an exemplary image mismatch between left and right images versus observer-screen distance for Percival's comfortable viewing zone. [Figure 4]FIG. 1 is a schematic diagram illustrating exemplary left-eye and right-eye images for a stereoscopic display for a viewer with a non-rotated orientation. [Figure 5] FIG. 1 is a schematic diagram illustrating exemplary left-eye and right-eye images bounded in glasses coordinates for a stereoscopic display for a viewer with a rotated orientation, where the horizon is tilted with the head. [Figure 6] FIG. 10 is a schematic diagram illustrating exemplary left-eye and right-eye images adjusted to match the orientation of the horizon for a stereoscopic display for a viewer with rotated orientation and unchanged disparity. [Figure 7] FIG. 1 is a system diagram illustrating an example set of interfaces for manipulating image mismatches. [Figure 8] FIG. 1 is a system diagram illustrating an example set of interfaces for a DIBR-based 3D transmission system. [Figure 9] 1A-1C are schematic plan views illustrating example scenarios showing the appearance of dis-occlusion when varying the source and target cameras; [Figure 10A] 1 is an illustration showing an exemplary 3D warping image before hole filling. [Figure 10B] 10 is an illustration showing an exemplary 3D warped image after hole filling. [Figure 11A] FIG. 1 is a schematic perspective view illustrating an exemplary vergence accommodation conflict (VAC) in many stereoscopic viewing systems. [Figure 11B] FIG. 1 is a schematic perspective view illustrating an exemplary vergence accommodation conflict (VAC) in many stereoscopic viewing systems. [Figure 11C] FIG. 1 is a schematic perspective view illustrating an exemplary vergence accommodation conflict (VAC) in many stereoscopic viewing systems. [Figure 12] FIG. 1 is a schematic plan view illustrating an exemplary multi-focal plane (MFP) near-eye display according to some embodiments. [Figure 13] 1 is a schematic plan view illustrating exemplary viewing of near-focus, intermediate-focus, and far-focus image planes according to some embodiments. FIG. [Figure 14] 1 is a schematic diagram illustrating an exemplary focal image plane viewed by an observer; [Figure 15A] FIG. 1 is a schematic diagram illustrating exemplary front and back focal planes for an exemplary image of a square. [Figure 15B] FIG. 1 is a schematic diagram illustrating exemplary front and back focal planes for an exemplary image of a square. [Figure 15C] FIG. 1 is a schematic diagram illustrating exemplary front and back focal planes for an exemplary image of a square. [Figure 16A] 1 is a schematic front view illustrating an exemplary focal plane displayed for right-eye and left-eye images; [Figure 16B] 1 is a schematic front view illustrating an exemplary focal plane displayed for right-eye and left-eye images; [Figure 17] FIG. 1 is a schematic plan view illustrating an exemplary viewing of two focal planes with a depth-based luminous intensity weighting function. [Figure 18] 1 is a schematic plan view illustrating exemplary viewing of three focal planes by the left and right eyes according to some embodiments. FIG. [Figure 19] 1 is a schematic plan view illustrating an example quantization of depth for an MFP display without depth blending (weighting), according to some embodiments. FIG. [Figure 20A] 10 is a graph illustrating an example of weight versus depth according to some embodiments. [Figure 20B] 10 is a graph illustrating an example of weight versus depth according to some embodiments. [Figure 21A] 10 is a graph illustrating an example blending function of weight versus depth according to some embodiments. [Figure 21B] 1 is an illustration showing an exemplary test image. [Figure 21C] 21C is an exemplary depth map illustrating pixel distances for the case of FIG. 21B. [Figure 21D]21B is an illustration showing an example focal plane image using the depth weights of FIG. 21A together with the test image of FIG. 21B and its example depth map of FIG. 21C, in accordance with some embodiments. [Figure 21E] 21B is an illustration showing an example focal plane image using the depth weights of FIG. 21A together with the test image of FIG. 21B and its example depth map of FIG. 21C, in accordance with some embodiments. [Figure 21F] 21B is an illustration showing an example focal plane image using the depth weights of FIG. 21A together with the test image of FIG. 21B and its example depth map of FIG. 21C, in accordance with some embodiments. [Figure 22A] 10 is a graph illustrating an example blending function of weight versus depth according to some embodiments. [Figure 22B] 1 is an illustration showing an exemplary test image. [Figure 22C] 22C is an exemplary depth map illustrating pixel distances for FIG. 22B. [Figure 22D] 22B is an illustration showing an example focal plane image using the depth weights of FIG. 22A with the test image of FIG. 22B and its example depth map of FIG. 22C, according to some embodiments. [Figure 22E] 22B is an illustration showing an example focal plane image using the depth weights of FIG. 22A with the test image of FIG. 22B and its example depth map of FIG. 22C, according to some embodiments. [Figure 22F] 22B is an illustration showing an example focal plane image using the depth weights of FIG. 22A with the test image of FIG. 22B and its example depth map of FIG. 22C, according to some embodiments. [Figure 23A] 1 is a schematic front view illustrating an example scenario for stereoscopic image mismatch, according to some embodiments. [Figure 23B] 1 is a schematic front view illustrating an example scenario for stereoscopic image mismatch, according to some embodiments. [Figure 23C]1 is a schematic front view illustrating an example scenario for stereoscopic image mismatch, according to some embodiments. [Figure 24A] 1A-1C are schematic front views illustrating example scenarios for image mismatch with viewer orientation direction, according to some embodiments. [Figure 24B] 1A-1C are schematic front views illustrating example scenarios for image mismatch with viewer orientation direction, according to some embodiments. [Figure 24C] 1A-1C are schematic front views illustrating example scenarios for image mismatch with viewer orientation direction, according to some embodiments. [Figure 25] FIG. 1 is a system diagram illustrating an example set of processing blocks and interfaces for generating left and right eye shifted and projected MFPs, according to some embodiments. [Figure 26] FIG. 1 is a system diagram illustrating an example set of processing blocks and interfaces for generating left-eye and right-eye shifted and projected MFPs for three scenarios of viewer orientation, according to some embodiments. [Figure 27A] 1A-1C are schematic plan views illustrating exemplary focal planes viewed under different viewer orientations, according to some embodiments. [Figure 27B] 1A-1C are schematic plan views illustrating exemplary focal planes viewed under different viewer orientations, according to some embodiments. [Figure 27C] 1A-1C are schematic plan views illustrating exemplary focal planes viewed under different viewer orientations, according to some embodiments. [Figure 27D] 1A-1C are schematic plan views illustrating exemplary focal planes viewed under different viewer orientations, according to some embodiments. [Figure 28A] 1 is an illustration showing an exemplary test image. [Figure 28B] 28B is an illustration showing an example weighted focal plane image using the test image of FIG. 28A in accordance with some embodiments. [Figure 28C] 28B is an illustration showing an example weighted focal plane image using the test image of FIG. 28A in accordance with some embodiments. [Figure 28D] 28B is an illustration showing an example weighted focal plane image using the test image of FIG. 28A in accordance with some embodiments. [Figure 29A] 28B-28D are illustrations showing corresponding exemplary right-eye images formed using the focal plane images of FIGS. 28B-28D, according to some embodiments. [Figure 29B] 28B-28D are illustrations showing corresponding exemplary left-eye images formed using the focal plane images of FIGS. 28B-28D, according to some embodiments. [Figure 30A] 1A-1C are schematic plan views illustrating exemplary focal planes viewed under different eye separation distances, according to some embodiments. [Figure 30B] 1A-1C are schematic plan views illustrating exemplary focal planes viewed under different eye separation distances, according to some embodiments. [Figure 30C] 1A-1C are schematic plan views illustrating exemplary focal planes viewed under different eye separation distances, according to some embodiments. [Figure 31A] 10 is an illustration showing an enlarged excerpt of an exemplary stereoscopic image for the right eye formed with sinusoidal depth weighting, according to some embodiments. [Figure 31B] 10 is an illustration showing an exemplary stereoscopic image for the right eye formed with corresponding sinusoidal depth weighting, according to some embodiments. [Figure 31C] 10 is an illustration showing an exemplary stereoscopic image for the left eye formed with corresponding sinusoidal depth weighting, according to some embodiments. [Figure 31D] 10 is an illustration showing an enlarged excerpt of an exemplary stereoscopic image for the right eye formed with polynomial depth weighting, according to some embodiments. [Figure 31E]10 is an illustration showing an exemplary right-eye image correspondingly formed with polynomial depth weighting, according to some embodiments. [Figure 31F] 10 is an illustration showing an exemplary left-eye image correspondingly formed with polynomial depth weighting, according to some embodiments. [Figure 32A] 1A-1C are schematic front views illustrating exemplary mismatches for a first stereoscopic display with different user orientations, according to some embodiments. [Figure 32B] 1A-1C are schematic front views illustrating exemplary mismatches for a first stereoscopic display with different user orientations, according to some embodiments. [Figure 32C] 1A-1C are schematic front views illustrating exemplary mismatches for a first stereoscopic display with different user orientations, according to some embodiments. [Figure 32D] 10A-10C are schematic front views illustrating exemplary mismatches for a second stereoscopic display with different user orientations, according to some embodiments. [Figure 32E] 10A-10C are schematic front views illustrating exemplary mismatches for a second stereoscopic display with different user orientations, according to some embodiments. [Figure 32F] 10A-10C are schematic front views illustrating exemplary mismatches for a second stereoscopic display with different user orientations, according to some embodiments. [Figure 33A] 1A-1C are schematic front views illustrating exemplary mismatches for stereoscopic displays with different user orientations, according to some embodiments. [Figure 33B] 1A-1C are schematic front views illustrating exemplary mismatches for stereoscopic displays with different user orientations, according to some embodiments. [Figure 33C] 1A-1C are schematic front views illustrating exemplary mismatches for stereoscopic displays with different user orientations, according to some embodiments. [Figure 34] 1 is a schematic user's diagram illustrating exemplary left-eye and right-eye images for a stereoscopic display for a viewer with a rotated orientation, according to some embodiments. [Figure 35]1 is a schematic processing diagram illustrating an exemplary process for generating multiple perspectives for a view by left-right warping a set of focal plane images, according to some embodiments. [Figure 36] 36 is an illustration showing an exemplary series of three stereoscopic images for stereoscopic display using the warped image of FIG. 35 according to some embodiments. [Figure 37] FIG. 1 is a system diagram illustrating an example set of processing blocks and interfaces for generating shifted and projected MFP images for flexible baseline stereoscopic display for three viewer orientations, according to some embodiments. [Figure 38A] 1 is a set of computer windows illustrating an exemplary set of interfaces for an image manipulation program showing a depth map image, according to some embodiments. [Figure 38B] 1 is a set of computer windows illustrating an exemplary set of interfaces for an image manipulation program showing a depth weighting graph, according to some embodiments. [Figure 38C] 1 is a set of computer windows illustrating an exemplary set of interfaces for an image manipulation program showing generated focal plane images, according to some embodiments. [Figure 39A] 1 is an illustration showing an exemplary set of generated focal plane images within a simulated environment. [Figure 39B] 1 is an illustration showing an exemplary set of generated focal plane images within a simulated environment. [Figure 39C] 1 is an illustration showing an exemplary set of generated focal plane images within a simulated environment. [Figure 40] FIG. 2 is a message sequence diagram illustrating an exemplary process for generating a multi-focal plane (MFP) display image, according to some embodiments. [Figure 41] FIG. 1 is a message sequence diagram illustrating an exemplary process for generating a stereoscopic 3D display image, according to some embodiments. [Figure 42A] 10 is a graph illustrating an example blending function of weight versus depth according to some embodiments. [Figure 42B] 1 is an illustration showing an exemplary test image. [Figure 42C] 42C is an exemplary depth map illustrating pixel distances for FIG. 42B. [Figure 42D] 42B is an illustration showing an example focal plane image using the depth weights of FIG. 42A together with the test image of FIG. 42B and its example depth map of FIG. 42C, according to some embodiments. [Figure 42E] 42B is an illustration showing an example focal plane image using the depth weights of FIG. 42A together with the test image of FIG. 42B and its example depth map of FIG. 42C, according to some embodiments. [Figure 42F] 42B is an illustration showing an example focal plane image using the depth weights of FIG. 42A together with the test image of FIG. 42B and its example depth map of FIG. 42C, according to some embodiments. [Figure 43A] 10 is a graph illustrating an example blending function of weight versus depth according to some embodiments. [Figure 43B] 1 is an illustration showing an exemplary test image. [Figure 43C] 43C is an exemplary depth map illustrating pixel distances for FIG. 43B. [Figure 43D] 43B is an illustration showing an example focal plane image using the depth weights of FIG. 43A together with the test image of FIG. 43B and its example depth map of FIG. 43C, in accordance with some embodiments. [Figure 43E] 43B is an illustration showing an example focal plane image using the depth weights of FIG. 43A together with the test image of FIG. 43B and its example depth map of FIG. 43C, in accordance with some embodiments. [Figure 43F] 43B is an illustration showing an example focal plane image using the depth weights of FIG. 43A together with the test image of FIG. 43B and its example depth map of FIG. 43C, in accordance with some embodiments. [Figure 44]1A-1C are schematic diagrams illustrating exemplary focus stack images captured at different depths from a user. [Figure 45A] 1 is an illustration showing an exemplary linearly blended focal plane. [Figure 45B] 1 is an illustration showing an exemplary linearly blended focal plane. [Figure 45C] 1 is an illustration showing an exemplary linearly blended focal plane. [Figure 45D] 1 is an illustration showing an exemplary linearly blended focal plane. [Figure 45E] 1 is an illustration showing an exemplary redistributed focal plane. [Figure 45F] 1 is an illustration showing an exemplary redistributed focal plane. [Figure 45G] 1 is an illustration showing an exemplary redistributed focal plane. [Figure 45H] 1 is an illustration showing an exemplary redistributed focal plane. [Figure 46A] 10 is an illustration showing an exemplary stereoscopic image for the right eye formed using corresponding unblended focal plane images, according to some embodiments. [Figure 46B] 10 is an illustration showing an exemplary stereoscopic image for the left eye formed using corresponding unblended focal plane images, according to some embodiments. [Figure 47] 10 is a flowchart illustrating an example process for generating redistributed focal plane images for front, mid, and rear depth ranges, according to some embodiments. [Figure 48A] 1 is an illustration showing an exemplary focal plane image without redelivery, according to some embodiments. [Figure 48B] 1 is an illustration showing an exemplary focal plane image without redelivery, according to some embodiments. [Figure 48C] 1 is an illustration showing an exemplary focal plane image without redelivery, according to some embodiments. [Figure 48D]1 is an illustration showing an exemplary focal plane image with redelivery, according to some embodiments. [Figure 48E] 1 is an illustration showing an exemplary focal plane image with redelivery, according to some embodiments. [Figure 48F] 1 is an illustration showing an exemplary focal plane image with redelivery, according to some embodiments. [Figure 49A] 10 is an illustration showing an exemplary right image for a crossed-eye stereogram formed using linearly blended focal planes, according to some embodiments. [Figure 49B] 10 is an illustration showing an exemplary left image for a crossed-eye stereogram formed using linearly blended focal planes, according to some embodiments. [Figure 50A] 10 is an illustration showing an exemplary right image for a crossed-eye stereogram formed using a redistributed focal plane, according to some embodiments. [Figure 50B] 10 is an illustration showing an exemplary left image for a crossed-eye stereogram formed using redistributed focal planes, according to some embodiments. [Figure 51A] 10 is an illustration showing an exemplary right image for a crossed-eye stereogram formed using selective Gaussian filtering in redistributing an MFP, according to some embodiments. [Figure 51B] 10 is an illustration showing an exemplary left image for a crossed-eye stereogram formed using selective Gaussian filtering in redistributing an MFP, according to some embodiments. [Figure 52A] 1 is a flowchart illustrating an exemplary process for generating a redistributed focal plane image using low-frequency filtering and high-frequency filtering, according to some embodiments. [Figure 52B] 1 is a flowchart illustrating an exemplary process for generating a high frequency focal plane image using low pass filtering, according to some embodiments. [Figure 53]10 is a flowchart illustrating an example process for generating redistributed focal plane images for a front depth range, a middle depth range, and a rear depth range, according to some embodiments. [Figure 54] 1 is a flowchart illustrating an exemplary process for generating a redistributed focal plane image using low-pass filtering, according to some embodiments. [Figure 55A] 1 is a schematic perspective view illustrating exemplary multi-focal planes rendered in space, according to some embodiments; [Figure 55B] 1 is a schematic perspective view illustrating exemplary multi-focal planes rendered in space, according to some embodiments; [Figure 56A] 1 is a schematic perspective view illustrating an example of shifted and scaled multi-focal planes for different viewpoints according to some embodiments; FIG. [Figure 56B] 1 is a schematic perspective view illustrating an example of shifted and scaled multi-focal planes for different viewpoints according to some embodiments; FIG. [Figure 56C] 1 is a schematic perspective view illustrating an example of shifted and scaled multi-focal planes for different viewpoints according to some embodiments; FIG. [Figure 57] FIG. 1 is a process diagram illustrating an exemplary process for generating multiple focal planes in a motion parallax system, according to some embodiments. [Figure 58] 10A-10C are schematic plan views illustrating exemplary focal plane size scaling adjustments for axial movement, according to some embodiments. [Figure 59A] 1A-1C are schematic plan views illustrating exemplary deformations of exemplary focal plane content during axial movement, according to some embodiments. [Figure 59B] 1A-1C are schematic plan views illustrating exemplary deformations of exemplary focal plane content during axial movement, according to some embodiments. [Figure 59C] 1A-1C are schematic plan views illustrating exemplary deformations of exemplary focal plane content during axial movement, according to some embodiments. [Figure 60] 1 is a graph illustrating exemplary tracking of gaze coordinates during user movement, according to some embodiments. [Figure 61] FIG. 1 is a process diagram illustrating an exemplary process for generating a multi-focal plane true volume set, according to some embodiments. [Figure 62] FIG. 1 is a process diagram illustrating an exemplary process for generating a multi-focal plane synthesized volume set, according to some embodiments. [Figure 63] FIG. 1 is a process diagram illustrating an exemplary process for generating and summing a set of multi-focal planes for a two-dimensional display, according to some embodiments. [Figure 64] FIG. 1 is a process diagram illustrating an exemplary process for generating and summing a set of multi-focal planes for a monocular 2D display, according to some embodiments. [Figure 65] FIG. 1 is a process diagram illustrating an exemplary process for generating and summing a set of multiple focal planes for a true stereoscopic three-dimensional (S3D) display, according to some embodiments. [Figure 66] FIG. 1 is a process diagram illustrating an exemplary process for generating and summing a set of multi-focal planes for a synthesized stereoscopic three-dimensional (S3D) display, according to some embodiments. [Figure 67] FIG. 1 is a process diagram illustrating an example process for generating a multi-focal plane with a front-end processing path and a display-end path, according to some embodiments. [Figure 68A] 68C and 68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 68B] 68C and 68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 68C]68C and 68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 68D] 68C and 68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 68E] 68C and 68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 68F] 68C and 68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 68G] 68C and 68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 68H] 68C and 68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 68I] 68C and 68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 68J] 68C and 68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 69] 1 is a flowchart illustrating an exemplary process according to some embodiments. [Figure 70] 1 is a flowchart illustrating an exemplary process, according to some embodiments. [Figure 71] FIG. 2C is a grayscale version of the native image corresponding to FIG. 2B. [Figure 72] FIG. 15 is a grayscale version of the native image corresponding to FIG. 14. [Figure 73] FIG. 16C is a grayscale version of the native image corresponding to FIGS. 16A-16B. [Figure 74] 21B-21F are grayscale versions of the native images corresponding to FIGS. 21B-21F. [Figure 75] 22B-22F are grayscale versions of the native images corresponding to FIGS. 22B-22F. [Figure 76] 28A-28D are grayscale versions of the native images corresponding to FIGS. 28A-28D. [Figure 77] FIG. 29C is a grayscale version of the native image corresponding to FIGS. 29A-29B. [Figure 78] 31A-31F are grayscale versions of the native images corresponding to FIGS. 31A-31F. [Figure 79] FIG. 36 is a grayscale version of the native image corresponding to FIG. 35. [Figure 80] FIG. 37 is a grayscale version of the native image corresponding to FIG. 36. [Figure 81] FIG. 38D is a grayscale version of the native image corresponding to FIG. 38C. [Figure 82] FIG. 39C is a grayscale version of the native image corresponding to FIGS. 39A-39C. [Figure 83] 42B-42F are grayscale versions of the native images corresponding to FIGS. 42B-42F. [Figure 84] 43B-43F are grayscale versions of the native images corresponding to FIGS. 43B-43F. [Figure 85] FIG. 45 is a grayscale version of the native image corresponding to FIG. 44. [Figure 86] 45A-45H are grayscale versions of the native images corresponding to FIGS. 45A-45H. [Figure 87] FIG. 46B is a grayscale version of the native image corresponding to FIGS. 46A-46B. [Figure 88] 48A-48F are grayscale versions of the native images corresponding to FIGS. 48A-48F. [Figure 89] FIG. 49B is a grayscale version of the native image corresponding to FIGS. 49A-49B. [Figure 90] 50A-50B are grayscale versions of the native images corresponding to FIGS. 50A-50B. [Figure 91] FIG. 51C is a grayscale version of the native image corresponding to FIGS. 51A-51B. [Figure 92] 68A-68J are grayscale versions of the native images corresponding to FIGS. 68A-68J. DETAILED DESCRIPTION OF THE INVENTION

[0034] The entities, connections, arrangements, etc. depicted in and described in connection with the various figures are offered by way of example, and not by way of limitation. Accordingly, any and all statements or other indications as to what a particular figure "depicts," what a particular element or entity within a particular figure "is" or "has," and any and all similar statements that, in isolation and from context, may be read as absolute and therefore limiting, may properly only be read when constructively preceded by a phrase such as "in at least one embodiment...." For brevity and clarity of presentation, this implied introductory phrase will not be repeated at length in the Detailed Description.

[0035] Exemplary Network for Implementation of the Embodiments A wireless transmit / receive unit (WTRU) may be used, for example, as an MFP display in some embodiments described herein.

[0036] 1A is a diagram illustrating an example communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple-access system that provides content, such as voice, data, video, messages, broadcasts, etc., to multiple wireless users. The communication system 100 may enable the multiple wireless users to access such content through sharing of system resources, including wireless bandwidth. For example, the communication system 100 may use one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tailed unique word DFT spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, filter bank multicarrier (FBMC), etc.

[0037] 1A, communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RANs 104 / 113, CNs 106 / 115, public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, although it will be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. For example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a “station” and / or “STA,” may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspot or Mi-Fi devices, IoT devices, watches or other wearable head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain contexts), consumer electronic devices, devices operating on commercial and / or industrial wireless networks, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be referred to interchangeably as a UE.

[0038] The communications system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communications networks, such as the CN 106 / 115, the Internet 110, and / or other networks 112. For example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNode B, a home Node B, a home eNode B, a gNB, an NR Node B, a site controller, an access point (AP), a wireless router, etc. Although the base stations 114a, 114b are each depicted as a single element, it will be understood that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0039] The base station 114a may be part of the RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide wireless service coverage for a particular geographic area, which may be relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, one for each sector of the cell. In an embodiment, the base station 114a may employ MIMO technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in desired spatial directions.

[0040] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over the air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0041] More specifically, as mentioned above, the communication system 100 may be a multiple-access system and may use one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base stations 114a and WTRUs 102a, 102b, 102c in the RAN 104 / 113 may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 115 / 116 / 117 using Wideband CDMA (WCDMA). WCDMA may include communication protocols such as High Speed Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed Downlink (DL) Packet Access (HSDPA) and / or High Speed UL Packet Access (HSUPA).

[0042] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-Pro).

[0043] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as New Radio (NR) radio access, which may establish the air interface 116 using NR.

[0044] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may jointly implement LTE and NR radio access, e.g., using a dual connectivity (DC) principle. Thus, the air interface utilized by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0045] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement a wireless technology such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), or the like.

[0046] 1A may be, for example, a wireless router, a Home Node B, a Home eNode B, or an access point and may utilize any suitable RAT to facilitate wireless connectivity within a local area, such as a business office, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, etc. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-Pro, NR, etc.) to establish a picocell or femtocell. 1A, base station 114b may have a direct connection to the Internet 110. Therefore, base station 114b may not be required to access the Internet 110 via CN 106 / 115.

[0047] The RAN 104 / 113 may communicate with the CN 106 / 115, which may be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have varying quality of service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 / 115 may provide call control, billing services, mobile location services, prepaid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions such as user authentication. Although not shown in FIG. 1A , it will be understood that the RAN 104 / 113 and / or the CN 106 / 115 may communicate directly or indirectly with other RANs using the same RAT as the RAN 104 / 113 or a different RAT. For example, in addition to being connected to the RAN 104 / 113, which may utilize NR radio technology, the CN 106 / 115 may also communicate with another RAN (not shown) that uses GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0048] The CN 106 / 115 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network providing plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) in the TCP / IP Internet protocol suite. The network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the network 112 may include another CN connected to one or more RANs that may use the same RAT as the RAN 104 / 113 or a different RAT.

[0049] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links.) For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with a base station 114a that may use a cellular-based wireless technology and with a base station 114b that may use an IEEE 802.2 wireless technology.

[0050] 1B is a system diagram illustrating an example WTRU 102. As shown in FIG. 1B, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a GPS chipset 136, and / or other peripherals 138. It will be understood that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0051] The processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. While FIG. 1B depicts the processor 118 and the transceiver 120 as separate components, it will be understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0052] The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive, for example, IR signals, UV signals, or visible light signals. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF signals and light signals. It will be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0053] 1B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0054] The transceiver 120 may be configured to modulate signals that can be transmitted by the transmit / receive element 122 and to demodulate signals received by the transmit / receive element 122. As mentioned above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11, for example.

[0055] The processor 118 of the WTRU 102 may be coupled to and may receive user input data from a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. Additionally, the processor 118 may access information from and store data in any type of suitable memory, such as non-removable memory 130 and / or removable memory 132. The non-removable memory 130 may include RAM, ROM, a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, etc. In other embodiments, the processor 118 may access information from and store data in memory that is not physically located on the WTRU 102, such as on a server or home computer (not shown).

[0056] The processor 118 may receive power from the power source 134 and may be configured to distribute and / or control the power to other components within the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0057] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or instead of, information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be understood that the WTRU 102 may acquire location information by way of any suitable location determination method while remaining consistent with an embodiment.

[0058] The processor 118 may be further coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or videos), a USB port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game player module, an internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, etc. The peripheral device 138 may include one or more sensors, which may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0059] The WTRU 102 may include a full-duplex radio where transmission and reception of some or all of the signals (e.g., associated with particular subframes for both the UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit that reduces and / or substantially eliminates self-interference either through hardware (e.g., chokes) or signal processing via a processor (e.g., a separate processor (not shown) or via the processor 118). In an embodiment, the WTRU 102 may include a half-duplex radio for transmission and reception of some or all of the signals (e.g., associated with particular subframes for either the UL (e.g., for transmission) or downlink (e.g., for reception)).

[0060] 1A-1B, one or more, or all, of the functions described herein with respect to one or more of the WTRUs 102a-d, the base stations 114a-b, and / or any other devices described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more, or all, of the functions described herein. For example, the emulation device may be used to test other devices and / or to simulate network and / or WTRU functionality.

[0061] The emulation device may be designed to perform one or more tests of other devices within a lab environment and / or an operator network environment. For example, one or more emulation devices may perform one or more or all functions while fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within the communication network. One or more emulation devices may perform one or more or all functions while temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation device may be directly coupled to another device for testing purposes and / or may perform tests using over-the-air wireless communication.

[0062] The one or more emulation devices may perform one or more functions, inclusive, while not implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation devices may be utilized in a test laboratory and / or in a test scenario within an undeployed (e.g., test) wired and / or wireless communication network to perform testing of one or more components. The one or more emulation devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (which may, for example, include one or more antennas) may be used by the emulation devices to transmit and / or receive data.

[0063] MFP-based method for stereoscopic viewing Systems and methods disclosed in some embodiments herein describe various exemplary MFP-based embodiments for cost-effective and flexible stereoscopic viewpoint generation within a depth-image-based rendering (DIBR) system. Exemplary methods in some embodiments for generating virtual viewpoints allow users to more freely vary their head tilt and viewing position without degrading the viewing experience. In addition, in some embodiments, users are provided with natural motion parallax and personal adjustment of stereoscopic vision, enabled by the systems and methods disclosed in some embodiments herein, to create stereoscopic disparity. Some embodiments enable the creation of high-quality stereoscopic content and disparity-based interaction with both wearable and external displays. Some embodiments and means of interaction may be used with emerging MFP displays.

[0064] In some embodiments, MFP-based synthetic disparity is used to produce high-quality stereoscopic content for DIBR. In some embodiments, the number of focal planes is not limited. In addition, creating a stereoscopic view at the receiver allows for flexible functionality. Using an MFP enables virtual viewpoint generation with adjustable disparity and orientation in real-time operation. This methodology enables functionality that might otherwise use more complex and bitrate-intensive methods (e.g., real-time 3D and bright-field systems). The depth-plus-texture input format is applicable to one or more embodiments that may use lower bitrates for transmission and storage.

[0065] In some embodiments, MFPs are used to generate high-quality stereoscopic content with large disparity. Large disparity means maintaining a large stereoscopic depth and / or a small viewing distance to the generated content. The desired accuracy can be achieved by using a large number of MFPs and by selecting and optimizing depth blending functions when generating the MFPs.

[0066] In some embodiments of the disclosed method and system, a virtual stereoscopic projection may be formed in real time at the receiving site, allowing for stereoscopic content to be viewed with maximum head tilt orientation of the stereoscopic baseline. Mismatch and baseline orientation changes may be generated independently at multiple receiving nodes, individually for several users per node. Additionally, some embodiments support generating motion parallax at any viewing orientation, allowing for more natural viewing.

[0067] The systems and methods disclosed herein in some embodiments may be used in creating content for stereoscopic displays, including wearable displays and external screens, using accommodation-invariant (parfocal) NEDs, a variant of stereoscopic display, that support natural accommodation (avoiding VAC) when viewing regular stereoscopic content. In some embodiments, virtual mismatch accommodation and related interactions may also be implemented for emerging MFP displays with natural accommodation. In some embodiments for creating content with large mismatches, using a polynomial blending function (or other function that weights pixels across the entire depth range) may result in better quality on object edges.

[0068] Systems and methods disclosed herein in some embodiments relate to forming stereoscopic image pairs (stereograms) in stereoscopic 3D (S3D) video transmission systems and in systems based on depth image-based rendering (DIBR). Some embodiments may use multi-focal plane (MFP) content, MFP capabilities, and number and location of planes. Some embodiments may use dynamic user behavior or content to reconcile discrepancies.

[0069] While the MFP can provide focal cues, many systems use only two focal planes to allow for disparity (a compromise image between the eyes is shown, creating the appearance of stereoscopic 3D). Many previous systems do not use larger MFP stacks (>2 planes) to perceive continuous depth (from both accommodation and disparity). In many previous stereoscopic viewing systems, a separate MFP stack is provided for each eye for stereoscopic viewing.

[0070] FIG. 2A is a schematic diagram 200 illustrating an example set of focal planes displayed at different depths from the user. FIG. 2B shows an illustration showing an example focal plane image. One system for addressing vergence accommodation conflict (VAC) uses a multi-focal plane (MFP) display to avoid VAC. FIGS. 2A and 2B are adapted from a paper on optimal rendering for MFP displays (see non-patent document 1). In FIG. 2A, a user 202 is shown viewing four presentation planes (e.g., focal planes) 204, 206, 208, and 210. Images 212, 214, 216, and 218 for each of the four planes 204, 206, 208, and 210 are shown in FIG. 2B. Such systems may not be responsive to the viewer. MFPs provide focal cues, but have previously been used with only two focal planes to allow for conflict (a compromise image is illustrated between the eyes, creating a stereoscopic 3D appearance). In many previous stereoscopic viewing systems, a separate MFP stack is provided for each eye for stereoscopic viewing.

[0071] 3 is a graph illustrating an exemplary image mismatch between left and right images versus observer-screen distance for Percival's comfortable viewing zone. FIG. 3 is a graph 300 of Percival's comfortable viewing zone. The graph shows image mismatch 302 between left and right images versus observer-screen distance 304 for viewer comfort based on Percival's comfortable viewing zone theorem. FIG. 3 is shown in the paper (Non-Patent Document 2).

[0072] Figure 4 is a schematic diagram illustrating exemplary left-eye and right-eye images for a stereoscopic display for a viewer with an unrotated orientation. Figure 4 shows left and right images displayed for a near-eye display viewer as seen by a user. Figure 4 illustrates a parallax scenario 400 when viewing a small boat in the distance behind a larger foreground boat. Right-eye image 404 shows a smaller boat 412 to the right of larger boat 410 than left-eye image 402, which shows the top of smaller boat 408 behind larger boat 406. A head icon 414 on the right side of Figure 4 indicates the viewer's original orientation.

[0073] FIG. 5 is a schematic diagram illustrating exemplary left-eye and right-eye images bound in glasses coordinates for a stereoscopic display for a viewer with a rotated orientation in which the horizon is tilted with the head. FIG. 5 shows left and right images displayed for a near-eye display with viewer rotation. In the scenario 500 of FIG. 5, a head icon 506 on the right side of FIG. 5 indicates the viewer's rotated orientation as seen by an external observer standing in front of the viewer. The glasses and near-eye display are illustrated from the perspective of the viewer worn by the viewer. In FIG. 5, the same images 502, 504 shown in FIG. 4 are illustrated regardless of the rotated orientation. This means that 3D perception remains appropriate, but the horizon rotates along the viewer's orientation, disrupting viewing in the physical world.

[0074] Figure 6 is a schematic diagram illustrating exemplary left-eye and right-eye images adjusted to match the orientation of the horizon for a stereoscopic display for a viewer with a rotated orientation and unchanged mismatch. Figure 6 shows left and right images displayed for a near-eye display with the same viewer rotation as shown in Figure 5. In scenario 600 of Figure 6, images 602, 604, with the same original orientation as in Figure 4, are rotated corresponding to the rotation shown for head icon 606 to keep the images horizontal. The mismatch between left image 602 and right image 604 is the same as in Figures 4 and 5 (staying along the horizontal baseline). Although the horizon for the rendering is aligned with the real world, the mismatch does not match the rotated orientation, distorting the 3D perception.

[0075] FIG. 7 is a system diagram illustrating an example set of interfaces for manipulating image mismatch. FIG. 7 is a system interface diagram for manipulating image mismatch, taken from Non-Patent Document 2. Many systems that produce stereo pairs during capture do not support individual user motion (occlusion, variable stereo baselines, non-horizontal orientation, etc.). One system 700 with some user customization of mismatch is described in Non-Patent Document 2, but it is understood that this is limited to variations in rendering the mismatch. Such a system 700 adjusts the mismatch based on user preference. It is understood that such a system provides only spacing adjustment.

[0076] Figure 8 is a system diagram illustrating an example set of interfaces for a DIBR-based 3D transmission system. Figure 8 shows an interface diagram for one configuration of a DIBR-based 3D transmission system 800, adapted from the academic paper (Non-Patent Document 3).

[0077] While many DIBR-based systems enable stereoscopic image formation as a post-processing step at the receiver, many previous stereoscopic video systems have limited flexibility: the amount of stereoscopic disparity and depth budget can be highly dependent on the captured content, and while control of disparity and depth budget can be advantageous in some cases, e.g., after the scene is shot, it can also be difficult due to fixed parameters in the production / front end.

[0078] In many DIBR systems, at the receiving end, virtual viewpoint generation 802 consists of a 3D warping 804 stage and a hole filling 806 stage (see virtual viewpoint generation block 802 in Figure 8). 3D warping 804 is used to create two virtual views into a textured depth map, as seen from the viewer's two eyepoints.

[0079] FIG. 9 is a schematic plan view illustrating an example scenario showing the appearance of disocclusion when changing source and target cameras. Often, the corresponding perspective change reveals portions of the view behind the occluding object. As illustrated by the schematic picture 900 of FIG. 9 and the examples of FIGS. 10A and 10B, disocclusion (i.e., disocclusion region 902) may appear as gaps or holes 904 in each new perspective image because the captured texture or depth does not contain information from those areas. In FIG. 9, when changing viewpoint from source to target as illustrated by the viewpoint lines for source camera 910 and target camera 912 and the respective source image 906 and target image 908, disocclusions related to near object 914 and far object 916 may be caused.

[0080] 10A and 10B are illustrations showing exemplary 3D warped images before and after hole filling. FIGS. 10A and 10B show examples of 3D warped images. The examples show only the left-eye image. The right-eye image is formed by the same means and has a similar defect, but it is on the right side of the object. FIGS. 10A and 10B show an example of disocclusion caused by 3D warping. The original monoscopic image 1000 is on the left, and details of the 3D warped image 1050 are on the right (linear interpolation-based hole filling 1052 for filling the white space hole 1002 shown on the left-side image 1000 as a black line through the use of black and white line art throughout this application). In some embodiments, the method may include processing the focal plane images to address gaps caused by shifting the focal plane images. Some embodiments may filter one or more focal plane images to address such gaps (e.g., to address image mismatches caused by shifting one or more focal plane images). 10A and 10B are line drawing versions adapted from slide 14 of the presentation (according to [4]).

[0081] In hole filling, various image and depth map filtering and inpainting algorithms may be used to fill holes or other forms of disocclusions. The type and number of occlusions are highly content-dependent, as are the distortion and their visibility. 3D warping may be performed by computer graphics algorithms. The complexity, speed, and results of these algorithms vary, depending, for example, on the use of perspective views and camera models. An example mathematical formulation for a shift sensor setup is given in [5].

[0082] The systems and methods disclosed herein by some embodiments are used for virtual viewpoint generation (DIBR), which differs from previous methods for 3D warping and hole filling. The systems and methods disclosed herein by some embodiments may, for example, enable adjusting discrepancies and corresponding stereoscopic depth budgets as post-processing more easily than some previous systems. The systems and methods disclosed herein by some embodiments may, for example, enable supporting stereoscopic viewing at arbitrary viewing orientations (e.g., head tilt) more easily than some previous devices (many previous devices only support horizon viewing). The systems and methods disclosed herein by some embodiments, for example, enable easier creation of virtual viewpoints to support motion parallax (e.g., translational and / or axial). The systems and methods disclosed herein by some embodiments may have more dispersed and less visible distortion than some previous DIBR systems.

[0083] Stereoscopic 3D (S3D) The use of stereoscopic displays is one means of presenting 3D information (sometimes called stereoscopic 3D or S3D). Stereoscopic viewing involves capturing parallel views—a stereo pair—by two cameras separated by a small distance (e.g., a stereo baseline). The capture setup mimics binocular image perception by two human eyes.

[0084] In recent years, stereoscopic systems have seen a resurgence in 3D movies, 3DTV, and augmented and virtual reality applications (AR and VR). Many AR / VR systems use wearable near-eye displays (or glasses).

[0085] In real-world space, the human eye can freely scan and glean information by focusing and accommodating at different distances or depths. When viewing, the eyes' convergence (convergence) is different when looking in a parallel direction (e.g., at an object infinitely far away) than when looking in a cross direction (e.g., at an object close to the eye). In general, convergence and accommodation are often very strongly coupled, so that in most cases, the accommodation / focus and convergence points of the two eyes naturally meet at the same 3D point.

[0086] In many previous stereoscopic viewing systems, the eyes are focused on the same image (or display) plane, but the human visual system (HVS) and brain create the 3D perception by detecting image discrepancies (e.g., small distances between corresponding pixels in two 2D projections).

[0087] 11A-11C are schematic perspective views illustrating exemplary vergence accommodation conflicts (VACs) in many stereoscopic viewing systems. FIGS. 11A-11C are based on Non-Patent Document 6. FIG. 11A illustrates stereoscopic viewing in which the convergence point 1102 is the same as the accommodation point 1102 (which may be typical when viewing with the naked eye). FIGS. 11B and 11C illustrate viewing with a stereoscopic display. FIGS. 11B and 11C illustrate viewing in which the convergence points 1132, 1162 and the accommodation points 1134, 1164 are different, causing a vergence accommodation conflict (VAC). FIG. 11A illustrates natural viewing in a real-world environment 1100 with a vergence distance 1106 equal to the focal length 1108 for a viewer 1110. Figure 11B shows an intersected stereoscopic view 1130 on an S3D display with a vergence distance 1136 that is shorter than the focal length 1138 for a viewer 1140. Figure 11C shows an unintersected stereoscopic view 1160 on an S3D display with a vergence distance 1166 that is longer than the focal length 1168 for a viewer 1170. Although VAC is known to cause visual distortion and other types of discomfort, stereoscopic video is still commonly used for near-eye displays due to its equipment and cost-effectiveness.

[0088] 3D display using multi-focal planes (MFP) Multi-focal plane (MFP) systems allow the viewer to focus on different objects and depths, avoiding the VAC typical of stereoscopic displays. A stack of natural or virtual focal plane images is rendered at different depths. The focal plane being observed is seen in focus, while other focal planes not being observed are blurred by the human visual system.

[0089] An MFP display presents a stack of different focal planes that make up a 3D scene from layers along the viewer's visual axis. The viewer is provided with a 3D view by projecting those pixels of the focal planes that are visible to the viewer's eyepoint at different depths and spatial angles.

[0090] The multiple focal planes are primarily complementary (not additive) to the viewing direction from the eyepoint, however the additive effect can eliminate quantization steps and contouring that would otherwise be perceived multiple times when viewing edited views from different focal planes.

[0091] Multiple focal planes can be displayed, for example, either by spatially multiplexing a stack of 2D displays, or by sequentially switching the focal length of a single 2D display—in a time-multiplexed manner—with a high-speed variable focus element (VFE) while spatially rendering the visible portion of the corresponding multi-focal image frame.

[0092] FIG. 12 is a schematic plan view illustrating an exemplary multi-focal-plane (MFP) near-eye display according to some embodiments. FIG. 12 shows a schematic diagram 1200 of a multi-focal-plane (MFP) near-eye display. Each image 1202, 1204 in a display stack 1208 of a virtual focal plane 1206 is rendered at a different depth, and the viewer's eyes 1210, 1212 blur those focal planes that are not observed through the eyepieces 1214, 1216. A paper (see non-patent document 7) describes viewing content on an external display. Non-patent document 7 describes the phenomenon of two specially shaped focal planes observed by a physical prototype and the resulting perception of 3D when viewed. This paper describes the specific case of an MFP display.

[0093] A larger number of focal planes may have better accuracy for MFP displays, but may also have difficult implementation. For MFP displays, the feasible number of focal planes may be a small number (such as 4 to 6), which may limit the quality achieved. The paper (Non-Patent Document 8) gives information on the number of focal planes.

[0094] Figure 13 is a schematic plan view illustrating exemplary viewing of near-focus, intermediate-focus, and far-focus image planes according to some embodiments. Figure 13 is a plan view schematic of view lines for the left and right eyes for a prototype (described in non-patent document 9). Figure 13 shows a plan view schematic 1300 (adapted from non-patent document 9 and using the exemplary numerical values described therein) that uses multiple MFPs 1302, 1304, and 1306 to capture 3D perception. In the case of Figure 13, each eye 1308, 1310 views the focal planes 1302, 1304, and 1306 from slightly different perspectives, creating a discrepancy in terms of 3D perception. The example of Figure 13 shows a far image plane 1302 of an object at a distance of 53.6 cm (1.87D) from the viewer's eyes 1308, 1310, a middle image plane 1304 of an object at a distance of 39.4 cm (2.54D) from the viewer's eyes 1308, 1310, and a near image plane 1306 of an object at a distance of 31.1 cm (3.21D) from the viewer's eyes 1308. For the example of Figure 13, ±4.4° vertical field of view.

[0095] 3D perception from focal planes is appropriate for flat objects that lie strictly on those planes. By using depth blending, disparities are created between focal planes, creating a continuous and realistic depth perception for objects between the planes. Further details regarding depth blending and the perception of disparities in MFP rendering are given below.

[0096] In previous MFP prototypes, the focal planes were shown on external displays at different distances from the viewer. When showing the MFP on a near-eye display (e.g., Figure 12), two MFP stacks are formed, seen separately by the left and right eyes.

[0097] Figure 14 is a schematic diagram illustrating exemplary focal image planes viewed by an observer. A version of this figure is included in [Publication ID: 7]. Figure 14 shows a top view 1400 and front views 1402, 1404 of two image planes 1406, 1408 viewed by an observer 1410, and a side view 1414 of how the planes are perceived as a 3D image by the observer 1412. In [Publication ID: 7], two focal planes 1406, 1408 are formed, placed at different distances, and viewed from two eyepoints. This setup creates disparity and 3D perception caused by the lateral displacement of the two focal planes viewed from the two eyepoints. The displacement causes a specific pixel value distribution at the object edge, which the brain interprets as disparity and depth variation, respectively. Figure 14 illustrates the perception of a 3D image by viewing two overlapping focal planes formed from a scene. The two eyes view the focal plane from slightly different angles, which creates a synthetic stereoscopic discrepancy between the focal planes.

[0098] 15A-15C are schematic diagrams illustrating exemplary front and back focal planes for an exemplary image of a square. Figures 15A-15C, a version of which is included in Non-Patent Document 7, show synthetic stereo disparity created at the edges of an object. In Non-Patent Document 7, synthetic stereo disparity is created at the edges of squares 1504, 1506 as shown in Figures 15A-15C. In Non-Patent Document 7, an exemplary view 1500 consisting of two focal planes (such as that shown in Figure 14) is visualized by an observer 1502 by capturing a left stereo image pair 1532, 1536 from a left eyepoint 1544 and a right stereo image pair 1534, 1538 from a right eyepoint 1546 on a prototype display (shown in Figures 16A-16B). Figure 15B is a perspective view 1530 of the scene. The retinal images 1540 have a common region 1542 that is seen in each retinal image. The planar schematic 1560 in Figure 15C shows the image luminance distributions perceived from a left eyepoint 1562 and a right eyepoint 1564. The disparity in these luminance distributions (see lateral shift) gives the perception of depth.

[0099] 16A-16B are schematic front views illustrating exemplary focal planes displayed for right-eye and left-eye images. Figures 16A-16B (versions of which are from [Publication ID: 7]) show a cross-eye stereogram of the view in Figure 14 formed by two focal planes. Figure 16A shows a right-eye image 1600, and Figure 16B shows a left-eye image 1650.

[0100] In Non-Patent Document 7, the luminance distribution for a focal plane is computationally achieved by weighting the image pixels (captured projections of the scene) by two linear weighting functions proportional to the pixel's distance (depth coordinate) and its complement. Correspondingly, a feasible image capture format that supports the formation of a focal plane is texture plus depth. Non-Patent Document 7 describes depth fusion or blending effects using two focal planes as seen from two eyepoints.

[0101] Disparity can be used to indicate depth perception. For example, if two focal planes are flat objects, the closer object will be slightly shifted relative to the more distant one as viewed from two eyepoints. Non-Patent Document 7 states that an object between two focal planes is perceived at a position between the two extremes. This phenomenon is caused by depth weighting (or depth blending) used to shape the two focal plane contents. Depth blending shifts the apparent edge location as illustrated in Figures 15A-15C, creating a synthetic yet realistic disparity and depth perception between the two focal planes.

[0102] Figure 17 is a schematic plan view illustrating an exemplary viewing of two focal planes using a depth-based luminosity weighting function. Figure 17 is a plan view schematic 1700 showing the principle of depth blending of two focal planes 1704, 1706 viewed from one eyepoint 1702. Non-Patent Documents 10 and 11 describe how depth blending effects are imparted when two or more focal planes are viewed monocularly by only one eye. Figure 17 is adapted from Non-Patent Document 11. Non-Patent Document 11 describes the perceived luminance of two overlapping pixels on two focal planes in Equation 1. L0=L1(z)+L2(z)=w1(z)L0+w2(z)L0 Formula 1 where w1 and w2 are depth-weighted fusion functions. The perceived depth of the fused pixel

[0103]

number

[0104] can be thought of as the weighted sum of the depths z1, z2 of the two focal planes 1704, 1706 in Equation 2.

[0105]

number

[0106] When viewing two or more MFPs monocularly, the phenomenon is of course not attempting to shift edges to change the mismatch (and depth perception) as described by [7], but rather to eliminate the abrupt step in perceived retinal blur when viewing around the transition between adjacent focal planes, as described by

[10] .

[0107] As described by

[10] , monocular depth blending effects have been shown to work with MFP stacks with more than two focal planes. Leveraging the depth-fused 3D (DFD) phenomenon in MFP displays leads to a variant of the MFP method (described in

[11] and denoted DFD-MFP) in which the focal planes are not strictly separated in angular (and xy) space, but also have some overlapping and additive properties along the depth (z) dimension. This phenomenon is sometimes called depth-based filtering or depth blending.

[0108] FIG. 18 is a schematic plan view illustrating exemplary viewing of three focal planes by the left and right eyes, according to some embodiments. Non-Patent Document 7 describes a method for supporting binocular viewing of a single (monocular) MFP stack despite it consisting of only two focal planes. This idea can be generalized to MFP displays with three or more focal planes. FIG. 18 extends the viewing setup of Non-Patent Document 7 to a stereoscopic viewing setup 1800 for a three-focal-plane display, in which a monocular stack of three MFPs 1802, 1804, 1806 is formed from an average eyepoint 1808, and the same focal planes 1802, 1804, 1806 are viewed by the left and right eyes from slightly different viewpoints 1810, 1812. Following Non-Patent Document 7, research reports and MFP displays may not precisely specify (or be understood to specify) whether the particular device in question displays one or two sets of MFPs to the viewer.

[0109] In the example described herein, according to some embodiments, the viewing situation for an MFP display is that shown in FIG. 18, where one monocular MFP stack is formed using monocular depth plus texture input, and two (stereoscopic) viewpoints are created for this MFP stack. In a near-eye display, however, it is difficult to provide a suitable viewing point for only one focal plane stack, and optics and reflectors may be used to enable the appropriate lateral position for the eyes. Furthermore, this adjustment will naturally differ for people with different eye separations (different viewing angles to the MFP stack). In practice, similar to a stereoscopic near-eye display that illustrates stereoscopic images side-by-side, the one MFP stack needs to be split into two stacks, one for each eye. As in FIG. 12, the viewing condition described by FIG. 18 and FIG. 27B needs to be matched with two parallel MFP stacks from slightly different viewpoints. Both of these two MFP stacks are different from the original monocular stack, whereas the viewing point is between the eyes (see FIG. 18).

[0110] Depth blending functions for forming MFP stacks The captured scene is decomposed into several focal planes (e.g., image layers at various depths or accommodation levels). The focal planes are formed using a depth blending or weighting function, which multiplies each image pixel by a weight depending on its distance from the formed focal plane (e.g., the distance indicated by a depth image or depth map). Two properties of the blending function are unity division and depth blending property. For unity division, the focal plane intensities sum to the original scene intensities. For depth blending property, the same pixel may contribute to multiple focal planes. This property is a result of the continuity and overlap of the blending function. The functions used or referenced herein according to some embodiments are examples and are non-limiting.

[0111] Box filtering (e.g., slicing image information based on pixel depth) is a good starting point for understanding depth blending. However, a box filter is not actually a blending function because each pixel contributes to only one of the focal planes to the slicing result. When using a box filter, an approximation of the focal plane images is formed by slicing the depth map corresponding to each image into narrow depth regions (slices) and projecting the corresponding pixels onto the central (flat) focal plane of each depth region.

[0112] When viewing a stack of focal planes, the resulting view is formed by information about the different focal planes visible from the viewer's eyepoint. In box filtering, slicing into the depth domain results in MFPs completing each other in the spatial (xy) direction, rather than summing along the depth dimension (z). Correspondingly, the focal planes generated by the box filter do not create a discrepancy between focal planes (the perception of 3D), but only for objects on the focal planes.

[0113] Figure 19 is a schematic plan view illustrating an example quantization of depth for an MFP display without depth blending (weighting), according to some embodiments. As illustrated by Figure 19, resulting from the use of a box filter, smooth 3D surfaces are quantized in the depth dimension. Figure 19 shows a schematic example 1900 of depth quantization when describing a view with five focal planes 1902, 1904, 1906, 1908, 1910. Arrow 1912 indicates the viewing direction.

[0114] The box filter separates image information in a rigorous manner in the spatial (xy) and depth (z-) dimensions. In practice, since only a small number of focal planes are used, the depth dimension is heavily quantized, which can lead to poor accuracy in presenting 3D shape and depth.

[0115] In some embodiments, the processing of the focal plane images may include determining depth blending weights for one or more of the plurality of focal plane images. The processing of the focal plane images may also include adjusting the focal plane images using the depth blending weights. For example, depth dimension values may be multiplied by the depth blending weights.

[0116] Depth-based blending may be used to reduce quantization errors in the depth dimension that would otherwise be visible to the human eye. Depth blending refers to using a depth-based function to weight the pixels used to construct each focal plane. One depth blending function is a tent filter, which is a piecewise linear, sawtooth blending function 2050 (FIG. 20B). For a box filter, the corresponding function 2000 is shown in FIG. 20A.

[0117] 20A and 20B are graphs illustrating an example of weight versus depth according to some embodiments, showing schematic diagrams of depth blending functions for four MFPs. FIG. 20A shows depth slicing without blending (box filter), and FIG. 20B shows a linear (tent) filter. In some embodiments, the depth values range between 0 and 255 (z in FIGS. 20A and 20B). min and z max (See

[0118] FIG. 21A is a graph illustrating an example blending function of weight versus depth according to some embodiments. FIG. 21B is an illustration showing an example test image. FIG. 21C is an example depth map illustrating pixel distance for FIG. 21B. FIGS. 21D-21F are illustrations showing example focal plane images using the depth weights of FIG. 21A with the test image of FIG. 21B and its example depth map of FIG. 21C according to some embodiments. FIGS. 21B and 22B are based on an image found on a webpage (Non-Patent Document 12), and FIGS. 21D-21F are modified versions of that image. The example unprocessed test image and its depth map shown in FIGS. 21B and 21C are merely example generic test images of flowers, correspondingly used for illustrative purposes many times throughout this disclosure. In many cases, modified versions of example test images are presented, e.g., from example processes performed on test images according to some embodiments. In some embodiments, a polynomial blending function extends across the entire depth range. Figure 21A shows three polynomial blending functions 2100, 2102, and 2104. Figure 21A and Figures 21B-21F show examples of the three polynomial blending functions 2100, 2102, and 2104 (Figure 21A) and the corresponding focal planes (Figures 21D-21F) for a test image 2110 (Figure 21B) and its corresponding depth map 2120 (Figure 21C).

[0119] In some embodiments, a depth map (e.g., FIG. 21C) may be generated, which may be similar to that shown in FIG. 19 by the smooth thin line 1914. The depth map is used to separate the input image, for example, into small, medium, and large focal plane distances from the viewer. A depth blending function (e.g., FIG. 21A) is used to generate a focal plane image for a particular focal plane distance. This process may be repeated for each focal plane image (e.g., FIGS. 21D through 21F). In some embodiments, the depth map (see FIG. 21C) may be generated by several methods, including depth from focus, depth from one or more RGB images, and depth from an RGB-D sensor (using structured light or ToF). It will be understood that there are various known means of acquiring a depth image.

[0120] Figure 22A is a graph illustrating an example blending function of weights versus depth according to some embodiments. Figure 22B is an illustration showing an example test image. Figure 22C is an example depth map illustrating pixel distance for Figure 22B. Figures 22D-22F are illustrations showing example focal plane images using the depth weights of Figure 22A along with the test image of Figure 22B and its example depth map of Figure 22C according to some embodiments. The example depth maps of Figures 22B and 22C are derived from an image found on a webpage (Non-Patent Document 12), and Figures 22D-22F are modified versions of that image. Figure 22A shows weights for three sinusoidal blending functions. 22A shows an example of three sinusoidal functions 2200, 2202, 2204 that result in three MFPs 2230, 2240, 2250 (FIGS. 22D-22F) for a test pattern image 2210 (FIG. 22B) and its depth map 2220 (FIG. 22C). Another variation of the blending filter is proposed by

[11] .

[0121] Depth blending or fusion, as explained by [7], is a means of increasing the apparent number of focal planes and therefore depth accuracy.

[10] extended the synthesis mismatch findings to cases with three or more focal planes, increasing the effective number of depth planes from the few allowed by practical MFP display implementations.

[10] further reported the benefits of depth blending when viewing a monocular MFP stack with one eye. According to

[10] , depth blending filters the gradual changes in focal length perceived by one eye at the transitions between sparse focal planes.

[0122] The creation and perception of synthetic disparity (depth) between two focal planes is described in Non-Patent Document 7. The creation of depth perception between three or more focal planes is described in Non-Patent Document 10 and Non-Patent Document 9. Filtering of gradual accommodation changes between focal planes is described in Non-Patent Document 10. Non-Patent Document 11 describes variations on depth blending functions.

[0123] Wearable glasses-based near-eye displays (NEDs) are popular in virtual reality and gaming, and more recently in augmented reality. Glasses that use a mobile phone as the display element are an example of a popular, low-cost device for viewing virtual or camera-captured images. Many near-eye displays (NEDs) present a sequence of stereoscopic image pairs. Stereoscopic rendering can also be used with external displays, including those for stereoscopic 3D (S3D) TVs.

[0124] Capturing stereoscopic video (a sequence of stereoscopic image pairs) can pose various limitations: capturing and transmitting stereoscopic content can, for example, prevent post-processing adjustments for mismatches and other important S3D parameters, both at content production time and when the content is rendered or viewed on a receiver.

[0125] The video-plus-depth (depth-plus-texture) format in capture and transmission is used in depth image-based rendering (DIBR). Much research has been done in the field, focusing on optimizing the rendering quality at the receiver, and more specifically, on removing disocclusions induced in DIBR systems when generating stereoscopic views from two virtual viewpoints.

[0126] MFP displays are based on the 3D perception of multiple focal planes, which can be formed using a depth-plus-texture format. Many MFPs can avoid VAC, which can occur for stereoscopic displays. For technical reasons, the maximum number of focal planes is generally limited to only a few in an MFP display. When using a few focal planes, rendering quality can be improved by depth blending, but the number of focal planes can still present a limitation on MFP rendering quality.

[0127] Many previous systems for both DIBR and MFP lack support for user motion-based interaction, which requires adjusting the viewpoint to either changing stereo disparity or motion parallax. For many DIBR-based systems, one reason is the computational complexity of the methods used in virtual viewpoint generation (e.g., 3D warping and hole filling).

[0128] Some embodiments use a depth-plus-texture content format in image capture and transmission. Some embodiments of viewpoint generation differ from the 3D warping and hole-filling methods used in previous depth image-based rendering (DIBR) methods. In some embodiments, a multi-focal plane (MFP) is formed within the receiver using depth and texture data. Stereoscopic content with a set of properties and supporting motion parallax is generated in real time using the formed MFP for multiple receivers individually at any baseline orientation.

[0129] Exemplary Processes and Structures An MFP-based method generates a stereoscopic viewpoint for a DIBR-based system. An exemplary method generates a virtual viewpoint to allow a user to vary their head tilt and viewing position. In addition, the user is provided with natural motion parallax and personal adjustment of stereoscopic vision, enabled by an exemplary method for creating stereoscopic disparity. The user's movements are tracked (e.g., by one or more head-mounted or external sensors) to generate disparities that adapt to the user's posture and movements.

[0130] Some embodiments enable the creation of stereoscopic content and disparity-based interactions for both wearable and external displays. Example implementations and means of interaction described herein, according to some embodiments, may also be used with MFP displays. Some embodiments use a depth-plus-texture type content format for capture and transmission, as well as a multi-focal-plane (MFP) formed therein within the receiver. In some embodiments, using data received once, stereoscopic content with a set of properties can be formed in real time for multiple local viewers, individually, at any baseline orientation. Formation of the MFP may be accomplished by several means, examples of which are disclosed herein according to some embodiments.

[0131] 23A-23C are schematic front views illustrating an example scenario for stereoscopic image disparity, according to some embodiments. FIGS. 23A-23C show three (binocular) viewpoints from which a user views stereoscopic content. FIG. 23A is an example 2300 illustrating the use of viewing depth effects while lying down. FIG. 23B is an example 2330 illustrating a more widely adjusted depth effect. FIG. 23C is an example 2360 in which the viewpoint follows the user's movement. The user views the content from different orientations and with varying amounts of stereoscopic disparity. The content may be viewed, for example, using either a wearable display or an external display. In some embodiments, the 3D perception is based on an approximated (e.g., virtual) disparity between the user viewing a monocular stack of multiple focal planes from two eye viewpoints.

[0132] Many stereoscopic 360° video systems respond only to orientation changes, not to user parallax movements. Many previous systems use orientation (pure orientation changes) to control the 360° video. Head rotation also imposes a translation of eye position. Avoiding VAC, for example, may provide comfortable stereoscopic viewing from a fixed position.

[0133] In some embodiments, virtual viewpoint generation for forming a stereoscopic image is performed by shifting and adding MFPs. Stereoscopic displays may be used to view the results, including near-eye S3D screens and external S3D screens. When using time-shared S3D displays, shutter glasses worn by the user may be equipped with sensors to track user movement to support motion parallax effects. To avoid VAC associated with stereoscopic displays, the stereoscopic display may be replaced by a display that supports natural accommodation, for example.

[0134] In some embodiments, a method may include receiving three-dimensional video content, calculating one or more multi-focal plane (MFP) images for a number and positions of multi-focal planes using the three-dimensional video content, measuring motion tracking sensor readings of a motion tracking sensor, generating a viewer orientation signal from the motion tracking sensor readings, calculating one or more disparity images (e.g., MFP stacks distorted by an amount indicated by a disparity vector, projected, and summed to a viewing point) for each view of a shutter glasses display using the one or more MFP images and the viewer orientation signal, and rendering the one or more disparity images to the shutter glasses display.

[0135] In some embodiments, a device may include one or more motion tracking sensors; a processor; and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform the following processes: receive three-dimensional video content; calculate one or more multi-focal plane (MFP) images for a number and positions of multi-focal planes using the three-dimensional video content; measure motion tracking sensor readings of the motion tracking sensors (e.g., one or more motion tracking sensors); generate a viewer orientation signal from the motion tracking sensor readings; calculate one or more mismatch images for each view of a shutter glasses display using the one or more MFP images and the viewer orientation signal; and render the one or more mismatch images on the shutter glasses display. For example, in some embodiments, measuring motion tracking sensor readings of a viewer's position relative to a real-world environment may include, for example, measuring motion tracking sensor readings of the viewer's position relative to a physical stationary point.

[0136] Inconsistent changes due to user movement 24A-24C are schematic front views illustrating an example scenario for image mismatch with the direction of viewer orientation, according to some embodiments. Stereoscopic mismatch adjustment can be performed through user motion and motion-related interactions. FIGS. 24A-24C show mismatch-related viewpoint changes due to user motion supported by some embodiments. As illustrated by FIGS. 24A-24C, the amount of synthetic mismatch (e.g., length, height, or width) 2400 ( FIG. 24A ), orientation (e.g., direction) 2430 ( FIG. 24B ), and position (e.g., lateral translation) 2460 (motion parallax) ( FIG. 24C ) can be adjusted, for example, by any combination of these user motions. The mismatch (amount of head tilt and lateral translation) can be adjusted based on tracking data from the user's eyewear or their manual input (e.g., indicating a personal preference for the amount of mismatch). The three mismatch-associated interactions illustrated in Figures 24A-24C, for example, may occur in any order and may form a continuous sequence.

[0137] 24A-24C illustrate mismatches created by some embodiments and expressed via bipolar vectors 2402, 2404, 2432, 2434, 2462, 2464 whose quantity (e.g., length, height, or width), orientation, and position relative to the origin (to support motion parallax effects) can be adjusted. The old mismatch values are indicated by dashed arrows 2402, 2432, 2462, and the new mismatch values are indicated by solid arrows 2404, 2434, 2464.

[0138] The disparity of a stereoscopic image pair (for the left and right eyes) may be represented by a tuple of vectors ("bipolar vectors") pointing in two opposite directions along the baseline between the eyes. The amount of disparity is given by the magnitude of the difference between the two vectors (Figure 24A). Head tilt is indicated by the angle between the disparity vector and the horizon (or a vector parallel to the horizon) (Figure 24B). Motion parallax (e.g., a momentary shifted viewpoint) may be represented by the asymmetry of the two vectors, and viewpoint is indicated by their average (Figure 24C).

[0139] The amount of mismatch is adjusted by scaling the mismatch vector (multiplying it by a scalar), the direction is adjusted by rotating the vector (or, for example, by modifying a numerical expression for the amount of rotation), and the translation is performed by a parallel shift of the vector. Instead of communicating the captured user movements between the system components (between the glasses and the system terminal) as motion vectors, the component interfaces may be based on modifying the mismatch vectors based on the captured user movements before transmitting them.

[0140] In some embodiments, shifting the focal plane images may include scaling the mismatch vector for each focal plane image (or, for example, by modifying a numerical representation of the amount of mismatch adjustment due to the shift). For example, scaling may be performed as shown in FIG. 24A to widen (or shrink) the horizontal (or vertical) dimension. In some embodiments, shifting the focal plane images may include translating each focal plane image. For example, translating the focal plane images may include rotating the focal plane image or translating the focal plane image horizontally (or vertically) by an offset relative to a fixed origin.

[0141] FIG. 25 is a system diagram illustrating an example set of processing blocks and interfaces for generating shifted and projected MFPs for the left and right eyes, according to some embodiments. The disclosed example method is a new example implementation of a virtual viewpoint generation stage in DIBR, according to some embodiments. DIBR was discussed above with respect to the virtual viewpoint generation stage illustrated in FIG. 8 (adapted from non-patent document 3). In FIG. 25, a block diagram 2500 for an example DIBR-based implementation according to some embodiments is shown. Depth map pre-processing 2506 may include a filtering operation on the depth map 2502. One operation is to smooth the depth map 2502 to reduce holes or maximum created discrepancies (e.g., for viewing comfort), as described in non-patent document 13.

[0142] Some embodiments for generating 2508 the MFP may be to use a blending function to generate a weight map (remapped depth map) for each MFP and form the MFP by multiplying each of the formed weight maps with the captured (texture 2504) image. Some embodiments may post-process 2510 the MFP before shifting and projecting 2512 the MFP. In some embodiments, filtering may be used to reduce holes when the MFP is shifted for large viewpoint changes or mismatches. In some embodiments, the MFP may be received as input. In some embodiments, post-processing 2510 the MFP may include forming the MFP (e.g., via redistribution).

[0143] DIBR-based methods according to some embodiments may include hole-filling and other inpainting methods to reduce disocclusion artifacts in 3D warped images. In some disclosed embodiments, focal plane shifts may produce similar distortion results, and equivalent processing (e.g., inpainting in either the depth or spatial dimensions) may be used on MFPs after they are formed by depth blending. Shifting the MFP and projecting it onto stereoscopic images 2516, 2518 2512 may be used to meet a desired stereoscopic disparity (or depth budget). Shifting the MFP may be performed along any chosen baseline orientation (e.g., not just horizontal) 2514 to support arbitrary head tilt (data from an IMU (inertial measurement unit) or similar in the NED). Additionally, motion parallax may be supported based on the user's head movement.

[0144] Note that forming one set of MFPs can serve both various baseline orientations, amounts of mismatch, and synthetic motion parallax for one or more local viewers (e.g., based on illustrating a series of stereograms with chancing viewpoints). A single derived MFP stack can serve all mentioned cases. The direction and amount of shift, as well as the display (or, e.g., person) to which the results are rendered, vary.

[0145] Figure 26 is a system diagram illustrating an example set of processing blocks and interfaces for generating left-eye and right-eye shifted and projected MFPs for three scenarios of viewer orientation, according to some embodiments. Figure 26 shows a process 2600 in which the generated MFP stack can be used, for example, for several local viewers 2602, 2604, 2606 with different disparities, orientations, and motion parallax. For example, a receiving terminal 2628 may receive a depth map 2630 and preprocess 2626 the depth map. A texture 2632 may be received by the receiving terminal 2628 to generate 2624 an MFP using the preprocessed depth map output. Some embodiments may post-process 2622 the MFP. The MFP may be shifted and projected 2616, 2618, 2620 for the left and right views for each viewer 2602, 2604, 2606. Head pose and movement data (NED with IMU or similar) 2608, 2610, 2612 may be received by a multiplexer and wireless access point 2614. The movement data may be used in shifting and projecting 2616, 2618, 2620 of the MFP.

[0146] Some embodiments serve multiple people individually per site, with some interaction features. Note that instead of sending motion data from the glasses to the receiving terminal (see the "Head Pose and Movement" block in FIG. 26), in some embodiments the data may be a mismatch vector derived from the motion data or provided as manual input (e.g., to determine a preferred mismatch).

[0147] Some embodiments incorporate mismatch adjustment by creating a visually natural mismatch using the MFP display when a single MFP stack is viewed from an offset position. Some embodiments incorporate mismatch adjustment through feedback from the viewer to the TV / HMD renderer to produce a mismatch-adjusted image. Orientation changes can be handled with glasses-based TVs and with 360° video HMD clients.

[0148] In some embodiments, orienting one or more focal plane images may include shifting the respective focal plane images. Such shifting may be performed to address motion parallax, for example, as shown in FIG. 26.

[0149] Forming stereo pairs from the focal plane 27A-27D are schematic plan views illustrating exemplary focal planes viewed under different viewer orientations, according to some embodiments. In addition to forming a stack of MFPs, a stereoscopic image pair (stereogram) is formed from the focal planes. A monocular stack of MFPs (FIG. 27A) is placed at a desired distance from the viewer's (mean) eyepoint 2702 and viewed by two eyes 2722 (FIG. 27B).

[0150] FIG. 27A shows a plan view illustration 2700 of a monocular MFP stack formed by depth blending. FIG. 27B shows a plan view illustration 2720 for viewing the MFP stack with two eyes 2722. FIG. 27C shows a plan view illustration 2740 for viewing a focal plane with the left eye 2742, so that the focal plane appears to shift to the right. FIG. 27D shows a plan view illustration 2760 for viewing a focal plane with the right eye 2762, so that the focal plane appears to shift to the left. Correspondingly, combining two (stereoscopic) MFP stacks from a monocular MFP stack for stereoscopic (or stereoscopic MFP) viewing can be done by shifting the MFPs in a certain direction and by a certain amount. The MFP shift may be an approximation of a more complex transformation due to a virtual viewpoint change (see 3D warping in DIBR systems).

[0151] In Figures 27C and 27D, the left and right eyes view the stack from slightly different perspectives. The apparent lateral shift of the focal planes in Figures 27C and 27D depends linearly on the distance of the focal planes. If the farthest plane is seen at the same position by both eyes (placed far from the eyes), the middle plane is shifted laterally by "1 unit" and the closest plane is shifted laterally by "2 units."

[0152] In some embodiments, the laterally shifted focal planes are collapsed by projecting them onto each eyepoint. When projecting, corresponding pixels on the focal planes may be summed along the depth axis. This may generally result in a stereo image pair of two images with different stereo disparities or parallax. When starting with a monocular MFP stack, the disparities may be interpreted as somewhat artificial or synthetic.

[0153] An example of a stereogram formed using MFP FIG. 28A is an illustration showing an exemplary test image. FIGS. 28B-28D are illustrations showing exemplary weighted focal plane images using the test image of FIG. 28A, according to some embodiments. FIG. 28A is derived from an image found on a webpage (Non-Patent Document 12), and FIGS. 28B-28D are modified versions of that image. Some embodiments may be illustrated using an exemplary set of three focal planes formed using a polynomial blending function (e.g., FIG. 21A). FIGS. 28B-28D show an exemplary test image 2800 (FIG. 28A) decomposed into three MFPs 2810, 2820, and 2830 using a polynomial blending function. Crossed-eye stereograms are used to illustrate the quality of the MFP stacks and stereo pairs. 29A and 29B show stereoscopic image pairs 2900, 2950 (for the right and left eyes, respectively) formed using the MFPs 2810, 2820, 2830 shown in FIGS. 28B-28D.

[0154] 29A and 29B are illustrations showing corresponding exemplary right-eye and left-eye images formed using the focal plane images of FIGS. 28B-28D, according to some embodiments. FIGS. 29A and 29B are examples of crossed-eye stereograms formed using three MFPs. The total mismatch corresponds to 2.76% of the image width (see MFPs shifted up to 6 pixels using the particular test image of FIG. 28A). The amount of shift in pixels depends on the resolution of the test image used. Unlike many MFP displays whose rendering precision is limited by the number of focal planes, in some embodiments, the precision and number of MFPs are similarly unlimited. Forming, shifting, and summing MFPs for a stereoscopic image may be performed with a greater number of focal planes than the mere three-focal plane example used to form FIGS. 29A and 29B.

[0155] Applying focal planes when creating stereoscopic content Similar to creating stereoscopic content, different accommodation distances may be chosen for the zero parallax setting (ZPS). In the real world, the ZPS plane is at infinity (far away from the viewer's viewpoint). In such a setup, the induced disparity is in front of the farthest plane (commonly interpreted as a negative disparity, although the sign convention is sometimes reversed). In many cases, for various reasons, it may be more beneficial to use a distance closer to the ZPS. Closer distances are used for many screens for stereoscopic displays.

[0156] 30A-30C are schematic plan views illustrating exemplary focal planes viewed under different eye separation distances, according to some embodiments. Figures 30A-30C show three plan view schematics 3000, 3030, 3060 illustrating that adjusting the length of the stereo baseline (e.g., eye separation or interpupillary distance (IPD)) 3002, 3032, 3062 can reduce the edge effect of overlapping / underlying focal planes. In Figures 30A-30C, the total disparity (e.g., amount of stereo depth and location of the MFP) and rendering distance remain the same. In Figures 30A-30C, this applies to three different viewers with correspondingly reduced interpupillary distances (IPD) / interocular distances (IOD). In fact, referring to Figures 30A-30C, individual variations of the IOD can affect perceived depth in a similar manner as selecting a ZPS-location relative to the rear (e.g., Figure 30A), middle (e.g., Figure 30B), or front (e.g., Figure 30C).

[0157] The interocular adjustment illustrated in Figures 30A-30C corresponds to choosing a convergence point / focal plane (e.g., also called a zero parallax setting, or ZPS) for the content. The focal plane at the ZPS divides the total disparity range (e.g., depth budget) into negative (near) disparity and positive (far) disparity. In Figure 30A, all content appears to be in front of the screen (due to all negative disparities 3004, 3006). In Figure 30B, content appears to be on either side of the screen (due to both negative disparity 3034 and positive disparity 3064). In Figure 30C, content appears to be behind the scene (due to only having positive disparities 3064, 3066). Note that the ZPS does not affect the perceived depth range or disparity range itself.

[0158] In some embodiments, to minimize effects caused by only partial overlay of the MFP on the focal plane edges, it is beneficial for the focal plane to be aligned with the centermost plane (FIG. 30B). With depth blending, the focal planes and their stereoscopic projections create a continuous depth perception for the image area and for objects between the focal planes.

[0159] The number of MFPs is a parameter that can affect the perceived quality. In the case of synthetic stereoscopic vision, the number of focal planes is not limited by display properties (such as transparency and multiplexing speed). In some embodiments, the number of focal planes can be chosen to be high enough to meet any desired depth accuracy. The computational complexity increases linearly with the number of focal planes, but this may not be an issue with a relatively large number of MFPs.

[0160] For the same reason, there may be less benefit in optimizing the focal plane positions. Shifting the focal planes for the left and right eyes is easier with a normal focal plane distance (e.g., it can be performed without sub-pixel operation), so equal displacement distances (e.g., intervals) may be used. Instead of optimizing the position by refractive interval, the number of MFPs with normal intervals may be increased. An odd number of focal planes may be used, with the most central plane at the alignment / pivot position.

[0161] The depth map of the captured scene may be relative and normalized. The depth range may be relative, not locked to real-world coordinates, or system variable. In some embodiments, the stereoscopic depth of the scene is determined by the amount of lateral shift used to form the (pseudo-)stereoscopic image pair.

[0162] A larger number of MFPs may correlate to greater synthetic disparity and stereoscopic depth. This process also depends on the image content. In scenes with only a few objects with large depth separation, depth blending over a large depth range may be beneficial instead of blending over a more compact range.

[0163] Formation of synthetic stereopairs A stereoscopic image pair is formed from the MFP by summing the projections from the focal plane toward each eye. This process can be viewed with stereoscopic glasses, resulting in a stereoscopic pair that can induce VAC, or an MFP version of stereoscopic glasses that can avoid VAC. The maximum disparity or parallax in the stereogram thus formed is the same as that observed for stereoscopic content. This maximum parallax recommended in [5], also called the depth range or (stereo) budget, depends on the content but is around 3% of the image width. After this limit, as explained in [5], the stereoscopic effect tends to break down, meaning that the (subjective) quality of the stereoscopic content deteriorates.

[0164] For example, in Figure 13, the maximum disparity / disparity is about 2.5%, below the recommended maximum of 3% from Non-Patent Document 5. Figure 13 shows that the corresponding maximum field of view for stereoscopic viewing is relatively small, about 20 degrees horizontally (compared to about 70 degrees for eye rotation without head movement). In the vertical dimension, the maximum field of view is up to 55 degrees without head rotation. In the vertical direction, the stereoscopic (or any other) display is smaller than what is allowed by the eyes.

[0165] Depending on the source, different recommendations are given for allocating the depth budget, which is divided into depths in front of and behind the stereoscopic display screen (negative and positive parallax). In addition to the content itself, this allocation can be affected by the screen size, which varies depending on the source.

[0166] According to Non-Patent Document 14, crossed discrepancies (for objects closer than the screen) should not exceed 2-3% of the screen width, and non-crossed discrepancies (for objects farther than the screen) should not exceed 1-2% of the screen width.

[0167] In some embodiments, limits or recommendations, such as those identified by, for example, non-patent literature 5 and non-patent literature 14, may be applied as design considerations and compensation may be provided.

[0168] Examples with large discrepancies Below, further examples with sinusoidal and polynomial blending functions are given. To make the printing distortion easier to see, a large total mismatch of 4.6% is chosen for these examples. This is much larger than the recommended maximum of 3% in [5]. The effect is illustrated in Figures 31A-31F.

[0169] FIG. 31A is an illustration showing an enlarged excerpt of an exemplary stereoscopic image for the right eye formed with sinusoidal depth weighting, according to some embodiments. FIGS. 31B-31C are illustrations showing corresponding exemplary stereoscopic images for the right and left eyes formed with sinusoidal depth weighting, according to some embodiments. FIG. 31D is an illustration showing enlarged excerpts of an exemplary stereoscopic image for the right eye formed with polynomial depth weighting, according to some embodiments. FIGS. 31E-31F are illustrations showing corresponding exemplary right-eye and left-eye images formed with polynomial depth weighting, according to some embodiments. FIGS. 31B-31C and 31E-31F are modified versions of the unprocessed flower test image, e.g., of FIG. 21B, by processing the test image using its depth map, FIG. 21C, according to some embodiments. Figures 31B-31C and 31E-31F show a comparison of crossed-eye stereograms with a large composite discrepancy (±2.3%) produced by shifting and summing three MFPs and formed using either sinusoidal and polynomial blending functions (see corresponding upper stereograms 3110, 3120 in Figures 31B and 31C and lower stereograms 3140, 3150 in Figures 31E and 31F). Left insets 3100 and 3130 (Figures 31A and 31D) show details of the corresponding right-eye images 3110, 3140 using both methods. The insets demonstrate the benefit of the polynomial function on object edges, which reduces disocclusion caused by the large discrepancy. The benefit may come from stronger depth blending with the polynomial function over the sinusoidal function.

[0170] In the case of the polynomial function, background image details are slightly blurrier. This is caused by a stronger spatial filtering influence that filters the result when the shifted focal plane components are summed to form the two final viewpoints. In the case of large mismatches, the polynomial blending function has a benefit to object edges. The benefit comes from a stronger blending inpainting influence in the depth direction, which reduces the visible disocclusion caused by large mismatches.

[0171] Free baseline solid support Images, videos, and virtual content can be viewed using stereoscopic VR glasses. These services may only be able to provide stereoscopic disparity in the horizontal dimension (when viewed with the head upright). To avoid interfering with the viewer's perception of their own pose and balance, the horizon of the viewed content is generally locked to the viewer's real-world horizon. This is done by sensing the user's head tilt with position sensors in the viewing device (e.g., a cell phone in "cardboard" VR glasses).

[0172] In previous systems, despite orienting content according to the user's head tilt, the content disparity generally does not track the user's head tilt but is horizontal regardless of head orientation. Increasing head tilt distorts, weakens, or even loses the horizontal disparity-based 3D perception. The systems and methods disclosed herein according to some embodiments may be used to provide stereoscopic content independent of head orientation. Stereoscopic (pseudo) disparity may be formed in real time at the receiving end according to the measured head tilt of the viewer. In some embodiments, if the viewer orientation signal indicates viewer tilt, a disparity image may be calculated using a rotated disparity vector corresponding to the indicated viewer tilt.

[0173] 32A-32C are schematic front views illustrating exemplary mismatches for a first stereoscopic display with different user orientations, according to some embodiments. 32D-32F are schematic front views illustrating exemplary mismatches for a second stereoscopic display with different user orientations, according to some embodiments. 32A-32F illustrate some of the differences between previous systems and the system disclosed herein according to some embodiments. In FIGS. 32A-32F, stereoscopic image pairs 3200, 3210, 3220, 3230, 3240, and 3250 are depicted as they are shown with VR glasses using a mobile phone as the display. Optimizing the image geometry for viewing on a flat display near the eyes results in stereoscopic images with a characteristic pincushion shape.

[0174] Figures 32A-32F show two methods for showing stereoscopic content on a near-eye display (e.g., a mobile phone). Figures 32A-32C show the content tilted to follow the viewer's head pose, resulting in a discrepancy between the real-world horizon and the one shown. This discrepancy can cause simulator sickness and disorientation. Figures 32D-32F show a method, in some embodiments, for aligning the content's horizon with the real-world horizon by tracking the user's head pose. A disadvantage of this system is that the baselines of the stereoscopic pair are misaligned, leading to an erroneous or lost perception of depth.

[0175] 33A-33C are schematic front views illustrating exemplary mismatches for stereoscopic displays with different user orientations, according to some embodiments. In some embodiments, such as those illustrated by FIGS. 33A-33C, stereoscopic pairs 3300, 3330, 3360 with appropriate baselines and mismatches may be formed in real time following changes in the user's head pose. Supported interactions may also include motion parallax. Note that head tilt and movement are captured in the virtual viewpoint generation (see FIGS. 25 and 26) to modify the mismatches accordingly.

[0176] Figure 34 is a schematic user's diagram illustrating exemplary left-eye and right-eye images for a stereoscopic display for a viewer with a rotated orientation, according to some embodiments. Figure 34 is a front view of left and right images displayed for a near-eye display viewer. A head icon 3414 on the right side of Figure 34 indicates the rotation of the viewer's orientation. The glasses on the left side of Figure 34 are shown from the perspective of a viewer worn by the viewer. After the viewer's head rotation, the right-eye image 3404 is viewed higher, from above, and the left-eye image 3402 is viewed from below. The same image shown in its original orientation in Figure 5 is rotated for scenario 3400 of Figure 34 to keep the image horizontal. The mismatch between the left and right images is changed by the rotated orientation (the location of the small boats shown in FIG. 5 is changed so that FIG. 34 shows a change in both vertical and horizontal mismatch such that small boat 3408 is closer to large boat 3406 in the left eye image 3402 compared to FIG. 5, and small boat 3412 is farther away from large boat 3410 in the right eye image 3404 compared to FIG. 5).

[0177] In some embodiments, orienting one or more focal plane images may include rotating the respective focal plane images. For example, one or more focal plane images may be rotated to show images relative to left eye image 3402 and right eye image 3404 shown in FIG. 34. In some embodiments, rotating the focal plane images may include rotating a mismatch vector for one or more of the focal plane images relative to a vector parallel to the horizon.

[0178] Motion parallax support Focal plane shifts based on synthetic disparity at any baseline orientation (e.g., head tilt) may also be used to support small amounts of motion parallax for the left and right eyes. This embodiment uses a large number of focal planes, and the span of disparity may be greatly extended so that a wide range of head positions (motion parallax) can be supported within that range.

[0179] FIG. 35 is a schematic processing diagram illustrating an exemplary process for generating multiple perspectives by warping a set of focal plane images left and right, according to some embodiments. FIG. 35 shows various modified versions of the unprocessed flower test image, e.g., of FIG. 21B, using its depth map, e.g., of FIG. 21C, according to some embodiments. In the example of FIG. 35, five (e.g., sinusoidal) MFPs and a maximum disparity / lateral shift (for either eye) of 12 units are used. Seven multi-view images 3502, 3504, 3506, 3508, 3510, 3512, and 3514 are formed 3516 for a chosen disparity / disparity range. Each of frames 3502-3514 is formed by first warping the MFP stack to varying degrees, then projecting and summing the MFPs toward the viewer. The original zero-disparity image is also put into a series. The multi-view image includes a series of cross-eye stereograms, where the right eye's viewpoint is on the left, and vice versa. By sampling every other frame within the multi-view series of Figure 35, five stereogram pairs 3502 / 3506, 3504 / 3508, 3506 / 3510, 3508 / 3512, and 3510 / 3514 may be formed, differing by 8 units of disparity / lateral shift. Correspondingly, linearly varying stereo disparity (with 8 units of stereo discrepancy) may be supported at five positions. Figure 35 shows an example 3500 for forming a multi-view image using synthetic discrepancy. The maximum discrepancy range for each eye (e.g., the maximum lateral shift range of the five MFP stacks) is 12 units (±2.75% of the test image width). For example, the MFP stack may be skewed to the right 3518 (or, e.g., MFP shifted), and the MFP stack may be skewed to the left 3520 (or, e.g., MFP shifted). The stack of MFPs is skewed (or, e.g., MFP shifted) by various amounts. The result is a stack of MFPs that is tilted to the left or right, depending on the shift direction.

[0180] Separating multiview images into stereo pairs is more convenient for perceiving parallax effects from printing. A compromise between motion parallax effects and perceived stereo disparity (depth) can be achieved by using either 1) two adjacent images at six positions (total disparity 0.46%, both positive and negative disparities 0.23%), 2) every other image at five positions (total disparity 0.92%), 3) every third image at four positions (total disparity 1.38%), 4) every third image at three positions (total disparity 1.84%), or every fourth image at two positions (total disparity 2.3%). A stereo pair with two end images provides a total maximum disparity of 2.76% without motion parallax.

[0181] Figure 36 is an illustration showing an exemplary series of three stereoscopic images for a stereoscopic display using the distorted images of Figure 35 (each of which is an MFP stack that is first distorted to various degrees and then projected and summed toward the viewer), according to some embodiments. Figure 36 shows three samples 3602 / 3604, 3606 / 3608, and 3610 / 3612 from the series formed by gleaning every other image (cross-eye stereo pairs with a 0.92% discrepancy). While the motion parallax effect from printing (3.7% overall) is somewhat difficult to perceive, it is evident when animations are formed using the multiview images of Figure 35. Figure 36 illustrates an example 3600 of three stereoscopic views 3602 / 3604, 3606 / 3608, and 3610 / 3612 with motion parallax (the viewpoint moves from left to right) formed using the multiview images shown in Figure 35. Figure 36 shows three (cross-eye) stereograms 3602 / 3604, 3606 / 3608, 3610 / 3612 with a left-to-right viewpoint change (see Figure 35). The images connected by the alignment are the same.

[0182] The amount of motion parallax in the example of Figure 36 can be estimated. The two viewpoints for the stereoscopic content are separated by an eye separation distance, typically about 60 mm. Correspondingly, the virtual view illustrated in Figure 36 allows the user to move their head 12 cm between the two poles. However, viewing the result as a small crossed-eye stereogram is likely to distort the effects compared to observing them in more realistic conditions. The head tilt and motion parallax support described may be provided individually to multiple local viewers, since MFP formation and processing can occur multiple times within the receiver for content that is transmitted only once (broadcast).

[0183] In the case of synthetic motion parallax, the stereo pair is formed not to have symmetrically left- and right-shifted versions, but, for example, to have two left-shifted versions with different amounts of shift. However, this is not expected to cause a noticeable degradation in the perception of the motion parallax effect. Another relevant detail is that the original unshifted texture image (without affecting both sides of the object edge) may also be used as part of the multi-view image series.

[0184] Although the choice of focal point or ZPS (see Figures 30A-30C and their discussion) does not affect the depth range perceived from a stereoscopic image (see Total Disparity), the choice has little consequence when creating synthetic motion parallax. When the focal point is central (e.g., as typically chosen for examples in this disclosure), a viewer moving laterally perceives the rendered volume as rotating around a pivot point (at the ZPS distance). However, it is more natural for motion parallax to have the volume rotate around its farthest point. In some embodiments, when creating motion parallax effects, the far plane (and any disparity in front of this far plane) may be used in virtual viewpoint generation.

[0185] Applicable source formats for forming MFP Video plus depth map is a format applicable to the disclosed method according to some embodiments, providing the distance of each pixel in the image. Analyzing a stereoscopic view from such a format using 3D warping or MFP-based design may cause some errors at object edges when changing the viewpoint reveals occluded areas of the view. One exemplary embodiment is a stereoscopic video with depth map. Both of the above formats are supported by video compression methods, such as H.264 MVC.

[0186] There are multiple processes for generating depth images. One method according to some embodiments is to derive the disparity between the two camera views (stereo pair) and map the disparity to depth. Finding correspondences between image pixels is done in parallel with finding a line connecting the two camera points (baseline). Finding true correspondences is an error-prone process, with potential false-positive correspondences and disparities. Various types of filtering and segmentation operations may be used to remove errors.

[0187] Some embodiments use a depth-plus-texture source format captured by a depth sensor (such as RGB-D or ToF (Time-of-Flight)). The depth information may be compressed before transmission. This uses a smaller bitrate than real-time transmission of a virtual (3D modeled) scene for a stereoscopic viewpoint or a corresponding depth map.

[0188] Exemplary Embodiments Using an External Display FIG. 37 is a system diagram illustrating an example set of processing blocks and interfaces for generating shifted and projected MFP images for a flexible baseline stereoscopic display for three viewer orientations, according to some embodiments. In some embodiments, the flexible baseline stereoscopic display may be, for example, a flexible view TV. While the context for describing some embodiments has primarily been near-eye displays, the same interactions may also be implemented using, for example, an external display, according to some embodiments. For example, when used for a time-shared S3D display, shutter glasses worn by the user may be equipped with sensors to track user motion and to receive 3718 data from the glasses and send 3720 data to the glasses. External tracking may be used in some embodiments. By tracking user pose and motion for the example of three users 3722, 3724, 3726, the systems and methods disclosed herein (including virtual disparity-related features) according to some embodiments may use an external display. Multi-user viewing (which may be implemented with time sharing 3716) may be used to serve multiple co-located viewers. Figure 37 illustrates some embodiments using an external display and two exemplary methods 3712, 3714 for virtual viewpoint generation 3710 within a receiver 3708 at a receiving site 3706 for input of a depth map 3702 and a texture 3704. Although the techniques are shown as alternatives ("or") in Figure 37, in some embodiments any combination of these may be used.

[0189] Exemplary embodiments using DIBR-based methods for viewpoint generation Some embodiments for virtual viewpoint generation with MFP may be used with DIBR systems based on 3D warping and hole filling. Some embodiments may use DIBR-based methods to generate free-baseline stereo and motion parallax for freely posed or moving users. This embodiment may calculate 3D geometric deformations for multiple users and viewpoint changes. This embodiment is illustrated in Figure 37 (see the two boxes inside "Virtual Viewpoint Generation" corresponding to methods 3712 and 3714).

[0190] Exemplary embodiments of free-baseline stereo and motion parallax in MFP NED The systems and methods disclosed herein in some embodiments create MFPs and use them to create selected (various) stereoscopic disparities at desired orientations (head tilts). Some embodiments shift the MFP to match the baseline orientation. Some embodiments create free-baseline stereo and motion parallax effects on the actual MFP display. When illustrated on the actual MFP display, natural accommodation is supported. The MFP stack may be viewed with head tilts, such as those illustrated for the stereoscopic display in FIG. 34, such that the stereoscopic perception supported by accommodation is virtually invariant to the viewer's head tilt (rotation "around the nose").

[0191] Exemplary Embodiments for Asymmetric Viewpoint Generation An alternative embodiment is to generate and plot one eye's information with a shifted MFP and plot the other eye's MFP without the shift (using a focal plane for the texture image directly received for one eye). According to

[15] , plotting one high-quality image and one low-quality image (here, the unoccluded area) simultaneously can provide quality benefits similar to monovision methods. However, this variant allows for smaller discrepancies due to asymmetric viewpoint generation and distortion.

[0192] Generation of MFP and virtual viewpoint Some embodiments for forming the MFP and virtual viewpoints calculate depth blending weights for each focal plane, form an MFP weight map by remapping the image depth map by the calculated depth blending weights for each focal plane, multiply the captured image or input image by each MFP weight map to form each focal plane image, perform spatial shifts on each focal plane image, and add the formed MFPs to generate a stereoscopic image pair for mismatch. In some embodiments, depth blending weights may be calculated, and image processing may be used to form the MFP weight map (illustrated in FIG. 38B), form each focal plane image (illustrated in FIGS. 39A-39C for one focal plane image), and perform spatial shifts and additions (e.g., for mismatches where stereoscopic images are selected).

[0193] 38A-38C are a set of computer windows illustrating an example set of interfaces for an image manipulation program illustrating a depth map image, a depth weighting graph, and a generated focal plane image, respectively, according to some embodiments. Figures 38A-38C show an example in which a depth map 3800 (FIG. 38A) is remapped with a sinusoidal weighting function 3830 (FIG. 38B) to create a weight map for one focal plane 3860 (FIG. 38C). The other weight maps (five in total in this example) may be formed in a similar manner.

[0194] 39A-39C are illustrations showing an example set of generated focal plane images within a simulated environment. Figures 39A-39C show test image 3900 (Figure 39A) multiplied by weight map 3930 (center graph of Figure 38) to form corresponding focal plane 3960 (Figure 39C). Other focal planes (five in total in this example) are generated in the same manner.

[0195] On the computational complexity of virtual viewpoint generation 3D warping for viewpoint changes may be performed by computer graphics algorithms. The complexity, speed, and results of these algorithms vary depending, for example, on the perspective and camera model used. An example of a mathematical formulation for a shift sensor setup can be found in non-patent document 5. The computational complexity for 3D warping can be high, such as with high-quality near-eye displays with a large FoV (field of view) and higher resolution than many external displays. In addition, when using 3D warping to form the viewpoint for the MFP display, the focal plane is formed after 3D warping, which can further increase the complexity for many previous systems.

[0196] On the other hand, forming a stack of MFPs is a procedure that uses pixel-based operations to weight each pixel value by its distance to each focal plane (e.g., obtained from a depth map). After the decomposition is performed, any change in viewpoint or stereo baseline orientation is simply a lateral (e.g., bulk) shift of the once-derived focal planes relative to each other and their pixel-by-pixel addition to form the projection to the new viewpoint. In some embodiments, there is no sub-pixel shift and operation, as focal plane shifts may occur in multiplication of pixel displacements without resolution changes due to interpolation, etc.

[0197] Instead, using traditional 3D warping procedures, changing the viewpoint typically requires multiple 3D geometry-based arithmetic operations (e.g., matrix calculations) at sub-pixel accuracy. Furthermore, inpainting and hole-filling operations to remove disocclusions may use operations with several pixel-wide kernels or windows instead of simple pixel-based operations.

[0198] Figure 40 is a message sequence diagram illustrating an example process for generating a multi-focal plane (MFP) display image, according to some embodiments. Figure 40 is a message sequence diagram 4000 for some embodiments that generate a stereoscopic MFP stack (e.g., two monocular stacks of MFPs, correspondingly, for the left and right eyes) by shifting and projecting focal plane images. The MFP display 4006 sends 4008 the MFP plane positions to the renderer control module 4002. The texture images and depth maps are received 4010 by the renderer control module 4002. The renderer control module 4002 generates 4012 an image for each MFP plane. The tracking module 4004 (or sensor) sends 4014 the viewer position / orientation to the renderer control module 4002. The MFP images are shifted and projected 4016 by the renderer control module 4002. The stereoscopic MFP stack (left / right pair) is sent 4018 from the renderer control module 4002 to the MFP display 4006, where it is displayed to the user.

[0199] Figure 41 is a message sequence diagram illustrating an exemplary process for generating stereoscopic 3D display images, according to some embodiments. Figure 41 is a message sequence diagram 4100 for some embodiments generating disjoint views of focal plane images for a stereoscopic 3D display (e.g., an external flat screen). Texture images and depth maps are received 4108 by a renderer control module 4102. The renderer control module 4102 generates 4110 images for each MFP plane. In some embodiments, the renderer control 4102 may be used to calculate 4114 disjoint views. The glasses / HMD sensor 4104 sends 4112 viewer position / orientation to the renderer control module 4102. The disjoint views are calculated by the renderer control module 4102. A stereoscopic left / right image pair is sent 4116 from the rendering control module 4102 to the stereoscopic 3D display 4106. The stereoscopic 3D display 4106 sends an active glasses signal 4118 from the stereoscopic display 4106 to the glasses / HMD 4104 and the image is displayed to the user.

[0200] In some embodiments, a method may include receiving three-dimensional video content, calculating one or more multi-focal plane (MFP) images for the number and position of multi-focal planes using the three-dimensional video content, receiving a viewer orientation signal, calculating one or more discrepancy images for a common view of a stereoscopic display using the one or more MFP images and the viewer orientation signal, and rendering the one or more discrepancy images on the stereoscopic display, where the common view of the stereoscopic display is rendered to each eye view of the stereoscopic display. In some embodiments, the common view may be a left-eye perspective and a right-eye perspective relative to an average-eye perspective of the monocular MFP stack. In the case of an MFP NED, the optical arrangement may use a periscope-type eyepiece that allows the monocular MFP stack to be viewed between the eyes in an uncrossed eye state.

[0201] In some embodiments, the method for calculating one or more mismatched images for a common view may calculate an average eye point that is the average of each eye view of a stereoscopic display (or the average of a left / right pair of images).

[0202] The systems and methods disclosed herein in some embodiments create MFPs and use them to generate selected (various) stereoscopic mismatches at desired orientations (head tilts). The MFPs may be shifted in a direction to match the baseline orientation. The resulting stereoscopic images may be displayed on wearable displays as well as, for example, external screens. Some embodiments may be used to generate stereoscopic content for services that use stereoscopic displays. Some embodiments enable the development of stereoscopic displays, enabling a simple and cost-effective means of generating head-tilt-independent (free-baseline) stereoscopic rendering with undistorted parallax and sensor-supported motion parallax effects. Such systems may support stereoscopic vision and motion parallax as users move their heads and bodies.

[0203] Some embodiments produce stereoscopic images that may cause VAC when viewed on older stereoscopic displays. VAC can be reduced by viewing the content with accommodation-invariant stereoscopic glasses. Alternative embodiments support disparity-based interaction with the MFP display.

[0204] In some embodiments, the method may include calculating image data for one or more focal planes of the MFP display using the three-dimensional video content and the viewer orientation signal, and rendering one or more MFP images on the MFP display using the calculated image data.

[0205] In some embodiments, a device may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform a process that calculates image data for one or more focal planes of an MFP display using three-dimensional video content and a viewer orientation signal, and a process that uses the calculated image data to render one or more MFP images on the MFP display.

[0206] Virtual Viewpoint Generation from Depth and Texture Systems and methods disclosed herein in some embodiments relate to forming stereoscopic image pairs (stereograms) in a stereoscopic 3D (S3D) display system. One method for forming stereograms is depth image-based rendering (DIBR). Systems and methods disclosed herein in some embodiments generate synthetic stereoscopic views from multiple focal planes (MFPs). One exemplary method in some embodiments uses virtual viewpoint generation with 3D warping used in a DIBR system. In both DIBR and MFP stereoscopic methods, a depth-plus-texture input format may be used in capturing the view. Some embodiments use post-processing of multiple focal planes (MFPs) for virtual viewpoint generation and for MFP rendering.

[0207] Depth blending used to improve depth perception in MFP-based methods is described in

[10] and

[11] , and a depth blending method based on redistribution of focal (representation) plane information in the frequency domain is described in [1]. Natural blur and occlusion are important factors for quality in stereoscopic and 3D image rendering. For information on their perception by the human visual system (HVS), see, for example,

[16] .

[0208] Many stereoscopic 3D systems use depth image-based rendering (DIBR). A schematic picture of a DIBR-based 3D transmission system is shown in Figure 8, which is adapted from Non-Patent Document 3. In many DIBR systems, at the receiving side, virtual viewpoint generation consists of a 3D warping stage and a hole-filling stage (see the virtual viewpoint generation block in Figure 8). 3D warping may be used to form two virtual views into a 3D view (textured depth map) seen from the two eyepoints of the viewer. 3D warping is performed by a computer graphics algorithm.

[0209] Stereoscopic 3D synthesized from stacks of MFPs Multi-focal plane (MFP) rendering is used to support natural eye accommodation in near-eye displays. As explained above, a stereoscopic view may be synthesized from multiple focal planes (see 3D warping in DIBR). MFP displays create a stack of different focal planes that make up a 3D scene from layers along the viewer's visual axis. The view into the 3D scene is formed by projecting all those volumetric pixels (e.g., more precisely, voxels) that appear to the user at different depths and spatial angles. Multi-focal planes can be rendered, for example, either by spatially multiplexing a stack of 2D views or by sequentially—in a time-multiplexed manner—switching the focal length of a single 2D view with a high-speed variable focus element (VFE) while rendering the visible portions of the corresponding multi-focal planes.

[0210] FIG. 12 shows a schematic diagram for a multi-focal plane near-eye display. The MFP display creates an approximation for the bright field of the displayed scene. Many near-eye displays support only one user viewpoint. Correspondingly, the approximation is simpler and may not use a full bright field capture as input. Rendering a large number of MFPs at different distances (in 3D space) supports something closer to natural accommodation by the viewer (avoiding VAC). By using depth blending when forming the MFPs, the rendering quality may be further improved and / or the number of MFPs may be reduced.

[0211] Depth blending refers to weighting pixels according to their distance from, typically, the two nearest focal planes. The distances may be obtained, for example, from a depth map. Depth blending may improve depth perception between focal planes. The focal planes are summed to (or approximately to) the original captured scene so as to fit into a division of one. There are many means of decomposing a scene into additional planes so that they meet this requirement. Correspondingly, the systems and methods disclosed herein are presented as illustrative examples according to some embodiments.

[0212] Many MFP rendering systems create and render different MFP stacks for each eye to support stereoscopic perception. Non-Patent Document 7 describes the depth-fused 3D (DFD) phenomenon between two spatially formed focal planes. Non-Patent Document 10 extends this phenomenon (described as depth blending) to setups with multiple (three or more) focal planes. However, Non-Patent Document 10 is understood to describe viewing a separate MFP stack by each eye, created from stereoscopic capture for depth and texture (rather than by shifting the MFP to synthesize discrepancies).

[0213] In some exemplary systems and methods disclosed herein according to some embodiments, both eyes view the same monocular stack of focal planes from slightly different viewpoints, creating a slight synthetic discrepancy for 3D perception. Thus, in some embodiments, instead of two sets of MFPs captured and formed separately for each eye, only one stack of MFPs is formed and aligned toward a point between the eyes, as shown in FIG. 18. FIG. 18 shows a stack of MFPs formed from an average eye point, with the same focal plane seen by the left and right eyes from slightly different viewpoints. In some embodiments, one stack of MFPs formed and aligned toward a point between the eyes is rendered to each eye view of a stereoscopic MFP display by forming two versions of the single monocular MFP stack from slightly different viewpoints, as shown in FIG. 18. In some embodiments, the projections of the two versions of the MFP stack are summed and displayed to each eye using a conventional stereoscopic display (S3D).

[0214] 3D perception from focal planes is appropriate for flat objects strictly on those planes. However, with depth blending, perception approaches appropriate across planes. Because MFP stacks can be formed separately for each eye, depth blending is developed by examining accommodation for two sets of stereoscopic MFPs. However, depth blending can provide a more continuous depth perception through (composite) stereoscopic disparity (when viewing one monocular depth-blended MFP stack with two eyes (see Figure 18)). This phenomenon can be seen by comparing cross-eye stereograms with depth blending, corresponding to Figures 29A, 29B, 46A, and 46B, with those without.

[0215] Synthesizing stereoscopic images from MFPs is described below. In addition to forming a stack of MFPs, stereoscopic image pairs (stereograms) are formed from the focal planes. For example, a monocular stack of MFPs (see FIG. 27A) is formed, placed at a set (or desired or fixed) distance from the viewer's (average) eyepoint, and viewed by two eyes (see FIG. 27B).

[0216] Figure 27A shows a monocular MFP stack formed by depth blending. Figure 27B shows viewing the viewing stack with two eyes. Figure 27C shows the focal plane for the left eye, which causes the focal plane to appear shifted to the right. Figure 27D shows the focal plane for the right eye, which causes the focal plane to appear shifted to the left. The left and right eyes view the stack from slightly different perspectives. Correspondingly, combining two (stereoscopic) MFP stacks from a monocular MFP stack for stereoscopic (or stereoscopic MFP) viewing can be achieved by shifting the MFPs by a certain amount in a certain direction. The MFP shift is actually an approximation of a more complex deformation due to a virtual viewpoint change (see 3D warping in conventional DIBR systems). The lateral position of each focal plane perceived by the eye depends linearly on the focal plane distance (see Figure 27B). When using the fixed focal plane interval described in Figure 27B, if the farthest surface is viewed in the same position by both eyes (e.g., placed far from the eyes), the intermediate surface will appear to be shifted laterally by "1 unit" and the nearest surface will appear to be shifted laterally by "2 units." If the focal plane separations are fixed on a refractive scale, their actual metric distances are used to derive the various amounts of lateral shift.

[0217] When forming stereoscopic image pairs corresponding to eye positions, the laterally shifted focal planes are eliminated by projecting them to each eye point. Projecting for a stereoscopic image pair effectively means that corresponding pixels on the focal planes are summed along the depth axis. This results in a stereoscopic image pair of two images with different stereo disparities / parallaxes. Some embodiments may start with a monocular MFP stack (such as those in Figures 28B-28D) and use synthetic disparity to generate a stereoscopic MFP stack.

[0218] Figures 29A and 29B provide an example of a stereoscopic image pair formed by the described procedure. Depth blending with a linear (tent) filter was used to form the three focal planes used in the examples of Figures 29A and 29B. Figures 29A and 29B show an example of a cross-eye stereogram formed by shifting and summing three linearly blended MFPs. The total discrepancy corresponds to 2.76% of the image width (see MFPs shifted up to 6 pixels using a specific test image). When the focal plane separation is determined by a refractive scale, their actual metric distance is used to derive various amounts of lateral shift.

[0219] Depth blending improves depth perception in MFP rendering In some embodiments, 3D video content may be filtered into high-frequency content and low-frequency content. The high-frequency content may be decomposed into one or more high-frequency focal planes. The low-frequency content may be redistributed to one or more high-frequency focal planes to generate one or more redistributed focal planes. The one or more redistributed focal planes may be used to render one or more multi-focal plane images.

[0220] According to Non-Patent Document 17, for any human, a maximum of 28 focal panes is sufficient to cover the depth range from infinity to 4 diopters (25 cm), which corresponds to a 1 / 7 diopter spacing for the focal planes. For a person with average vision, Non-Patent Document 17 states that 14 focal planes cover the range. Therefore, a large number of focal planes may be used for high-quality depth perception.

[0221] Rendering a large number of focal planes is limited for various technical reasons. The number can be reduced by using depth blending. When using only a few focal planes, optimal placement may be beneficial to the human eye. Since the accuracy of depth perception decreases inversely with distance from the viewer, greater accuracy may be obtained by making the depth plane a function of refractive depth.

[0222] A mapping between linear and refractive scales can be determined. Many examples are obtained by using a fixed spacing between focal planes on a linear scale. Depth-based blending may be used to reduce quantization errors in the depth dimension that would otherwise be visible to the human eye. Depth blending means using a depth-based function to weight the pixels used to construct each focal plane.

[0223] FIG. 42A is a graph illustrating example blending functions 4200, 4202, 4204 of weights versus depth, according to some embodiments. One depth blending function is a box filter (slicing without blending) shown in FIG. 42A. For comparison, a tent filter, which is a piecewise linear, sawtooth blending function, is shown in FIG. 43A. In some embodiments, the depth value range is between 0 and 255, which is used in some of the examples herein, such as FIGS. 42A and 42C. FIG. 42B is an illustration 4210 showing an example test image. FIG. 42C is an example depth map 4220 illustrating pixel distances for FIG. 42B. FIGS. 42D-42F are illustrations 4230, 4240, 4250 showing example focal plane images using the depth weights of FIG. 42A with the test image 4210 of FIG. 42B and its example depth map of FIG. 42C, according to some embodiments. Figures 42B and 42C are derived from an image found on a web page (Non-Patent Document 12), and Figures 42D to 42F are modified versions of that image.

[0224] Figure 43A is a graph illustrating example blending functions 4300, 4302, 4304 of weight versus depth, according to some embodiments. Figure 43B is an illustration 4310 showing an example test image. Figure 43C is an example depth map 4320 illustrating pixel distance for Figure 43B. Figures 43D-43F are illustrations 4330, 4340, 4350 showing example focal plane images using the depth weights of Figure 43A with the test image 4310 of Figure 43B and its example depth map of Figure 43C, according to some embodiments. Figures 43B and 43C are derived from images found on a webpage (Non-Patent Document 12), and Figures 43D-43F are modified versions of that image.

[0225] In some embodiments, the polynomial blending functions extend across the entire depth range. Figures 21A-21F show an example of forming focal planes using three polynomial blending functions (Figure 21A). The corresponding focal planes are shown in Figures 21D-21F for the test image (Figure 21B) and its depth map in Figure 21C.

[0226] Figures 22A-22F show an example of forming a focal plane using three sinusoidal blend weight functions (Figure 22A). The corresponding focal planes are shown in Figures 22D-22F for a test image (Figure 22B) and its depth map in Figure 22C.

[0227] In addition to the blending filter described above, another variation has been proposed by

[11] , who describes a method for forming a dioptrically positioned MFP. Instead of continuously capturing depth and texture from a scene, this method captures a series of focus stacks, i.e., a consecutive set of multiple images with various focal lengths.

[0228] Using focal stacks, the MFP can be more accurately optimized with the human visual system (HVS). Specifically, according to [1], this method applies to the effects of occlusion, reflections, and other non-Lambertian phenomena in the captured view. In addition to cameras with different focal lengths, the set of input images may be derived from bright-field images captured from the scene. In [1], a first-generation Lytro Illum camera was experimented with for this purpose.

[0229] Figure 44 is a schematic diagram illustrating exemplary focus stack images captured at different depths from a user. Figure 44 shows an example 4400 of a focus stack (four images 4412, 4414, 4416, 4418 with different focal lengths) captured from a real-world view and processed to form focal (presentation) planes 4404, 4406, 4408, 4410 for a user's eye 4402 (adapted from non-patent document 1). Insets 4420, 4422, 4424, 4426, 4428, 4430, 4432, 4434 show details of the focus stack image 4400 with various defocus (adapted from non-patent document 1).

[0230] Figures 45A-45D are illustrations showing exemplary linearly blended focal planes. Figures 45E-45H are illustrations showing exemplary focal (presentation) planes processed from focal stacks. Figures 45A-45D show a comparison of MFPs formed by linear depth blending, and Figures 45E-45H show a comparison of MFPs formed by the optimized procedure in Non-Patent Document 1. Figures 45A-45H compare focal planes 4500, 4510, 4520, and 4530 (Figures 45A-45D) formed using conventional linear blending with focal (e.g., presentation) planes 4540, 4550, 4560, and 4570 (Figures 45E-45H) formed from focal stacks using optimized blending. Note that the brightness of the images in Figures 45E-45H has been increased for better visibility when printed. Figures 45A-45H are line art versions of the images published in Non-Patent Document 1.

[0231] The approach in [1] does not use a depth-plus-texture format for scene capture. Instead, the system captures a set of images with different focal lengths, processes them in the frequency domain, and assigns information to different focal planes (or presentation planes as described in [1]) using a model of the human visual system (e.g., retinal blur).

[0232] As explained in [1], the brightness of the optimized focal planes (FIGS. 45E-45H) appears to be increased by a factor of (approximately) 2, presumably for better visibility when printed. When these focal planes are used as test images later in this disclosure, an inverse scaling by 0.5 is performed on them to prevent saturation of the results (violation of the division of 1).

[0233] In particular, there are also some horizontal brightness artifacts in the linearly blended focal planes shown in Figures 45A-45D. These artifacts are noticeable when summing the focal planes to recreate the original test image (demonstrating the division of 1). [Illegible text - likely part of a separate document] is not understood to provide depth maps for the test images in their paper or its supplementary material, so we are unable to recreate the MFP. Despite the observed artifacts, these focal planes are used to visualize the impact of some embodiments to enable a comparison of their performance with the approach in [Illegible text - likely part of a separate document].

[0234] According to Non-Patent Document 1, high-frequency (generated edge and texture) components benefit from being assigned to their nearest focal distance. Low-frequency components may also be assigned to more distant focal planes. To quote Non-Patent Document 1, "High-frequency image content must be allocated to its nearest plane, while low-frequency content may be under-constrained and redistributed between planes. It is these under-constrained degrees of freedom that algorithms exploit to satisfy constraints on display intensity." High-frequency components dominate in perceiving depth from focality. Low-frequency components, such as slowly varying luminance or color, generate only a few cues for depth perception. In some embodiments, focal planes generated by known methods (e.g., Figures 45A-45D generated using linear depth blending) are processed by redistributing their low-frequency content between the original focal plane positions. The redistributed focal planes have been shown to have advantages over the original, unprocessed focal planes when used to create stereoscopic or changing perspectives (the latter due to motion parallax).

[0235] The apparent distance detected by a depth sensor does not necessarily apply to reflected or refracted (non-Lambertian) image content. When capturing image information as a function of focal length, similar to the system of [1], the focal properties and distances will match such content if the images are captured with a small enough separation in the depth dimension (a small enough difference in focal length).

[0236] For many 3D display systems, new viewpoints benefit from being derived to support stereoscopic or multi-view rendering. For example, when using a depth-plus-texture capture format, a stereoscopic view may be derived by forming a virtual viewpoint symmetrically around the capture point, such that both views are composited.

[0237] Using depth plus texture as a capture format has benefits in stereoscopic 3D: the stereoscopic baseline is not fixed at the front end, but the amount and direction of the discrepancy may be synthesized within the receiver, even for stored content.

[0238] In many cases, the corresponding viewpoint change reveals portions of the view behind the occluding object, as explained in Non-Patent Document 3. As illustrated by the schematic picture in Figure 9 and the examples in Figures 10A and 10B, these disocclusions appear as gaps or holes in each new perspective image, since the captured texture or depth does not include information from those areas.

[0239] FIG. 9 shows that disocclusion can occur when changing viewpoint, as illustrated when changing the camera viewpoint from a source position / image to a target position / image.

[0240] The mechanisms for creating disocclusions may be similar to DIBR and MFP-based virtual viewpoint generation. These disocclusions may be created in viewpoint generation or if the MFP viewer moves far enough away from the viewing point. Larger disparities or viewpoint changes may be achieved by using measures that reduce the visibility of the disocclusions. In many previous systems, disocclusion still limits the maximum disparity.

[0241] 10A and 10B provide an example of a 3D warped image. This example shows only the left-eye image. The right-eye image is formed in the same way and has a similar defect, but it is on the right side of the object. 10A and 10B show an example of disocclusion caused by 3D warping. The original monocular image after 3D warping is on the left, and details of the 3D warped image are on the right (when using linear interpolation-based hole filling).

[0242] In hole filling, various image and depth map filtering and inpainting algorithms are used to fill holes or other forms of disocclusions. The type and amount of occlusions are highly content-dependent, as are the distortion and their visibility. Disocclusions can degrade the perceived stereoscopic quality around object edges. They can limit the amount of synthetic disparity and depth perception. Procedures for virtual viewpoint generation based on shifting the MFP can produce disocclusions near vertical edges, or more generally, edges perpendicular to the chosen stereo baseline.

[0243] 46A and 46B are illustrations showing exemplary stereoscopic images for the right and left eyes, respectively, formed using unblended focal plane images, according to some embodiments. In some embodiments, box filtering (e.g., a box filter using a tooth-shaped depth weighting function, such as the depth weighting function of FIG. 20A ) is used to form the unblended focal plane images. FIGS. 46A and 46B provide an example of disocclusion in a stereoscopic image formed by deriving two viewpoints 4600, 4650 on a stack of MFPs. In some embodiments, the unblended focal plane images may be generated by box filtering the input images. In this example, the MFPs were not depth blended because a box filter was used to form them. FIGS. 46A and 46B show an example of disocclusion in a cross-eye stereogram formed by shifting and summing three unblended (box-filtered) MFPs. The total discrepancy corresponds to 2.76% of the image width (see the MFPs shifted by up to 6 pixels for both images). When depth blending is used, there is less disocclusion. Depth blending also creates depth values between MFPs. Both of these improvements can be seen by comparing Figures 46A and 46B to Figures 29A and 29B, where depth blending was used.

[0244] In addition to reducing quantization effects in the depth dimension, depth blending can reduce disocclusion when synthesizing viewpoints from an MFP. To reduce disocclusion, content from neighboring depth planes is used for depth blending. If there is no content at the depth corresponding to a focal plane, shifting the near (occluding) focal plane to the side reveals empty areas that lack any texture or color information (and appear transparent). These empty areas may appear as black stripes near the object edges (due to transparency through all focal planes). If there is no content in neighboring focal planes, there is no depth blending to reduce disocclusion. Therefore, disocclusion reduction by depth blending is content dependent.

[0245] In creating synthetic volumes from MFPs, one challenge with previous depth blending methods is that they produce MFPs that can have disocclusions due to concentrating most of the low-frequency content (image areas where brightness and color vary slowly) in one focal plane (of many). Correspondingly, large areas within the MFP are left without any information.

[0246] Some embodiments post-process multiple focal planes (MFPs) to make them better suited for virtual viewpoint generation or MFP rendering. Some embodiments are based on removing typical holes or non-textured (e.g., white background) areas of MFPs by redistributing their low-frequency content. In particular, redistribution reduces disocclusion when using MFPs to generate virtual (e.g., stereoscopic) viewpoints, even in cases of large mismatches.

[0247] Instead of concentrating the content within the MFP, redistribution distributes low-frequency content more uniformly across all focal planes. This method may produce more uniformly distributed luminance and color values than other MFP systems. Low-frequency content can be redistributed without degrading 3D perception. This procedure preserves a division of 1 for the MFP, which means that they sum (e.g., close enough) to return to the original image.

[0248] Viewed from two virtual viewpoints, monoscopic MFP stacks are attractive for their ability to support disparity and motion parallax at any baseline orientation. This benefit rivals that of more conventional depth image-based rendering (DIBR) in stereoscopic 3D.

[0249] If the stereo viewpoint, disparity, and baseline are chosen before capturing the scene, the receiving terminal may not allow these parameters to be changed after the scene is captured. Problems may arise if the viewer wishes to set these parameters, or if the system attempts to combine renderings of stereoscopic content that occur with different parameters.

[0250] A DIBR system may be used in place of traditional stereoscopic video, as stereoscopic image formation using DIBR may enable additional options in the receiver's post-processing stage. The amount of stereoscopic disparity (using a depth budget) may depend on the captured content, which may be beneficial to control later rather than during the shooting of the scene, for example. Similarly, using a depth-plus-texture format to create multiple focal planes may be easier to control later than during the shooting. Post-processing at the receiving end of the monocular stack of MFPs enables the stereoscopic MFP to render at a baseline orientation with variable disparity.

[0251] Many 3D display systems or near-eye displays with accommodation support do not support or tolerate even smaller changes in viewpoint. Some embodiments for producing a (redistributed) MFP support relatively large viewpoint changes, disparities, and motion parallax, improving visual quality in, for example, 3DoF+ applications. Relatively large disocclusions can be reduced by distributing MFP luminance and color values more uniformly than in previous MFP systems. This uniform distribution is based on allocating low-frequency content over several or all focal planes instead of concentrating it in one of the MFPs.

[0252] Some embodiments perform post-processing to improve the quality of MFPs formed using some existing methods, which may use depth and texture inputs as well as focal plane stacks as inputs, and may work with multiple capture and transmission formats.

[0253] In previous MFP displays, disocclusions related to stereo mismatch may not be an issue by rendering separate MFP stacks for each eye, acquired using stereoscopic capture for texture and depth. However, by using focal planes with high brightness and contrast differences, even smaller viewpoint changes caused by eye rotation may appear as small but visible disocclusions. These artifacts may be removed under some scenarios. The systems and methods disclosed herein in some embodiments enable the generation of virtual viewpoints with large mismatches.

[0254] MFP redistribution to support viewpoint changes Described below is an exemplary procedure for generating an optimized MFP for viewpoint generation with viewpoint variation. Some embodiments for generating a redistributed MFP include receiving an MFP input that may be generated by a depth blending method. Low-frequency image areas are detected, which may be generated by low-pass filtering the input image and by low-pass filtering each MFP. The low-pass components of each MFP may be replaced by an equal share of the low-pass image filter output, which may be generated by removing the corresponding low-pass component from each MFP and adding an integer fraction of the low-pass filtered input image, where the integer fraction is the reciprocal of the number of MFPs used. The MFPs may be redistributed to generate an MFP for synthesizing stereoscopic views with a reduced amount of disocclusion and / or to render an MFP with a depth dimension to support natural accommodation (removing VAC). High-frequency image content may not be derived as such but may be supplemental for the low-frequency components.

[0255] FIG. 47 is a flowchart illustrating an example process for generating redistributed focal plane images for front, mid, and rear depth ranges, according to some embodiments. FIG. 47 is a block diagram 4700 of a redistribution process for three MFPs. In some embodiments, redistribution 4708 is a post-process for MFPs formed in a conventional manner. In some embodiments, a set of (e.g., conventional) focal planes (MFPs) is used as input. The received image (picture) 4704 may be summed from those focal planes using picture depth data 4702 or may be received by the MFP block 4706. The MFP block 4706 may output front, mid, and rear depth 4710, 4712, 4714 planes. The low-pass component of each focal plane (one out of N) is replaced by the low-pass component of the input image by a scaling factor of 1 / N, where N is the number of focal planes. For example, N is 3. This process may preserve the unity partitioning nature of the MFPs.

[0256] FIGS. 48A-48C are illustrations showing example focal plane images without redelivery, according to some embodiments. FIGS. 48D-48F are illustrations showing example focal plane images with redelivery, according to some embodiments. FIGS. 48A-48F are modified versions of the unprocessed flower test image and its respective depth maps, e.g., of FIGS. 21B and 21C, according to some embodiments. FIGS. 48A-48F show example MFPs before and after redelivery. FIGS. 48A-48C show linearly blended MFPs 4800, 4810, and 4820 before redelivery. FIGS. 48D-48F show redelivered versions of the same MFPs 4830, 4840, and 4850. A Gaussian filter with a radius of 20 pixels (both horizontally and vertically) was used for low-pass filtering. Figures 48A-48F reveal that redelivery preserves the high frequency components (details) of each focal plane while bringing low frequency components into areas that were originally devoid of information (such as the white background in Figures 48D-48F).

[0257] Both MFP examples in Figures 48A-48C and 48D-48F split the captured information into different accommodation levels (focal lengths). However, they differ in their ability to repair disocclusion. While the MFP stacks resulting from linear blending in Figures 48A-48C have non-textured / uncolored (e.g., white background) areas that can cause disocclusion, the redistributed versions of the same MFPs in Figures 48D-48F show an improvement and have more uniform characteristics. When shifting the second set of MFPs for a new viewpoint or small viewer movements, disocclusion is less visible.

[0258] The MFPs in Figures 48D-48F are selectively low-pass filtered so that high-contrast transitions between textured and non-textured (e.g., white background) areas are not filtered. Because low-frequency image components are insensitive to being distributed at various depths, in some exemplary embodiments, low-frequency image components are distributed uniformly across all focal planes. Low frequencies may be distributed to just one focal plane, or to extra focal planes between or outside (not belonging to) any of the original MFP distances.

[0259] There may be various reasons for preferring a particular allocation: for example, the allocation ratio may vary depending on the nature of the content, and viewers may expect the resulting MFP to fit (or be close to) a division of 1.

[0260] Figures 49A-49B are illustrations showing exemplary right and left images for a crossed-eye stereogram formed using linearly blended focal planes, according to some embodiments. Figures 49A and 49B show an example of a crossed-eye stereogram formed using the linearly blended focal planes shown in Figures 45A-45D. Figures 49A and 49B show crossed-eye stereograms 4900, 4950 formed by shifting the linearly blended focal planes (Figures 45A-45D, which are line-art versions of the images appearing in Non-Patent Document 1), according to some embodiments. The total discrepancy is 4% (2 x 12 pixels in a 600 x 600 pixel image). Figures 49A and 49B show exemplary disocclusion errors 4902, 4952 near the object edges.

[0261] 50A-50B are illustrations showing exemplary right and left images for a crossed-eye stereogram formed using redistributed focal planes, according to some embodiments. FIGS. 50A and 50B show an embodiment of the same but redistributed focal planes of FIGS. 49A and 49B. In FIGS. 50A and 50B, the exemplary disocclusions 5002, 5052 are much smaller. FIGS. 50A and 50B show a crossed-eye stereogram formed using redistributed focal planes using the simplified procedure described. The total discrepancy is 4%. The disocclusion error is reduced relative to that in FIGS. 49A and 49B. FIGS. 50A and 50B are processed versions of the unprocessed images ( FIGS. 45A-45D ) from Non-Patent Document 1.

[0262] In Figures 49A, 49B, 50A, and 50B, four MFPs were used with a synthetic mismatch amount corresponding to 4.0% of the image width (600x600 pixel MFPs shifted by up to 12 pixels for both viewpoints). In the examples of Figures 50A and 50B, the MFPs are selectively low-pass filtered so that high-contrast transitions between textured and non-textured (e.g., white background) areas are not filtered.

[0263] Redistribution of low frequency components of MFP As shown by [1], the adjustment can be more natural for the re-distributed MFP. This is especially true for non-Lambertian effects in the scene. [1] should not be understood as re-distributing the MFP as a post-process, nor as being used to allow large synthesis discrepancies in stereoscopic views, even at arbitrary baseline orientations.

[0264] According to some embodiments, redistribution of low-frequency components is used for multi-focal-plane (MFP) virtual viewpoint generation, where discrepancies are large. By (re)distributing low-frequency components, large variations in pixel values within and between conventional focal planes can be avoided. This, in turn, reduces high-contrast effects and visible disocclusions when shifting focal planes for new viewpoints (see 3D warping).

[0265] Regarding the execution of filtering operations The redistribution may result in a modified MFP such that each focal plane is the sum of a specified integer fraction of the low-frequency content of the entire scene and those high-frequency components specific to the focal plane (at the corresponding accommodation distance). This process replaces the MFP's specific low-frequency components with a specific portion of the total low-frequency components.

[0266] The high-frequency and low-frequency components are complementary such that, in deriving either one, the other component can be determined by subtraction. In the exemplary process shown in FIG. 47, the low-frequency components are derived and the high-frequency components are obtained as their complements (by subtraction) from the unfiltered image. This process can be reversed, or both components can be obtained directly using two filters with complementary transfer functions in the frequency domain. Because the user's accommodation distance is not determined or stored when rendering, many systems may use the same filter function for all focal planes.

[0267] Note that the original MFP is selectively filtered so that high-contrast transitions between textured and non-textured (e.g., white background) areas are not filtered (see, e.g., Figures 45A-45D). Without selective filtering, high-contrast areas would produce high-frequency components that were not part of the original image content, induce ringing around object edges, and degrade the quality of redistributed MFPs and renderings that use them.

[0268] In the disclosed examples according to some embodiments, a Gaussian low-pass filter was used with an adjustable radius (filter window). In some embodiments, it is beneficial to the final result that the filtering operation is applied only to pixels that do not belong to abrupt transitions between non-zero-value (colored) areas and zero-value (transparent or void) areas. The selective Gaussian filter does not filter image areas with color differences greater than a set threshold. Due to the generally high contrast between colored and black areas, their edges can be reliably detected, and the detection is not sensitive to the threshold.

[0269] 51A-51B are illustrations showing exemplary right and left images for a crossed-eye stereogram formed using selective Gaussian filtering when redistributing an MFP, according to some embodiments. FIGS. 51A and 51B show a crossed-eye stereogram using selective Gaussian filtering when redistributing an MFP. The parameters are typically the same as in FIGS. 50A-50B, except that filtering is not applied if the filtering changes pixel values by more than a threshold (e.g., 60 here). The exemplary disocclusion errors 5102 and 5152 are further reduced from those in FIGS. 50A-50B. The results show some contouring in smooth parts of the scene, which may be induced by some imprecision used in the implementation. FIGS. 51A and 51B are processed versions of the unprocessed images (FIGS. 45A-45D) from Non-Patent Document 1.

[0270] Support for non-Lambertian content Non-Patent Document 1 reports a method in which non-Lambertian effects for reflection or refraction are correctly presented. Many previous methods do not support this type of content, but this lack of support can be a drawback for only certain types of content.

[0271] As the disclosed method can use multiple capture and transmission formats, non-Lambertian content can be used as well, for example, by selecting an input format. Many previous systems do not support depth map generation using a depth sensor, but may support focal stacking (see non-patent document 1) or some other camera-based (disparity-to-depth) method that may be able to capture the correct pixel distance even for non-Lambertian content. Thus, the MFP received as input can cause or correct reflected or refracted image content to appear as shown at the correct distance.

[0272] In some embodiments, a focal stack-derived MFP (presentation plane) may be received as input at the front end and used without an MFP formation stage, for example, to synthesize stereo or motion parallax. In some embodiments, a method may include receiving a plurality of focal plane images, orienting the plurality of focal plane images using head information to provide stereo disparity between the left and right eyes, and displaying the oriented plurality of focal plane images.

[0273] Exemplary Embodiments for Forming a Redistributed Focal Plane A procedure for redistributing low-frequency components has been described above. This procedure may be based on decomposing a texture image into different MFPs and interpolating between the different planes using a depth blending method (e.g., linear filtering). The filtering operation for redistributing low frequencies between MFPs may be performed as a post-process.

[0274] FIG. 52A is a flowchart illustrating an example process for generating a redistributed focal plane image using low-frequency and high-frequency filtering, according to some embodiments. FIG. 52B is a flowchart illustrating an example process for generating a high-frequency focal plane image using low-pass filtering, according to some embodiments. An example process 5200 (see FIG. 52A) separates high-frequency and low-frequency content of a picture 5202 using a filter 5206, and then, at 5208, resolves the content into focal planes 5210 using picture depth data 5204. Low-frequency redistribution 5212 and MFP rendering 5214 also occur. Changing the filtering stage generates alternative MFPs. FIG. 52A is a block diagram of another redistribution process for an MFP, according to some embodiments. FIG. 52B illustrates an example process 5250 for implementing complementary low-pass and high-pass filters 5254 and 5256 of picture data 5252. The exemplary process 5200 may avoid, for example, using a selective Gaussian filter and optimizing its threshold parameters when low-pass filtering is not performed on focal planes that often contain high-contrast transitions. Correspondingly, some embodiments using the exemplary process 5200 may improve quality.

[0275] Post-process filtering may be used by the MFP process and output. However, some exemplary methods and systems disclosed herein according to some embodiments may improve quality. Some embodiments avoid the use of selective or guided filtering to exclude void areas (pixels that do not belong to the actual focal plane content) from filtering. Void areas may appear as white or black regions in the MFP and may not be correctly detected when filtered. Depending on the image content, this may appear as defects in the filtering results and reduce the quality of the redistributed focal plane. Some embodiments avoid optimizing thresholds for selective filtering, since no parameters are used.

[0276] In some embodiments, the process for forming the redistributed focal plane may include filtering an input image for low-frequency content and redistributing the low-frequency content into multiple focal plane images. In some embodiments, the process for forming the redistributed focal plane may include filtering an input image for high-frequency content and decomposing the high-frequency content into multiple focal plane images. In some embodiments, when decomposing, the high-frequency components are not moved from their original depth, as is the case when, for example, redistributing low-frequency content. An exemplary process for forming the redistributed focal plane may be as shown in FIG. 52A. In some embodiments, the high-frequency content HF shown in FIG. 52A may be derived from the low-frequency content LF shown in FIG. 52A, or vice versa.

[0277] Avoiding signed arithmetic FIG. 53 is a flowchart illustrating an exemplary process for generating redistributed focal plane images for the front, mid, and rear depth ranges according to some embodiments. When performing redistribution for the given example, a slightly modified process flow shown in FIG. 53 is used. FIG. 53 shows a block diagram of the redistribution process 5300. Three MFP redistributions 5320 are shown as an example. To prevent saturation in the image processing software, a scaling operation using constants (0.5 and 2) is used. Some embodiments use a set of focal planes (MFPs) as input. The received image (picture) 5304 may be summed from or received by the MFP block 5306 from those focal planes 5306. The MFP block 5306 may receive picture depth data 5302 in addition to the picture 5304. The MFP may be divided into a front range plane 5314, a mid-range plane 5316, and a rear range plane 5318. The redelivery process 5300 may output the forward, middle, and rear redelivered pictures. In some embodiments, arithmetic operations and components that perform arithmetic operations (e.g., ALUs) may handle signed calculations using signed arithmetic with sufficient bits.

[0278] In Figure 53, the low-pass (low frequency) components of each focal plane (one out of N) are replaced by the low-pass components of the input image by a scaling factor of 1 / N, where N is the number of focal planes used. In this example, N is 3, which corresponds to a scaling by 1 / 3.

[0279] To prevent saturation in the image processing software, a scaling operation using other constants (0.5 and multiplication by 2) is used. The downscaling by 0.5 is performed in the first addition operation, which is done by applying 50% transparency to the overlaying image. The subtraction of low frequency components from each focal plane requires downscaling by 0.5 and correspondingly multiplying the result by 2 after that operation.

[0280] Note that the low frequency components of the MFP sum (e.g., close enough) to the low frequency components of the original image, and therefore the addition and subtraction operations maintain (e.g., close enough) a division of 1 for the MFP decomposition.

[0281] The implementation illustrated in Figure 53 may cause some loss of calculation precision. In particular, some pixels may saturate to 0 in the final subtraction phase and may not scale back when multiplied by 2. This may appear as slight contouring in some parts of the image, which is not caused by the disclosed exemplary method of redistribution, according to some embodiments.

[0282] Avoiding the use of signed pixels in transformations Figure 54 is a flowchart illustrating an example process for generating a redistributed focal plane image using low-pass filtering, according to some embodiments. In performing the simulation corresponding to Figure 54, procedure 5400 was applied to avoid using signed integers per pixel (or (signed) high-frequency representations). Figure 54 shows an alternative embodiment. In some embodiments, a picture 5402 and a picture depth 5404 are received by procedure 5400. The picture 5402 may be filtered into high- and low-frequency components at 5406. The high- and low-frequency components may be used in combination with the picture depth data to generate inputs to MFP decomposition processes 5408, 5410. The picture may be decomposed into N layers 5412, 5414 for low- and high-frequency components. The decomposed layers 5412, 5414 may be redistributed 5416 and rendered 5418.

[0283] To avoid saturation in the sum (keeping the result within the 8-bit value reserved for image pixels), the redistributed LF 5416 is summed at 50% transparency. When subtracting low frequency faces (to result in a redistributed face), a corresponding scale-up is performed. One alternative embodiment to the process shown in Figures 52A and 52B avoids working with high frequency faces with signed representations. Some embodiments may use (signed) full precision arithmetic for the redistributed LF 5416.

[0284] Systems and methods disclosed herein in some embodiments create monocular MFP stacks that exhibit reduced amounts of disocclusion when used to generate stereoscopic 3D views. Some embodiments redistribute the MFPs and shift the post-processed MFPs to generate disparity and baseline orientation. The resulting stereoscopic images may be displayed, for example, on external screens and wearable displays, including those that support natural accommodation.

[0285] The systems and methods disclosed herein according to some embodiments may be used to generate stereoscopic content for services that use stereoscopic displays.

[0286] Some embodiments enable the development of stereoscopic displays that allow an easy and cost-effective way to produce baseline-free (head-tilt independent) stereoscopic rendering with relatively large, adjustable disparity and motion parallax effects. These properties may be used with 3DoF+ applications.

[0287] Many MFP display implementations use temporal or spatial (image) MFP multiplexing. Redistribution algorithms can be recognized by designing suitable test scenes with objects at different depths and by exhibiting characteristic behavior at object edges. Some embodiments can be recognized based on their ability to support stereoscopic vision and motion parallax for users moving their heads and themselves.

[0288] Some embodiments produce stereoscopic images that may cause VAC when viewed on a front stereoscopic display. MFPs formed by the systems and methods disclosed herein according to some embodiments may be rendered using MFP displays that support natural accommodation.

[0289] A synthetic mismatch (e.g., based on processing the MFP (e.g., by shifting the MFP for both S3D display and MFP display) is used in some embodiments. The systems and methods disclosed herein in some embodiments direct accommodation-dependent properties of the MFP (such as properties that are problematic for visualization in print) into spatial mismatches that are visible when printed and that indicate many properties of the corresponding MFP are in the depth dimension.

[0290] axial motion parallax A common problem with many previous systems is that motion parallax is supported only for small lateral (typically horizontal) movements and not for axial movements. Multi-focal plane (MFP) is an economical approximation of light field. Many previous MFP displays support only a single precise viewpoint and do not support viewpoint changes as the viewer moves in the depth (axial) dimension. Therefore, the use of such previous systems is generally limited to fixed viewer locations.

[0291] Many previous 360-degree panoramic rendering systems use orientation sensors to support viewing in various directions, but do not support axial motion parallax. These three-degree-of-freedom (3DOF) systems constrain users only to changes in viewing direction. Many six-degree-of-freedom (6DOF) systems, such as computed and automated virtual environments (CAVEs), are typically costly and not feasible for broadcast systems. The ability to bring small, general user movements to a 3DOF system is described as 3DOF+. In this application, according to some embodiments, the term "natural 3D movement" refers to this 3DOF+ capability.

[0292] Enabling 3DOF+ systems with reduced complexity and legacy 3DOF or 2D stereoscopic content is a challenge addressed by the systems and methods disclosed herein in several embodiments. An additional challenge is to enable natural accommodation of the displayed content (avoiding vergence accommodation conflict (VAC)) in these new 3DOF+ solutions.

[0293] 55A-55B are schematic perspective views 5500, 5550 illustrating exemplary multi-focal planes rendered in space, according to some embodiments. FIGS. 55A and 55B illustrate a nominal viewpoint 5504, shown by a dashed line, relative to a focal plane (MFP) 5506, shown by a solid rectangle. If the viewer 5502 moves, instead of viewing information (focal planes) from the nominal origin viewpoint, new viewpoints may be generated. Focal planes are approximations of a 3D scene, and new viewpoints may be generated in a manner similar in some respects to, for example, warping a 3D model to a new orientation (see the Virtual Viewpoint Generation block). In FIG. 55A, the user views focal planes rendered in space by the user's MFP glasses. In FIG. 55B, motion parallax is supported for a user inside an exemplary volume (dx, dy, dz) 5554 around the nominal viewpoint (0,0,0) 5552, such as the exemplary cube shown.

[0294] Motion parallax varies the perspective relative to the focal plane around a nominal viewpoint (0,0,0). If the user moves away from the nominal point, the focal plane image is seen from a slightly varying viewpoint and can simply be translated (shifted) and scaled accordingly to create a correct MFP rendering. Each coordinate (x, y, and z) can be varied within some limits to keep the distortion created acceptable. Motion parallax can thus be supported inside a volume around the nominal viewpoint (such as the cube shown in FIG. 55B). The cube around the viewer's nominal position can be selected to support 3D motion parallax (and capture user motion). In some embodiments, a sphere or other shape can be selected to support the capture of user motion for motion parallax.

[0295] 56A-56C are schematic perspective views illustrating three examples of multiple focal planes that are shifted and scaled for different viewpoints, according to some embodiments. A user's movement can change his or her viewpoint relative to the focal planes, as indicated by the change in dashed lines. FIGS. 56A-56C show several example views 5600, 5630, 5660 for changing viewpoints 5602, 5632, 5662 inside motion parallax cubes 5604, 5634, 5664. The dashed lines indicate viewpoint changes relative to the focal plane stack. New viewpoints relative to the MFP stack can be created by deforming the MFP accordingly. The mathematics for performing the deformations can be similar in some respects to warping a 3D model. In some embodiments, the deformations can be approximated using other operations, as described below.

[0296] Each 3D coordinate or voxel within the cube represents a different viewpoint for the focal plane stack (or pair of stacks for a stereoscopic display). For each of these viewpoints, a transformation of the original MFP stack (viewed from the nominal viewpoint) is performed. Thus, for each 3D viewpoint / voxel, there may be a unique MFP stack (reoriented and scaled by the transformation). In some embodiments, a set of MFP stacks may be pre-derived and stored. These derived MFP stacks may be categorized by the captured motion information.

[0297] FIG. 57 is a process diagram illustrating an example process for generating multiple focal planes within a motion parallax system, according to some embodiments. FIG. 57 illustrates a process 5700 for monoscopic MFP formation, processing (e.g., viewpoint synthesis for stereoscopic video and motion parallax), and rendering. For simplicity, creating the motion parallax effect is illustrated for only one viewer. In some embodiments, the system may include one or more wearable user terminals (e.g., head-mounted displays, HMDs, etc.) that allow each user to individually perceive motion parallax. Some embodiments may use, for example, a more distributed or more centralized architecture.

[0298] Texture 5702 and depth (image plus depth) 5704 signals are received as input to MFP formation block 5706, which forms a multi-focal plane (MFP) using a depth map (image) captured from the scene. The MFP is sent to mismatch formation block 5708.

[0299] The mismatch formation block 5708 calculates the vector r i5710. Create a synthesized stereo viewpoint (or disparity) using the specified baseline orientation and length (denoted by ). Some embodiments may use the disparity direction and amount (≧0%) 5710. Some embodiments may omit this step, for example, when a stereo input is used where separate texture and depth signals are used for each eye.

[0300] Motion detection and tracking (e.g., using an inertial measurement unit (IMU) 5720 and / or Kinect type 5718 of external sensors 5714) may be used to capture user movement relative to the user's initial viewing position using a nominal viewing setup (focal plane size and distance) for MFP rendering. In the exemplary process 5700, and in other processes, other motion detection and tracking devices may be used instead or in addition. The lateral motion parallax block 5712 may shift the two stereoscopic viewpoints in both the x and y dimensions. The focal planes in each MFP stack may be shifted by different amounts corresponding to their distance in depth.

[0301] The axial motion parallax module 5716 may adjust the focal planes by scaling and / or sizing them according to the measured axial motion relative to the initial / nominal position and viewpoint. Different focal planes may be scaled differently because they correspond to different distances.

[0302] When viewing broadcast types of content, user motion may generally not affect the input stream (the viewpoint used to capture the view). Thus, after the user 5724 experiences motion parallax due to body movement, the changed viewpoint can be translated back to the nominal viewpoint by the selected mechanism (delay, rate, trajectory, and time).

[0303] In some embodiments, motion tracking sensor readings of the viewer's position may be measured relative to the focal plane display to generate head orientation information. In some embodiments, in response to generating the head orientation information, motion parallax may be synthesized by altering one or more of the focal plane images.

[0304] Some previous devices use MFPs for viewpoint changes when supporting (baseline-less) stereoscopic vision and lateral motion parallax. The following section describes axial motion parallax. Focal plane adjustments for intermediate or compound motion can be derived by applying these two cases. For example, axial motion parallax can be created in some embodiments by scaling one or more focal planes.

[0305] FIG. 58 is a schematic plan view illustrating exemplary focal plane scaling for axial movement, according to some embodiments. While FIG. 58 is described with respect to a monoscopic viewpoint relative to an MFP stack during axial movement, corresponding processes may be performed with each viewpoint transformation in a stereoscopic system or method. Similar to the case of translational movement, a focal image transformation may be performed to handle axial movement. In the exemplary monoscopic viewpoint environment 5800, when a viewer 5810 approaches the MFP stack axially, the edge of the focal plane 5808 may be adjusted to move (or shift) outward. When a viewer 5810 moves axially away from the MFP stack, the edge of the focal plane 5806 may be adjusted to move (or shift) inward. The edge of the focal plane 5804, located at the zero parallax setting (ZPS) 5802, remains in place (or unadjusted) as the viewer 5810 moves axially toward or away from the MFP stack. The change in the viewer's 5810 viewpoint due to axial movement is shown as a dashed line for the two examples. Figure 58 illustrates how the apparent size of an MFP changes during axial movement. Three MFPs are shown with equal amounts of separation. The zero parallax setting (ZPS) 5802 is located at the farthest focal plane. This farthest focal plane does not shift when creating stereoscopic disparity or lateral parallax, and this farthest focal plane is not scaled when creating axial motion parallax. While single-step axial movement is shown in the exemplary monoscopic viewpoint environment 5800, multiple axial step movement can be supported using a similar process of scaling the MFP by an amount proportional to the number of axial steps moved by the viewer 5810.

[0306] If the viewer 5810 moves one step (e.g., one distance unit) axially closer to the MFP stack, the size of the focal plane 5808 may be adjusted one unit (e.g., one distance unit) larger. If the viewer 5810 moves one step axially away from the MFP stack, the size of the focal plane 5806 may be adjusted one unit smaller. The size of the focal plane 5804 located at the zero parallax setting (ZPS) 5802 remains the same size (e.g., or unchanged) as the viewer 5810 moves axially closer to or farther from the MFP stack. The change in the viewpoint of the viewer 5810 due to axial movement is shown as a dashed line for the two examples.

[0307] In some embodiments, the scaling transformation of one or more focal plane images may be approximated by adjusting the size of the focal plane symmetrically around its center so that the aspect ratio of the image is preserved. Small changes to motion parallax, such as in creating stereoscopic mismatch, are captured by small changes in focal plane image resolution (increments or decrements up to a threshold, such as a 5-pixel change). In some embodiments using higher resolution images, a higher threshold may be used.

[0308] When adjusting the image size, the top and bottom edges of the image may be moved in opposite directions by an amount (the vertical size of the image is changed by a number of pixels). The scaling factor of the MFP may vary linearly as a function of depth, as shown in the example of FIG. 58. This linear variation results in a set of deformed MFPs and a set of applicable viewing positions (focal points) along the viewing axis. The amount of adjustment depends on the position (distance) of the focal plane in depth, such that the farthest plane is not adjusted (as if the farthest plane were located infinitely far from the viewer). In some embodiments, the focal plane next to the farthest focal plane is scaled by one unit (or pixel multiplication). The next closest focal plane may be scaled by two units in some embodiments. The amount of scaling for the remaining focal planes may, in some embodiments, follow this paradigm for progressively closer focal planes.

[0309] An exemplary process for scaling an MFP according to some embodiments is described below for scaling N focal planes (MFPs). Herein, the focal planes are separated by a uniform distance and use linear increments and decrements as a function of distance. In some embodiments, non-uniform separation distances may be used with non-linear increments and decrements as a function of distance. Non-linear scaling (linear in the dioptic scale) may be used to produce a linear perception by the user. The size of the focal planes (or set of MFPs) is obtained for several unequal steps along the viewing axis. This exemplary process may be used, for example, for size adjustment validation. In some embodiments, more rigorous geometric operations may be performed.

[0310] If the viewer moves one step closer to the MFP stack, the farthest focal plane (FP1) is the zero parallax plane ("at infinity") and is not scaled (or adjusted in size). Therefore, FP1 is adjusted by 0 units. The remaining focal planes (FP2, FP3, ..., FP N ) for each focal plane FP i is adjusted up by (i-1) units, with i ∈ (2, 3, ..., N). This adjustment process may be repeated for each closer step to the MFP stack.

[0311] If the viewer moves one step further away from the MFP stack, the farthest focal plane (FP1) is the zero parallax plane ("at infinity") and is not scaled (or adjusted in size). Therefore, FP1 is adjusted by 0 units. The remaining focal planes (FP2, FP3, ..., FP N ) for each focal plane FP i is adjusted down by (i-1) units, with i ∈ (2, 3, ..., N). This adjustment process may be repeated for each step further away from the MFP stack.

[0312] In some embodiments, the height of the focal plane may be increased in size by scaling the height by the incremental scaling factor shown in Equation 3.

[0313]

number

[0314] where V res is the vertical resolution of the focal plane. Similarly, the height of the focal plane can be reduced in size by scaling the height by the reduction scaling factor shown in Equation 4.

[0315]

number

[0316] where V res is the vertical resolution of the focal plane. The width of the focal plane for both increments and decrements is adjusted to preserve the aspect ratio before adjustment.

[0317] In some embodiments, moving in steps using the process described above is not linear on a metric scale. The physical distance may depend on the nominal viewing conditions, including the position, size, and distance of each focal plane (MFP). In addition, the physical distance may depend on the (input) content properties, such as the chosen zero parallax setting (ZPS), and the correspondence between the depth map and the physical distance.

[0318] Using an exemplary process according to some embodiments, the focal plane image resolution and unit size used may determine the position along the viewing axis where the focal plane is seen to be aligned (focused on the viewpoint) after the transformation. These points may be used to render content during user movement.

[0319] The higher the image resolution and the smaller the unit size, the smaller the difference between the determined axial (focused) positions. The unit size used in conjunction with the image resolution can be experimentally verified and / or calculated from geometry to ensure that the density of focal points supports the perception of smooth motion parallax.

[0320] Shifting and scaling artifacts may limit axial parallax (motion) to a small amount that the user determines is reasonable. This amount may be set to a small number of steps (or axial positions). Step positions and sizes (physical distances) may be approximated from a geometric shape (such as the geometric shape shown in FIG. 58).

[0321] In some embodiments, the nominal viewing position and focal plane distance are not set. In some embodiments, setting the unit size used for MFP size adjustment may determine the step position and size (focal point and associated distance) in axial direction. In some embodiments, the steps may be linearly mapped to distance without degradation of user-determined naturalness.

[0322] In some embodiments, the motion parallax effect may not be calibrated to physical location, such as when conditions for 3D content production (which may include mismatches and depth ranges) are not standardized or set. Some users may be relatively insensitive to depth information in terms of the perception of naturalness of (axial) motion parallax.

[0323] 59A-59C are schematic plan views illustrating exemplary deformations of exemplary focal plane content during axial motion, according to some embodiments. FIG. 59A shows an MFP configuration 5900 for a nominal "start" point of exemplary mismatch shift 5902 for a viewer's left eye. A process similar to that using the monoscopic view of FIG. 59A may be applied to scale the right eye view of a stereoscopic viewer. Distance d k denotes the focal plane distance of the vector with k equal to 1, 2, …, N. The distance de denotes the disparity distance vector for an exemplary MFP transformation (or translation). k ,y k ) indicates the original coordinates of the pixel on the focal plane k. k’ ,y k’ ) 5904 shows the coordinates of a pixel in focal plane k after the transformation (or translation in some embodiments) but before scaling. k’ ,y k’ ) 5904 is an exemplary left-eye viewpoint focal plane F of three focal planes. N 59A to 59C show the focal planes F1, F2, ..., F for an example of N equal to 3. N Shows.

[0324] Figure 59B shows an MFP configuration 5930 for axial motion of a viewer approaching the MFP stack. The viewer moves in a direction parallel to the motion vector d z 5932. The left eye viewpoint of the three focal planes shows that each focal plane is larger in Figure 59B than in Figure 59A. The left eye viewpoint is translated due to the viewer's proximity to the MFP stack.

[0325] FIG. 59C shows an MFP configuration 5960 for focal plane deformation that approximates axial motion without changing the focal plane distance. The left eye viewpoints of the three focal planes show the changes to pixel coordinates after deformation. Coordinates (x k’’ ,y k’’ ) 5962 shows the coordinates of the pixel in focal plane k after the transformation (or translation in some embodiments) and after scaling.

[0326] Keeping the previous notion of relevant parameters, (x k’’ ,y k’’ ) denote the coordinates of the pixel on the focal plane k after the scaling transformation, the scaling can be written as shown in Equation 5. (x k’’ ,y k’’ )=S(x k’ ,y k’ ,d z ,d e) Equation 5 where S is a uniform scaling transformation using each focal plane center as a fixed point.

[0327] For axial motion, the focal plane size may be adjusted to provide the user with the illusion of a deformation of physical space and coordinates through their view of the rendered environment. The focal plane rendering process may not change accommodation distance when experiencing axial motion parallax. Correspondingly, there is no adjustment of the optics or mechanics of the MFP display. This process differs from the real-world case, as the viewer's accommodation distance changes corresponding to their distance from the focused object. However, no VAC occurs, and if the (supported) movement to experience motion parallax is small, the lack of accommodation change does not degrade the user's naturalness experience.

[0328] Some embodiments may support motion parallax effects for a traditional 2D display (as an alternative or addition to an MFP display). Using a 2D display may avoid the complexity of an MFP display, but the user may experience VAC. Some embodiments may support a 2D display by combining two or more MFPs. Support for 2D displays is described in more detail below.

[0329] If support for axial movement is provided along with support for lateral (side-to-side) movement and baseline-free (head-tilt-free) motion parallax, realistic 3D motion parallax around a nominal viewpoint using focal plane rendering can be provided without VAC.

[0330] In some embodiments, relative motion around a primarily fixed (nominal) viewpoint does not change the viewpoint without receiving new data for the new nominal viewpoint. Although the viewing setup may be comparable to watching broadcast content, the viewing setup disclosed herein according to some embodiments may support user interaction by allowing small movements and corresponding viewpoint / parallax changes (such as 3DOF+ functionality).

[0331] Supported user features may not affect the nominal or intrinsic viewpoint of the received content. Small motion parallax may be supported in the same manner for each viewing position in the user's environment, for example, when the user changes his or her seat or walks around the room. If the user moves, the nominal viewpoint follows his or her average location. In some embodiments, the viewpoint returns to the nominal viewpoint (which in some embodiments may involve a delay after creating motion parallax). Returning to viewing from the nominal viewpoint may be based on manual input by the user and / or an automatic policy.

[0332] FIG. 60 is a graph illustrating exemplary viewpoint coordinate tracking during user movement, according to some embodiments. Large position changes may be treated, for example, as a change in the user's seating location and may not be followed by a small adjustment in parallax. Instead, the viewpoint may be returned to the nominal viewpoint of the received content without any change in parallax. Small movements may be handled with a corresponding virtual viewpoint change. If the small movement change is more permanent, for example, if the user is resting their body on a chair back, the nominal viewpoint will return to the nominal viewpoint at some rate of change in some embodiments.

[0333] The graph in FIG. 60 illustrates an example control policy for controlling viewpoint coordinates (e.g., axial directions) during user movement. FIG. 60 shows the change in coordinate compared to a nominal value at various time steps. In example graph 6000, the zero line represents the nominal value, and the trace on the graph measures the difference value 6004 compared to the nominal value. Between time step 28 and time step 40 (6002), the coordinate value drops toward zero. At time step 67 (6006), user movement moves outside the maximum and minimum disparity limits 6008, 6010, and tracking of user movement is reset to zero. A viewpoint control process similar to the example viewpoint control process used with respect to FIG. 60 may be used for each axis of the coordinate system (e.g., one axial coordinate value and two lateral coordinate values).

[0334] In some embodiments, coordinates may be saturated to a maximum (or minimum) value, allowing each coordinate system axis to drift toward 0 after user motion stops (e.g., after the viewer sits in a different chair). In some embodiments, if user motion goes outside the maximum and minimum limits, and if the user's viewing orientation goes outside the maximum and minimum limits, the user's viewpoint may be returned to the nominal viewpoint. In some embodiments, a user changing his or her viewing orientation appears as a lateral (xy) shift of the focal plane, and the user's viewpoint may be returned to the nominal viewpoint after motion stops or exceeds a maximum or minimum value. It will be understood that the process of returning the user's viewpoint to the nominal viewpoint may be a manual interaction process, an automatic process, or a combination of manual and automatic processes. Applicable policies for controlling 3D motion parallax may be developed according to some embodiments.

[0335] In some embodiments, the described process for synthesizing a stereoscopic or shifted viewpoint from a monoscopic MFP stack may be extended to other configurations, such as motion parallax using a true (e.g., as opposed to synthesized) stereoscopic MFP stack, motion parallax using a synthesized stereoscopic MFP stack, motion parallax using projection and rendering of an MFP on a 2D display, and MFP formations that are combinations of a true stereoscopic MFP stack, a synthesized stereoscopic MFP stack, and an MFP used with a 2D display.

[0336] FIG. 61 is a process diagram illustrating an example process for generating a true stereo set of multi-focal planes, according to some embodiments. FIG. 61 shows a motion parallax MFP generation and rendering process 6100 using true stereo input. A right eye MFP creation process 6110 receives a texture 6102 and a depth map 6104 for the right eye. A left eye MFP creation process 6112 receives a texture 6102 and a depth map 6108 for the left eye. In some embodiments, the MFPs for the right and left eyes are sent to a lateral disparity adjustment process 6114. A disparity adjustment 6116 for direction and amount (≧0%) may be received by the lateral disparity adjustment process 6114 to adjust the viewpoint separation of the MFPs. The use of stereo MFP input may limit the reorientation and adjustment of the baseline MFP. The rendering of the MFP is anchored to a position and direction in space. The adjusted MFPs for the left and right eyes are sent to a lateral parallax adjustment process 6118. The motion tracking process 6120 may send user motion data to a lateral parallax adjustment process 6118 and to scale adjustment processes for axial parallax 6122, 6124. The motion data may be represented as x(t) and y(t) data captured relative to a nominal viewpoint and orientation for the MFP anchored in space. In some embodiments, for example, an inertial motion unit (IMU) 6130 and / or a Kinect-type unit 6132 may be used to measure the motion data. The lateral parallax adjustment process 6118 may translate the MFP for lateral motion parallax (e.g., horizontal viewpoint change for parallax). The focal planes may be scaled relative to each other for axial parallax from the new viewpoint. The lateral parallax-adjusted MFPs for the right and left eyes may be received by scale adjustment processes for axial parallax 6122, 6124. The scaled MFP for the right eye may be received by an MFP rendering process 6126. Similarly, a scaled MFP for the left eye may be received by MFP rendering process 6128. The MFP is rendered and displayed for a new viewpoint / perspective for true stereoscopic viewing by user 6134.In some embodiments, synthetic motion parallax may involve scaling multiple focal planes relative to one another using information indicative of the axial movement of the user / viewer (such as motion tracking data).

[0337] Figure 62 is a process diagram illustrating an example process for generating a multi-focal plane synthesized stereo set, according to some embodiments. Figure 62 shows a motion parallax MFP generation and rendering process 6200 for generating a synthesized stereoscopic MFP stack. An MFP creation process 6206 receives a texture 6202 and a depth map 6204 for the monoscopic view. The MFP is sent to a lateral mismatch creation process 6208. A mismatch adjustment 6210 for direction and amount (≧0%) may be received by the lateral mismatch adjustment process 6208 to adjust the viewpoint separation of the MFP. The rendering of the MFP is anchored to a position and orientation in space. The lateral mismatch creation process 6208 generates MFPs for the left and right eyes (e.g., using a baseline orientation), which are sent to a lateral parallax adjustment process 6212. In some embodiments, some baseline orientation before lateral or axial viewpoint changes may be selected to support mismatch creation. The motion tracking process 6214 may send user motion data to a lateral parallax adjustment process 6212 and to scale adjustment for axial parallax processes 6216, 6218. The motion data may be represented as x(t) and y(t) data captured relative to a nominal viewpoint and orientation for the MFP anchored in space. In some embodiments, for example, an inertial motion unit (IMU) 6224 and / or a Kinect-type unit 6226 may be used to measure the motion data. The lateral parallax adjustment process 6212 may translate the MFP for lateral motion parallax (e.g., horizontal viewpoint change for parallax). The focal planes may be scaled relative to each other for axial parallax from the new viewpoint. The lateral parallax-adjusted MFPs for the right and left eyes may be received by scale adjustment for axial parallax processes 6216, 6218. The scaled MFP for the right eye may be received by an MFP rendering process 6220. Similarly, a scaled MFP for the left eye may be received by MFP rendering process 6222. The MFP is rendered and displayed for a new viewpoint / perspective for synthesized stereoscopic viewing by user 6228.Capturing a monoscopic view may use less bandwidth in transmission. Furthermore, when combining two viewpoints within a receiver, these two viewpoints may be formed at a selected baseline orientation and amount of disparity to support personal preferences for stereoscopic and depth perception at any head tilt. While some artifacts may be generated when combining the views, these artifacts may be reduced by using focal plane redistribution and / or expansion.

[0338] Figure 63 is a process diagram illustrating an example process for generating and summing a set of multi-focal planes for a two-dimensional display, according to some embodiments. Figure 63 shows a motion parallax MFP generation and rendering process 6300 for generating and summing an MFP stack for a 2D display. An MFP, disparity, and disparity formation process 6306 receives a texture 6302 and a depth map 6304 for the monoscopic view. A disparity adjustment 6308 for orientation and amount (≧0%) may be received by the MFP, disparity, and disparity formation process 6306 to adjust the viewpoint separation of the MFP. The MFP may be generated for a position and orientation in space. A motion tracking process 6310 may send user motion data to the MFP, disparity, and disparity formation process 6306. The motion data may be represented as x(t), y(t), and z(t) data captured relative to a nominal viewpoint and orientation for the MFP anchored in space. In some embodiments, an inertial motion unit (IMU) 6316, e.g., with near-eye display (NED) glasses, and / or a Kinect-type unit 6318, e.g., with a flat screen, may be used to measure the motion data. In some embodiments, an MFP, mismatch, and parallax formation process 6306 may perform one or more of the following processes: forming the MFP; adjusting the MFP for lateral mismatch; adjusting the MFP for lateral motion parallax (horizontal viewpoint change in parallax); and scaling for axial parallax. The focal planes may be scaled relative to each other for axial parallax from the new viewpoint. The MFPs are summed by an MFP summation process 6312. The summed MFP is rendered and displayed by a 2D NED or flat screen 6314 for viewing by a user 6320. In some embodiments, the 2D display lacks support for natural accommodation but may exhibit motion parallax effects that follow user movement. In some embodiments, a stereoscopic three-dimensional (S3D) display may be used to render and display the summed MFP.

[0339] Figures 64, 65, and 66 illustrate some embodiments for projecting and summing MFPs for a 2D display. Figure 64 is a process diagram illustrating an example process for generating and summing a set of multi-focal planes for a monocular 2D display, according to some embodiments. Figure 64 shows a motion parallax MFP generation and summation process 6400. An MFP formation process 6406 receives a texture 6402 and a depth map 6404 for the monoscopic view. The MFP is sent to a lateral parallax adjustment process 6408. The adjusted MFP is sent to a lateral parallax adjustment process 6408. A motion tracking process 6410 may send user motion data to the lateral parallax adjustment process 6408 and to a scale adjustment process 6412 for axial parallax. The motion data may be represented as x(t) and y(t) data captured relative to a nominal viewpoint and bearing for the MFP anchored in space. In some embodiments, an inertial motion unit (IMU) 6418 and / or a Kinect-type unit 6416 may be used to measure the motion data. A lateral parallax adjustment process 6408 may translate the MFP for lateral motion parallax (horizontal viewpoint change in parallax). The focal planes may be scaled relative to each other for axial parallax from the new viewpoint. The lateral parallax adjusted MFP may be received by a scale adjustment for axial parallax process 6412 for scaling the MFP. The scaled MFP may be received by an MFP summation process 6414. The MFPs are summed by the MFP summation process 6414. The summed MFP is rendered and displayed by a 2D NED or flat screen 6420 (monocular 2D display) for viewing by a user 6422.

[0340] FIG. 65 is a process diagram illustrating an exemplary process for generating and summing a set of multiple focal planes for a true stereoscopic three-dimensional (S3D) display, according to some embodiments. In some embodiments, baseline changes may be supported along with limitations on true stereoscopic 3D input. The input stereoscopic viewpoint (e.g., baseline orientation) may impose some restrictions on subsequent viewpoint changes (e.g., lateral or axial changes). FIG. 65 shows a motion parallax MFP generation and summation process 6500 using true stereoscopic input. A right eye MFP creation process 6510 receives a texture 6502 and a depth map 6504 for the right eye. A left eye MFP creation process 6512 receives a texture 6502 and a depth map 6508 for the left eye. In some embodiments, the MFPs for the right and left eyes are sent to a lateral mismatch adjustment process 6514. A mismatch adjustment 6516 for direction and amount (≧0%) may be received by the lateral mismatch adjustment process 6514 to adjust the viewpoint separation of the MFPs. The use of stereoscopic MFP input may limit reorientation and adjustment of the baseline MFP. MFPs may be generated relative to a position and orientation in space. The adjusted MFPs for the left and right eyes are sent to a lateral parallax adjustment process 6518. A motion tracking process 6520 may send user motion data to the lateral parallax adjustment process 6518 and to scale adjustment processes 6522, 6524 for axial parallax. The motion data may be represented as x(t) and y(t) data captured relative to a nominal viewpoint and orientation for the MFP anchored in space. In some embodiments, an inertial motion unit (IMU) 6530 and / or a Kinect-type unit 6532 may be used to measure the motion data. The lateral parallax adjustment process 6518 may translate the MFP for lateral motion parallax (horizontal viewpoint change in parallax). The focal planes may be scaled relative to each other for axial parallax from the new viewpoint. The lateral parallax adjusted MFPs for the right and left eyes may be received by scale adjustment processes for axial parallax 6522, 6524. The scaled MFP for the right eye may be received by a sum MFP for right eye process 6526.The MFP for the right eye is summed by a MFP sum for right eye process 6526. Similarly, the scaled MFP for the left eye may be received by a MFP sum for left eye process 6528. The MFP for the left eye is summed by a MFP sum for left eye process 6528. The summed MFP for both the right and left eyes is sent to a 2D NED or flat screen 6534 (monocular 2D display) for rendering and display to a user 6536.

[0341] Figure 66 is a process diagram illustrating an example process for generating and summing a set of multiple focal planes for a synthesized stereoscopic three-dimensional (S3D) display, according to some embodiments. Figure 66 shows a motion parallax MFP generation and summation process 6600. An MFP formation process 6606 receives a texture 6602 and a depth map 6604 for the monoscopic view. The MFP is sent to a lateral mismatch formation process 6608. A mismatch adjustment 6610 for direction and amount (≧0%) may be received by the lateral mismatch adjustment process 6608 to adjust the viewpoint separation of the MFP. The MFP may be generated for position and direction in space. The lateral mismatch formation process 6608 generates MFPs for the left and right eyes, which are sent to a lateral parallax adjustment process 6612. The motion tracking process 6614 may send user motion data to a lateral parallax adjustment process 6612 and to scale adjustment for axial parallax processes 6616, 6618. The motion data may be represented as x(t) and y(t) data captured relative to a nominal viewpoint and orientation for the MFP anchored in space. In some embodiments, an inertial motion unit (IMU) 6626 and / or a Kinect-type unit 6628 may be used to measure the motion data. The lateral parallax adjustment process 6612 may translate the MFP for lateral motion parallax (horizontal viewpoint change in parallax). The focal planes may be scaled relative to each other for axial parallax from the new viewpoint. The lateral parallax-adjusted MFPs for the right and left eyes may be received by scale adjustment for axial parallax processes 6616, 6618. The scaled MFP for the right eye may be received by an MFP sum for right eye process 6620. The MFP for the right eye is summed by a sum MFP for right eye process 6620. Similarly, the scaled MFP for the left eye may be received by a sum MFP for left eye process 6622. The MFP for the left eye is summed by a sum MFP for left eye process 6622. The summed MFP for both the right and left eyes is sent to an S3D NED or flat screen 6624 for rendering and display to a user 6630.Support for motion parallax effects using 2D display data may be provided by summing MFPs to produce 2D display data that can be displayed by an external stereoscopic or monoscopic display.

[0342] In some embodiments using a monoscopic display, external motion tracking sensors may be used to track user motion, and the user may not be wearing shutter glasses. In some embodiments using a stereoscopic three-dimensional (S3D) display, sensors (e.g., IMUs) that track user motion may be embedded within the shutter glasses, etc. In some embodiments using an S3D display, some vergence accommodation (VAC) conflict may occur. This conflict is small when the viewing distance is long compared to the display size (e.g., see normal viewing distances for stereoscopic TVs). In some embodiments, displaying the oriented focal plane images may include combining (e.g., summing) shifted focal plane images to generate a stereoscopic image pair.

[0343] FIG. 67 is a process diagram illustrating an example process for generating multiple focal planes with a front-end processing path and a display-end path, according to some embodiments. FIG. 67 shows a motion parallax MFP generation and summation process 6700 using true stereo input. In some embodiments of the front-end process 6702, the texture 6706 and depth map 6708 may be received by an MFP formation 6714 using a depth blending process. The MFP may be generated using depth blending (weighting) to reduce the quantization effects of the MFP in the depth dimension, as previously described. In some embodiments of the front-end process 6704, the texture 6710 and depth map 6712 may be received by an MFP formation 6716 using a redistribution process. The MFP may be generated using MFP redistribution, as previously described.

[0344] In some embodiments, the depth blended and / or redistributed MFP may be sent to a mismatch formation process 6718. A mismatch adjustment 6720 for direction and amount (≧0%) may be received by the mismatch formation process 6718 to adjust the viewpoint separation of the MFP. The adjusted MFP may be sent to a lateral parallax formation process 6722. A motion tracking process 6724 may send user motion data to the lateral parallax formation process 6722 and to a scale adjustment process 6726 for axial parallax. The motion data may be represented as x(t) and y(t) data captured relative to a nominal viewpoint and direction for the MFP anchored in space. In some embodiments, an inertial motion unit (IMU) 6730 and / or a Kinect-type unit 6732 may be used to measure the motion data. The lateral parallax formation process 6722 may translate the MFP for lateral motion parallax (horizontal viewpoint change of parallax). A scale adjustment process 6726 for axial parallax may receive the translated MFP for scaling. The focal planes may be scaled relative to each other for axial parallax from the new viewpoint. In some embodiments, the front end may include a direct input of focal planes 6742 (e.g., without a depth map) formed using known methods, such as the method disclosed in Non-Patent Document 1, and one or more of the processing stages 6718, 6720, 6722, 6724, 6726 may be used as described above.

[0345] In some embodiments of display path 1, the scaled MFP may be received by an MFP rendering process 6728 for display to a user 6738. In some embodiments of display path 2, the scaled MFP may be received by an MFP summing process 6734 for summing the received MFP. The summed MFP may be received by a 2D NED or flat screen display 6736 for rendering and displaying the content to a user 6740.

[0346] Figures 68A-68J are illustrations showing a series of exemplary right and left images for axial movement between each set image pair (e.g., Figures 68C and 68D), according to some embodiments. Figures 68A-68J show a series of crossed-eye stereograms during axial movement using the arrangement shown in Figure 58 and the focal plane size adjustment process described above. Figures 68A-68J are modified versions of the images and their depth maps found in Non-Patent Document 12. For the exemplary sequence of Figures 68A-69J, three MFPs with linear depth blending are used. A unit size of two pixels is used for image scaling adjustment. The test image used is a flower with a resolution of 157 x 436 pixels. For low-resolution focal plane images, users may find a unit size of two pixels reasonable. Higher-resolution focal plane images may use a larger unit size.

[0347] After forming the monocular MFP stack, stereoscopic disparity is synthesized by shifting the focal plane horizontally. In the exemplary sequence of crossed-eye stereograms, the three planes from rear to front (rear, middle, and front) are shifted by 0, 2, and 4 pixels, respectively. Shifting occurs to the right for left-eye views 6810, 6830, 6850, 6870, and 6890, and to the left for right-eye views 6800, 6820, 6840, 6860, and 6880. The synthesized total disparity corresponds to 1.8% of the image width. After shifting for stereoscopic disparity, the focal plane is scaled to create axial motion parallax, as illustrated in FIG. 67. Viewing the sequence from top to bottom in FIGS. 68A through 68J corresponds to moving away from the scene. All views in FIGS. 68A through 68J are formed by shifting from a single monoscopic MFP stack. Lateral parallax is not illustrated in this example.

[0348] Applying the exemplary procedure described above for scaling the three focal plane test images, the vertical resolution heights listed in Tables 1 and 2 are obtained for five axial movements (+2, +1, 0, -1, -2 steps) around the nominal viewing point d0. The actual scaling factor can be obtained by dividing the table value by Vres.

[0349] [Table 1]

[0350] [Table 2]

[0351] Table 3 shows the axial position d i shows the vertical resolution adjustment for PFback. The horizontal resolution is adjusted to preserve the aspect ratio before the adjustment. The unit size of scaling is 2 pixels (with the corresponding focal plane edge symmetrically shifted by 1 pixel). The size of both the left eye view (focal plane image) and the right eye view (focal plane image) is the same as at the nominal viewpoint d0 (157x436 pixels in this example). During movement, the size remains unchanged (157x436 pixels) for the farthest focal plane PFback, which is treated as being at infinity.

[0352] As shown in Table 3, the axial movement (position d i ), the scaling is the same for both the left and right eye focal planes (shifted due to mismatch). In the original test image, Vres = 157 pixels. After scaling, the vertical resolutions for both the left and right focal planes (FP1, FP2, FP3) for five different viewpoints are shown in Table 3. Position d0 is the nominal viewpoint.

[0353] [Table 3]

[0354] The actual scaling factor for focal plane size is V, which is the value in Table 3. res = 157. By applying these scaling values to the three focal planes of the test image, we obtain the sequence of crossed-eye stereograms for axial motion shown in Figures 68A-68J. Viewing the sequence from top to bottom in Figures 68A-68J corresponds to moving away from the scene. To provide a reference for scaling, black rectangles are shown in Figures 68A and 68I. Each row pair (e.g., Figures 68C and 68D) corresponds to positions along the same viewing direction (or axis). Figures 68A-68J demonstrate the motion parallax effect.

[0355] The axial step size (the physical distance corresponding to each row in Figures 68A-68J, such as Figures 68C and 68D, forming a row showing one cross-eye stereoscopic view of the sequence) may be determined by the resolution and unit size used. This example shows axial disparity (based on focal plane scaling) for a stereo pair with a compounded disparity (based on shifting the focal plane by a chosen amount). Lateral disparity can be created by applying the same (or similar) transformation as for creating the (lateral) disparity. Any 3D motion that combines lateral and axial components can be determined by applying the transformation (or approximation in some embodiments) used in this example.

[0356] When viewing focal planes from a nominal position, they will all be seen within the same solid / spatial angle. Because the unadjusted focal plane resolution is the same, one pixel occupies the same solid angle, and focal planes for perceived MFP rendering (or projection, as shown in Figures 68A-68J) may be summed. When adjusting focal plane size for synthesized motion parallax, the resolution and size may be adjusted before summing. Pixels may be shifted by sub-pixel amounts relative to each other. To ensure accurate summation of corresponding pixels, image interpolation may be performed to obtain sub-pixel values.

[0357] In the sequences shown in Figures 68A-68J, the zero parallax setting for the stereogram, and the vanishing point of axial motion parallax, is the farthest focal plane. Axial motion parallax can be extended by adjusting the size of all focal planes. This situation corresponds to placing the zero parallax plane behind all of the focal planes and using a scaling factor derived for one lower focal plane, e.g., using the scaling factor of FP2 for FP1 as shown in Table 2. The result is a stronger motion parallax effect with smaller user movements.

[0358] FIG. 69 is a flowchart illustrating an example process, according to some embodiments. Some embodiments of process 6900 may include receiving 6902 an input image with depth information. Process 6900 may further include mapping 6904 the input image to multiple focal plane images. Process 6900 may further include 6906 orienting the multiple focal plane images using head orientation information to provide stereo disparity between the left and right eyes. Process 6900 may further include 6908 displaying the oriented multiple focal plane images. Some embodiments may include a device with one or more motion tracking sensors, a processor, and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform process 6900. In some embodiments, process 6900 may be performed by a device having a multi-focal plane (MFP) display.

[0359] FIG. 70 is a flowchart illustrating an example process according to some embodiments. Some embodiments of process 7000 may include receiving 7002 a description of three-dimensional (3D) content. Process 7000 may further include receiving 7004 information from a tracker indicative of a viewer's movement relative to a real-world environment. Process 7000 may further include synthesizing 7006 motion parallax by, for example, altering multi-focal planes formed using the 3D content, including texture and depth information, in response to receiving the information indicative of the viewer's movement. Process 7000 may further include rendering 7008 an image on a multi-focal plane display using the altered multi-focal plane rendering. Some embodiments may include a device with one or more motion tracking sensors, a processor, and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform process 7000. In some embodiments, process 7000 may be performed by a device having a multi-focal plane (MFP) display.

[0360] While methods and systems according to some embodiments have been discussed in the context of virtual reality (VR) and augmented reality (AR), some embodiments may be applied to mixed reality (MR), augmented reality (AR), and virtual reality (VR) contexts as well. Also, while the term "head-mounted display (HMD)" is used herein in accordance with some embodiments, some embodiments may be applied to, for example, wearable devices (which may or may not be head-mounted) that are capable of VR, AR, and / or MR in some embodiments.

[0361] In some embodiments, the term "mismatch vector" refers to the maximum separation (along with direction) of corresponding pixels in a synthetic stereo pair formed, for example, by shifting and projecting an MFP. Due to the geometry, the maximum shift occurs relative to the focal plane closest to the viewer. The amount and direction of this maximum shift is called the "mismatch vector."

[0362] In the literature, the spatial separation of each corresponding pixel in two images (left and right) can be represented by a brightness value. This is usually called a "discrepancy image." In some embodiments of the present application, unlike the literature, the "discrepancy image" is, for example, an MFP stack that is skewed by an amount (which can be indicated by a discrepancy vector), projected, and summed to form a viewing point.

[0363] In some embodiments, the 3D content includes, for example, a depth map and a texture image. In some embodiments, the texture image includes a projection of the 3D model and / or content to a selected viewpoint. In some embodiments, rendering to the MFP display may be in monoscopic (e.g., a monoscopic MFP stack), true stereo (e.g., two monoscopic MFP stacks for the right and left eyepoints, respectively) or composite stereo (e.g., two MFP stacks composited with a shift) format.

[0364] An exemplary method according to some embodiments may include receiving an input image with depth information, mapping the input image to a plurality of focal plane images using the depth information, orienting the plurality of focal plane images using head orientation information to provide stereoscopic disparity between the left and right eyes, and displaying the oriented plurality of focal plane images.

[0365] In some embodiments of the exemplary method, at least receiving the input image and displaying the oriented multiple focal plane images is performed by a multi-focal plane (MFP) display.

[0366] In some embodiments of the example method, orienting the plurality of focal plane images may include rotating one or more of the plurality of focal plane images.

[0367] In some embodiments of the example method, rotating one or more of the plurality of focal plane images may include rotating a mismatch vector for one or more of the focal plane images relative to a vector parallel to the horizon.

[0368] In some embodiments of the example method, orienting the plurality of focal plane images may include shifting one or more of the plurality of focal plane images.

[0369] In some embodiments of the example method, shifting one or more of the plurality of focal plane images may include scaling a disparity vector for the respective one or more focal plane images.

[0370] In some embodiments of the example method, shifting one or more of the plurality of focal plane images may include rotating one or more of the plurality of focal plane images relative to a fixed origin.

[0371] In some embodiments of the example method, shifting one or more of the plurality of focal plane images includes moving one or more of the plurality of focal plane images by an offset relative to a fixed origin.

[0372] Some embodiments of the example method may further include processing the multiple focal plane images to address image mismatch caused by shifting the focal plane images (e.g., to reduce disocclusions or holes).

[0373] In some embodiments of the example method, processing the plurality of focal plane images may include filtering one or more of the focal plane images.

[0374] In some embodiments of the example method, processing the plurality of focal plane images includes determining depth blending weights for at least one of the plurality of focal plane images and generating at least one of the plurality of focal plane images using the respective depth blending weights.

[0375] In some embodiments of the example method, generating at least one of the plurality of focal plane images includes multiplying at least one of the plurality of textures with a respective depth blending weight.

[0376] In some embodiments of the example method, displaying the oriented multiple focal plane images may include combining the shifted focal plane images to generate a stereoscopic image pair.

[0377] Some embodiments of the example method may further include measuring motion tracking sensor readings of the viewer's position relative to the real-world environment to generate head orientation information.

[0378] Some embodiments of the example method may further include synthesizing motion parallax by altering one or more of the plurality of focal plane images in response to generating the head orientation information.

[0379] In some embodiments of the exemplary method, the synthesized motion parallax may include scaling one or more of the multiple focal plane images relative to each other using head orientation information, which may indicate axial motion.

[0380] Some embodiments of the example method may further include filtering the input image for low frequency content and redistributing the low frequency content into multiple focal plane images.

[0381] Some embodiments of the example method may further include filtering the input image for high frequency content and decomposing the high frequency content into multiple focal plane images.

[0382] An exemplary apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform one of the exemplary methods described above.

[0383] An exemplary method according to some embodiments may include receiving a description of three-dimensional (3D) content, receiving information from a tracker indicative of a viewer's movement relative to a real-world environment, synthesizing motion parallax by varying multi-focal planes of the 3D content in response to receiving the information indicative of the viewer's movement, and rendering an image on a multi-focal plane display using the varied multi-focal plane rendering.

[0384] In some embodiments of the exemplary method, the synthesized motion parallax may include scaling the multi-focal planes relative to each other using information indicative of the viewer's motion, which may be axial.

[0385] An exemplary apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform one of the exemplary methods described above.

[0386] An exemplary method according to some embodiments may include determining a plurality of focal plane images to be mapped to an input image, orienting the plurality of focal plane images using head orientation information to provide stereoscopic disparity between the left and right eyes, and displaying the oriented plurality of focal plane images.

[0387] In some embodiments of the example method, determining the plurality of focal plane images includes receiving the plurality of focal plane images.

[0388] In some embodiments of the example method, determining the multiple focal plane images includes mapping the input image to the multiple focal plane images to generate the multiple focal plane images.

[0389] An exemplary apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform one of the exemplary methods described above.

[0390] An exemplary method according to some embodiments may include receiving three-dimensional (3D) video content, receiving a viewer orientation signal, determining image data for one or more focal planes of a display using the 3D video content and the viewer orientation signal, and rendering one or more focal plane images on the display using the image data.

[0391] An exemplary apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform the exemplary methods described above.

[0392] An exemplary method according to some embodiments may include receiving a description of the capabilities of a multi-focal plane (MFP) display, receiving three-dimensional (3D) video content, receiving a viewer orientation signal, calculating image data for one or more focal planes of the MFP display using the 3D video content and the viewer orientation signal, and rendering one or more MFP images on the MFP display using the calculated image data.

[0393] In some embodiments of the example method, the 3D video content may include one or more texture images and corresponding depth maps.

[0394] Some embodiments of the example method may further include filtering the depth map.

[0395] In some embodiments of the example method, calculating image data for one or more focal planes may include extracting one or more images from the 3D video content, calculating depth blending weights for the one or more focal planes, generating one or more MFP weight maps by remapping the depth map by the depth blending weights, multiplying the one or more images by each MFP weight map to form a respective focal plane image, and performing a spatial shift for each focal plane image.

[0396] In some embodiments of the exemplary method, the description of the capabilities of the MFP display may include the number and location of focal planes.

[0397] In some embodiments of the example method, calculating image data for one or more focal planes may include calculating one or more MFP images for one or more focal planes of the MFP display using 3D video content and shifting the one or more calculated MFP images.

[0398] In some embodiments of the example method, shifting the one or more calculated MFP images may include scaling a disparity vector for one or more of the MFP images.

[0399] In some embodiments of the example method, shifting the one or more calculated MFP images may include rotating a mismatch vector for one or more of the MFP images relative to a vector parallel to the horizon.

[0400] In some embodiments of the example method, shifting one or more calculated MFP images may include translating one or more of the MFP images.

[0401] In some embodiments of the example method, calculating image data for one or more focal planes may include calculating one or more MFP images for one or more focal planes of an MFP display using 3D video content, 3D warping the one or more calculated MFP images, and performing hole filling on the one or more calculated MFP images.

[0402] An exemplary method according to some embodiments may include receiving three-dimensional video content, calculating one or more multi-focal plane (MFP) images for a number and positions of multi-focal planes using the three-dimensional video content, receiving a viewer orientation signal, calculating one or more mismatch images for each view of a stereoscopic display using the one or more MFP images and the viewer orientation signal, and rendering the one or more mismatch images on the stereoscopic display.

[0403] In some embodiments of the exemplary method, the three-dimensional video content may include one or more texture images and corresponding depth maps.

[0404] Some embodiments of the example method may further include filtering the depth map.

[0405] In some embodiments of the example method, calculating image data for one or more focal planes may include extracting one or more images from the 3D video content, calculating depth blending weights for the one or more focal planes, generating one or more MFP weight maps by remapping the depth map by the depth blending weights, multiplying the one or more images by each MFP weight map to form a respective focal plane image, and performing a spatial shift for each focal plane image.

[0406] In some embodiments of the exemplary method, rendering the one or more mismatched images on a stereoscopic display includes summing each spatially shifted focal plane to generate a stereoscopic image pair for each of the one or more mismatched images.

[0407] Some embodiments of the example method may further include receiving one or more pixel values corresponding to one or more pixels in one of the multi-focal plane (MFP) images and adjusting the one or more pixel values by a corresponding distance in the depth map.

[0408] In some embodiments of the exemplary method, the description of the capabilities of the MFP display may include the number and location of focal planes.

[0409] In some embodiments of the exemplary method, the viewer orientation signal may indicate a viewer tilt, and calculating one or more disparity images for each view of the stereoscopic display may use a rotated disparity vector corresponding to the indicated viewer tilt.

[0410] In some embodiments of the example method, calculating one or more MFP images for the number and positions of multiple focal planes may include calculating one or more MFP images for each focal plane position using 3D video content, 3D warping the one or more calculated MFP images, and performing hole filling on the one or more calculated MFP images.

[0411] An exemplary method according to some embodiments may include receiving three-dimensional video content, calculating one or more multi-focal plane (MFP) images for a number and positions of the multi-focal planes using the three-dimensional video content, measuring motion tracking sensor readings of a motion tracking sensor, generating a viewer orientation signal from the motion tracking sensor readings, calculating one or more mismatch images for each view of a shutter glasses display using the one or more MFP images and the viewer orientation signal, and rendering the one or more mismatch images on the shutter glasses display.

[0412] An exemplary device according to some embodiments may include one or more motion tracking sensors; a processor; and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform the processes of receiving three-dimensional video content, calculating one or more multi-focal plane (MFP) images for the number and positions of the multi-focal planes using the three-dimensional video content, measuring motion tracking sensor readings of the motion tracking sensor, generating a viewer orientation signal from the motion tracking sensor readings, calculating one or more mismatch images for each view of a shutter glasses display using the one or more MFP images and the viewer orientation signal, and rendering the one or more mismatch images on the shutter glasses display.

[0413] An exemplary device according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform the processes of receiving a description of the capabilities of a multi-focal plane (MFP) display, receiving three-dimensional video content, receiving a viewer orientation signal, calculating image data for one or more focal planes of the MFP display using the three-dimensional video content and the viewer orientation signal, and rendering one or more MFP images on the MFP display using the calculated image data.

[0414] An exemplary method according to some embodiments may include receiving three-dimensional (3D) video content, filtering the 3D video content into high-frequency content and low-frequency content, decomposing the high-frequency content into one or more high-frequency computed images, redistributing the low-frequency content into the one or more high-frequency computed images to generate one or more redistributed focal planes, and rendering one or more multi-focal plane images corresponding to the one or more redistributed focal planes.

[0415] In some embodiments of the example method, the 3D video content may include one or more texture images and corresponding depth maps.

[0416] In some embodiments of the example method, decomposing the high-frequency content into one or more high-frequency calculated images may include calculating image data for one or more focal planes of the MFP display using the high-frequency content, and decomposing the calculated image data into one or more high-frequency calculated images.

[0417] In some embodiments of the example method, calculating image data for one or more focal planes may include extracting one or more images from the 3D video content, calculating depth blending weights for the one or more focal planes, generating one or more MFP weight maps by remapping the depth map by the depth blending weights, multiplying the one or more images by each MFP weight map to form a respective focal plane image, and performing a spatial shift for each focal plane image.

[0418] In some embodiments of the example method, the depth blend weights may be box filters.

[0419] In some embodiments of the exemplary method, the depth blend weights may be a linear ramp.

[0420] In some embodiments of the example method, the depth blending weight may be a polynomial function.

[0421] In some embodiments of the exemplary method, the depth blend weight may be a sinusoidal function.

[0422] In some embodiments of the example method, redelivering the low-frequency content into one or more high-frequency computed images may include dividing pixel values of the low-frequency content by N and combining the divided low-frequency content with each of the N high-frequency computed images to generate one or more redelivered focal planes, where N is a positive integer.

[0423] An exemplary method according to some embodiments may include receiving three-dimensional (3D) video content, calculating image data for N focal planes of an MFP display using the 3D video content, low-pass filtering the 3D video content to generate low-frequency 3D video content, dividing pixel values of the low-frequency 3D video content by N, re-distributing the divided low-frequency 3D video content to the N focal planes of the calculated image data to generate N re-distributed images, and rendering N sets of MFP images on the MFP display using the N re-distributed images, where N is a positive integer.

[0424] In some embodiments of the exemplary method, the three-dimensional video content may include one or more texture images and corresponding depth maps.

[0425] In some embodiments of the example method, redelivering the divided low frequency 3D video content may include low pass filtering each of the N focal planes of the calculated image data, summing the divided low frequency video content with each of the N focal planes of the calculated image data to generate N sets of summed image data, and subtracting each of the N sets of low pass filtered calculated image data from the respective summed image data to generate the N sets of redelivered image data.

[0426] In some embodiments of the example method, calculating image data for the N focal planes may include extracting N images from the 3D video content, calculating depth blending weights for the N focal planes, generating N multi-focal plane (MFP) weight maps by remapping the depth map with the depth blending weights, multiplying the N images by each MFP weight map to form a respective focal plane image, and performing a spatial shift for each of the N focal plane images, where N is a positive integer.

[0427] In some embodiments of the example method, the depth blend weights may be box filters.

[0428] In some embodiments of the exemplary method, the depth blend weights may be a linear ramp.

[0429] In some embodiments of the example method, the depth blending weight may be a polynomial function.

[0430] In some embodiments of the exemplary method, the depth blend weight may be a sinusoidal function.

[0431] In some embodiments of the exemplary method, redelivering the low-frequency content to one or more high-frequency focal planes may include dividing pixel values of the low-frequency content by N and combining the divided low-frequency content with each of the N sets of high-frequency calculated image data to generate one or more sets of redelivered image data, where N is a positive integer.

[0432] An exemplary device according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform the processes of receiving three-dimensional (3D) video content, filtering the 3D video content into high-frequency content and low-frequency content, calculating image data for one or more focal planes of an MFP display using the high-frequency content, decomposing the calculated image data into N sets of high-frequency calculated image data, dividing pixel values of the low-frequency content by N, redistributing the divided low-frequency content into N high-frequency calculated images, and rendering N sets of MFP images on the MFP display using the N redistributed images, where N is a positive integer.

[0433] An exemplary method according to some embodiments may include receiving three-dimensional (3D) video content, filtering the 3D video content into high-frequency content and low-frequency content, decomposing the high-frequency content into one or more high-frequency focal planes, redistributing the low-frequency content to the one or more high-frequency focal planes to generate one or more redistributed focal planes, and rendering one or more multi-focal plane images using the one or more redistributed focal planes.

[0434] An exemplary method according to some embodiments may include receiving an image represented as texture plus depth; forming a first decomposition into a plurality of N focal plane images based on depth plane values of the image; creating a scaled low-frequency representation by applying a low-pass filter to the texture image and scaling pixel values by dividing by N; and, for each of the plurality of N focal plane images, calculating a plurality of re-distributed focal plane images, which may include calculating the low-pass representation of each focal plane image; adding the scaled low-frequency representation to each focal plane image to generate an adjusted focal plane image; and subtracting the calculated low-pass representation of each focal plane image from the adjusted focal plane image to generate the re-distributed focal plane image.

[0435] In some embodiments of the exemplary method, the multiple redistributed focal plane images may be rendered on a multi-focal plane display.

[0436] In some embodiments of the exemplary method, the multiple redistributed focal plane images may be used as input to generate a stereo pair projection on a stereoscopic display.

[0437] An exemplary method according to some embodiments may include receiving an image represented as a multi-focal plane stack of images composed of a plurality of N focal plane images based on depth plane values of the image; creating a scaled low-frequency representation by applying a low-pass filter to the image and scaling pixel values by dividing by N; and calculating a plurality of re-distributed focal plane images for each of the plurality of N focal plane images, which may include calculating the low-pass representation of each focal plane image; adding the scaled low-frequency representation to each focal plane image to generate an adjusted focal plane image; and subtracting the calculated low-pass representation of each focal plane image from the adjusted focal plane image to generate the re-distributed focal plane image.

[0438] In some embodiments of the exemplary method, the multiple redistributed focal plane images may be rendered on a multi-focal plane display.

[0439] In some embodiments of the exemplary method, the multiple redistributed focal plane images may be used as input to generate a stereo pair projection on a stereoscopic display.

[0440] An exemplary method according to some embodiments may include calculating image data for one or more focal planes of an MFP display using three-dimensional video content and a viewer orientation signal, and rendering one or more MFP images on the MFP display using the calculated image data.

[0441] An exemplary device according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform a process of calculating image data for one or more focal planes of an MFP display using three-dimensional video content and a viewer orientation signal, and rendering one or more MFP images on the MFP display using the calculated image data.

[0442] An exemplary method according to some embodiments may include rendering to a multi-focal plane display and creating stereoscopic video in response to a user's head orientation.

[0443] An exemplary method according to some embodiments may include receiving 3D video content that includes a depth map.

[0444] An exemplary method according to some embodiments may include receiving 3D video content comprising texture images.

[0445] An exemplary method according to some embodiments may include receiving an orientation signal from a viewer.

[0446] An exemplary method according to some embodiments may include receiving a description of multi-focal plane display capabilities (including the number and positions of focal planes).

[0447] An exemplary method according to some embodiments may include calculating image data for a plane of a multi-focal plane display using the orientation signal.

[0448] An exemplary method according to some embodiments may include calculating image data for a selected number and positions of multiple focal planes.

[0449] An exemplary method according to some embodiments may include computing a disparity image (or, for example, an image with disparity) for each view of a stereoscopic display using multiple focal planes and orientation signals.

[0450] An exemplary method according to some embodiments may include rendering a multi-focal plane / mismatch image to a display.

[0451] Table 4 shows the mapping of grayscale versions of the native images to the black and white line drawings of Figures 71-92.

[0452] [Table 4]

[0453] It should be noted that one or more of the various hardware elements of the described embodiments are referred to as “modules” that perform (i.e., implement, execute, etc.) the various functions described herein with respect to the respective modules. As used herein, a module includes hardware deemed suitable by one of ordinary skill in the art for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more ASICs, one or more FPGAs, one or more memory devices). It should be noted that each described module may also include instructions executable to perform one or more functions described as being performed by the respective module, which may take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., and may be stored on any suitable non-transitory computer-readable medium or media, such as those commonly referred to as RAM, ROM, etc.

[0454] Although features and elements have been described above in particular combinations, those skilled in the art will appreciate that each feature or element can be used alone or in any combination with the other features and elements. The methods described herein can be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, ROM, RAM, registers, cache memory, semiconductor memory devices, internal and removable disks, magnetic media such as magneto-optical media, and optical media such as CD-ROM disks and DVDs. A processor in association with software can be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. 1. A method performed by a head-mounted display (HMD) with a multi-focal plane (MFP) display, comprising: Arranging the plurality of focal plane images in a virtual space; determining a user's head movement in terms of tilt of the user's head relative to the horizon in a roll direction; shifting and rotating one or more of the plurality of focal plane images in response to the determined head movement of the user to generate a stereoscopic view of the user, the stereoscopic view including motion parallax information, and shifting and rotating one or more of the plurality of focal plane images in the same direction as the determined head movement of the user; displaying, using the MFP display, the multiple focal plane images that have been shifted and rotated to generate a stereoscopic view of the user; A method for providing the above.

2. The method of claim 1 , wherein shifting and rotating one or more of the plurality of focal plane images provides stereo disparity between the left and right eyes.

3. 2. The method of claim 1 , wherein rotating one or more of the plurality of focal plane images comprises rotating a disparity vector for one or more of the focal plane images about a vector parallel to the horizon.

4. 4. The method of claim 1, further comprising processing the plurality of focal plane images to reduce image mismatch caused by rotating one or more of the plurality of focal plane images, the image mismatch comprising a mismatch between one or more of the plurality of focal plane images in an un-rotated orientation before rotation and a respective one or more of the plurality of focal plane images in a rotated orientation after rotation.

5. The method of claim 1 , wherein shifting one or more of the plurality of focal plane images comprises scaling a disparity vector for the respective one or more focal plane images.

6. 10. The method of claim 1, wherein rotating one or more of the plurality of focal plane images comprises rotating one or more of the plurality of focal plane images about a fixed origin.

7. 10. The method of claim 1, wherein shifting one or more of the plurality of focal plane images comprises moving one or more of the plurality of focal plane images by an offset relative to a fixed origin.

8. 8. The method of claim 1, further comprising processing the plurality of focal plane images to address image mismatch caused by shifting one or more of the plurality of focal plane images, the image mismatch comprising a mismatch between one or more of the plurality of focal plane images before a shift and a respective one or more of the plurality of focal plane images after a shift.

9. The method of claim 8 , wherein processing the plurality of focal plane images comprises filtering image information of one or more of the focal plane images.

10. 10. The method of claim 1, wherein displaying the multiple focal plane images comprises summing shifted focal plane images along a depth axis to generate a stereoscopic image pair.

11. The method of claim 1 , further comprising measuring motion tracking sensor readings of a viewer's position relative to a real-world environment to determine the user's head movement.

12. The method of claim 11 , further comprising synthesizing motion parallax by altering one or more of the plurality of focal plane images.

13. 13. The method of claim 12, wherein synthesizing the motion parallax comprises spatially scaling one or more of the plurality of focal plane images relative to one another using determined head movement of the user, wherein the determined head movement of the user indicates axial movement.

14. a processor; a non-transitory computer-readable medium storing instructions; 14. An apparatus comprising: instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 13.

Images