Multifocal plane based method to produce stereoscopic viewpoints in dibr system (mfp-dibr)

Abstract

To keep 3D perception in a proper state in a case where a near-eye display, which is mounted on the head of a viewer and respectively displays multi-focal images corresponding to right and left eyes, is tilted from a horizontal line of the real world.SOLUTION: A method may include generating a plurality of focal plane images using a description of a 3D content; receiving motion information including information indicating motion of a user relative to a real-world environment; generating a lateral direction motion parallax by shifting one or more focal plane images in the plurality of focal plane images in response to motion information indicating lateral motion of the user; generating an axial motion parallax by scaling one or more focal plane images in the plurality of focal plane images in response to motion information indicating axial motion of the user; and rendering a view of the 3D content using the shifted and scaled focal plane images.SELECTED DRAWING: Figure 67

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional application claiming the benefit under 35 U.S.C. §119(e) of U.S. Provisional Patent Application No. 62 / 647,568, entitled “Multifocal Plane Based Method to Produce Stereoscopic Viewpoints in a DIBR System (MFP-DIBR),” filed March 23, 2018, which is incorporated by reference in its entirety. [Background technology]

[0002] Wearable glasses-type near-eye displays (NEDs) are popular in virtual reality and gaming, and more recently in augmented reality. Glasses that use a mobile phone as the display element are an example of a popular low-cost device for viewing virtual or camera-captured images. In general, all of these displays present a sequence of stereoscopic image pairs. Stereoscopic rendering may be used with external displays, including those for stereoscopic 3D (S3D) TVs.

[0003] Many previous MFP-based or 3D-based viewpoint generation systems may have lower visual quality because the viewpoint may change slightly with user movement. Many near-eye displays that support natural accommodation lack support for viewpoint changes, even small viewpoint changes relative to the display when the eye is turned (changing the viewpoint slightly inside the eyebox). Disocclusion is of course content-dependent, but large viewpoint changes or inconsistencies are more noticeable. Creating a virtual viewpoint for stereoscopic vision or motion parallax may be used, for example, in 3DoF+ extensions of 360° vision and other 3DoF applications. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Rahul Narain et al., Optimal Presentation of Imagery with Focus Cues on Multi-Plane Displays, 34(4) ACM TRANSACTIONS ON GRAPHICS 59:1–12 (August 2015) (“Narain”) [Non-Patent Document 2] Pan et al., 3D Video Disparity Scaling for Preference and Prevention of Discomfort, 7863 PROCEEDINGS OF SPIE (2011) (“Pan”) [Non-Patent Document 3] Linwei Zhu et al., View-Spatial-Temporal Post-Refinement for View Synthesis in 3D Video Systems, 28 Signal Processing: Image Communication 1342~1357 (2013) (“Zhu”) [Non-Patent Document 4] Sergey Sukhanov, 3D Content Production, ST. PETERSBURG ELECTROTECHNICAL UNIVERSITY (undated presentation, available at www.slideshare.net / mikhailvink / sergey-a-sukhanov-3d-content-production, website indicates publication date March 12, 2011) (last accessed February 26, 2019) [Non-Patent Document 5] Kristoph Fehn, A 3D-TV Approach Using Depth-Image-Based Rendering (DIBR), Proceedings of VIIP '03 (September 2003) (Benalmadena, Spain) ("Fehn") [Non-Patent Document 6] Shibata, Takashi et al., Visual Discomfort with Stereo Displays: Effects of Viewing Distance and Direction of Vergence-Accommodation Conflict, PROCEEDINGS OF THE INT'L SOC. FOR OPTICS AND PHOTONICS (SPIE) 7863 (2011) (78630P1~78630P9) ("Shibata") [Non-Patent Document 7] Shiro Suyama et al., Apparent 3-D Image Perceived from Luminance-Modulated Two 2-D Images Displayed at Different Depths, 44 (8) Vision Research 785-793 (April 2004) (“Suyama”) [Non-Patent Document 8] Simon J. Watt et al., Real-World Stereoscopic Performance in Multiple-Focal-Plane Displays: How Far Apart Should the Image Planes Be?, SPIE: Stereoscopic Displays and Applications XXIII (February 2012) [Non-Patent Document 9] Simon J. Watt et al., Achieving Near-Correct Focus Cues in a 3-D Display Using Multiple Image Planes, 5666 Proceedings of SPIE 393~401 (2005) (``Watt 2005'') [Non-Patent Document 10] Kurt Akeley et al., A Stereo Display Prototype with Multiple Focal Distances, 23(3) ACM TRANSACTIONS ON GRAPHICS (TOG) 804–813 (2004) (“Akeley”) [Non-Patent Document 11] X. Hu & H. Hua, Design and Assessment of a Depth-Fused Multi-Focal-Plane Display Prototype, 10(4) IEEE / OSA Journal of Display Technology 308~316 (2014) (``Hu&Hua'') [Non-Patent Document 12] File:2D plus depth.png, WIKIPEDIA, / / commons.wikimedia.org / w / index.php?curid=5489771 (last accessed February 22, 2019) [Non-Patent Document 13] Xuyuan Xu et al., Depth Map Misalignment Correction and Dilation for DIBR View Synthesis, 28 Signal Processing: Image Communication 1023~1045 (2013) [Non-Patent Document 14] Shibata et al., The Zone of Comfort: Predicting Visual Discomfort with Stereo Displays, 11(8):11 J. OF VISION 1–29 (2011) (“Shibata II”) [Non-Patent Document 15] Paul V. Johnson et al., Dynamic Lens and Monovision 3D Displays to Improve Viewer Comfort, 24(11) Opt. Exp. 11808~11827 (2016) [Non-Patent Document 16] Zannoli et al., Blur and the Perception of Depth at Occlusion, 16(6):17 Journal of Vision 1–25 (2016) [Non-Patent Document 17] JP Rolland et al., Multifocal Planes Head-Mounted Displays, 39 Appl. Opt. 3209-3215 (2000) ("Rolland") Summary of the Invention

[0005] An exemplary method according to some embodiments may include receiving an input image with depth information, mapping the input image to multiple focal plane images using the depth information, orienting the multiple focal plane images using head orientation information to provide stereoscopic disparity between the left and right eyes, and displaying the oriented multiple focal plane images.

[0006] In some embodiments of the example method, at least receiving the input image and displaying the oriented multiple focal plane images are performed by a multi-focal plane (MFP) display.

[0007] In some embodiments of the example method, orienting the multiple focal plane images may include rotating one or more of the multiple focal plane images.

[0008] In some embodiments of the example method, rotating one or more of the multiple focal plane images may include rotating a mismatch vector for one or more of the focal plane images relative to a vector parallel to the horizon.

[0009] In some embodiments of the example method, orienting the multiple focal plane images may include shifting one or more of the multiple focal plane images.

[0010] In some embodiments of the example method, shifting one or more of the multiple focal plane images may include scaling a disparity vector for the respective one or more focal plane images.

[0011] In some embodiments of the example method, shifting one or more of the multiple focal plane images may include rotating one or more of the multiple focal plane images relative to a fixed origin.

[0012] In some embodiments of the example method, shifting one or more of the multiple focal plane images includes moving one or more of the multiple focal plane images by an offset relative to a fixed origin.

[0013] Some embodiments of the example method may further include processing the multiple focal plane images to address image mismatch caused by shifting the focal plane images.

[0014] In some embodiments of the example method, processing the multiple focal plane images may include filtering one or more of the multiple focal plane images.

[0015] In some embodiments of the example method, processing the multiple focal plane images may include determining depth blending weights for at least one of the multiple focal plane images and generating at least one of the multiple focal plane images using the respective depth blending weights.

[0016] In some embodiments of the example method, generating at least one of the plurality of focal plane images includes multiplying at least one of the plurality of textures with a respective depth blending weight.

[0017] In some embodiments of the example method, displaying the oriented multiple focal plane images may include combining the shifted focal plane images to generate a stereoscopic image pair.

[0018] Some embodiments of the example method may further include measuring motion tracking sensor readings of the viewer's position relative to the real-world environment to generate head orientation information.

[0019] Some embodiments of the example method may further include synthesizing motion parallax by altering one or more of the plurality of focal plane images in response to generating the head orientation information.

[0020] In some embodiments of the example method, the synthesized motion parallax may include scaling one or more of the multiple focal plane images relative to each other using head orientation information, which may indicate axial motion.

[0021] Some embodiments of the example method may further include filtering the input image for low frequency content and redistributing the low frequency content into multiple focal plane images.

[0022] Some embodiments of the example method may further include filtering the input image for high frequency content and decomposing the high frequency content into multiple focal plane images.

[0023] An exemplary apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform one of the exemplary methods described above.

[0024] An exemplary method according to some embodiments may include receiving a description of three-dimensional (3D) content, receiving information from a tracker indicative of a viewer's movement relative to a real-world environment, synthesizing motion parallax by altering multi-focal planes of the 3D content in response to receiving the information indicative of the viewer's movement from the tracker, and rendering an image on a multi-focal plane display using the altered multi-focal plane rendering.

[0025] In some embodiments of the exemplary method, the synthesized motion parallax may include scaling the multi-focal planes relative to each other using information indicative of the viewer's motion, which may be axial.

[0026] An exemplary apparatus according to some embodiments may include a processor (e.g., one or more processors) and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform one of the exemplary methods described above.

[0027] An exemplary method according to some embodiments may include determining a plurality of focal plane images to be mapped to an input image, orienting the plurality of focal plane images using head orientation information to provide stereoscopic disparity between the left and right eyes, and displaying the oriented plurality of focal plane images.

[0028] In some embodiments of the example method, determining the multiple focal plane images includes receiving the multiple focal plane images.

[0029] In some embodiments of the example method, determining the multiple focal plane images includes mapping the input image to the multiple focal plane images to generate the multiple focal plane images.

[0030] An exemplary apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform one of the exemplary methods described above.

[0031] An exemplary method according to some embodiments may include receiving three-dimensional (3D) video content, receiving a viewer orientation signal, determining image data for one or more focal planes of a display using the 3D video content and the viewer orientation signal, and rendering the one or more focal plane images on the display using the image data.

[0032] An exemplary apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform the exemplary methods described above. [Brief description of the drawings]

[0033] A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings, in which like reference numbers indicate like elements and in which: [Figure 1A] 1 is a system diagram of an example system illustrating an example communication system, according to some embodiments. [Figure 1B] 1A is a system diagram of an example system illustrating an example wireless transmit / receive unit (WTRU) that may be used within the communications system illustrated in FIG. 1A, in accordance with some embodiments. [Figure 2A] 1 is a schematic diagram illustrating an example set of focal planes displayed at different depths from a user. [Figure 2B] 1 is an illustration showing an exemplary focal plane image. [Diagram 3] 1 is a graph illustrating an exemplary image mismatch between left and right images versus observer-screen distance for Percival's comfortable viewing zone; [Figure 4]1 is a schematic diagram illustrating exemplary left-eye and right-eye images for a stereoscopic display for a viewer with a non-rotated orientation. [Diagram 5] FIG. 1 is a schematic diagram illustrating exemplary left-eye and right-eye images bounded in glasses coordinates for a stereoscopic display for a viewer with a rotated orientation in which the horizon is tilted with the head. [Figure 6] FIG. 1 is a schematic diagram illustrating exemplary left-eye and right-eye images adjusted to match the orientation of the horizon for a stereoscopic display for a viewer with rotated orientation and unchanged disparity. [Figure 7] FIG. 1 is a system diagram illustrating an example set of interfaces for manipulating image mismatch. [Figure 8] FIG. 1 is a system diagram illustrating an example set of interfaces for a DIBR-based 3D transmission system. [Figure 9] FIG. 1 is a schematic plan view illustrating an example scenario showing the appearance of dis-occlusion when varying the source and target cameras; [Figure 10A] 1 is an illustration showing an example 3D warping image before hole filling. [Figure 10B] 13 is an illustration showing an example 3D warping image after hole filling. [Figure 11A] FIG. 1 is a schematic perspective view illustrating an exemplary vergence accommodation conflict (VAC) in many stereoscopic viewing systems. [Figure 11B] FIG. 1 is a schematic perspective view illustrating an exemplary vergence accommodation conflict (VAC) in many stereoscopic viewing systems. [Figure 11C] FIG. 1 is a schematic perspective view illustrating an exemplary vergence accommodation conflict (VAC) in many stereoscopic viewing systems. [Figure 12] FIG. 1 is a schematic plan view illustrating an exemplary multi-focal plane (MFP) near-eye display according to some embodiments. [Figure 13] 1 is a schematic plan view illustrating exemplary viewing of near, intermediate, and far focus image planes in accordance with some embodiments; [Figure 14] 1 is a schematic diagram illustrating an exemplary focal image plane viewed by an observer; [Figure 15A] 1 is a schematic diagram illustrating exemplary front and back focal planes for an exemplary image of a square. [Figure 15B] 1 is a schematic diagram illustrating exemplary front and back focal planes for an exemplary image of a square. [Figure 15C] 1 is a schematic diagram illustrating exemplary front and back focal planes for an exemplary image of a square. [Figure 16A] 2 is a schematic front view illustrating an exemplary focal plane displayed for right and left eye images; FIG. [Figure 16B] 2 is a schematic front view illustrating an exemplary focal plane displayed for right and left eye images; FIG. [Figure 17] FIG. 2 is a schematic plan view illustrating an exemplary viewing of two focal planes with a depth-based luminous intensity weighting function. [Figure 18] FIG. 2 is a schematic plan view illustrating an exemplary viewing of three focal planes by left and right eyes according to some embodiments. [Figure 19] 1 is a schematic plan view illustrating an example quantization of depth for an MFP display without depth blending (weighting) in accordance with some embodiments. [Figure 20A] 1 is a graph illustrating an example of weight versus depth according to some embodiments. [Figure 20B] 1 is a graph illustrating an example of weight versus depth according to some embodiments. [Figure 21A] 1 is a graph illustrating an example blending function of weight versus depth according to some embodiments. [Figure 21B] 1 is an illustration showing an exemplary test image. [Figure 21C] 21C is an exemplary depth map illustrating pixel distances for FIG. 21B. [Figure 21D]21B is an illustration showing an example focal plane image using the depth weights of FIG. 21A along with the test image of FIG. 21B and its example depth map of FIG. 21C in accordance with some embodiments. [Figure 21E] 21B is an illustration showing an example focal plane image using the depth weights of FIG. 21A along with the test image of FIG. 21B and its example depth map of FIG. 21C in accordance with some embodiments. [Figure 21F] 21B is an illustration showing an example focal plane image using the depth weights of FIG. 21A along with the test image of FIG. 21B and its example depth map of FIG. 21C in accordance with some embodiments. [Figure 22A] 1 is a graph illustrating an example blending function of weight versus depth according to some embodiments. [Figure 22B] 1 is an illustration showing an exemplary test image. [Figure 22C] 22C is an exemplary depth map illustrating pixel distances for FIG. 22B. [Figure 22D] 22B is an illustration showing an example focal plane image using the depth weights of FIG. 22A along with the test image of FIG. 22B and its example depth map of FIG. 22C in accordance with some embodiments. [Figure 22E] 22B is an illustration showing an example focal plane image using the depth weights of FIG. 22A along with the test image of FIG. 22B and its example depth map of FIG. 22C in accordance with some embodiments. [Figure 22F] 22B is an illustration showing an example focal plane image using the depth weights of FIG. 22A along with the test image of FIG. 22B and its example depth map of FIG. 22C in accordance with some embodiments. [Figure 23A] 1 is a schematic front view illustrating an example scenario for stereoscopic image mismatch, according to some embodiments; [Figure 23B] 1 is a schematic front view illustrating an example scenario for stereoscopic image mismatch, according to some embodiments; [Figure 23C]1 is a schematic front view illustrating an example scenario for stereoscopic image mismatch, according to some embodiments; [Figure 24A] 1 is a schematic front view illustrating an example scenario for image mismatch with viewer orientation direction, according to some embodiments; [Figure 24B] 1 is a schematic front view illustrating an example scenario for image mismatch with viewer orientation direction, according to some embodiments; [Figure 24C] 1 is a schematic front view illustrating an example scenario for image mismatch with viewer orientation direction, according to some embodiments; [Diagram 25] FIG. 2 is a system diagram illustrating an example set of processing blocks and interfaces for generating left and right eye shifted and projected MFPs in accordance with some embodiments. [Figure 26] FIG. 1 is a system diagram illustrating an example set of processing blocks and interfaces for generating left-eye and right-eye shifted and projected MFPs for three scenarios of viewer orientation, according to some embodiments. [Figure 27A] 1A-1D are schematic plan views illustrating exemplary focal planes viewed under different viewer orientations, according to some embodiments. [Figure 27B] 1A-1D are schematic plan views illustrating exemplary focal planes viewed under different viewer orientations, according to some embodiments. [Figure 27C] 1A-1D are schematic plan views illustrating exemplary focal planes viewed under different viewer orientations, according to some embodiments. [Figure 27D] 1A-1D are schematic plan views illustrating exemplary focal planes viewed under different viewer orientations, according to some embodiments. [Figure 28A] 1 is an illustration showing an exemplary test image. [Figure 28B] 28B is an illustration showing an example weighted focal plane image using the test image of FIG. 28A in accordance with some embodiments. [Figure 28C] 28B is an illustration showing an example weighted focal plane image using the test image of FIG. 28A in accordance with some embodiments. [Figure 28D] 28B is an illustration showing an example weighted focal plane image using the test image of FIG. 28A in accordance with some embodiments. [Figure 29A] 28B-28D are corresponding illustrations of exemplary right-eye images formed using the focal plane images of FIGS. 28B-28D, in accordance with some embodiments. [Figure 29B] 28B-28D are corresponding illustrations of exemplary left-eye images formed using the focal plane images of FIGS. 28B-28D, in accordance with some embodiments. [Figure 30A] 1A-1C are schematic plan views illustrating exemplary focal planes viewed under different eye separation distances according to some embodiments. [Figure 30B] 1A-1C are schematic plan views illustrating exemplary focal planes viewed under different eye separation distances according to some embodiments. [Figure 30C] 1A-1C are schematic plan views illustrating exemplary focal planes viewed under different eye separation distances according to some embodiments. [Figure 31A] 1 is an illustration showing an enlarged excerpt of an exemplary stereoscopic image for the right eye formed with sinusoidal depth weighting in accordance with some embodiments. [Figure 31B] 11 is an illustration showing an example stereoscopic image for the right eye correspondingly formed with sinusoidal depth weighting in accordance with some embodiments. [Figure 31C] 11 is an illustration showing an example stereoscopic image for the left eye correspondingly formed with sinusoidal depth weighting in accordance with some embodiments. [Figure 31D] 1 is an illustration showing an enlarged excerpt of an exemplary stereoscopic image for the right eye formed with polynomial depth weighting in accordance with some embodiments. [Figure 31E]11 is an illustration showing an exemplary right-eye image correspondingly formed with polynomial depth weighting in accordance with some embodiments. [Fig. 31F] 11 is an illustration showing an exemplary left-eye image correspondingly formed with polynomial depth weighting in accordance with some embodiments. [Figure 32A] 1A-1C are schematic front views illustrating exemplary mismatches for a first stereoscopic display with different user orientations, according to some embodiments. [Figure 32B] 1A-1C are schematic front views illustrating exemplary mismatches for a first stereoscopic display with different user orientations, according to some embodiments. [Figure 32C] 1A-1C are schematic front views illustrating exemplary mismatches for a first stereoscopic display with different user orientations, according to some embodiments. [Fig. 32D] 11A-11C are schematic front views illustrating exemplary mismatches for a second stereoscopic display with different user orientations, according to some embodiments. [Figure 32E] 11A-11C are schematic front views illustrating exemplary mismatches for a second stereoscopic display with different user orientations, according to some embodiments. [Fig. 32F] 11A-11C are schematic front views illustrating exemplary mismatches for a second stereoscopic display with different user orientations, according to some embodiments. [Figure 33A] 1A-1C are schematic front views illustrating exemplary mismatches for stereoscopic displays with different user orientations, according to some embodiments. [Figure 33B] 1A-1C are schematic front views illustrating exemplary mismatches for stereoscopic displays with different user orientations, according to some embodiments. [Figure 33C] 1A-1C are schematic front views illustrating exemplary mismatches for stereoscopic displays with different user orientations, according to some embodiments. [Diagram 34] 1 is a schematic user's diagram illustrating exemplary left-eye and right-eye images for a stereoscopic display for a viewer with a rotated orientation, according to some embodiments. [Diagram 35]1 is a schematic processing diagram illustrating an exemplary process for generating multiple perspectives for a view by left-right warping a set of focal plane images according to some embodiments. [Diagram 36] 36 is an illustration showing an exemplary series of three stereoscopic images for stereoscopic display using the warped image of FIG. 35 according to some embodiments. [Figure 37] FIG. 1 is a system diagram illustrating an example set of processing blocks and interfaces for generating shifted and projected MFP images for flexible baseline stereoscopic display for three viewer orientations, according to some embodiments. [Figure 38A] 1 is a set of computer windows illustrating an exemplary set of interfaces for an image manipulation program showing a depth map image, according to some embodiments. [Figure 38B] 1 is a set of computer windows illustrating an exemplary set of interfaces for an image manipulation program showing a depth weighting graph, according to some embodiments. [Figure 38C] 1 is a set of computer windows illustrating an exemplary set of interfaces for an image manipulation program showing generated focal plane images according to some embodiments. [Figure 39A] 1 is an illustration showing an exemplary set of generated focal plane images within a simulated environment. [Figure 39B] 1 is an illustration showing an exemplary set of generated focal plane images within a simulated environment. [Figure 39C] 1 is an illustration showing an exemplary set of generated focal plane images within a simulated environment. [Diagram 40] 1 is a message sequence diagram illustrating an exemplary process for generating a multi-focal plane (MFP) display image according to some embodiments. [Diagram 41] 1 is a message sequence diagram illustrating an example process for generating a stereoscopic 3D display image according to some embodiments. [Figure 42A] 1 is a graph illustrating an example blending function of weight versus depth according to some embodiments. [Figure 42B] 1 is an illustration showing an exemplary test image. [Figure 42C] 42C is an exemplary depth map illustrating pixel distances for FIG. 42B. [Fig.42D] 42B is an illustration showing an example focal plane image using the depth weights of FIG. 42A along with the test image of FIG. 42B and its example depth map of FIG. 42C in accordance with some embodiments. [Figure 42E] 42B is an illustration showing an example focal plane image using the depth weights of FIG. 42A along with the test image of FIG. 42B and its example depth map of FIG. 42C in accordance with some embodiments. [Fig.42F] 42B is an illustration showing an example focal plane image using the depth weights of FIG. 42A along with the test image of FIG. 42B and its example depth map of FIG. 42C in accordance with some embodiments. [Figure 43A] 1 is a graph illustrating an example blending function of weight versus depth according to some embodiments. [Figure 43B] 1 is an illustration showing an exemplary test image. [Figure 43C] 43C is an exemplary depth map illustrating pixel distances for FIG. 43B. [Fig. 43D] 43B is an illustration showing an example focal plane image using the depth weights of FIG. 43A along with the test image of FIG. 43B and its example depth map of FIG. 43C in accordance with some embodiments. [Figure 43E] 43B is an illustration showing an example focal plane image using the depth weights of FIG. 43A along with the test image of FIG. 43B and its example depth map of FIG. 43C in accordance with some embodiments. [Figure 43F] 43B is an illustration showing an example focal plane image using the depth weights of FIG. 43A along with the test image of FIG. 43B and its example depth map of FIG. 43C in accordance with some embodiments. [Diagram 44]1A-1C are schematic diagrams illustrating exemplary focal stack images captured at different depths from a user. [Figure 45A] 1 is an illustration showing an example linearly blended focal plane. [Figure 45B] 1 is an illustration showing an example linearly blended focal plane. [Figure 45C] 1 is an illustration showing an example linearly blended focal plane. [Figure 45D] 1 is an illustration showing an example linearly blended focal plane. [Figure 45E] 1 is an illustration showing an example redistributed focal plane. [Fig.45F] 1 is an illustration showing an example redistributed focal plane. [Figure 45G] 1 is an illustration showing an example redistributed focal plane. [Fig. 45H] 1 is an illustration showing an example redistributed focal plane. [Figure 46A] 11 is an illustration showing an example stereoscopic image for the right eye formed using correspondingly unblended focal plane images in accordance with some embodiments. [Figure 46B] 11 is an illustration showing an example stereoscopic image for the left eye formed using correspondingly unblended focal plane images in accordance with some embodiments. [Figure 47] 1 is a flowchart illustrating an example process for generating redistributed focal plane images of front, mid, and rear depth ranges in accordance with some embodiments. [Figure 48A] 1 is an illustration showing an exemplary focal plane image without redelivery in accordance with some embodiments. [Figure 48B] 1 is an illustration showing an exemplary focal plane image without redelivery in accordance with some embodiments. [Figure 48C] 1 is an illustration showing an exemplary focal plane image without redelivery in accordance with some embodiments. [Figure 48D]1 is an illustration showing an example focal plane image with redelivery according to some embodiments. [Figure 48E] 1 is an illustration showing an example focal plane image with redelivery according to some embodiments. [Fig.48F] 1 is an illustration showing an example focal plane image with redelivery according to some embodiments. [Figure 49A] 1 is an illustration showing an example right image for a crossed-eye stereogram formed using linearly blended focal planes, in accordance with some embodiments. [Figure 49B] 1 is an illustration showing an example left image for a crossed-eye stereogram formed using linearly blended focal planes, in accordance with some embodiments. [Figure 50A] 1 is an illustration showing an exemplary right image for a crossed-eye stereogram formed using a redistributed focal plane, in accordance with some embodiments. [Figure 50B] 1 is an illustration showing an exemplary left image for a crossed-eye stereogram formed using a redistributed focal plane, in accordance with some embodiments. [Figure 51A] 13 is an illustration showing an exemplary right image for a crossed-eye stereogram formed using selective Gaussian filtering in redistributing an MFP in accordance with some embodiments. [Figure 51B] 13 is an illustration showing an exemplary left image for a crossed-eye stereogram formed using selective Gaussian filtering in redistributing an MFP in accordance with some embodiments. [Figure 52A] 1 is a flowchart illustrating an example process for generating a redistributed focal plane image using low and high frequency filtering, in accordance with some embodiments. [Figure 52B] 1 is a flowchart illustrating an exemplary process for generating a high frequency focal plane image using low pass filtering, according to some embodiments. [Figure 53]1 is a flowchart illustrating an example process for generating redistributed focal plane images for a front depth range, a mid depth range, and a rear depth range, according to some embodiments. [Figure 54] 1 is a flowchart illustrating an example process for generating a redistributed focal plane image using low pass filtering, in accordance with some embodiments. [Figure 55A] 1 is a schematic perspective view illustrating exemplary multi-focal planes rendered in space according to some embodiments; [Figure 55B] 1 is a schematic perspective view illustrating exemplary multi-focal planes rendered in space according to some embodiments; [Figure 56A] 1 is a schematic perspective view illustrating an example of shifted and scaled multiple focal planes for different viewpoints according to some embodiments; [Figure 56B] 1 is a schematic perspective view illustrating an example of shifted and scaled multiple focal planes for different viewpoints according to some embodiments; [Figure 56C] 1 is a schematic perspective view illustrating an example of shifted and scaled multiple focal planes for different viewpoints according to some embodiments; [Figure 57] FIG. 1 is a process diagram illustrating an example process for generating multiple focal planes in a motion parallax system, according to some embodiments. [Figure 58] 1A-1C are schematic plan views illustrating exemplary focal plane size scaling adjustments for axial movement, according to some embodiments. [Figure 59A] 1A-1C are schematic plan views illustrating exemplary deformations of exemplary focal plane content during axial movement, according to some embodiments. [Figure 59B] 1A-1C are schematic plan views illustrating exemplary deformations of exemplary focal plane content during axial movement, according to some embodiments. [Figure 59C] 1A-1C are schematic plan views illustrating exemplary deformations of exemplary focal plane content during axial movement, according to some embodiments. [Figure 60] 1 is a graph illustrating exemplary eye-gaze coordinate tracking during user movement, according to some embodiments. [Figure 61] FIG. 1 is a process diagram illustrating an example process for generating a multi-focal plane true volume set, according to some embodiments. [Figure 62] FIG. 1 is a process diagram illustrating an example process for generating a multi-focal plane synthesized volume set, according to some embodiments. [Figure 63] FIG. 1 is a process diagram illustrating an example process for generating and summing a set of multiple focal planes for a two-dimensional display, according to some embodiments. [Figure 64] FIG. 1 is a process diagram illustrating an example process for generating and summing a set of multi-focal planes for a monocular 2D display, according to some embodiments. [Figure 65] FIG. 1 is a process diagram illustrating an example process for generating and summing a set of multiple focal planes for a true stereoscopic three-dimensional (S3D) display, according to some embodiments. [Figure 66] FIG. 1 is a process diagram illustrating an example process for generating and summing a set of multiple focal planes for a synthesized stereoscopic three-dimensional (S3D) display, according to some embodiments. [Figure 67] FIG. 1 is a process diagram illustrating an example process for generating a multi-focal plane with a front-end processing path and a display-end path, according to some embodiments. [Figure 68A] 68A-68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 68B] 68A-68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 68C]68A-68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 68D] 68A-68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 68E] 68A-68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Fig. 68F] 68A-68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 68G] 68A-68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Fig. 68H] 68A-68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Fig.68I] 68A-68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Fig. 68J] 68A-68D are illustrations showing an exemplary series of right and left images for axial movement between each set image pair (e.g., FIGS. 68C and 68D) according to some embodiments. [Figure 69] 1 is a flowchart illustrating an example process according to some embodiments. [Figure 70] 1 is a flowchart illustrating an example process according to some embodiments. [Figure 71] FIG. 2C is a grayscale version of the native image corresponding to FIG. 2B. [Figure 72] FIG. 15 is a grayscale version of the native image corresponding to FIG. [Figure 73] FIG. 16C is a grayscale version of the native image corresponding to FIGS. 16A-16B. [Figure 74] 21B-21F are grayscale versions of the native images corresponding to FIG. 21B-21F. [Figure 75] 22B-22F are grayscale versions of the native images corresponding to FIG. 22B-F. [Figure 76] FIG. 28C is a grayscale version of the native image corresponding to FIGS. 28A-28D. [Figure 77] FIG. 29C is a grayscale version of the native image corresponding to FIGS. 29A-29B. [Figure 78] 31A-31F are grayscale versions of the native images corresponding to FIG. 31A-31F. [Figure 79] FIG. 36 is a grayscale version of the native image corresponding to FIG. 35. [Figure 80] FIG. 37 is a grayscale version of the native image corresponding to FIG. 36. [Figure 81] FIG. 38D is a grayscale version of the native image corresponding to FIG. 38C. [Figure 82] FIG. 39C is a grayscale version of the native image corresponding to FIG. 39A-FIG. 39C. [Figure 83] 42B-42F are grayscale versions of the native images corresponding to FIG. [Figure 84] 43B-43F are grayscale versions of the native images corresponding to FIG. [Figure 85] FIG. 45 is a grayscale version of the native image corresponding to FIG. 44. [Figure 86] 45A-45H are grayscale versions of the native images corresponding to the FIGS. [Figure 87] FIG. 46C is a grayscale version of the native image corresponding to FIGS. 46A-46B. [Figure 88] 48A-48F are grayscale versions of the native images corresponding to the FIGS. [Figure 89] FIG. 49C is a grayscale version of the native image corresponding to FIGS. 49A-49B. [Figure 90] FIG. 50C is a grayscale version of the native image corresponding to FIGS. 50A-B. [Figure 91] FIG. 51C is a grayscale version of the native image corresponding to FIGS. 51A-51B. [Figure 92] 68A-68J are grayscale versions of the native images corresponding to FIGS. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] The entities, connections, arrangements, etc. depicted in and described in connection with the various figures are presented by way of example and not by way of limitation. Thus, any and all statements or other indications as to what a particular figure "depicts" or what a particular element or entity within a particular figure "is" or "has," as well as any and all similar statements that may be read in isolation and from the context as absolute and therefore limiting, may properly only be read when constructively preceded by a phrase such as "in at least one embodiment...". For brevity and clarity of presentation, this implied introductory phrase will not be repeated at length in the Detailed Description.

[0035] Exemplary Network for Implementation of the Embodiments A wireless transmit / receive unit (WTRU) may be used, in some embodiments described herein, as, for example, an MFP display.

[0036] 1A is a diagram illustrating an example communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content, such as voice, data, video, messages, broadcasts, etc., to multiple wireless users. The communication system 100 may enable the multiple wireless users to access such content through sharing of system resources, including wireless bandwidth. For example, the communication system 100 may use one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero-tailed Unique Word DFT Spread OFDM (ZT UW DTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, Filter Bank Multi-Carrier (FBMC), etc.

[0037] 1A, the communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104 / 113, a CN 106 / 115, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, although it will be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. For example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a “station” and / or “STA,” may be configured to transmit and / or receive wireless signals, and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspot or Mi-Fi devices, IoT devices, watches or other wearable head mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in an industrial and / or automated processing chain context), home electronic devices, devices operating on commercial and / or industrial wireless networks, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be referred to interchangeably as a UE.

[0038] The communication system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106 / 115, the Internet 110, and / or other networks 112. For example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNode B, a home Node B, a home eNode B, a gNB, a NR Node B, a site controller, an access point (AP), a wireless router, etc. Although the base stations 114a, 114b are each depicted as a single element, it will be understood that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0039] The base station 114a may be part of the RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and / or base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide wireless service coverage to a particular geographic area, which may be relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, one for each sector of the cell. In an embodiment, the base station 114a may use MIMO technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0040] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0041] More specifically, as mentioned above, the communication system 100 may be a multiple access system and may use one or more channel access schemes such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base stations 114a and the WTRUs 102a, 102b, 102c in the RAN 104 / 113 may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 115 / 116 / 117 using Wide Area CDMA (WCDMA®). WCDMA may include communication protocols such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High Speed ​​UL Packet Access (HSUPA).

[0042] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-Pro).

[0043] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology, such as New Radio (NR) radio access, which may establish the air interface 116 using NR.

[0044] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may implement LTE and NR radio access together, e.g., using a dual connectivity (DC) principle. Thus, the air interface utilized by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0045] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement a wireless technology such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), or the like.

[0046] The base station 114b in FIG. 1A may be, for example, a wireless router, a Home Node B, a Home eNode B, or an access point, and may utilize any suitable RAT to facilitate wireless connectivity within a localized area, such as a business office, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by a drone), a road, etc. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology, such as IEEE 802.11, to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology, such as IEEE 802.15, to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-Pro, NR, etc.) to establish a picocell or femtocell. 1A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not be required to access the Internet 110 via the CN 106 / 115.

[0047] The RAN 104 / 113 may communicate with the CN 106 / 115, which may be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have varying Quality of Service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 / 115 may provide call control, billing services, mobile location services, prepaid calling, Internet connectivity, video distribution, etc., and / or perform high level security functions such as user authentication. Although not shown in FIG. 1A, it will be understood that the RAN 104 / 113 and / or the CN 106 / 115 may communicate directly or indirectly with other RANs using the same RAT as the RAN 104 / 113 or a different RAT. For example, in addition to being connected to the RAN 104 / 113, which may utilize NR radio technology, the CN 106 / 115 may also communicate with another RAN (not shown) that uses GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0048] The CN 106 / 115 may also act as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network providing plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) in the TCP / IP Internet protocol suite. The network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the network 112 may include another CN connected to one or more RANs that may use the same RAT as the RAN 104 / 113 or a different RAT.

[0049] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with a base station 114a that may use a cellular-based wireless technology and with a base station 114b that may use an IEEE 802 wireless technology.

[0050] Figure 1B is a system diagram illustrating an example WTRU 102. As shown in Figure 1B, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a GPS chipset 136, and / or other peripherals 138. It will be understood that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0051] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although FIG. 1B depicts the processor 118 and the transceiver 120 as separate components, it will be understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0052] The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF and light signals. It will be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0053] 1B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may use MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0054] The transceiver 120 may be configured to modulate signals that can be transmitted by the transmit / receive element 122 and to demodulate signals received by the transmit / receive element 122. As mentioned above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11, for example.

[0055] The processor 118 of the WTRU 102 may be coupled to and may receive user input data from a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from and store data in any type of suitable memory, such as a non-removable memory 130 and / or a removable memory 132. The non-removable memory 130 may include RAM, ROM, a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, or the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0056] The processor 118 may receive power from the power source 134 and may be configured to distribute and / or control the power to other components within the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0057] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to or instead of information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of when signals are received from two or more nearby base stations. It will be understood that the WTRU 102 may acquire location information by way of any suitable location determination method while remaining consistent with an embodiment.

[0058] The processor 118 may be further coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or video), a USB port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, and the like. The peripherals 138 may include one or more sensors, which may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0059] The WTRU 102 may include a full-duplex radio where transmission and reception of some or all of the signals (e.g., associated with a particular subframe for both the UL (e.g., for transmission) and the downlink (e.g., for reception)) may occur simultaneously and / or simultaneously. The full-duplex radio may include an interference management unit that reduces and / or substantially eliminates self-interference either via hardware (e.g., chokes) or signal processing via a processor (e.g., a separate processor (not shown), or via the processor 118). In an embodiment, the WRTU 102 may include a half-duplex radio for transmission and reception of some or all of the signals (e.g., associated with a particular subframe for either the UL (e.g., for transmission) or the downlink (e.g., for reception)).

[0060] 1A-1B and the corresponding description thereof, one or more, or all of the functions described herein with respect to one or more of the WTRUs 102a-d, the base stations 114a-b, and / or any other devices described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more, or all of the functions described herein. For example, the emulation device may be used to test other devices and / or to simulate network and / or WTRU functions.

[0061] The emulation device may be designed to perform one or more tests of other devices in a lab environment and / or an operator network environment. For example, the one or more emulation devices may perform one or more or all functions while fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices in the communication network. The one or more emulation devices may perform one or more or all functions while temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation device may be directly coupled to another device for testing purposes and / or may perform tests using over-the-air wireless communication.

[0062] The emulation device or devices may perform one or more functions, inclusive, while not being implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation device may be utilized in a test laboratory and / or in a test scenario within a non-deployed (e.g., test) wired and / or wireless communication network to perform testing of one or more components. The emulation device or devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (which may include, for example, one or more antennas) may be used by the emulation device to transmit and / or receive data.

[0063] MFP-Based Method for Stereoscopic Viewpoint The systems and methods disclosed herein according to some embodiments describe various exemplary MFP-based embodiments for cost-effective and flexible stereoscopic viewpoint generation within a depth-image-based rendering (DIBR) system. The exemplary methods according to some embodiments for generating virtual viewpoints allow users to vary their head tilt and viewing position more freely without degrading the viewing experience. In addition, in some embodiments, users are provided with natural motion parallax and personal adjustment of stereoscopic vision, enabled by the systems and methods disclosed herein according to some embodiments, to create stereoscopic mismatch. Some embodiments enable the creation of high-quality stereoscopic content and mismatch-based interaction of both wearable and external displays. Some embodiments and means of interaction may be used with emerging MFP displays.

[0064] In some embodiments, MFP-based synthetic disparity is used to produce high-quality stereoscopic content for DIBR. In some embodiments, the number of focal planes is not limited. In addition, forming a stereoscopic view at the receiver allows flexible functionality. Using MFP allows virtual viewpoint generation with adjustable disparity and orientation in real-time operation. This methodology enables functionality that might otherwise use more complex and bitrate-intensive methods (e.g., real-time 3D and bright-field systems). The depth-plus-texture input format is applicable to one or more embodiments that may use lower bitrates for transmission and storage.

[0065] In some embodiments, MFPs are used to generate high quality stereoscopic content with large discrepancies, meaning large stereoscopic depth and / or small viewing distance to the generated content. The desired accuracy can be achieved by using a large number of MFPs, as well as by selecting and optimizing depth blending functions when generating the MFPs.

[0066] In some embodiments of the disclosed method and system, a virtual stereoscopic projection may be formed in real time at the receiving site, which allows making the stereoscopic content to be viewed with maximum head tilt orientation of the stereoscopic baseline. Mismatch and baseline orientation modification may be performed independently at multiple receiving nodes, individually for several users per node. In addition, some embodiments support creating motion parallax at any viewing orientation, allowing for more natural viewing.

[0067] The systems and methods disclosed herein according to some embodiments may be used in creating content for stereoscopic displays, including wearable displays and external screens, using accommodation invariant (parfocal) NEDs, a variant of stereoscopic display, which supports natural accommodation (avoiding VAC) when viewing normal stereoscopic content. In some embodiments, virtual mismatch accommodation and related interactions may also be implemented to emerging MFP displays with natural accommodation. In some embodiments for creating content with large mismatch, using a polynomial blending function (or other function that weights pixels for the entire depth range) may result in better quality on object edges.

[0068] Systems and methods disclosed herein according to some embodiments relate to forming stereoscopic image pairs (stereograms) in stereoscopic 3D (S3D) video transmission systems and in systems based on depth image-based rendering (DIBR). Some embodiments may use multi-focal plane (MFP) content, MFP capabilities, and number and location of planes. Some embodiments may use dynamic user behavior or content to reconcile discrepancies.

[0069] While the MFP can provide focal cues, many systems use only two focal planes to allow for disparity (a compromise image between the eyes is shown, creating the appearance of stereoscopic 3D). Many previous systems do not perceive continuous depth (from both accommodation and disparity) using larger MFP stacks (>2 planes). In many previous stereoscopic viewing systems, a separate MFP stack is provided for each eye for stereoscopic viewing.

[0070] FIG. 2A is a schematic diagram 200 illustrating an example set of focal planes displayed at different depths from the user. FIG. 2B shows an illustration showing an example focal plane image. One system to address vergence accommodation conflict (VAC) uses a multi-focal plane (MFP) display to avoid VAC. FIG. 2A and FIG. 2B are diagrams adapted from a paper on optimal rendering for MFP displays (see non-patent document 1). In FIG. 2A, a user 202 is shown looking at four presentation planes (e.g., focal planes) 204, 206, 208, 210. Images 212, 214, 216, 218 for each of the four planes 204, 206, 208, 210 are shown in FIG. 2B. Such a system may not be responsive to the viewer. MFPs provide focal cues, but have previously been used with only two focal planes to allow for conflict (a compromise image is illustrated between the eyes, which creates the appearance of a stereoscopic 3D). In many previous stereoscopic viewing systems, a separate MFP stack is provided for each eye for stereoscopic viewing.

[0071] 3 is a graph illustrating an exemplary image mismatch between left and right images versus observer-screen distance for Percival's comfortable viewing zone. FIG. 3 is a graph 300 of Percival's comfortable viewing zone. The graph shows image mismatch 302 between left and right images versus observer-screen distance 304 for viewer comfort based on Percival's comfortable viewing zone theorem. FIG. 3 is shown in the paper (Non-Patent Document 2).

[0072] FIG. 4 is a schematic diagram illustrating exemplary left and right eye images for a stereoscopic display for a viewer with a non-rotated orientation. FIG. 4 shows left and right images displayed for a near-eye display viewer as seen by a user. FIG. 4 shows a parallax scenario 400 when viewing a small boat in the distance behind a large foreground boat. The right eye image 404 shows a smaller boat 412 to the right of the larger boat 410 than the left eye image 402, which shows the top of the smaller boat 408 behind the larger boat 406. The head icon 414 on the right side of FIG. 4 indicates the original orientation of the viewer.

[0073] FIG. 5 is a schematic diagram illustrating exemplary left and right eye images bound in glasses coordinates for a stereoscopic display for a viewer with a rotated orientation where the horizon is tilted with the head. FIG. 5 shows left and right images displayed for a near-eye display with viewer rotation. In the scenario 500 of FIG. 5, the head icon 506 on the right side of FIG. 5 indicates the rotated orientation of the viewer as seen by an external observer standing in front of the viewer. The glasses and near-eye display are illustrated from the perspective of the viewer worn by the viewer. In the case of FIG. 5, the same images 502, 504 shown in FIG. 4 are illustrated without regard to the rotated orientation. This means that the 3D perception remains proper, but the horizon rotates along the viewer's orientation, disrupting viewing in the physical world.

[0074] FIG. 6 is a schematic diagram illustrating an example left-eye image and right-eye image adjusted to match the orientation of the horizon for a stereoscopic display for a viewer with a rotated orientation and unchanged mismatch. FIG. 6 shows left and right images displayed for a near-eye display with the same viewer rotation as shown in FIG. 5. In the scenario 600 of FIG. 6, images 602, 604 with the same original orientation as in FIG. 4 are rotated corresponding to the rotation shown for the head icon 606 to keep the image horizontal. The mismatch between the left image 602 and the right image 604 is the same as in FIG. 4 and FIG. 5 (keep along the horizontal baseline). Although the horizon for the rendering is aligned with the real world, the mismatch does not match the rotated orientation, so the 3D perception is distorted.

[0075] FIG. 7 is a system diagram illustrating an example set of interfaces for manipulating image mismatch. FIG. 7 is a system interface diagram for manipulating image mismatch, taken from Non-Patent Document 2. Many systems that produce stereo pairs during capture do not support individual user motion (occlusion, variable stereo baselines, non-horizontal orientation, etc.). One system 700 with some user customization of mismatch is described in Non-Patent Document 2, but it is understood that this is limited to variations in rendering the mismatch. Such a system 700 adjusts the mismatch based on user preferences. It is understood that such a system provides only spacing adjustment.

[0076] 8 is a system diagram illustrating an example set of interfaces for a DIBR-based 3D transmission system. Figure 8 shows an interface diagram for one configuration of a DIBR-based 3D transmission system 800, adapted from an academic paper (Non-Patent Document 3).

[0077] Many DIBR-based systems enable stereoscopic image formation as a post-processing step at the receiver, but many previous stereoscopic video systems are less flexible: the amount of stereoscopic disparity and depth budget can be highly dependent on the captured content, and while control of disparity and depth budget can be advantageous in some cases, e.g., later than during shooting of the scene, it can also be difficult due to fixed parameters, e.g., in the production / front end.

[0078] In many DIBR systems, at the receiving end, virtual viewpoint generation 802 consists of a 3D warping 804 stage and a hole filling 806 stage (see virtual viewpoint generation block 802 in FIG. 8). 3D warping 804 is used to create two virtual views into a textured depth map, as seen from the viewer's two eyepoints.

[0079] FIG. 9 is a schematic plan view illustrating an example scenario showing the appearance of disocclusion when changing source and target cameras. In many cases, the corresponding perspective change reveals parts of the view behind the occluding object. As illustrated by the schematic picture 900 of FIG. 9 and the examples of FIGS. 10A and 10B, disocclusion (i.e., disocclusion area 902) may be visible as a gap or hole 904 in each new perspective image, since the captured texture or depth does not contain information from those areas. In FIG. 9, when changing the viewpoint from source to target, as illustrated by the viewpoint lines for source camera 910 and target camera 912 and the respective source image 906 and target image 908, disocclusions related to near object 914 and far object 916 may be caused.

[0080] 10A and 10B are illustrations showing an example 3D warped image before and after hole filling. FIG. 10A and 10B show an example of a 3D warped image. The example shows only the left eye image. The right eye image is formed by the same means and has a similar defect, but it is the right side of the object. FIG. 10A and 10B show an example of disocclusion caused by 3D warping. The original monoscopic image 1000 is on the left, and details of the 3D warped image 1050 are on the right (linear interpolation-based hole filling 1052 to fill the blank hole 1002 shown on the left image 1000 as a black line with the use of black and white line art with this application). In some embodiments, the method may include processing the focal plane images to handle gaps caused by shifting the focal plane images. Some embodiments may filter one or more focal plane images to handle such gaps (e.g., to handle image mismatch caused by shifting one or more focal plane images). 10A and 10B are line drawing versions adapted from slide 14 of the presentation (by ).

[0081] In hole filling, various image and depth map filtering and inpainting algorithms may be used to fill holes or other forms of disocclusions. The type and number of occlusions depends heavily on the content, and the distortion and their visibility also depends heavily on the content. 3D warping may be performed by computer graphics algorithms. The complexity, speed and results of these algorithms vary, depending for example on using perspective views and camera models. An example of a mathematical formulation for a shift sensor setup is given in Non-Patent Document 5.

[0082] The systems and methods disclosed herein according to some embodiments are used for virtual viewpoint generation (DIBR), which is different from previous methods for 3D warping and hole filling. The systems and methods disclosed herein according to some embodiments may, for example, allow for adjusting the mismatch and the corresponding stereoscopic depth budget as post-processing more easily than some previous systems. The systems and methods disclosed herein according to some embodiments may, for example, allow for supporting stereoscopic viewing at arbitrary viewing orientations (e.g., head tilt, etc.) more easily than some previous devices (many previous devices only support horizon viewing). The systems and methods disclosed herein according to some embodiments, for example, allow for easier formation of virtual viewpoints to support (e.g., translational and / or axial) motion parallax. The systems and methods disclosed herein according to some embodiments may have more distributed and less visible distortion than some previous DIBR systems.

[0083] Stereoscopic 3D (S3D) The use of stereoscopic displays is one means of presenting 3D information (sometimes called stereoscopic 3D or S3D). Stereoscopic viewing captures parallel views - a stereo pair - by two cameras separated by a small distance (e.g. the stereo baseline). The capture setup mimics binocular image perception by two human eyes.

[0084] In recent years, stereoscopic systems have seen a resurgence in 3D movies, 3DTV, and augmented and virtual reality applications (AR and VR). For many AR / VR systems, wearable near-eye displays (or glasses) are used.

[0085] In real-world space, human eyes can freely scan and glean information by focusing and accommodating at different distances or depths. When viewing, the convergence (convergence) of the eyes is different when looking in a parallel direction (e.g., an object at an infinite distance away) than when looking in a cross direction (e.g., an object close to the eye). In general, convergence and accommodation are often very strongly coupled, so that in most cases, the accommodation / focus and convergence points of the two eyes naturally meet at the same 3D point.

[0086] In many previous stereoscopic viewing systems, the eyes are focused on the same image (or display) plane, but the human visual system (HVS) and the brain form the 3D perception by detecting image discrepancies (e.g., small distances between corresponding pixels in the two 2D projections).

[0087] 11A-11C are schematic perspective views illustrating exemplary vergence accommodation conflicts (VACs) in many stereoscopic viewing systems. 11A-11C show vergence accommodation conflicts (VACs) in many stereoscopic viewing systems. 11A-11C are based on Non-Patent Document 6. 11A shows stereoscopic viewing where the vergence point 1102 is the same as the accommodation point 1102 (which may be typical when viewing with the naked eye). 11B and 11C show viewing with a stereoscopic display. 11B and 11C show viewing where the vergence points 1132, 1162 and the accommodation points 1134, 1164 are different, which causes vergence accommodation conflicts (VACs). 11A shows natural viewing in a real-world environment 1100 with a vergence distance 1106 equal to the focal distance 1108 for the viewer 1110. Figure 11B shows an intersected stereoscopic view 1130 on an S3D display with a vergence distance 1136 that is shorter than the focal length 1138 for a viewer 1140. Figure 11C shows a non-intersected stereoscopic view 1160 on an S3D display with a vergence distance 1166 that is longer than the focal length 1168 for a viewer 1170. Although VACs are known to cause visual distortions and other types of discomfort, stereoscopic imaging is still commonly used for near-eye displays due to its facilities and cost-effectiveness.

[0088] 3D display using multi-focal plane (MFP) In the case of multi-focal plane (MFP) systems, the viewer can focus on different objects and depths, which avoids the VAC typical of stereoscopic displays. A stack of natural or virtual focal plane images is rendered at different depths. The focal plane being observed is seen in focus, while other focal planes not being observed are blurred by the human visual system.

[0089] MFP displays present a stack of different focal planes that make up a 3D scene from layers along the viewer's visual axis. The viewer is provided with a 3D view by projecting those pixels of the focal planes that are visible to the viewer's eyepoint at different depths and spatial angles.

[0090] Multiple focal planes are primarily complementary (not additive) to the viewing direction from the eyepoint, however the additive effect can eliminate quantization steps and contouring that would otherwise be perceived multiple times when viewing edited views from different focal planes.

[0091] Multiple focal planes can be displayed, for example, either by spatially multiplexing a stack of 2D views, or by sequentially switching the focal length of a single 2D view by a high-speed variable focus element (VFE) while spatially rendering the visible portion of the corresponding multi-focal image frame.

[0092] FIG. 12 is a schematic plan view illustrating an exemplary multi-focal-plane (MFP) near-eye display according to some embodiments. FIG. 12 shows a schematic diagram 1200 of a multi-focal-plane (MFP) near-eye display. Each image 1202, 1204 in a display stack 1208 of a virtual focal plane 1206 is rendered at a different depth, and the viewer's eyes 1210, 1212 blur those focal planes that are not observed through the eyepieces 1214, 1216. The paper (see Non-Patent Document 7) describes viewing content on an external display. Non-Patent Document 7 describes the phenomenon of perceiving in 3D when viewing two specially formed focal planes and the result observed by a physical prototype. The paper describes the specific case of MFP display.

[0093] A larger number of focal planes may have better accuracy of the MFP display, but may have difficult implementation. For MFP displays, the feasible number of focal planes may be a small number (such as 4 to 6), which may limit the quality achieved. The paper (Non-Patent Document 8) gives information on the number of focal planes.

[0094] Figure 13 is a schematic plan view illustrating an exemplary viewing of near, intermediate, and far focus image planes according to some embodiments. Figure 13 is a plan view schematic of the view lines for the left and right eyes for a prototype (described in Non-Patent Document 9). Figure 13 shows a plan view schematic 1300 (adapted from Non-Patent Document 9 and using the exemplary values ​​described therein) that uses multiple MFPs 1302, 1304, 1306 to capture 3D perception. In the case of Figure 13, each eye 1308, 1310 sees the focal planes 1302, 1304, 1306 from slightly different perspectives, creating a discrepancy regarding 3D perception. The example of Figure 13 shows a far image plane 1302 of an object at a distance of 53.6 cm (1.87D) from the viewer's eye 1308, 1310, a mid image plane 1304 of an object at a distance of 39.4 cm (2.54D) from the viewer's eye 1308, 1310, and a near image plane 1306 of an object at a distance of 31.1 cm (3.21D) from the viewer's eye 1308. For the example of Figure 13, ±4.4° vertical field of view.

[0095] 3D perception from focal planes is appropriate for flat objects that lie strictly on those planes. By using depth blending, disparities are created between focal planes, creating a continuous and realistic depth perception for objects between the planes. Further details regarding depth blending and the perception of disparities in MFP rendering are given below.

[0096] In previous MFP prototypes, the focal planes were shown on an external display at different distances from the viewer. When showing the MFP on a near-eye display (such as in Figure 12), two MFP stacks are formed, seen separately by the left and right eyes.

[0097] FIG. 14 is a schematic diagram illustrating exemplary focal image planes viewed by an observer. A version of this is included in Non-Patent Document 7. FIG. 14 shows a top view 1400 and a front view 1402, 1404 of two image planes 1406, 1408 viewed by an observer 1410, and a side view 1414 of how the planes are perceived as a 3D image by an observer 1412. In Non-Patent Document 7, two focal planes 1406, 1408 are formed and placed at different distances and viewed from two eyepoints. This setup creates a mismatch and 3D perception caused by the lateral displacement of the two focal planes seen from the two eyepoints. The displacement gives rise to a specific pixel value distribution at the object edge, which the brain interprets as a mismatch and as a depth variation, respectively. FIG. 14 shows the perception of a 3D image by viewing two overlapping focal planes formed from a scene. The two eyes view the focal plane from slightly different angles, which creates a synthetic stereo discrepancy between the focal planes.

[0098] 15A-15C are schematic diagrams illustrating exemplary front and back focal planes for an exemplary image of a square. 15A-15C, a version of which is included in Non-Patent Document 7, show a synthetic stereo discrepancy created at the edge of an object. In Non-Patent Document 7, a synthetic stereo discrepancy is created at the edge of a square 1504, 1506 as shown in Figs. 15A-15C. In Non-Patent Document 7, an exemplary view 1500 consisting of two focal planes (such as that shown in Fig. 14) is visualized by an observer 1502 by capturing a left stereo image pair 1532, 1536 from a left eye point 1544 and a right stereo image pair 1534, 1538 from a right eye point 1546 on a prototype display (shown in Figs. 16A-16B). Fig. 15B is a perspective view 1530 of the scene. The retinal images 1540 have a common area 1542 that is seen in each retinal image. The planar schematic 1560 of Figure 15C shows the image luminance distributions as perceived from a left eyepoint 1562 and a right eyepoint 1564. The disparity in these luminance distributions (see lateral shift) gives the perception of depth.

[0099] 16A-16B are schematic front views illustrating exemplary focal planes displayed for right-eye and left-eye images. Figures 16A-16B (these versions are from "Patent Document 7") show a crossed-eye stereogram of the view of Figure 14 with two focal planes. Figure 16A shows a right-eye image 1600 and Figure 16B shows a left-eye image 1650.

[0100] In Non-Patent Document 7, the luminance distribution for the focal plane is computationally achieved by weighting the image pixels (captured projections of the scene) by two linear weighting functions proportional to the pixel's distance (depth coordinate) and its complement. Correspondingly, a feasible image capture format that supports the formation of focal planes is texture plus depth. Non-Patent Document 7 describes the depth fusion or blending effect with two focal planes, as seen from two eyepoints.

[0101] Disparity may be used to indicate depth perception. For example, if the two focal planes are flat objects, the closer object is slightly shifted relative to the more distant one as seen from the two eyepoints. Non-Patent Document 7 states that an object between two focal planes is perceived at a position between the two extremes. This phenomenon is caused by the depth weighting (or depth blending) used to shape the two focal plane contents. Depth blending shifts the apparent edge location as illustrated in Figures 15A-15C, creating a synthetic but realistic disparity and depth perception between the two focal planes.

[0102] FIG. 17 is a schematic plan view illustrating an example viewing of two focal planes with a depth-based luminosity weighting function. FIG. 17 is a plan view schematic 1700 showing the principle of depth blending of two focal planes 1704, 1706 seen from one eyepoint 1702. Non-Patent Documents 10 and 11 explain how depth blending effects are informed when working with two or more focal planes viewed monocularly by only one eye. FIG. 17 is adapted from Non-Patent Document 11. Non-Patent Document 11 explains the perceived luminance of two overlapping pixels on two focal planes in Eq. L0=L1(z)+L2(z)=w1(z)L0+w2(z)L0 Formula 1 where w1 and w2 are depth-weighted fusion functions. The perceived depth of the fused pixel

[0103]

number

[0104] can be considered as the weighted sum of the depths z1, z2 of the two focal planes 1704, 1706 in Equation 2.

[0105]

number

[0106] When viewing two or more MFPs monocularly, the phenomenon is of course not attempting to shift edges to change the mismatch (and depth perception) as described by [7], but rather to eliminate the abrupt step in perceived retinal blur when viewing around the transition between adjacent focal planes, as described by

[10] .

[0107] As described by

[10] , monocular depth blending effects have been noted to work with MFP stacks with more than two focal planes. Exploiting the Depth-Fused 3D (DFD) phenomenon in MFP displays leads to a variant of the MFP method (described in

[11] and denoted DFD-MFP) where the focal planes are not strictly separated in angular (and xy) space, but also have some overlapping and additive properties along the depth (z) dimension. This phenomenon is sometimes called depth-based filtering or depth blending.

[0108] FIG. 18 is a schematic plan view illustrating an exemplary viewing of three focal planes by the left and right eyes, according to some embodiments. Non-Patent Document 7 describes a method to support binocular viewing of one (monocular) MFP stack, despite it being composed of only two focal planes. This idea can be generalized to MFP displays with three or more focal planes. FIG. 18 extends the viewing setup of Non-Patent Document 7 to a stereoscopic viewing setup 1800 for three focal plane displays, where a monocular stack of three MFPs 1802, 1804, 1806 is formed from an average eyepoint 1808, and the same focal planes 1802, 1804, 1806 are viewed from slightly different viewpoints 1810, 1812 by the left and right eyes. Following Non-Patent Document 7, research reports and MFP displays may not specify (or be understood to specify) precisely whether the particular device in question displays one or two sets of MFPs to the viewer.

[0109] In the example described herein according to some embodiments, the viewing situation of the MFP display is that shown in FIG. 18, where one monocular MFP stack is formed using monocular depth plus texture input, and two (stereoscopic) viewpoints are created on this MFP stack. In a near-eye display, however, it is difficult to provide a suitable viewing point for only one focal plane stack, and optical components and reflectors may be used to allow a suitable lateral position for the eyes. Furthermore, this adjustment will of course be different for people with different eye separations (different viewing angles to the MFP stack). In reality, as with a stereoscopic near-eye display that illustrates stereoscopic images side-by-side, one MFP stack needs to be split into two stacks, one for each eye. As in FIG. 12, the viewing conditions described by FIG. 18 and FIG. 27B need to be matched with two parallel MFP stacks from slightly different viewpoints. Both of these two MFP stacks are different from the original monocular stack, for which the viewing point is between the eyes (see FIG. 18).

[0110] Depth blending function for forming MFP stacks The captured scene is decomposed into several focal planes (e.g., image layers at various depths or accommodation levels). The focal planes are formed using a depth blending or weighting function, which multiplies each image pixel by a weight depending on its distance from the formed focal plane (e.g., the distance indicated by a depth image or depth map). Two properties of the blending function are a division of one and a depth blending property. For a division of one, the focal plane luminance sums to the original scene luminance. For the depth blending property, the same pixel may contribute to multiple focal planes. This property is a result of the continuity and overlap of the blending function. The functions used or referenced herein according to some embodiments are examples and are non-limiting.

[0111] Box filtering (e.g., slicing image information based on pixel depth) is a good starting point for understanding depth blending. However, a box filter is not actually a blending function because each pixel contributes to only one of the focal planes to the slicing result. When using a box filter, an approximation of the focal plane images is formed by slicing the depth map corresponding to each image into narrow depth regions (slices) and projecting the corresponding pixels onto a central (flat) focal plane in each depth region.

[0112] When viewing a stack of focal planes, the constructed view is formed by the information about the different focal planes visible from the viewer's eyepoint. In box filtering, slicing into the depth domain results in MFPs completing each other in the spatial (xy) directions, rather than summing along the depth dimension (z dimension). Correspondingly, focal planes generated by box filters do not create discrepancies between focal planes (perception of 3D), but only for objects on the focal planes.

[0113] Fig. 19 is a schematic plan view illustrating an example quantization of depth for an MFP display without depth blending (weighting), according to some embodiments. As illustrated by Fig. 19, resulting from the use of a box filter, smooth 3D surfaces are quantized in the depth dimension. Fig. 19 shows a schematic example 1900 of depth quantization when describing a view with five focal planes 1902, 1904, 1906, 1908, 1910. Arrow 1912 indicates the viewing direction.

[0114] The box filter separates image information in a rigorous manner in the spatial (xy) and depth (z-) dimensions. In practice, since only a small number of focal planes are used, the depth dimension is heavily quantized, which can lead to poor accuracy in presenting 3D shape and depth.

[0115] In some embodiments, the processing of the focal plane images may include determining depth blending weights for one or more of the focal plane images. The processing of the focal plane images may also include adjusting the focal plane images using the depth blending weights. For example, the depth dimension values ​​may be multiplied by the depth blending weights.

[0116] Depth-based blending may be used to reduce quantization errors in the depth dimension that would otherwise be visible to the human eye. Depth blending means using a depth-based function to weight the pixels used to construct each focal plane. One depth blending function is the tent filter, which is a piecewise linear, sawtooth blending function 2050 (FIG. 20B). For the box filter, the corresponding function 2000 is shown in FIG. 20A.

[0117] 20A and 20B are graphs illustrating an example of weight versus depth according to some embodiments, showing schematic diagrams of depth blending functions for four MFPs. FIG. 20A shows depth slicing without blending (box filter) and FIG. 20B shows a linear (tent) filter. In some embodiments, the range of depth values ​​is between 0 and 255 (z in FIG. 20A and FIG. 20B ). min and z max (See ).

[0118] FIG. 21A is a graph illustrating an example blending function of weights versus depth according to some embodiments. FIG. 21B is an illustration showing an example test image. FIG. 21C is an example depth map illustrating pixel distance for FIG. 21B. FIG. 21D-FIG. 21F are illustrations showing an example focal plane image using the depth weights of FIG. 21A with the test image of FIG. 21B and its example depth map of FIG. 21C according to some embodiments. FIG. 21B and FIG. 22B are based on an image found on a webpage (Non-Patent Document 12), and FIG. 21D-FIG. 21F are modified versions of that image. The example unprocessed test image and its depth map shown in FIG. 21B and FIG. 21C are merely example general test images of flowers that are used for illustrative purposes many times throughout this disclosure. In many cases, modified versions of example test images are presented, for example from an example process performed on the test image according to some embodiments. In some embodiments, a polynomial blending function extends over the entire depth range. Figure 21A shows three polynomial blending functions 2100, 2102, 2104. Figure 21A and Figures 21B-21F show examples of the three polynomial blending functions 2100, 2102, 2104 (Figure 21A) and corresponding focal planes (Figures 21D-21F) for a test image 2110 (Figure 21B) and its corresponding depth map 2120 (Figure 21C).

[0119] In some embodiments, a depth map (e.g., FIG. 21C) may be generated, which may be similar to that shown in FIG. 19 by the smooth thin line 1914. The depth map is used to separate the input image into, for example, small, medium, and large focal plane distances from the viewer. A depth blending function (e.g., FIG. 21A) is used to generate a focal plane image for a particular focal plane distance. This process may be repeated for each focal plane image (e.g., FIG. 21D through FIG. 21F). In some embodiments, the depth map (see FIG. 21C) may be generated by several methods, including depth from focus, depth from one or more RGB images, and depth from an RGB-D sensor (using structured light or ToF). It will be appreciated that there are various known means of acquiring a depth image.

[0120] FIG. 22A is a graph illustrating an example blending function of weights versus depth according to some embodiments. FIG. 22B is an illustration showing an example test image. FIG. 22C is an example depth map illustrating pixel distance for FIG. 22B. FIG. 22D-F are illustrations showing an example focal plane image using the depth weights of FIG. 22A with the test image of FIG. 22B and its example depth map of FIG. 22C according to some embodiments. The example depth map of FIG. 22B and FIG. 22C is derived from an image found on a webpage (Non-Patent Document 12), and FIG. 22D-F are modified versions of that image. FIG. 22A shows weights for three sinusoidal blending functions. Fig. 22A shows an example of three sinusoidal functions 2200, 2202, 2204 that result in three MFPs 2230, 2240, 2250 (Figs. 22D-22F) for a test pattern image 2210 (Fig. 22B) and its depth map 2220 (Fig. 22C). Another variation of the blending filter has been proposed by

[11] .

[0121] Depth blending or fusion, as explained by [7], is a means of increasing the apparent number of focal planes and therefore also the depth precision.

[10] extended the synthesis mismatch findings to cases with three or more focal planes, increasing the effective number of depth planes from those few allowed by practical MFP display implementations.

[10] further reported the benefits of depth blending when viewing a monocular MFP stack with one eye. According to

[10] , depth blending filters the gradual changes in focal length perceived by one eye at the transitions between sparse focal planes.

[0122] The creation and perception of synthetic disparity (depth) between two focal planes is described in Non-Patent Document 7. The creation of depth perception between three or more focal planes is described in Non-Patent Document 10 and Non-Patent Document 9. Filtering of gradual accommodation changes between focal planes is described in Non-Patent Document 10. Non-Patent Document 11 describes variations on the depth blending function.

[0123] Wearable glasses-type near-eye displays (NEDs) are popular in virtual reality and gaming, and more recently in augmented reality. Glasses that use a mobile phone as the display element are one example of a popular, low-cost device for viewing virtual or camera-captured images. Many near-eye displays (NEDs) present a sequence of stereoscopic image pairs. Stereoscopic rendering may also be used with external displays, including those for stereoscopic 3D (S3D) TVs.

[0124] Capturing stereoscopic video (a sequence of stereoscopic image pairs) can pose various limitations: capturing and transmitting stereoscopic content can, for example, prevent post-processing adjustments for mismatches and other important S3D parameters, both at content production time and when the content is rendered or viewed at a receiver.

[0125] The video-plus-depth (depth-plus-texture) format in capture and transmission is used in depth image-based rendering (DIBR). Much research has been done in the field, focusing on optimizing the rendering quality at the receiver, and more specifically, on removing disocclusion induced in DIBR systems when generating stereoscopic views from two virtual viewpoints.

[0126] MFP displays are based on the 3D perception of multiple focal planes, which can be formed using depth plus texture formats. Many MFPs can avoid VAC, which can occur for stereoscopic displays. For technical reasons, the maximum number of focal planes is generally limited to only a few in an MFP display. When using a few focal planes, the rendering quality can be improved by depth blending, but the number of focal planes can still present a limitation in the MFP rendering quality.

[0127] Many previous systems for both DIBR and MFP lack support for user motion-based interaction, which requires adjustment of the viewpoint to either changing stereo disparity or motion parallax. For many DIBR-based systems, one reason is the computational complexity of the methods used in virtual viewpoint generation (e.g., 3D warping and hole filling).

[0128] Some embodiments use a depth-plus-texture content format in image capture and transmission. Some embodiments of viewpoint generation differ from the 3D warping and hole-filling methods used in previous depth image-based rendering (DIBR) methods. In some embodiments, a multi-focal plane (MFP) is formed within the receiver using the depth and texture data. Stereoscopic content with a set of properties and supporting motion parallax is generated in real time using the formed MFP for multiple receivers individually at any baseline orientation.

[0129] Exemplary Processes and Structures The MFP-based method generates a stereoscopic viewpoint for a DIBR-based system. The exemplary method generates a virtual viewpoint to allow the user to vary his / her head tilt and viewing position. In addition, the user is provided with natural motion parallax and personal adjustment of stereoscopic vision, enabled by the exemplary method for creating stereoscopic disparity. The user's movements are tracked (e.g., by one or more head-mounted or external sensors) to generate a disparity that matches the user's posture and movements.

[0130] Some embodiments allow for stereoscopic content and non-coincidence-based interaction of both wearable and external displays. Exemplary implementations and means of interaction described herein according to some embodiments may also be used with MFP displays. Some embodiments use a depth plus texture type content format in the capture and transmission, as well as a multi-focal plane (MFP) formed therein within the receiver. In some embodiments, using data received once, stereoscopic content with a set of properties may be formed in real time, individually for multiple local viewers at any baseline orientation. The formation of the MFP may be done by several means, examples of which are disclosed herein according to some embodiments.

[0131] 23A-C are schematic front views illustrating an example scenario for stereoscopic image mismatch, according to some embodiments. FIGS. 23A-C show three (binocular) viewpoints from which a user views stereoscopic content. FIG. 23A is an example 2300 showing the use of viewing depth effects while lying down. FIG. 23B is an example 2330 showing a more widely adjusted depth effect. FIG. 23C is an example 2360 in which the viewpoint follows the user's movements. The user views the content from different orientations and varying amounts of stereoscopic mismatch. The content may be viewed, for example, using either a wearable display or an external display. In some embodiments, the 3D perception is based on an approximated (e.g., virtual) mismatch with the user viewing a monocular stack of multiple focal planes from two eye viewpoints.

[0132] Many stereoscopic 360° video systems only respond to orientation changes, not to user parallax movements. Many previous systems use orientation (pure orientation changes) to control the 360° video. Head rotation also imposes a translation of eye position. Avoiding VAC may provide comfortable stereoscopic viewing from a fixed position, for example.

[0133] In some embodiments, virtual viewpoint generation to form a stereoscopic image is performed by shifting and adding MFPs. Stereoscopic displays may be used to view the results, including near-eye S3D screens and external S3D screens. When using time-multiplexed S3D displays, the shutter glasses worn by the user may be equipped with sensors to track user motion to support motion parallax effects. To avoid VAC associated with stereoscopic displays, the stereoscopic display may be replaced by a display that supports natural accommodation, for example.

[0134] In some embodiments, a method may include receiving three-dimensional video content, calculating one or more multi-focal plane (MFP) images for a number and position of multiple focal planes using the three-dimensional video content, measuring motion tracking sensor readings of a motion tracking sensor, generating a viewer orientation signal from the motion tracking sensor readings, calculating one or more mismatch images (e.g., an MFP stack distorted, projected, and summed to a viewing point by an amount indicated by a mismatch vector) for each view of a shutter glasses display using the one or more MFP images and the viewer orientation signal, and rendering the one or more mismatch images to the shutter glasses display.

[0135] In some embodiments, a device may include one or more motion tracking sensors, a processor, and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform the following processes: receive three-dimensional video content, calculate one or more multi-focal plane (MFP) images for a number and position of the multi-focal planes using the three-dimensional video content, measure motion tracking sensor readings of a motion tracking sensor (e.g., one or more motion tracking sensors), generate a viewer orientation signal from the motion tracking sensor readings, calculate one or more mismatch images for each view of a shutter glasses display using the one or more MFP images and the viewer orientation signal, and render the one or more mismatch images on the shutter glasses display. For example, in some embodiments, measuring the motion tracking sensor readings of a viewer's position relative to a real-world environment may include, for example, measuring the motion tracking sensor readings of a viewer's position relative to a physical stationary point.

[0136] Inconsistent changes due to user actions 24A-C are schematic front views illustrating an example scenario for image mismatch with the direction of viewer orientation, according to some embodiments. Stereoscopic mismatch adjustments can be made by user motion and motion-related interactions. FIGS. 24A-C show mismatch-related viewpoint changes due to user motion supported by some embodiments. As illustrated by FIGS. 24A-C, the amount of synthetic mismatch (e.g., length, height, or width) 2400 (FIG. 24A), the orientation (e.g., direction) 2430 (FIG. 24B), and the position (e.g., lateral translation) 2460 (motion parallax) (FIG. 24C) can be adjusted, or any combination of these user motions, for example. The mismatch (amount of head tilt and lateral translation) can be adjusted based on tracking data from the user's eyewear or based on their manual input (e.g., indicating a personal preference for the amount of mismatch). The three mismatch-associated interactions illustrated in Figures 24A-24C, for example, may occur in any order and may form a consecutive sequence.

[0137] 24A-C illustrate mismatches created by some embodiments and expressed via bipolar vectors 2402, 2404, 2432, 2434, 2462, 2464 whose quantity (e.g., length, height, or width), orientation, and position relative to the origin (to support motion parallax effects) may be adjusted. The old mismatch values ​​are indicated by dashed arrows 2402, 2432, 2462, and the new mismatch values ​​are indicated by solid arrows 2404, 2434, 2464.

[0138] The disparity of a stereoscopic image pair (for the left and right eyes) may be represented by a tuple of vectors ("bipolar vectors") pointing in two opposite directions along the baseline between the eyes. The amount of disparity is given by the magnitude of the difference between the two vectors (Figure 24A). Head tilt is indicated by the angle between the disparity vector and the horizon (or a vector parallel to the horizon) (Figure 24B). Motion parallax (e.g., momentary shifted viewpoint) may be represented by the asymmetry of the two vectors, and the viewpoint is indicated by their average (Figure 24C).

[0139] The amount of mismatch is adjusted by scaling the mismatch vector (multiplying it by a scalar), the direction is adjusted by rotating the vector (or, for example, by modifying a numerical expression for the amount of rotation), and the translation is performed by a translation shift of the vector. Instead of conveying the captured user motion between the system components (between the glasses and the system terminal) as motion vectors, the component interfaces may be based on transmitting the mismatch vectors after modifying them based on the captured user motion.

[0140] In some embodiments, shifting the focal plane images may include scaling the mismatch vector for each focal plane image (or, for example, by modifying a numerical representation of the amount of mismatch adjustment due to the shift). For example, scaling may be performed as shown in FIG. 24A to widen (or shrink) the horizontal (or vertical) dimension. In some embodiments, shifting the focal plane images may include translating each focal plane image. For example, translating the focal plane images may include rotating the focal plane images or translating the focal plane images horizontally (or vertically) by an offset relative to a fixed origin.

[0141] FIG. 25 is a system diagram illustrating an example set of processing blocks and interfaces for generating left-eye and right-eye shifted and projected MFPs, according to some embodiments. The disclosed example method is a new example implementation of a virtual viewpoint generation stage in DIBR, according to some embodiments. DIBR was discussed above with respect to the virtual viewpoint generation stage illustrated in FIG. 8 (adapted from non-patent document 3). In FIG. 25, a block diagram 2500 for an example DIBR-based implementation according to some embodiments is shown. Depth map pre-processing 2506 may include a filtering operation on the depth map 2502. One operation is to smooth the depth map 2502 to reduce holes or maximum created discrepancies (e.g., for viewing comfort), as described in non-patent document 13.

[0142] Some embodiments for generating 2508 the MFP may be to use a blending function to produce a weight map (remapped depth map) for each MFP and form the MFP by multiplying each of the formed weight maps with the captured (texture 2504) image. Some embodiments may post-process 2510 the MFP before shifting and projecting 2512 the MFP. In some embodiments, filtering may be used to reduce holes when the MFP is shifted for large viewpoint changes or mismatches. In some embodiments, the MFP may be received as an input. In some embodiments, post-processing 2510 the MFP may include forming the MFP (e.g., via redistribution).

[0143] DIBR-based methods according to some embodiments may include hole-filling and other inpainting methods that reduce dis-occlusion artifacts in the 3D warped image. In some disclosed embodiments, focal plane shifts may produce similar distortions in the results, and equivalent processing (e.g., inpainting in either the depth or spatial dimensions) may be used on the MFPs after they are formed by depth blending. Shifting the MFPs to project 2512 into stereoscopic images 2516, 2518 may be used to meet the desired stereoscopic disparity (or depth budget). Shifting the MFPs may be done along any chosen baseline orientation (e.g., not just horizontal) 2514 to support arbitrary head tilt (data from an IMU (Inertial Measurement Unit) or similar in the NED). Additionally, motion parallax may be supported based on the user's head motion.

[0144] Note that forming one set of MFPs can serve both various baseline orientations, amounts of mismatch, and synthetic motion parallax for one or more local viewers (e.g., based on illustrating a series of stereograms with chance viewpoints). The MFP stack, once derived, can serve all mentioned cases. The direction and amount of shift, as well as the display (or, e.g., person) on which the results are rendered, vary.

[0145] FIG. 26 is a system diagram illustrating an example set of processing blocks and interfaces for generating left-eye and right-eye shifted and projected MFPs for three scenarios of viewer orientation, according to some embodiments. FIG. 26 shows a process 2600 in which the generated MFP stack can be used for several local viewers 2602, 2604, 2606, for example, with different disparities, orientations, and motion parallax. For example, a receiving terminal 2628 may receive a depth map 2630 and pre-process 2626 the depth map. A texture 2632 may be received by the receiving terminal 2628 to generate 2624 an MFP using the pre-processed depth map output. Some embodiments may post-process 2622 the MFP. The MFP may be shifted and projected 2616, 2618, 2620 for left and right views for each viewer 2602, 2604, 2606. Head pose and movement data (NED with IMU or similar) 2608, 2610, 2612 may be received by a multiplexer and wireless access point 2614. The movement data may be used in shifting and projecting 2616, 2618, 2620 of the MFP.

[0146] Some embodiments serve multiple people individually per site, with some interaction features. Note that instead of sending motion data from the glasses to the receiving terminal (see the "Head Pose and Movement" block in FIG. 26), in some embodiments the data may be a mismatch vector derived from the motion data or provided as manual input (e.g., to determine a preferred mismatch).

[0147] Some embodiments incorporate mismatch adjustment by creating a visually natural mismatch with the MFP display when a single MFP stack is viewed from an offset position. Some embodiments incorporate mismatch adjustment through feedback from the viewer to the TV / HMD renderer to produce a mismatch-adjusted image. Orientation changes can be handled with a glasses-based TV and with a 360° video HMD client.

[0148] In some embodiments, orienting one or more focal plane images may include shifting the respective focal plane images. Such shifting may be performed to address motion parallax, such as, for example, as shown in FIG.

[0149] Forming stereo pairs from focal planes 27A-27D are schematic plan views illustrating exemplary focal planes viewed under different viewer orientations, according to some embodiments. In addition to forming a stack of MFPs, a stereoscopic image pair (stereogram) is formed from the focal planes. A monocular stack of MFPs (FIG. 27A) is placed at a desired distance from the viewer's (mean) eyepoint 2702 and viewed by two eyes 2722 (FIG. 27B).

[0150] FIG. 27A shows a plan view illustration 2700 of a monocular MFP stack formed by depth blending. FIG. 27B shows a plan view illustration 2720 for viewing the MFP stack with two eyes 2722. FIG. 27C shows a plan view illustration 2740 for viewing a focal plane with the left eye 2742, so that the focal plane appears to shift to the right. FIG. 27D shows a plan view illustration 2760 for viewing a focal plane with the right eye 2762, so that the focal plane appears to shift to the left. Correspondingly, compositing two (stereo) MFP stacks from a monocular MFP stack for stereoscopic (or stereoscopic MFP) viewing can be done by shifting the MFPs in a certain direction and by a certain amount. The shifting of the MFPs may be an approximation of a more complex transformation to the virtual viewpoint change (see 3D warping in DIBR systems).

[0151] In Figures 27C and 27D, the left and right eyes view the stack from slightly different perspectives. The apparent lateral shift of the focal planes in Figures 27C and 27D depends linearly on the distance of the focal planes. If the farthest plane is seen at the same position by both eyes (placed farther from the eye), the middle plane is shifted laterally by "1 unit" and the nearest plane is shifted laterally by "2 units".

[0152] In some embodiments, the laterally shifted focal planes are collapsed by projecting them onto each eye point. When projecting, corresponding pixels on the focal planes may be summed along the depth axis. This may generally result in a stereo image pair of two images with different stereo disparities or parallax. When starting with a monocular MFP stack, the disparities may be interpreted as somewhat artificial or synthetic.

[0153] An example of a stereogram formed using MFP FIG. 28A is an illustration showing an example test image. FIG. 28B-D are illustrations showing example weighted focal plane images using the test image of FIG. 28A, according to some embodiments. FIG. 28A is from an image found on a webpage (Non-Patent Document 12), and FIG. 28B-D are modified versions of that image. Some embodiments may be illustrated using an example set of three focal planes formed using a polynomial blending function (e.g., FIG. 21A). FIG. 28B-D show an example test image 2800 (FIG. 28A) decomposed into three MFPs 2810, 2820, 2830 using a polynomial blending function. Crossed-eye stereograms are used to illustrate the quality of the MFP stacks and stereo pairs. 29A and 29B show stereoscopic image pairs 2900, 2950 (for the right and left eyes, respectively) formed using the MFPs 2810, 2820, 2830 shown in FIGS. 28B-28D.

[0154] 29A and 29B are illustrations showing corresponding exemplary right-eye and left-eye images formed using the focal plane images of FIGS. 28B-D, according to some embodiments. FIGS. 29A and 29B are examples of crossed-eye stereograms formed using three MFPs. The total mismatch corresponds to 2.76% of the image width (see MFPs shifted up to 6 pixels with the particular test image of FIG. 28A). The amount of shift in pixels depends on the resolution of the test image used. Unlike many MFP displays where the rendering accuracy is limited by the number of focal planes, in some embodiments, the accuracy and number of MFPs are similarly not limited. Forming, shifting, and summing MFPs for stereoscopic images may be performed with a greater number of focal planes than just the three focal plane example used to form FIGS. 29A and 29B.

[0155] Applying focal planes when creating stereoscopic content Similar to forming stereoscopic content, different accommodation distances may be chosen for the Zero Parallax Setting (ZPS). In the real world, the ZPS plane is at infinity (far away from the viewer's viewpoint). In such a setup, the induced disparity is in front of the farthest plane (commonly interpreted as a negative disparity, although the sign convention is sometimes reversed). In many cases, for various reasons, it may be more beneficial to use a distance closer to the ZPS. Closer distances are used for many screens for stereoscopic displays.

[0156] 30A-30C are schematic plan views illustrating exemplary focal planes viewed under different eye separation distances, according to some embodiments. FIGS. 30A-30C show three plan view schematics 3000, 3030, 3060 illustrating that adjusting the length of the stereo baseline (e.g., eye separation or interpupillary distance (IPD)) 3002, 3032, 3062 can reduce edge effects of overlapping / underlapping focal planes. In FIGS. 30A-30C, the total mismatch (e.g., amount of stereo depth and location of the MFP) and rendering distance remain the same. In FIGS. 30A-30C, this applies to three different viewers with correspondingly reduced interpupillary distance (IPD) / interocular distance (IOD). In fact, referring to Figures 30A-30C, individual variations in the IOD can affect the perceived depth as can selecting a ZPS-location to the rear (e.g., Figure 30A), middle (e.g., Figure 30B), or front (e.g., Figure 30C).

[0157] The interocular adjustment illustrated in Figures 30A-30C corresponds to choosing a convergence point / focal plane (e.g., also called a zero parallax setting or ZPS) for the content. The focal plane at the ZPS divides the total mismatch range (e.g., depth budget) into negative (near) and positive (far) mismatch. In Figure 30A, all content appears to be in front of the screen (due to all negative mismatches 3004, 3006). In Figure 30B, content appears to be on either side of the screen (due to both negative mismatches 3034 and positive mismatches 3064). In Figure 30C, content appears to be behind the scene (by having only positive mismatches 3064, 3066). Note that the ZPS does not affect the perceived depth range or the mismatch range itself.

[0158] In some embodiments to minimize effects caused by only partial overlay of the MFP on the focal plane edges, it is beneficial for the focal planes to be aligned with the centermost plane (FIG. 30B). In the case of depth blending, the focal planes and their stereoscopic projections create a continuous depth perception for the image area and for objects between the focal planes.

[0159] The number of MFPs is a parameter that can affect the perceived quality. For synthetic stereoscopic vision, the number of focal planes is not limited by display properties (such as transparency and multiplexing speed). In some embodiments, the number of focal planes can be chosen high enough to meet any desired depth accuracy, for example. The computational complexity increases linearly with the number of focal planes, but may not be an issue with a relatively large number of MFPs.

[0160] For the same reasons, there may be less benefit in optimizing the focal plane positions. Since shifting the focal planes for the left and right eyes is easier with a regular focal plane distance (e.g., can be performed without sub-pixel operation), equal displacement distances (e.g., intervals) may be used. Instead of optimizing the position by refractive interval, the number of MFPs with regular intervals may be increased. An odd number of focal planes may be used, with the most central plane at the alignment / pivot position.

[0161] The depth map of the captured scene may be relative and normalized. The depth range may be relative, not locked to real-world coordinates, and may be system variable. In some embodiments, the stereoscopic depth of the scene is determined by the amount of lateral shift used to form the (pseudo-)stereoscopic image pair.

[0162] A larger number of MFPs may correlate to a larger synthetic disparity and stereoscopic depth that may be produced. This process also depends on the image content. In a scene with only a few objects with large depth separation, depth blending over a large depth range may be beneficial instead of blending over a more compact range.

[0163] Formation of synthetic stereopairs A stereoscopic image pair is formed from the MFP by summing the projections from the focal plane towards each eye. The process can be viewed with stereoscopic glasses, resulting in a stereoscopic pair that can induce VAC, or a MFP version of stereoscopic glasses that can avoid VAC. The maximum disparity or parallax in the stereogram thus formed is the same that is observed for stereoscopic content. This recommended maximum parallax in Non-Patent Document 5, also called depth range or (stereo) budget, is content-dependent, but is around 3% of the image width. After this limit, as explained in Non-Patent Document 5, the stereoscopic effect tends to break, meaning that the (subjective) quality of the stereoscopic content is degraded.

[0164] For example, in Fig. 13, the maximum disparity / mismatch is about 2.5%, below the recommended maximum of 3% from Non-Patent Document 5. Fig. 13 shows that the corresponding maximum field of view for stereoscopic viewing is relatively small, about 20 degrees horizontal (compared to about 70 degrees for eye rotation without head movement). In the vertical dimension, the maximum field of view is up to 55 degrees without head rotation. In the vertical direction, the stereoscopic (or any other) display is smaller than that permitted by the eyes.

[0165] Depending on the source, different recommendations are given for the allocation of the depth budget, which is divided into depth in front of and behind the stereoscopic display screen (negative and positive parallax). In addition to the content itself, this allocation can be affected by the screen size, which varies depending on the source.

[0166] According to non-patent document 14, crossed discrepancies (for objects closer than the screen) should not exceed 2-3% of the screen width, and non-crossed discrepancies (for objects farther away) should not exceed 1-2% of the screen width.

[0167] In some embodiments, limits or recommendations, such as those identified by, for example, non-patent literature 5 and non-patent literature 14, may be applied as design considerations and compensation may be provided.

[0168] Examples with large discrepancies In the following, further examples with sinusoidal and polynomial blending functions are given. In order to make the printing distortion easier to see, a large total mismatch of 4.6% is chosen for these examples. This is much larger than the recommended maximum of 3% in Non-Patent Document 5. The effect is illustrated in Figures 31A-31F.

[0169] FIG. 31A is an illustration showing a magnified excerpt of an exemplary stereoscopic image for the right eye formed with sinusoidal depth weighting, according to some embodiments. FIGs. 31B-31C are illustrations showing corresponding exemplary stereoscopic images for the right and left eyes formed with sinusoidal depth weighting, according to some embodiments. FIG. 31D is an illustration showing a magnified excerpt of an exemplary stereoscopic image for the right eye formed with polynomial depth weighting, according to some embodiments. FIGs. 31E-31F are illustrations showing corresponding exemplary right and left eye images formed with polynomial depth weighting, according to some embodiments. FIGs. 31B-31C and 31E-31F are modified versions of the unprocessed flower test image, e.g., of FIG. 21B, by processing the test image using its depth map, FIG. 21C, according to some embodiments. 31B-C and 31E-F show a comparison of crossed-eye stereograms with large synthetic discrepancies (±2.3%) produced by shifting and summing three MFPs and formed using either sinusoidal and polynomial blending functions (see corresponding upper stereograms 3110, 3120 in Figs. 31B and 31C and lower stereograms 3140, 3150 in Figs. 31E and 31F). Left insets 3100 and 3130 (Figs. 31A and 31D) show details of corresponding right-eye images 3110, 3140 using both methods. The insets show the benefit of the polynomial function on object edges, which reduces disocclusions caused by large discrepancies. The benefit may come from stronger depth blending with the polynomial function over the sinusoidal function.

[0170] In the case of the polynomial function, the background image details are slightly blurred. This is caused by a stronger spatial filtering influence that filters the result when the shifted focal plane components are summed to form the two final viewpoints. In the case of large discrepancies, the polynomial blending function has a benefit on object edges. The benefit comes from a stronger blending inpainting influence in the depth direction that reduces the visible disocclusion caused by large discrepancies.

[0171] Free baseline volume support Images, videos, and virtual content may be viewed using stereoscopic VR glasses. These services may only be able to provide stereoscopic disparities in the horizontal dimension (when viewed with head upright). In order not to disturb the viewer's perception of his or her pose and balance, the horizon of the viewed content is generally locked to the viewer's real-life horizon. This is done by sensing the user's head tilt with a position sensor in the viewing device (e.g., a cell phone in "cardboard" VR glasses).

[0172] In previous systems, despite orienting the content with the user's head tilt, the content disparity generally does not follow the user's head tilt, but is horizontal regardless of head orientation. Increasing head tilt distorts, weakens, or even loses the horizontal disparity-based 3D perception. The systems and methods disclosed herein according to some embodiments may be used to provide stereoscopic content independent of head orientation. Stereoscopic (pseudo) disparity may be formed in real-time at the receiving end according to the measured head tilt of the viewer. In some embodiments, if the viewer orientation signal indicates a viewer tilt, a disparity image may be calculated using a rotated disparity vector corresponding to the indicated viewer tilt.

[0173] 32A-32C are schematic front views illustrating an example mismatch for a first stereoscopic display with different user orientations, according to some embodiments. FIGs. 32D-32F are schematic front views illustrating an example mismatch for a second stereoscopic display with different user orientations, according to some embodiments. FIGs. 32A-32F illustrate some of the differences between previous systems and the system disclosed herein according to some embodiments. In FIGs. 32A-32F, stereoscopic image pairs 3200, 3210, 3220, 3230, 3240, 3250 are depicted as they are illustrated with VR glasses using a mobile phone as a display. Optimizing the image geometry for viewing on a flat display close to the eyes results in stereoscopic images with a characteristic pincushion shape.

[0174] 32A-32F show two methods for showing stereoscopic content on a near-eye display (e.g., a mobile phone). 32A-32C show the content tilted along the viewer's head pose, which leads to a discrepancy between the real-world horizontal tilt line and what is shown. This discrepancy can cause simulator sickness and disorientation. 32D-32F show a method, in some embodiments, for aligning the content's horizon with the real-world horizon by tracking the user's head pose. A disadvantage of this system is that the baselines of the stereo pairs are misaligned, leading to an erroneous or lost perception of depth.

[0175] 33A-33C are schematic front views illustrating an example mismatch for a stereoscopic display with different user orientations, according to some embodiments. In some embodiments, such as illustrated by Figs. 33A-33C, stereo pairs 3300, 3330, 3360 with appropriate baseline and mismatch may be formed in real-time following changes in the user's head pose. Supported interactions may also include motion parallax. Note that head tilt and movement are captured in the virtual viewpoint generation (see Figs. 25 and 26) to modify the mismatch accordingly.

[0176] FIG. 34 is a schematic user's diagram illustrating exemplary left and right eye images for a stereoscopic display for a viewer with a rotated orientation, according to some embodiments. FIG. 34 is a front view of the left and right images displayed for a near-eye display viewer. The head icon 3414 on the right side of FIG. 34 indicates the rotation of the viewer's orientation. The glasses on the left side of FIG. 34 are shown from the viewer's perspective as worn by the viewer. After the viewer's head rotation, the right eye image 3404 is seen higher, from above, and the left eye image 3402 is seen lower. The same image shown in the original orientation of FIG. 5 is rotated for the scenario 3400 of FIG. 34 to keep the image horizontal. The mismatch between the left and right images is altered by the rotated orientation (the location of the small boats shown in FIG. 5 is altered such that FIG. 34 shows a change in both vertical and horizontal mismatch such that small boat 3408 is closer to large boat 3406 in the left eye image 3402 compared to FIG. 5, and small boat 3412 is farther away from large boat 3410 in the right eye image 3404 compared to FIG. 5).

[0177] In some embodiments, orienting one or more focal plane images may include rotating the respective focal plane images. For example, one or more focal plane images may be rotated to show images relative to left eye image 3402 and right eye image 3404 shown in FIG. 34. In some embodiments, rotating the focal plane images may include rotating a mismatch vector for one or more of the focal plane images relative to a vector parallel to the horizon.

[0178] Motion parallax support Focal plane shifts based on synthetic mismatch at any baseline orientation (e.g., head tilt) may also be used to support small amounts of motion parallax for the left and right eyes. This embodiment uses a large number of focal planes, and the span of mismatch may be greatly extended so that a variety of head positions (motion parallax) within that range can be supported.

[0179] FIG. 35 is a schematic processing diagram illustrating an example process of creating multiple perspectives to a view by warping a set of focal plane images left and right, according to some embodiments. FIG. 35 shows various modified versions of an unprocessed flower test image, e.g., of FIG. 21B, using its depth map, e.g., of FIG. 21C, according to some embodiments. In the example of FIG. 35, five (e.g., sinusoidal) MFPs and a maximum disparity / lateral shift (for either eye) of 12 units are used. Seven frames 3502, 3504, 3506, 3508, 3510, 3512, 3514 multi-view images are formed 3516 for a chosen disparity / disparity range. Each of the frames 3502-3514 is formed by first warping the MFP stack to varying degrees, then projecting and summing the MFPs towards the viewer. The original zero disparity images are also put into a series. The multi-view image includes a series of crossed-eye stereograms, where the right eye viewpoint is on the left and vice versa. By picking up every other frame in the multi-view series of FIG. 35, five stereogram pairs 3502 / 3506, 3504 / 3508, 3506 / 3510, 3508 / 3512, 3510 / 3514 may be formed, which differ by 8 units of disparity / lateral shift. Correspondingly, linearly varying stereo disparity (with 8 units of stereo discrepancy) may be supported at five positions. FIG. 35 shows an example 3500 for forming a multi-view image using synthetic discrepancy. The maximum discrepancy range for each eye (e.g., maximum lateral shift range of five MFP stacks) is 12 units (±2.75% of the test image width). For example, the MFP stack may be skewed (or, e.g., MFP shifted) to the right 3518 and the MFP stack may be skewed (or, e.g., MFP shifted) to the left 3520. The stack of MFPs is skewed (or, e.g., MFP shifted) by various amounts. The result is a stack of MFPs that is tilted to the left or right, depending on the shift direction.

[0180] Splitting the multi-view images into stereo pairs is more convenient for perceiving the parallax effects from printing. A compromise between the motion parallax effects and the perceived stereo mismatch (depth) may be made by using either 1) two adjacent images at six positions (total mismatch 0.46%, both positive and negative mismatches 0.23%), 2) every other image at five positions (total mismatch 0.92%), 3) every other image at four positions (total mismatch 1.38%), 4) every third image at three positions (total mismatch 1.84%), or every fourth image at two positions (total mismatch 2.3%). A stereo pair with two end images provides a total maximum mismatch of 2.76% without motion parallax.

[0181] FIG. 36 is an illustration showing an example series of three stereoscopic images for stereoscopic display using the distorted images of FIG. 35 (each of them is an MFP stack that is first distorted to various degrees and then projected and summed towards the viewer) according to some embodiments. FIG. 36 shows three samples 3602 / 3604, 3606 / 3608, 3610 / 3612 (crossed eye stereo pairs with 0.92% discrepancy) from the above series formed by picking every other image. The motion parallax effect from the print (3.7% in total) is somewhat difficult to perceive, but it is evident when animation is formed using the multiview images of FIG. 35. FIG. 36 illustrates an example 3600 of three stereoscopic views 3602 / 3604, 3606 / 3608, 3610 / 3612 with motion parallax (the viewpoint moves from left to right) formed using the multiview images shown in FIG. 35. Figure 36 shows three (cross-eye) stereograms 3602 / 3604, 3606 / 3608, 3610 / 3612 with a left to right viewpoint change (see Figure 35). The images connected by the alignment are the same.

[0182] The amount of motion parallax in the example of FIG. 36 can be estimated. The two viewpoints for the stereoscopic content are separated by an eye separation distance, typically about 60 mm. Correspondingly, the virtual view illustrated in FIG. 36 allows the user to move his head 12 cm between the two poles. However, considering the result as a small crossed-eye stereogram is likely to distort the effects compared to observing them in more realistic conditions. The head tilt and motion parallax support described may be provided individually to multiple local viewers, since the formation and processing of the MFP may be performed multiple times within the receiver for content that is transmitted only once (broadcast).

[0183] In the case of synthetic motion parallax, the stereo pair is formed not to have symmetrically left and right shifted versions, but e.g. to have two left shifted versions with different amounts of shift. However, this is not expected to cause a noticeable degradation in the perception of the motion parallax effect. Another relevant detail is that the original unshifted texture image (without effects on both sides of the object edge) may also be used as part of the multi-view image series.

[0184] Choosing the focal point or ZPS (see Figures 30A-30C and their discussion) does not affect the depth range perceived from the stereoscopic image (see total disparity), but the consequences of the choice are small when creating synthetic motion parallax. When the focal point is in the center (e.g., as is typically chosen for the examples in this disclosure), a viewer moving laterally perceives the rendered volume as rotating around a pivot point (at the ZPS distance). However, it is more natural for motion parallax to rotate the volume around its farthest point. In some embodiments, when creating motion parallax effects, the far plane (and all disparities in front of this far plane) may be used in virtual viewpoint generation.

[0185] Applicable source formats for forming MFP Video plus depth map is a format applicable to the disclosed method according to some embodiments, which gives the distance of each pixel in the image. Analyzing a stereoscopic view from such a format using 3D warping or MFP-based design may cause some errors at object edges when changing the viewpoint reveals occluded areas of the view. One exemplary embodiment is a stereoscopic video with depth map. Both of the above mentioned formats are supported by video compression methods, e.g., H.264 MVC.

[0186] There are multiple processes to generate a depth image. One method according to some embodiments is to derive the disparity of the two camera views (stereo pair) and map the disparity to depth. Finding the correspondence between image pixels is done in parallel with the line connecting the two camera points (baseline). Finding the true correspondence is an error-prone process, with potential false positive correspondences and disparities. Various types of filtering and segmentation operations may be used to remove the errors.

[0187] Some embodiments use a source format of depth plus texture captured by a depth sensor (such as RGB-D or ToF (time of flight)). The depth information may be compressed before transmission. This uses a smaller bit rate than real-time transmission of a virtual (3D modeled) scene to a stereoscopic viewpoint or a corresponding depth map.

[0188] Exemplary embodiments using an external display FIG. 37 is a system diagram illustrating an example set of processing blocks and interfaces for generating shifted and projected MFP images for flexible baseline stereoscopic display for three viewer orientations according to some embodiments. In some embodiments, the flexible baseline stereoscopic display may be, for example, a flexible view TV. Although the context for describing some embodiments has been primarily near-eye displays, the same interaction may also be implemented using, for example, an external display according to some embodiments. For example, when used for time-shared S3D displays, shutter glasses worn by the user may be equipped with sensors to track user motion and to receive 3718 data from the glasses and to send 3720 data to the glasses. External tracking may be used in some embodiments. By tracking user poses and movements for an example of three users 3722, 3724, 3726, the systems and methods disclosed herein according to some embodiments (including virtual discordance related features) may use an external display. Multi-user views (which may be implemented with time-shared 3716) may be used to serve multiple co-located viewers. Figure 37 illustrates some embodiments using an external display and two example methods 3712, 3714 for virtual viewpoint generation 3710 within a receiver 3708 at a receiving site 3706 for input of a depth map 3702 and texture 3704. Although the techniques are shown to be alternatives ("or") in Figure 37, in some embodiments any combination of these may be used.

[0189] Exemplary embodiments using a DIBR-based approach for viewpoint generation Some embodiments for virtual viewpoint generation by MFP may be used with DIBR systems based on 3D warping and hole filling. Some embodiments may use DIBR-based methods to generate free-baseline stereo and motion parallax for freely posed or moving users. This embodiment may calculate 3D geometric deformations for multiple users and viewpoint changes. This embodiment is illustrated in FIG. 37 (see the two boxes inside "Virtual Viewpoint Generation" corresponding to methods 3712 and 3714).

[0190] Exemplary embodiments of free-baseline stereo and motion parallax in MFP NED The systems and methods disclosed herein according to some embodiments create MFPs and use them to create selected (various) stereo mismatches at desired orientations (head tilts). Some embodiments shift the MFPs to match the baseline orientation. Some embodiments create free-baseline stereo and motion parallax effects on the actual MFP display. When illustrated on the actual MFP display, natural accommodation is supported. The MFP stack may be viewed with head tilts such as those illustrated for the stereo display in FIG. 34 such that the stereoscopic perception supported by the accommodation is in fact invariant to the viewer's head tilt (rotation "around the nose").

[0191] Exemplary embodiments for asymmetric viewpoint generation An alternative embodiment is to form and plot one eye information with a shifted MFP and plot the other eye MFP without shift (using a focal plane for the texture image received directly for one eye). According to non-patent document 15, plotting one high quality image and one low quality image (here, the unoccluded area) simultaneously can provide similar quality benefits as monovision methods. However, this variant allows for smaller discrepancies due to asymmetric viewpoint generation and distortion.

[0192] Generation of MFP and virtual viewpoint Some embodiments for forming the MFP and virtual viewpoints include calculating depth blending weights for each focal plane, forming an MFP weight map by remapping the image depth map by the calculated depth blending weights for each focal plane, multiplying the captured image or input image by each MFP weight map to form each focal plane image, performing a spatial shift of each focal plane image, and adding the formed MFP to generate a stereoscopic image pair for mismatch. In some embodiments, depth blending weights may be calculated and image processing may be used to form the MFP weight map (illustrated in FIG. 38B), form each focal plane image (illustrated in FIG. 39A-C for one focal plane image), and perform the spatial shift and add (e.g., for mismatch where a stereoscopic image is chosen).

[0193] 38A-C are a set of computer windows illustrating an example set of interfaces for an image manipulation program illustrating a depth map image, a depth weighting graph, and a generated focal plane image, respectively, according to some embodiments. Figures 38A-C show an example in which a depth map 3800 (Fig. 38A) is remapped by a sinusoidal weighting function 3830 (Fig. 38B) to create a weight map for one focal plane 3860 (Fig. 38C). The other weight maps (in the example, five in total) may be formed in a similar manner.

[0194] 39A-39C are illustrations showing an exemplary set of generated focal plane images within a simulated environment. Figures 39A-39C show a test image 3900 (Figure 39A) multiplied by a weight map 3930 (middle graph of Figure 38) to form a corresponding focal plane 3960 (Figure 39C). The other focal planes (in the example, five in total) are generated in the same manner.

[0195] On the computational complexity of generating virtual viewpoints 3D warping for viewpoint changes may be performed by computer graphics algorithms. The complexity, speed, and results of these algorithms vary depending, for example, on the perspective and camera model used. An example of a mathematical formulation for a shift sensor setup can be found in Non-Patent Document 5. The computational complexity for 3D warping can be high, such as with high-quality near-eye displays with large FoV (field of view) and larger resolution than in many external displays. In addition, when using 3D warping to form a viewpoint for an MFP display, the focal plane is formed after 3D warping, which can further increase the complexity for many previous systems.

[0196] On the other hand, forming a stack of MFPs is a procedure that uses pixel-based operations to weight each pixel value by its distance to each focal plane (e.g., obtained from a depth map). After the decomposition is performed, any change in viewpoint or stereo baseline orientation is simply a lateral (e.g., bulk) shift of the focal planes once derived relative to each other, and their pixel-by-pixel addition to form the projection to the new viewpoint. In some embodiments, there is no sub-pixel shift and operation, since focal plane shifts may be performed in the multiplication of pixel displacements without resolution change due to interpolation, etc.

[0197] Instead, using traditional 3D warping procedures, changing the viewpoint typically requires multiple 3D geometry-based arithmetic operations (e.g., matrix calculations) at sub-pixel accuracy. Furthermore, inpainting and hole-filling operations to remove disocclusions may use operations with several pixel-wide kernels or windows instead of simple pixel-based operations.

[0198] FIG. 40 is a message sequence diagram illustrating an example process for generating a multi-focal plane (MFP) display image, according to some embodiments. FIG. 40 is a message sequence diagram 4000 for some embodiments that generate a stereoscopic MFP stack (e.g., two monocular stacks of MFPs for the left and right eyes, correspondingly) by shifting and projecting focal plane images. The MFP display 4006 sends 4008 the MFP plane position to the renderer control module 4002. The texture image and depth map are received 4010 by the renderer control module 4002. The renderer control module 4002 generates 4012 an image for each MFP plane. The tracking module 4004 (or sensor) sends 4014 the viewer position / orientation to the renderer control module 4002. The MFP images are shifted and projected 4016 by the renderer control module 4002. The stereoscopic MFP stack (left / right pair) is sent 4018 from the renderer control module 4002 to the MFP display 4006, where it is displayed to the user.

[0199] FIG. 41 is a message sequence diagram illustrating an example process for generating a stereoscopic 3D display image, according to some embodiments. FIG. 41 is a message sequence diagram 4100 for some embodiments of generating inconsistent views of focal plane images for a stereoscopic 3D display (e.g., an external flat screen). Texture images and depth maps are received 4108 by the renderer control module 4102. The renderer control module 4102 generates 4110 images for each MFP plane. In some embodiments, the renderer control 4102 may be used to calculate 4114 the inconsistent views. The glasses / HMD sensor 4104 sends 4112 the viewer position / orientation to the renderer control module 4102. The inconsistent views are calculated by the renderer control module 4102. A stereoscopic image left / right pair is sent 4116 from the rendering control module 4102 to the stereoscopic 3D display 4106. The stereoscopic 3D display 4106 sends an active glasses signal 4118 from the stereoscopic display 4106 to the glasses / HMD 4104 so that the image is displayed to the user.

[0200] In some embodiments, a method may include receiving three-dimensional video content, calculating one or more multi-focal plane (MFP) images for a number and position of multi-focal planes using the three-dimensional video content, receiving a viewer orientation signal, calculating one or more mismatch images for a common view of a stereoscopic display using the one or more MFP images and the viewer orientation signal, and rendering the one or more mismatch images to the stereoscopic display, where the common view of the stereoscopic display is rendered to each eye view of the stereoscopic display. In some embodiments, the common view may be a left eye perspective and a right eye perspective relative to an average eye perspective of the monocular MFP stack. In the case of an MFP NED, the optical arrangement may use a periscope type eyepiece that allows the monocular MFP stack to be viewed between the eyes in an uncrossed eye state.

[0201] In some embodiments, the method for calculating one or more mismatched images for a common view may calculate an average eye point that is the average of each eye view of a stereoscopic display (or the average of a left / right pair of images).

[0202] The systems and methods disclosed herein according to some embodiments create MFPs and use them to generate selected (various) stereoscopic mismatches at desired orientations (head tilts). The MFPs may be shifted in a direction that matches the baseline orientation. The resulting stereoscopic images may be displayed on wearable displays as well as, for example, external screens. Some embodiments may be used to generate stereoscopic content for services that use stereoscopic displays. Some embodiments enable the development of stereoscopic displays, enabling a simple and cost-effective means of generating head-tilt independent (free baseline) stereoscopic rendering with undistorted parallax and sensor-supported motion parallax effects. Such a system may support stereoscopic vision and motion parallax for users moving their heads and bodies.

[0203] Some embodiments produce stereoscopic images that may cause VAC when viewed on previous stereoscopic displays. VAC can be reduced by illustrating the content with accommodation-invariant stereoscopic glasses. In alternative embodiments, disparity-based interaction features with the MFP display are supported.

[0204] In some embodiments, the method may include calculating image data for one or more focal planes of the MFP display using the three-dimensional video content and the viewer orientation signal, and rendering one or more MFP images on the MFP display using the calculated image data.

[0205] In some embodiments, a device may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform a process for calculating image data for one or more focal planes of an MFP display using three-dimensional video content and a viewer orientation signal, and for rendering one or more MFP images on the MFP display using the calculated image data.

[0206] Virtual Viewpoint Generation from Depth and Texture Systems and methods disclosed herein according to some embodiments relate to forming stereoscopic image pairs (stereograms) in stereoscopic 3D (S3D) display systems. One method for forming stereograms is depth image-based rendering (DIBR). Systems and methods disclosed herein according to some embodiments generate synthetic stereoscopic views from multiple focal planes (MFPs). One exemplary method according to some embodiments uses virtual viewpoint generation with 3D warping used in DIBR systems. In both DIBR and MFP stereo methods, a depth plus texture input format may be used in capturing the view. Some embodiments use post-processing of multiple focal planes (MFPs) for virtual viewpoint generation and for MFP rendering.

[0207] Depth blending used to improve depth perception in MFP-based methods is described in

[10] and

[11] , and a depth blending method based on redistribution of focal (representation) plane information in the frequency domain is described in [1]. Natural blur and occlusion are important factors for quality in stereoscopic and 3D image rendering. For information on their perception by the human visual system (HVS), see, for example,

[16] .

[0208] Many stereoscopic 3D systems use depth image-based rendering (DIBR). A schematic picture of a DIBR-based 3D transmission system is shown in Fig. 8, adapted from Non-Patent Document 3. In many DIBR systems, at the receiving side, virtual viewpoint generation consists of a 3D warping stage and a hole-filling stage (see the virtual viewpoint generation block in Fig. 8). 3D warping may be used to form two virtual views into a 3D view (textured depth map) seen from the two eyepoints of the viewer. The 3D warping is performed by a computer graphics algorithm.

[0209] Stereoscopic 3D synthesized from stacks of MFPs Multi-focal plane (MFP) rendering is used to support natural eye accommodation in near-eye displays. As explained above, a stereoscopic view may be synthesized from multiple focal planes (see 3D warping in DIBR). MFP displays create a stack of different focal planes that make up a 3D scene from layers along the viewer's visual axis. A view into the 3D scene is formed by projecting all those volume pixels (e.g., more precisely, voxels) that are visible to the user at different depths and spatial angles. Multi-focal planes can be rendered, for example, either by spatially multiplexing a stack of 2D views, or by sequentially switching the focal length of a single 2D view by a fast variable focus element (VFE) - in a time-multiplexed manner - while rendering the visible parts of the corresponding multi-focal planes.

[0210] FIG. 12 shows a schematic diagram for a multi-focal plane near-eye display. The MFP display creates an approximation for the bright field of the displayed scene. Many near-eye displays support only one user viewpoint. Correspondingly, the approximation may be simpler and not use a full bright field capture as input. When rendering a large number of MFPs at different distances (in 3D space), something close to natural accommodation by the viewer is supported (VAC is avoided). By using depth blending when forming the MFP, the rendering quality may be further improved and / or the number of MFPs may be reduced.

[0211] Depth blending means weighting pixels according to their distance from, typically, the two nearest focal planes. The distance may be obtained, for example, from a depth map. Depth blending may improve the depth perception between focal planes. The focal planes are summed to (or approximately to) the original captured scene so as to fit into a division of one. There are many ways to decompose the scene into additional planes so that they meet this requirement. Correspondingly, the systems and methods disclosed herein are presented as illustrative examples according to some embodiments.

[0212] Many MFP rendering systems form and render different MFP stacks for each eye, supporting stereoscopic perception. Non-Patent Document 7 describes the depth-fused 3D (DFD) phenomenon between two spatially formed focal planes. Non-Patent Document 10 extends this phenomenon (described as depth blending) to a setup with multiple (three or more) focal planes. However, Non-Patent Document 10 is understood to describe viewing a separate MFP stack by each eye, formed from stereoscopic capture for depth and texture (not by shifting the MFP to synthesize discrepancies).

[0213] In some exemplary systems and methods disclosed herein according to some embodiments, both eyes are viewing the same monocular stack of focal planes from slightly different viewpoints, creating a somewhat synthetic mismatch for 3D perception. Thus, in some embodiments, instead of two sets of MFPs captured and formed separately for each eye, only one stack of MFPs is formed and aligned towards a point between the eyes, as shown in FIG. 18. FIG. 18 shows a stack of MFPs formed from an average eye point, where the same focal plane is seen by the left and right eyes from slightly different viewpoints. In some embodiments, one stack of MFPs formed and aligned towards a point between the eyes is rendered to each eye view of a stereoscopic MFP display by forming two versions of one monocular MFP stack from slightly different viewpoints, as shown in FIG. 18. In some embodiments, the projections of the two versions of the MFP stack are summed and displayed to each eye using a conventional stereoscopic display (S3D).

[0214] 3D perception from focal planes is adequate for flat objects strictly on those planes. However, with depth blending, the perception is close to adequate between planes. Depth blending is developed by examining the accommodation to stereoscopic MFPs (the two sets of them), since MFP stacks can be formed separately for each eye. However, depth blending can provide a more continuous depth perception through (composite) stereoscopic disparity (when viewing one monocular depth-blended MFP stack with two eyes (see FIG. 18)). This phenomenon can be seen by comparing cross-eye stereograms with depth blending, corresponding to FIG. 29A, FIG. 29B, FIG. 46A, and FIG. 46B, with those without it.

[0215] Synthesizing stereoscopic images from MFPs is described below. In addition to forming a stack of MFPs, a stereoscopic image pair (stereogram) is formed from the focal plane. For example, a monocular stack of MFPs (see FIG. 27A) is formed, placed at a set (or desired or fixed) distance from the viewer's (average) eyepoint, and viewed by two eyes (see FIG. 27B).

[0216] FIG. 27A shows a monocular MFP stack formed by depth blending. FIG. 27B shows viewing the viewing stack with two eyes. FIG. 27C shows the focal plane for the left eye, which makes it appear to have shifted to the right. FIG. 27D shows the focal plane for the right eye, which makes it appear to have shifted to the left. The left and right eyes view the stack from slightly different perspectives. Correspondingly, compositing two (stereoscopic) MFP stacks from a monocular MFP stack for stereoscopic (or stereoscopic MFP) viewing can be done by shifting the MFPs in a certain direction and by a certain amount. The shifting of the MFPs is actually an approximation of a more complex transformation for virtual viewpoint changes (see 3D warping in conventional DIBR systems). The lateral position of each focal plane perceived by the eye depends linearly on the distance of the focal plane (see FIG. 27B). When using the fixed focal plane interval described in Figure 27B, if the farthest surface is viewed in the same position by both eyes (e.g., placed far away from the eyes), the intermediate surface will appear to be shifted laterally by "1 unit" and the nearest surface will appear to be shifted laterally by "2 units." If the focal plane separations are fixed on a refractive scale, their actual metric distances are used to derive the various amounts of lateral shift.

[0217] When forming stereoscopic image pairs corresponding to eye positions, the laterally shifted focal planes are eliminated by projecting them to each eye point. Projecting for stereoscopic image pairs actually means that corresponding pixels on the focal planes are summed along the depth axis. This results in a stereoscopic image pair of two images with different stereo disparity / parallax. Some embodiments may start with a monocular MFP stack (such as, for example, Figures 28B to 28D) and use synthetic disparity to generate a stereoscopic MFP stack.

[0218] Figures 29A and 29B provide examples of stereoscopic image pairs formed by the described procedure. In forming the three focal planes used in the examples of Figures 29A and 29B, depth blending with a linear (tent) filter was used. Figures 29A and 29B show examples of crossed-eye stereograms formed by shifting and summing three linearly blended MFPs. The total mismatch corresponds to 2.76% of the image width (see MFPs shifted up to 6 pixels with a specific test image). When the focal plane separations are fixed in refractive scale, their actual metric distances are used to derive the various amounts of lateral shift.

[0219] Depth blending improves depth perception in MFP rendering In some embodiments, 3D video content may be filtered into high frequency content and low frequency content. The high frequency content may be decomposed into one or more high frequency focal planes. The low frequency content may be redistributed to one or more high frequency focal planes to generate one or more redistributed focal planes. The one or more redistributed focal planes may be used to render one or more multi-focal plane images.

[0220] According to Non-Patent Document 17, for any human being, up to 28 focal panes are sufficient to cover the depth range from infinity to 4 diopters (25 cm), which corresponds to a 1 / 7 diopter interval for the focal planes. For a person with average vision, Non-Patent Document 17 states that 14 focal planes cover the range. Therefore, a large number of focal planes may be used for high quality depth perception.

[0221] Rendering a large number of focal planes is limited for various technical reasons. The number can be reduced by using depth blending. When using only a few focal planes, optimal placement may be beneficial for human eye properties. As the accuracy of depth perception decreases inversely proportional to the distance from the viewer, greater accuracy may be obtained by making the depth plane a function of refractive depth.

[0222] A mapping between linear and refractive scales can be determined. Many examples are obtained by using a fixed spacing between focal planes on a linear scale. Depth-based blending may be used to reduce quantization errors in the depth dimension that would otherwise be visible to the human eye. Depth blending means using a depth-based function to weight the pixels used to construct each focal plane.

[0223] FIG. 42A is a graph illustrating example blending functions 4200, 4202, 4204 of weights versus depth according to some embodiments. One depth blending function is a box filter (slicing without blending) shown in FIG. 42A. For comparison, a tent filter, which is a piecewise linear, sawtooth blending function, is shown in FIG. 43A. In some embodiments, the range of depth values ​​is between 0 and 255, which is used in some of the examples herein, such as in FIG. 42A and FIG. 42C. FIG. 42B is an illustration 4210 showing an example test image. FIG. 42C is an example depth map 4220 illustrating pixel distances for FIG. 42B. FIG. 42D-F are illustrations 4230, 4240, 4250 showing example focal plane images using the depth weights of FIG. 42A with the test image 4210 of FIG. 42B and its example depth map of FIG. 42C according to some embodiments. 42B and 42C are derived from an image found on a web page (Non-Patent Document 12), and 42D to 42F are modified versions of that image.

[0224] FIG. 43A is a graph illustrating an example blending function 4300, 4302, 4304 of weights versus depth according to some embodiments. FIG. 43B is an illustration 4310 showing an example test image. FIG. 43C is an example depth map 4320 illustrating pixel distance for FIG. 43B. FIG. 43D-F are illustrations 4330, 4340, 4350 showing example focal plane images using the depth weights of FIG. 43A with the test image 4310 of FIG. 43B and its example depth map of FIG. 43C according to some embodiments. FIG. 43B and FIG. 43C are from an image found on a webpage (Non-Patent Document 12), and FIG. 43D-F are modified versions of that image.

[0225] In some embodiments, the polynomial blending functions extend across the entire depth range. Figures 21A-F show an example of forming a focal plane using three polynomial blending functions (Figure 21A). The corresponding focal planes are shown in Figures 21D-F for a test image (Figure 21B) and its depth map in Figure 21C.

[0226] Figures 22A-F show an example of forming a focal plane using three sinusoidal blend weight functions (Figure 22A). The corresponding focal planes are shown in Figures 22D-F for a test image (Figure 22B) and its depth map in Figure 22C.

[0227] In addition to the blending filter mentioned above, another variant has been proposed by Non-Patent Document 11, which describes a method for forming a dioptrically placed MFP. Instead of continuously capturing depth and texture from the scene, this method captures a series of focal stacks, i.e., a consecutive set of multiple images with various focal lengths.

[0228] Using focal stacks, the MFP can be more accurately optimized with respect to the human visual system (HVS). Specifically, according to [1], the method applies to the effects of occlusion, reflections, and other non-Lambertian phenomena in the captured view. In addition to cameras with different focal lengths, the set of input images may be derived from light fields captured from the scene. In [1], a first generation Lytro Illum camera was experimented with for this purpose.

[0229] Fig. 44 is a schematic diagram illustrating an example focus stack image captured at different depths from a user. Fig. 44 shows an example 4400 of a focus stack (four images 4412, 4414, 4416, 4418 with different focal lengths) captured from a real-world view and processed to form focal (presentation) planes 4404, 4406, 4408, 4410 for a user's eye 4402 (adapted from non-patent document 1). Insets 4420, 4422, 4424, 4426, 4428, 4430, 4432, 4434 show details of the focus stack image 4400 with various defocus (adapted from non-patent document 1).

[0230] 45A-45D are illustrations showing exemplary linearly blended focal planes. 45E-45H are illustrations showing exemplary focal (presentation) planes processed from focal stacks. 45A-45D show a comparison of MFPs formed by linear depth blending, and 45E-45H show a comparison of MFPs formed by the optimized procedure in Non-Patent Document 1. 45A-45H compare focal (e.g., presentation) planes 4500, 4510, 4520, 4530 (FIGS. 45A-45D) formed using conventional linear blending with focal (e.g., presentation) planes 4540, 4550, 4560, 4570 (FIGS. 45E-45H) formed from focal stacks using optimized blending. Note that the brightness of the images in FIGS. 45E-45H has been increased for better visibility when printed. 45A-45H are line drawing versions of the images published in Non-Patent Document 1.

[0231] The technique in [1] does not use a depth-plus-texture format for scene capture. Instead, the system captures a set of images with different focal lengths, processes them in the frequency domain, and assigns information to different focal planes (or presentation planes as presented in [1]) according to a model of the human visual system (e.g., retinal blur).

[0232] As explained in Non-Patent Document 1, the brightness of the optimized focal planes (FIGS. 45E-45H) is likely to be increased by a factor of (approximately) 2, presumably for better visibility when printed. When these focal planes are used as test images later in this disclosure, inverse scaling by 0.5 is performed on them to prevent saturation of the results (violation of the division of 1).

[0233] In particular, there are also some horizontal brightness artifacts in the linearly blended focal planes shown in Figures 45A-45D. These artifacts are noticed when summing the focal planes to re-create the original test image (demonstrating the division of 1). It is understood that [1] does not give depth maps for the test images in their collection of papers or their supplementary materials, so we are unable to regenerate the MFP. Despite the observed artifacts, these focal planes are used to visualize the impact of some embodiments to enable a comparison of their performance with the approach in [1].

[0234] According to Non-Patent Document 1, high frequency (generated edges and textures) components benefit from being assigned to their nearest focal distance. Low frequency components may also be assigned to more distant focal planes. To quote Non-Patent Document 1, "High frequency image content must be allocated to its nearest plane, and low frequencies may be under-constrained and redistributed between planes. It is these under-constrained degrees of freedom that the algorithm exploits to meet constraints on display intensity." High frequency components dominate in perceiving depth from focality. Low frequency components, e.g., slowly changing luminance or color, generate only a few cues for depth perception. In some embodiments, focal planes generated by known methods (e.g., Figs. 45A-45D generated using linear depth blending) are processed by redistributing their low frequency content between the original focal plane positions, and the redistributed focal planes are shown to have benefits over the original unprocessed focal planes when using them to create stereoscopic or changing viewpoints (the latter due to motion parallax).

[0235] The apparent distance detected by a depth sensor does not necessarily apply to reflected or refracted (non-Lambertian) image content. When capturing image information as a function of focal length, as in the system of Non-Patent Document 1, the focal properties and distances match such content if the images are captured with a small enough separation in the depth dimension (small enough difference in focal length).

[0236] For many 3D display systems, new viewpoints benefit from being derived to support stereoscopic or multi-view rendering. For example, when using a depth-plus-texture capture format, a stereoscopic view may be derived by forming a virtual viewpoint symmetrically around the capture point such that both of the two views are composited.

[0237] Using depth plus texture as a capture format has benefits in stereoscopic 3D: the stereoscopic baseline is not fixed at the front end, but the amount and direction of discrepancy may be synthesized within the receiver, even for stored content.

[0238] In many cases, the corresponding viewpoint change will reveal parts of the view behind the occluding object, as explained in Non-Patent Document 3. As illustrated by the schematic picture in Figure 9 and the examples in Figures 10A and 10B, these disocclusions appear as gaps or holes in each new perspective image, since the captured texture or depth does not contain information from those areas.

[0239] FIG. 9 shows that disocclusion can occur when changing viewpoint, as illustrated when changing the camera viewpoint from a source position / image to a target position / image.

[0240] The mechanisms for creating disocclusions may be similar to DIBR and MFP-based virtual viewpoint generation. These disocclusions may be created in viewpoint generation or if the MFP viewer moves far enough away from the viewing point. Larger dislocations or viewpoint changes may be achieved by using measures that reduce the visibility of the disocclusions. For many previous systems, disocclusion still limits the maximum dislocation.

[0241] 10A and 10B provide an example of a 3D warped image. This example shows only the left eye image. The right eye image is formed by the same means and has a similar defect, but it is the right side of the object. 10A and 10B show an example of disocclusion caused by 3D warping. The original monocular image after 3D warping is on the left, and the details of the 3D warped image are on the right (when using linear interpolation-based hole filling).

[0242] In hole filling, various image and depth map filtering and inpainting algorithms are used to fill holes or other forms of disocclusions. The type and amount of occlusions are highly content-dependent, as are the distortions and their visibility. Disocclusions can degrade the perceived stereoscopic quality around object edges. They can limit the amount of synthetic disparity and the perception of depth. Procedures for virtual viewpoint generation based on shifting the MFP can produce disocclusions near vertical edges, or more generally, edges perpendicular to the chosen stereoscopic baseline.

[0243] 46A and 46B are illustrations showing example stereoscopic images for the right and left eyes, respectively, formed using unblended focal plane images, according to some embodiments. In some embodiments, box filtering (e.g., a box filter using a tooth-shaped depth weighting function, such as the depth weighting function of FIG. 20A) is used to form the unblended focal plane images. FIGS. 46A and 46B give an example of disocclusion in a stereoscopic image formed by deriving two viewpoints 4600, 4650 on a stack of MFPs. In some embodiments, the unblended focal plane images may be generated by box filtering the input images. In this example, the MFPs were not depth blended, since a box filter was used to form them. FIGS. 46A and 46B show an example of disocclusion in a crossed-eye stereogram formed by shifting and summing three unblended (box-filtered) MFPs. The total mismatch corresponds to 2.76% of the image width (see the MFPs shifted up to 6 pixels for both images). There is less disocclusion when depth blending is used. Depth blending also creates depth values ​​between the MFPs. Both of these improvements can be seen by comparing Figures 46A and 46B to Figures 29A and 29B, where depth blending was used.

[0244] In addition to reducing quantization effects in the depth dimension, depth blending may reduce disocclusion when synthesizing viewpoints from an MFP. To reduce disocclusion, content from adjacent depth planes is used for depth blending. If there is no content at the depth corresponding to a focal plane, shifting the near (occluding) focal plane to the side reveals empty areas that do not have any texture or color information (and appear transparent). These empty areas may be visible as black stripes near the object edges (due to transparency through all focal planes). If there is no content in adjacent focal planes, there is no depth blending to reduce disocclusion. Thus, the reduction in disocclusion by depth blending is content dependent.

[0245] In creating synthetic volumes from MFPs, one challenge with previous depth blending methods is that they produce MFPs that can have disocclusions due to concentrating most of the low frequency content (areas of the image where brightness and color vary slowly) in one focal plane (of many). Correspondingly, large areas within the MFP are left without any information.

[0246] Some embodiments post-process multiple focal planes (MFPs) so that they have better properties for virtual viewpoint generation or MFP rendering. Some embodiments are based on removing typical holes or non-textured (e.g., white background) areas of MFPs by redistributing their low frequency content. In particular, the redistribution reduces disocclusion when using MFPs to generate virtual (e.g., stereoscopic) viewpoints, even in the case of large mismatches.

[0247] Instead of concentrating the content within the MFP, redistribution distributes the low-frequency content more uniformly across all focal planes. This method may produce more uniformly distributed luminance and color values ​​than other MFP systems. The low-frequency content may be redistributed without degrading the 3D perception. This procedure preserves a division of 1 for the MFP, which means that they can be summed (e.g., close enough) to return to the original image.

[0248] Viewed from two virtual viewpoints, monoscopic MFP stacks are attractive for their ability to support disparity and motion parallax at any baseline orientation. This benefit rivals that of more conventional depth image-based rendering (DIBR) in stereoscopic 3D.

[0249] If the stereo viewpoint, disparity, and baseline are chosen before capturing the scene, the receiving terminal may not allow these parameters to be changed after the scene is captured. Problems may arise if the viewer wishes to set these parameters, or if the system attempts to combine renderings of stereoscopic content that occur with different parameters.

[0250] DIBR systems may be used in place of traditional stereoscopic vision, as stereoscopic image formation using DIBR may enable additional options in the post-processing stages of the receiver. The amount of stereoscopic disparity (using a depth budget) may depend on the captured content, which may be beneficial to control later than during the shooting of the scene, for example. Similarly, using a depth-plus-texture format to produce multiple focal planes may be easier to control later than during the shooting. Post-processing at the receiving end of a monocular stack of MFPs allows a stereoscopic MFP to render at a baseline orientation with variable disparity.

[0251] Many 3D display systems or near-eye displays with accommodation support do not support or tolerate even smaller changes in viewpoint. Some embodiments for producing a (redistributed) MFP support relatively large viewpoint changes, disparities, and motion parallax, improving visual quality in, for example, 3DoF+ applications. Larger disocclusions can be reduced by distributing the MFP luminance and color values ​​more uniformly than in previous MFP systems. This uniform distribution is based on distributing low-frequency content over several or all focal planes instead of concentrating it in one of the MFPs.

[0252] Some embodiments perform post-processing to improve the quality of MFPs created using some existing methods, which may use depth and texture inputs as well as focal plane stacks as inputs, and may work with multiple capture and transmission formats.

[0253] In previous MFP displays, disocclusions related to stereo mismatch may not be an issue by rendering separate MFP stacks for each eye, acquired using stereo capture for texture and depth. However, by using focal planes with high brightness and contrast differences, even smaller viewpoint changes caused by eye rotation may appear as small but visible disocclusions. These artifacts may be removed under some scenarios. The systems and methods disclosed herein according to some embodiments allow for the generation of virtual viewpoints with large mismatches.

[0254] Redistribution of MFP to support viewpoint changes Described below is an exemplary procedure for generating an optimized MFP for viewpoint generation with viewpoint variation. Some embodiments for generating a redistributed MFP include receiving an MFP input that may be generated by a depth blending method. Low frequency image areas are detected, which may occur by low pass filtering the input image and by low pass filtering each MFP. The low pass components of each MFP may be replaced by an equal share of the low pass image filter output, which may occur by removing the corresponding low pass components from each MFP and adding an integer fraction of the low pass filtered input image, where the integer fraction is the reciprocal of the number of MFPs used. The MFPs may be redistributed to generate MFPs for synthesizing stereoscopic views with a reduced amount of disocclusion and / or to render MFPs with a depth dimension to support natural accommodation (removing VAC). High frequency image content may not be derived as such, but may be a supplement for the low frequency components.

[0255] FIG. 47 is a flow chart illustrating an example process for generating redistributed focal plane images of front, mid, and back depth ranges according to some embodiments. FIG. 47 is a block diagram 4700 of a redistribution process for three MFPs. In some embodiments, the redistribution 4708 is a post-process for MFPs formed in a conventional manner. In some embodiments, a set of (e.g., conventional) focal planes (MFPs) is used as input. The received image (picture) 4704 may be summed from those focal planes using picture depth data 4702 or may be received by the MFP block 4706. The MFP block 4706 may output the front, mid, and back depth 4710, 4712, 4714 planes. The low-pass components of each focal plane (one out of N) are replaced by the low-pass components of the input image by a scaling factor 1 / N, where N is the number of focal planes. For example, N is 3. This process may preserve the one-part nature of the MFP.

[0256] 48A-48C are illustrations showing example focal plane images without redelivery, according to some embodiments. FIG. 48D-48F are illustrations showing example focal plane images with redelivery, according to some embodiments. FIG. 48A-48F are modified versions of the unprocessed flower test image and correspondingly its respective depth maps, e.g., of FIG. 21B and FIG. 21C, according to some embodiments. FIG. 48A-48F show examples of MFPs before and after redelivery. FIG. 48A-48C show linearly blended MFPs 4800, 4810, 4820 before redelivery. FIG. 48D-48F show redelivered versions of the same MFPs 4830, 4840, 4850. A Gaussian filter with a radius of 20 pixels (in both horizontal and vertical directions) was used for low-pass filtering. Figures 48A-48F reveal that the redistribution preserves the high frequency components (details) of each focal plane while bringing low frequency components into areas that were originally devoid of information (such as the white background in Figures 48D-48F).

[0257] Both MFP examples in Figures 48A-C and 48D-F split the captured information into different accommodation levels (focal distances). However, they have differences in their ability to repair disocclusion. The MFP stack resulting from linear blending in Figures 48A-C has non-textured / uncolored (e.g., white background) areas that can cause disocclusion, while the re-distributed version of the same MFP in Figures 48D-F shows an improvement and has more uniform characteristics. When shifting the second set of MFPs for a new viewpoint or for a small movement of the viewer, disocclusion is less visible.

[0258] The MFPs in Figures 48D-48F are selectively low-pass filtered so that high-contrast transitions between textured and non-textured (e.g., white background) areas are not filtered. Because low-frequency image components are not sensitive to being allocated at different depths, in some exemplary embodiments, the low-frequency image components are allocated uniformly to all focal planes. Low frequencies may be allocated to just one focal plane, or to extra focal planes between or outside (not belonging to) any of the original MFP distances.

[0259] There may be various reasons for preferring a particular allocation: for example, the ratio of allocations may vary depending on the nature of the content, and viewers may expect the resulting MFP to fit (or be close to) a division of 1.

[0260] 49A-49B are illustrations showing exemplary right and left images for a crossed-eye stereogram formed using linearly blended focal planes, according to some embodiments. FIG. 49A and FIG. 49B show an example of a crossed-eye stereogram formed using the linearly blended focal planes shown in FIG. 45A-45D. FIG. 49A and FIG. 49B show crossed-eye stereograms 4900, 4950 formed by shifting the linearly blended focal planes (FIG. 45A-45D, which are line-drawing versions of the images appearing in Non-Patent Document 1), according to some embodiments. The total discrepancy is 4% (2×12 pixels in a 600×600 pixel image). FIG. 49A and FIG. 49B show exemplary disocclusion errors 4902, 4952 near the object edges.

[0261] 50A-50B are illustrations showing exemplary right and left images for a crossed-eye stereogram formed using a redistributed focal plane, according to some embodiments. FIG. 50A and FIG. 50B show an embodiment of the same but redistributed focal plane of FIG. 49A-49B. In FIG. 50A and FIG. 50B, the exemplary disocclusions 5002, 5052 are much smaller. FIG. 50A and FIG. 50B show a crossed-eye stereogram formed using a redistributed focal plane with the simplified procedure described. The total discrepancy is 4%. The disocclusion error is reduced relative to that in FIG. 49A-49B. FIG. 50A and FIG. 50B are processed versions of the unprocessed images (FIG. 45A-45D) from Non-Patent Document 1.

[0262] In Figures 49A, 49B, 50A and 50B, four MFPs were used with an amount of synthetic mismatch corresponding to 4.0% of the image width (a 600x600 pixel MFP shifted by up to 12 pixels for both viewpoints). In the example of Figures 50A and 50B, the MFPs are selectively low-pass filtered so that high-contrast transitions between textured and non-textured (e.g., white background) areas are not filtered.

[0263] Redistribution of low-frequency components of MFP As shown by [1], the adjustment can be more natural for the re-delivered MFP. In particular, this holds for non-Lambertian effects in the scene. [1] is not to be understood as re-delivering the MFP as a post-process, nor as being used to allow large synthesis discrepancies in stereoscopic views, even at arbitrary baseline orientations.

[0264] According to some embodiments, the redistribution of low-frequency components is used for multi-focal plane (MFP) virtual viewpoint generation where the discrepancy is large. By (re)distributing low-frequency components, large variations in pixel values ​​within and between conventional focal planes can be avoided. This in turn reduces high-contrast effects and visible disocclusions when shifting focal planes for new viewpoints (see 3D warping).

[0265] Regarding the execution of filtering operations The redistribution may result in a modified MFP such that each focal plane receives a specified integer fraction of the low-frequency content of the entire scene, plus the sum of those high-frequency components specific to the focal plane (at the corresponding adjustment distance). This process replaces the MFP's specific low-frequency components with a particular portion of the total low-frequency components.

[0266] The high and low frequency components are complementary such that in deriving either one, the other component can be determined by subtraction. In the exemplary process shown in FIG. 47, the low frequency components are derived and the high frequency components are obtained as their complement (by subtraction) from the unfiltered image. This process may be reversed, or both components may be obtained directly using two filters with complementary transfer functions in the frequency domain. Since the user's accommodation distance is not determined or stored when rendering, many systems may use the same filter function for all focal planes.

[0267] Note that the original MFP is selectively filtered such that high contrast transitions between textured and non-textured (e.g., white background) areas are not filtered (see, e.g., Figures 45A-45D). Without selective filtering, high contrast areas may result in high frequency components that are not part of the original image content, induce ringing around object edges, and degrade the quality of redistributed MFPs and renderings that use them.

[0268] In the disclosed examples according to some embodiments, a Gaussian low-pass filter was used with an adjustable radius (filter window). In some embodiments, it is beneficial for the final result that the filtering operation is applied only to pixels that do not belong to the sharp transition between non-zero value (colored) and zero value (transparent or void) areas. The selective Gaussian filter does not filter image areas that have a color difference larger than a set threshold. Due to the generally high contrast between colored and black areas, their edges can be reliably detected and the detection is not sensitive to the threshold.

[0269] 51A-51B are illustrations showing exemplary right and left images for a crossed-eye stereogram formed using selective Gaussian filtering in re-distributing an MFP, according to some embodiments. FIG. 51A and FIG. 51B show a crossed-eye stereogram using selective Gaussian filtering in re-distributing an MFP. The parameters are the same as in FIG. 50A-50B, except that filtering is not applied if the filtering changes pixel values ​​more than a threshold (e.g., here 60). The exemplary disocclusion errors 5102, 5152 are further reduced from those in FIG. 50A-50B. The results show some contouring in smooth parts of the scene, which may be induced by some inaccuracies used in the implementation. FIG. 51A and FIG. 51B are processed versions of the unprocessed images (FIG. 45A-45D) from Non-Patent Document 1.

[0270] Support for non-Lambertian content Non-Patent Document 1 reports a method in which non-Lambertian effects for reflection or refraction are correctly presented. Many previous methods do not support this type of content, but this lack of support can be a drawback for only certain types of content.

[0271] As the disclosed method may use multiple capture and transmission formats, non-Lambertian content may be used as well, for example, by selecting an input format. Many previous systems do not support depth map generation using a depth sensor, but may support focal stacking (see non-patent document 1) or some other camera-based (depth from mismatch) method that may be able to capture the correct pixel distance even for non-Lambertian content. Thus, the MFP received as input may be caused or corrected to show reflected or refracted image content at the correct distance.

[0272] In some embodiments, a focal stack derived MFP (presentation plane) may be received as an input at the front end and used without a MFP formation stage, for example to synthesize stereo or motion parallax. In some embodiments, a method may include receiving a plurality of focal plane images, orienting the plurality of focal plane images using head information to provide stereo disparity between the left and right eyes, and displaying the oriented plurality of focal plane images.

[0273] Exemplary embodiments for forming a redistributed focal plane A procedure for redistributing low-frequency components has been described above. This procedure may be based on decomposing the texture image into different MFPs and interpolating between the different planes using a depth blending method (e.g., linear filtering). The filtering operation for redistributing low frequencies between the MFPs may be performed as a post-process.

[0274] FIG. 52A is a flow chart illustrating an example process for generating a redistributed focal plane image using low and high frequency filtering, according to some embodiments. FIG. 52B is a flow chart illustrating an example process for generating a high frequency focal plane image using low pass filtering, according to some embodiments. An example process 5200 (see FIG. 52A) separates high and low frequency content of a picture 5202 using a filter 5206, and then resolves the content into focal planes 5210 at 5208 using picture depth data 5204. A low frequency redistribution 5212 and a rendering 5214 of the MFP are also performed. Changing the filtering stage generates an alternative MFP. FIG. 52A is a block diagram of another redistribution process for an MFP, according to some embodiments. FIG. 52B illustrates an example process 5250 for the implementation of complementary low pass filters 5254 and high pass filters 5256 of picture data 5252. The exemplary process 5200 may, for example, avoid using a selective Gaussian filter and optimizing its threshold parameters when low-pass filtering is not performed for focal planes that often contain high-contrast transitions. Correspondingly, some embodiments using the exemplary process 5200 may improve quality.

[0275] Filtering as a post-process may be used by the MFP process and output. However, some exemplary methods and systems disclosed by some embodiments herein may improve quality. Some embodiments avoid the use of selective or guided filtering to exclude void areas (pixels that do not belong to the actual focal plane content) from filtering. Void areas may be visible as white or black areas in the MFP and may not be detected correctly when filtered. Depending on the image content, this may be visible as defects in the filtering results and reduce the quality of the redistributed focal plane. Some embodiments avoid optimizing thresholds for selective filtering, since no parameters are used.

[0276] In some embodiments, the process for forming the redistributed focal plane may include filtering the input image for low frequency content and redistributing the low frequency content into multiple focal plane images. In some embodiments, the process for forming the redistributed focal plane may include filtering the input image for high frequency content and decomposing the high frequency content into multiple focal plane images. In some embodiments, when decomposing, the high frequency components are not moved from their original depth, as in the case of redistributing the low frequency content, for example. An exemplary process for forming the redistributed focal plane may be as shown in FIG. 52A. In some embodiments, the high frequency content HF shown in FIG. 52A may be derived from the low frequency content LF shown in FIG. 52A, or vice versa.

[0277] Avoid using signed arithmetic FIG. 53 is a flow chart illustrating an example process for generating redistributed focal plane images for the front, mid and rear depth ranges according to some embodiments. When performing redistribution for the given example, a slightly modified process flow shown in FIG. 53 is used. FIG. 53 shows a block diagram of the redistribution process 5300. Redistribution 5320 of three MFPs is shown as an example. Scaling operations with constants (0.5 and 2) are used to prevent saturation in the image processing software. Some embodiments use a set of focal planes (MFPs) as input. The received image (picture) 5304 may be summed from those focal planes 5306 or may be received by the MFP block 5306. The MFP block 5306 may receive picture depth data 5302 in addition to the picture 5304. The MFP may be divided into a front range plane 5314, a mid range plane 5316 and a rear range plane 5318. The redelivery process 5300 may output the forward, middle, and rear redelivered pictures. In some embodiments, arithmetic operations and components that perform arithmetic operations (e.g., ALUs) may handle signed calculations using signed arithmetic with sufficient bits.

[0278] In Figure 53, the low-pass (low frequency) components of each focal plane (one out of N) are replaced by the low-pass components of the input image by a scaling factor of 1 / N, where N is the number of focal planes used. In this example, N is 3, which corresponds to a scaling by 1 / 3.

[0279] To prevent saturation in the image processing software, a scaling operation with other constants (0.5 and multiplication by 2) is used. The downscaling by 0.5 is done in the first addition operation, which is done by applying 50% transparency to the overlaying image. The subtraction of low frequency components from each focal plane requires downscaling by 0.5 and correspondingly multiplying the result by 2 after that operation.

[0280] Note that the low frequency components of the MFP sum (e.g., sufficiently close) to the low frequency components of the original image, and thus the addition and subtraction operations preserve (e.g., sufficiently close) a division of 1 for the MFP decomposition.

[0281] The implementation illustrated in Fig. 53 may cause some loss of computational precision. In particular, some pixels may saturate to 0 in the final subtraction phase and may not scale back when multiplied by 2. This may appear as slight contouring in some parts of the image, which is not caused by the disclosed exemplary method of redelivery according to some embodiments.

[0282] Avoiding the use of signed pixels in transformations FIG. 54 is a flow chart illustrating an example process of generating a redistributed focal plane image using low pass filtering, according to some embodiments. In performing the simulation corresponding to FIG. 54, procedure 5400 was applied to avoid using signed integers / pixel (or (signed) high frequency representation). FIG. 54 shows an alternative embodiment. In some embodiments, a picture 5402 and a picture depth 5404 are received by procedure 5400. The picture 5402 may be filtered into high and low frequency components at 5406. The high and low frequency components may be used in combination with the picture depth data to generate inputs to the MFP decomposition process 5408, 5410. The picture may be decomposed into N layers 5412, 5414 for low and high frequency components. The decomposed layers 5412, 5414 may be redistributed 5416 and rendered 5418.

[0283] To avoid saturation in the sum (keeping the result within the 8-bit value reserved for image pixels), the redistributed LF 5416 is summed with 50% transparency. When subtracting low frequency faces (to result in a redistributed face), a scale-up is performed correspondingly. One alternative embodiment to the process shown in Figures 52A and 52B avoids working with high frequency faces with a signed representation. Some embodiments may use (signed) full precision arithmetic for the redistributed LF 5416.

[0284] The systems and methods disclosed herein according to some embodiments create monocular MFP stacks that exhibit a reduced amount of disocclusion when used to generate a stereoscopic 3D view. Some embodiments redistribute the MFPs and shift the post-processed MFPs to generate mismatch and baseline orientation. The resulting stereoscopic images may be displayed, for example, on external screens and wearable displays, including those that support natural accommodation.

[0285] The systems and methods disclosed herein according to some embodiments may be used to generate stereoscopic content for services that use stereoscopic displays.

[0286] Some embodiments enable the development of stereoscopic displays that allow an easy and cost-effective way to produce baseline-free (head-tilt independent) stereoscopic rendering with relatively large, adjustable disparity and motion parallax effects. These properties may be used with 3DoF+ applications.

[0287] Many MFP display implementations use temporal or spatial (image) multiplexing of the MFP. Redistribution algorithms can be recognized by designing suitable test scenes with objects at different depths and by exhibiting characteristic behavior at object edges. Some embodiments can be recognized based on their ability to support stereoscopic vision and motion parallax for users who are moving their heads and themselves.

[0288] Some embodiments produce stereoscopic images that may cause VAC when viewed on a front stereoscopic display. MFPs formed by the systems and methods disclosed herein according to some embodiments may be rendered using MFP displays that support natural accommodation.

[0289] A synthetic mismatch (e.g., based on processing the MFP (e.g., by shifting the MFP for both S3D display and MFP display) is used in some embodiments. The systems and methods disclosed herein in some embodiments channel accommodation-dependent properties of the MFP (such as properties that are problematic to visualize in print) into spatial mismatches that are visible when printed and that indicate many properties of the corresponding MFP are in the depth dimension.

[0290] axial motion parallax A common problem with many previous systems is that motion parallax is only supported for small lateral (typically horizontal) movements, and not for axial movements. Multi-focal planes (MFPs) are an economical approximation of a clear field. Many previous MFP displays support only a single precise viewpoint, and do not support viewpoint changes as the viewer moves in the depth (axial) dimension. Thus, the use of such previous systems is generally limited to fixed viewer locations.

[0291] Many previous 360-degree panoramic rendering systems support viewing in various directions using orientation sensors, but do not support axial motion parallax. These three degree of freedom (3DOF) systems constrain the user only to changes in viewing direction. Many six degree of freedom (6DOF) systems, such as computed autonomous virtual environments (CAVEs), are typically costly and not feasible for broadcast systems. The ability to bring small general user movements to a 3DOF system is described as 3DOF+. In this application, according to some embodiments, the term "natural 3D movement" refers to this 3DOF+ capability.

[0292] Enabling 3DOF+ systems with reduced complexity and legacy 3DOF or 2D stereoscopic content is a challenge addressed by the systems and methods disclosed herein in some embodiments. An additional challenge is to enable natural accommodation of the displayed content (avoiding vergence accommodation conflict (VAC)) in these new 3DOF+ solutions.

[0293] 55A-55B are schematic perspective views 5500, 5550 illustrating exemplary multi-focal planes rendered in space, according to some embodiments. FIG. 55A and FIG. 55B illustrate a nominal viewpoint 5504, shown by a dashed line, relative to a focal plane (MFP) 5506, shown by a solid rectangle. If the viewer 5502 moves, instead of viewing information (focal planes) from the nominal origin viewpoint, new viewpoints may be generated. Focal planes are approximations of the 3D scene, and new viewpoints may be generated in a manner similar in some respects to, for example, warping a 3D model to a new orientation (see virtual viewpoint generation block). In FIG. 55A, the user views focal planes rendered in space by the user's MFP glasses. In FIG. 55B, motion parallax is supported for the user inside an exemplary volume (dx, dy, dz) 5554 around the nominal viewpoint (0,0,0) 5552, such as the exemplary cube shown.

[0294] Motion parallax varies the perspective relative to the focal plane around a nominal viewpoint (0,0,0). If the user moves away from the nominal point, the focal plane image is seen from a slightly varying viewpoint and can simply be translated (shifted) and scaled accordingly to create a correct MFP rendering. Each coordinate (x, y, and z) can be varied within some limits to keep the distortion created acceptable. Motion parallax can thus be supported inside a volume around the nominal viewpoint (such as the cube shown in FIG. 55B). A cube around the viewer's nominal position can be selected to support 3D motion parallax (and capture user motion). In some embodiments, a sphere or other shape can be selected to support capture of user motion for motion parallax.

[0295] 56A-C are schematic perspective views illustrating three examples of multiple focal planes that are shifted and scaled for different viewpoints, according to some embodiments. The user's movements may change his or her viewpoint relative to the focal planes as indicated by the change in dashed lines. FIGS. 56A-C show several example views 5600, 5630, 5660 for changing viewpoints 5602, 5632, 5662 inside motion parallax cubes 5604, 5634, 5664. The dashed lines indicate the viewpoint change relative to the focal plane stack. New viewpoints relative to the MFP stack may be created by deforming the MFP accordingly. The mathematics for performing the deformations may in some respects be similar to warping a 3D model. In some embodiments, the deformations may be approximated using other operations described below.

[0296] Each 3D coordinate or voxel within the cube represents a different viewpoint for the focal plane stack (or pair of stacks for stereoscopic displays). For each of these viewpoints, a transformation of the original MFP stack (viewed from the nominal viewpoint) is performed. Thus, for each 3D viewpoint / voxel, there may be a unique MFP stack (reoriented and scaled by the transformation). In some embodiments, a set of MFP stacks may be pre-derived and stored. These derived MFP stacks may be categorized by the captured motion information.

[0297] FIG. 57 is a process diagram illustrating an example process for generating multiple focal planes in a motion parallax system, according to some embodiments. FIG. 57 illustrates a process 5700 for monoscopic MFP formation, processing (e.g., synthesis of viewpoints for stereoscopic vision and motion parallax), and rendering. For simplicity, creating a motion parallax effect is illustrated for only one viewer. In some embodiments, the system may include one or more wearable user terminals (head mounted displays, HMDs, etc.) that allow each user to individually perceive motion parallax. Some embodiments may use, for example, a more distributed or more centralized architecture.

[0298] Texture 5702 and depth (image plus depth) 5704 signals are received as input to the MFP formation block 5706, which forms a multi-focal plane (MFP) using a depth map (image) captured from the scene. The MFP is sent to the mismatch formation block 5708.

[0299] The mismatch formation block 5708 is a vector r i5710. Create a synthesized stereo viewpoint (or disparity) using the specified baseline orientation and length (denoted by

[0300] Motion detection and tracking (e.g., using an inertial measurement unit (IMU) 5720 and / or Kinect type 5718 of external sensor 5714) may be used to capture user movement relative to the user's initial viewing position with a nominal viewing setup (focal plane size and distance) for MFP rendering. In the exemplary process 5700, and in other processes, other motion detection and tracking devices may be used instead or in addition. The lateral motion parallax block 5712 may shift the two stereoscopic viewpoints in both the x and y dimensions. The focal planes in each MFP stack may be shifted by different amounts corresponding to their distance in depth.

[0301] The axial motion parallax module 5716 may adjust the focal planes by scaling and / or sizing according to the measured axial motion relative to the initial / nominal position and viewpoint. Different focal planes may be scaled differently since they correspond to different distances.

[0302] When viewing broadcast type of content, user motion may not generally affect the input stream (the viewpoint used in capturing the view), so after the user 5724 experiences motion parallax due to body motion, the changed viewpoint can be shifted back to the nominal viewpoint by the selected mechanism (delay, rate, trajectory, and time).

[0303] In some embodiments, motion tracking sensor readings of the viewer's position may be measured relative to the focal plane display to generate head orientation information. In some embodiments, in response to generating head orientation information, motion parallax may be synthesized by altering one or more of the focal plane images.

[0304] Some previous devices use MFP for viewpoint changes when supporting (baseline-free) stereoscopic vision and lateral motion parallax. The section below describes axial motion parallax. Focal plane adjustments for intermediate or compound motion can be derived by applying these two cases. For example, axial motion parallax can be created in some embodiments by scaling one or more focal planes.

[0305] FIG. 58 is a schematic plan view illustrating an example focal plane scaling for axial movement, according to some embodiments. Although FIG. 58 is described with respect to a monoscopic viewpoint relative to the MFP stack during axial movement, a corresponding process may be performed with each viewpoint transformation in a stereoscopic system or method. Similar to the case of translational movement, a transformation of the focal image may be performed to handle axial movement. In the example monoscopic viewpoint environment 5800, when the viewer 5810 approaches the MFP stack axially, the edge of the focal plane 5808 may be adjusted to move (or shift) outward. When the viewer 5810 moves axially away from the MFP stack, the edge of the focal plane 5806 may be adjusted to move (or shift) inward. The edge of the focal plane 5804 located at the zero parallax setting (ZPS) 5802 remains in place (or unadjusted) when the viewer 5810 moves axially toward or away from the MFP stack. The change in the viewer's 5810 viewpoint due to axial movement is shown as a dashed line for two examples. FIG. 58 illustrates how the apparent size of an MFP changes during axial motion. Three MFPs with equal amounts of separation are shown. The zero parallax setting (ZPS) 5802 is located at the farthest focal plane. This farthest focal plane does not shift when stereoscopic disparity or lateral parallax is created, and this farthest focal plane is not scaled when creating axial motion parallax. Although a single-step axial movement is shown in the exemplary monoscopic viewpoint environment 5800, multiple axial step movements can be supported using a similar process of scaling the MFP by an amount proportional to the number of axial steps moved by the viewer 5810.

[0306] If the viewer 5810 moves one step (e.g., one distance unit) axially closer to the MFP stack, the size of the focal plane 5808 may be adjusted one unit (e.g., one distance unit) larger. If the viewer 5810 moves one step axially away from the MFP stack, the size of the focal plane 5806 may be adjusted one unit smaller. The size of the focal plane 5804 located at the zero parallax setting (ZPS) 5802 remains the same size (e.g., or unchanged) as the viewer 5810 moves axially closer to or farther from the MFP stack. The change in the viewer's 5810 viewpoint due to axial movement is shown as a dashed line for the two examples.

[0307] In some embodiments, the scaling transformation of one or more focal plane images may be approximated by adjusting the size of the focal plane symmetrically around its center such that the aspect ratio of the image is preserved. Small changes to motion parallax, such as in creating stereoscopic mismatches, are captured by small changes in focal plane image resolution (increments or decrements up to a threshold, such as a change of 5 pixels). In some embodiments using higher resolution images, a higher threshold may be used.

[0308] When adjusting the image size, the top and bottom edges of the image may be moved in opposite directions by an amount (the vertical size of the image is changed by a number of pixels). The scaling factor of the MFP may vary linearly as a function of depth, as shown in the example of FIG. 58. This linear change results in a set of deformed MFPs and a set of applicable viewing positions (focal points) along the viewing axis. The amount of adjustment depends on the position (distance) of the focal plane in depth, such that the farthest plane is not adjusted (as if it were located infinitely far away from the viewer). In some embodiments, the focal plane next to the farthest focal plane is scaled by one unit (or pixel multiplication). The next closest focal plane may be scaled by two units in some embodiments. The amount of scaling for the remaining focal planes may follow this paradigm for progressively closer focal planes in some embodiments.

[0309] An exemplary process for scaling an MFP according to some embodiments is described below for scaling N focal planes (MFPs). Herein, the focal planes are separated by a uniform distance and use linear increments and decrements as a function of distance. In some embodiments, non-uniform separation distances may be used with non-linear increments and decrements as a function of distance. Non-linear scaling (linear in dioptic scale) may be used to produce a linear perception by the user. The size of the focal planes (or set of MFPs) is taken for several unequal steps along the viewing axis. This exemplary process may be used, for example, for validation of size adjustments. In some embodiments, more rigorous geometric operations may be performed.

[0310] If the viewer moves one step closer to the MFP stack, the farthest focal plane (FP1) is the zero parallax plane ("at infinity") and is not scaled (or adjusted in size). Therefore, FP1 is adjusted by 0 units. The remaining focal planes (FP2, FP3, ..., FP N ) for each focal plane FP i is adjusted up by (i-1) units, for i ∈ (2, 3, ..., N). This adjustment process may be repeated for each closer step to the MFP stack.

[0311] If the viewer moves one step further away from the MFP stack, the farthest focal plane (FP1) is the zero parallax plane ("at infinity") and is not scaled (or adjusted in size). Therefore, FP1 is adjusted by 0 units. The remaining focal planes (FP2, FP3, ..., FP N ) for each focal plane FP i is adjusted down by (i-1) units, for i ∈ (2, 3, ..., N). This adjustment process may be repeated for each step further away from the MFP stack.

[0312] In some embodiments, the focal plane height may be increased in size by scaling the height by the incremental scaling factor shown in Equation 3.

[0313]

number

[0314] Here, V res is the vertical resolution of the focal plane. Similarly, the height of the focal plane can be reduced in size by scaling the height by the reduction scaling factor shown in Equation 4.

[0315]

number

[0316] Here, V res is the vertical resolution of the focal plane. The width of the focal plane for both increments and decrements is adjusted to preserve the aspect ratio before adjustment.

[0317] In some embodiments, moving in steps using the above process is not linear on a metric scale. The physical distance may depend on the nominal viewing conditions, including the position, size, and distance of each focal plane (MFP). In addition, the physical distance may depend on the (input) content properties, such as the zero parallax setting (ZPS) chosen, and the correspondence between the depth map and the physical distance.

[0318] Using an exemplary process according to some embodiments, the focal plane image resolution and unit size used may determine the position along the viewing axis where the focal plane is seen to be aligned (focused on the viewpoint) after the transformation. These points may be used to render content during user movement.

[0319] The higher the image resolution and the smaller the unit size, the smaller the difference between the determined axial (focused) positions. The unit size used in conjunction with the image resolution can be experimentally verified and / or calculated from the geometry such that the density of focal points supports the perception of smooth motion parallax.

[0320] Shifting and scaling artifacts may limit the axial parallax (motion) to a small amount that the user determines to be reasonable. This amount may be set to a small number of steps (or axial positions). The step positions and sizes (physical distances) may be approximated from a geometric shape (such as the geometric shape shown in FIG. 58).

[0321] In some embodiments, the nominal viewing position and focal plane distance are not set. In some embodiments, setting the unit size used for MFP size adjustment may determine the step position and size in axial direction (focus and associated distance). In some embodiments, the steps may be linearly mapped to distance without degradation of naturalness as determined by the user.

[0322] In some embodiments, the motion parallax effect may not be calibrated to physical position, such as when conditions for 3D content production (which may include mismatches and depth ranges) are not standardized or set. Some users may be relatively insensitive to depth information with respect to the perception of naturalness of (axial) motion parallax.

[0323] 59A-C are schematic plan views illustrating an example deformation of an example focal plane content during axial motion, according to some embodiments. FIG. 59A shows an MFP configuration 5900 for a nominal "start" point of an example mismatch shift 5902 for a viewer's left eye. A process similar to that using the monoscopic view of FIG. 59A may be applied to scale the right eye view of a stereoscopic viewer. The distance d k denotes the focal plane distance of the vector with k equal to 1, 2, …, N. The distance de shows the disparity distance vector for an exemplary MFP transformation (or translation). k ,y k ) denotes the original coordinates of a pixel on the focal plane k. k’ ,y k’ ) 5904 shows the coordinates of a pixel in focal plane k after the transformation (or translation in some embodiments) but before scaling. k’ ,y k’ ) 5904 is an exemplary left-eye viewpoint focal plane F of the three focal planes. N 59A to 59C show the focal planes F1, F2, ..., F for an example of N equal to 3. N Shows.

[0324] FIG. 59B shows an MFP configuration 5930 for axial motion of a viewer approaching the MFP stack. The viewer is aligned with a motion vector d z The left eye viewpoint is translated due to the viewer being closer to the MFP stack.

[0325] FIG. 59C shows an MFP configuration 5960 for focal plane deformation that approximates axial motion without changing the focal plane distance. The left eye viewpoints of the three focal planes show the changes to pixel coordinates after deformation. k’’ ,y k’’ ) 5962 shows the coordinates of a pixel in focal plane k after transformation (or translation in some embodiments) and after scaling.

[0326] Keeping the previous notion of relevant parameters, (x k’’ ,y k’’ ), the scaling can be written as shown in Equation 5. (x k’’ ,y k’’ )=S(x k’ ,y k’ ,d z ,d e) Equation 5 where S is a uniform scaling transformation using each focal plane center as a fixed point.

[0327] For axial motion, the focal plane size may be adjusted to provide the user with the illusion of deformation of physical space and coordinates through the view of the rendered environment. The focal plane rendering process may not change the accommodation distance when experiencing axial motion parallax. Correspondingly, there is no adjustment of the optics or mechanics of the MFP display. This process differs from the real-world case, as the viewer's accommodation distance changes corresponding to his or her distance from the focused object. However, no VAC occurs, and if the (supported) movement to experience motion parallax is small, the lack of accommodation change does not degrade the user's experience of naturalness.

[0328] Some embodiments may support motion parallax effects for a traditional 2D display (as an alternative or in addition to an MFP display). The use of a 2D display may avoid the complexity of an MFP display, but the user may be subject to VAC. Some embodiments may support a 2D display by summing two or more MFPs. Support for 2D displays is described in more detail below.

[0329] If support for axial movement is provided along with support for lateral (side-to-side) movement and baseline-free (head-tilt-free) motion parallax, realistic 3D motion parallax around a nominal viewpoint using focal plane rendering can be provided without VAC.

[0330] In some embodiments, relative motion around a primarily fixed (nominal) viewpoint does not change the viewpoint without receiving new data for the new nominal viewpoint. The viewing setup may be comparable to watching broadcast content, but the viewing setup disclosed herein according to some embodiments may support user interaction by allowing small movements (such as 3DOF+ capabilities) and corresponding viewpoint / parallax changes.

[0331] The supported user features may not affect the nominal or inherent viewpoint of the received content. Small motion parallax may be supported in the same manner for each viewing position in the user's environment, for example, when the user changes his or her seat or walks around the room. If the user moves, the nominal viewpoint follows his or her average location. In some embodiments, the viewpoint returns to the nominal viewpoint (which in some embodiments may involve a delay after inducing motion parallax). Returning to viewing from the nominal viewpoint may be based on manual input by the user and / or an automatic policy.

[0332] FIG. 60 is a graph showing an example of tracking of viewpoint coordinates during user movement, according to some embodiments. Large position changes may be treated as, for example, a change in the user's seating location, and may not be followed by a small adjustment in parallax. Instead, the viewpoint may be returned to the nominal viewpoint of the received content without any change in parallax. Small movements may be treated with a corresponding virtual viewpoint change. If the small motion changes are more permanent, for example, if the user rests his or her body on the backrest, the nominal viewpoint returns to the nominal viewpoint at some rate of change in some embodiments.

[0333] The graph in FIG. 60 illustrates an exemplary control policy for controlling viewpoint coordinates (e.g., axial) during user motion. FIG. 60 shows the change in coordinates compared to nominal values ​​at various time steps. In the exemplary graph 6000, the zero line represents the nominal value, and the traces of the graph measure the difference values ​​6004 compared to the nominal value. Between time step 28 and time step 40 (6002), the coordinate values ​​drop toward 0. At time step 67 (6006), the user motion goes outside the maximum and minimum disparity limits 6008, 6010, and tracking of the user motion is reset to 0. A viewpoint control process similar to the exemplary viewpoint control process used with respect to FIG. 60 may be used for each axis of the coordinate system (e.g., one axial coordinate value and two lateral coordinate values).

[0334] In some embodiments, the coordinates may be saturated to a maximum (or minimum) value, and each coordinate system axis may drift toward 0 after user motion stops (e.g., after the viewer sits in a different chair). In some embodiments, the user's viewpoint may be returned to a nominal viewpoint if the user motion goes outside of the maximum and minimum limits, and if the user's viewing orientation goes outside of the maximum and minimum limits. In some embodiments, a user changing his or her viewing orientation appears as a lateral (xy) shift of the focal plane, and the user's viewpoint may be returned to the nominal viewpoint after the motion stops or exceeds a maximum or minimum value. It will be understood that the process of returning the user's viewpoint to the nominal viewpoint may be a manual interaction process, an automatic process, or a combination of manual and automatic processes. Applicable policies for controlling 3D motion parallax may be developed according to some embodiments.

[0335] In some embodiments, the described process for synthesizing a stereoscopic or shifted viewpoint from a monoscopic MFP stack may be extended to other configurations, such as motion parallax using a true (e.g., as opposed to synthesized) stereoscopic MFP stack, motion parallax using a synthesized stereoscopic MFP stack, motion parallax using projection and rendering of an MFP on a 2D display, and MFP formations that are combinations of a true stereoscopic MFP stack, a synthesized stereoscopic MFP stack, and an MFP used in conjunction with a 2D display.

[0336] FIG. 61 is a process diagram illustrating an example process for generating a true stereo set of multiple focal planes, according to some embodiments. FIG. 61 shows a motion parallax MFP generation and rendering process 6100 using true stereo input. A right eye MFP creation process 6110 receives a texture 6102 and a depth map 6104 for the right eye. A left eye MFP creation process 6112 receives a texture 6102 and a depth map 6108 for the left eye. In some embodiments, the MFPs for the right and left eyes are sent to a lateral mismatch adjustment process 6114. A mismatch adjustment 6116 for orientation and amount (≧0%) may be received by the lateral mismatch adjustment process 6114 to adjust the viewpoint separation of the MFPs. The use of stereo MFP input may limit the reorientation and adjustment of the baseline MFP. The rendering of the MFP is anchored to a position and orientation in space. The adjusted MFPs for the left and right eyes are sent to a lateral parallax adjustment process 6118. The motion tracking process 6120 may send user motion data to a lateral parallax adjustment process 6118 and to a scale adjustment process for axial parallax 6122, 6124. The motion data may be represented as x(t) and y(t) data captured relative to a nominal viewpoint and orientation for the MFP anchored in space. In some embodiments, for example, an inertial motion unit (IMU) 6130 and / or a Kinect type unit 6132 may be used to measure the motion data. The lateral parallax adjustment process 6118 may translate the MFP for lateral motion parallax (e.g., horizontal viewpoint change in parallax). The focal planes may be scaled relative to each other for axial parallax from the new viewpoint. The lateral parallax adjusted MFPs for the right and left eyes may be received by the scale adjustment process for axial parallax 6122, 6124. The scaled MFP for the right eye may be received by the MFP rendering process 6126. Similarly, a scaled MFP for the left eye may be received by MFP rendering process 6128. The MFP is rendered and displayed for a new viewpoint / perspective for true stereoscopic viewing by user 6134.In some embodiments, synthetic motion parallax may involve scaling multiple focal planes relative to one another using information indicative of the axial motion of the user / viewer (such as motion tracking data).

[0337] FIG. 62 is a process diagram illustrating an example process for generating a composite stereo set of multiple focal planes, according to some embodiments. FIG. 62 shows a motion parallax MFP generation and rendering process 6200 for generating a composite stereo MFP stack. An MFP creation process 6206 receives a texture 6202 and a depth map 6204 for a monoscopic view. The MFP is sent to a lateral mismatch creation process 6208. A mismatch adjustment 6210 for orientation and amount (≧0%) may be received by the lateral mismatch adjustment process 6208 to adjust the viewpoint separation of the MFP. The rendering of the MFP is anchored to a position and orientation in space. The lateral mismatch creation process 6208 generates MFPs for the left and right eyes (e.g., using a baseline orientation), which are sent to a lateral parallax adjustment process 6212. In some embodiments, some baseline orientation before lateral or axial viewpoint changes may be selected to support mismatch creation. The motion tracking process 6214 may send user motion data to a lateral parallax adjustment process 6212 and to a scale adjustment process for axial parallax 6216, 6218. The motion data may be represented as x(t) and y(t) data captured relative to a nominal viewpoint and orientation for the MFP anchored in space. In some embodiments, for example, an inertial motion unit (IMU) 6224 and / or a Kinect type unit 6226 may be used to measure the motion data. The lateral parallax adjustment process 6212 may translate the MFP for lateral motion parallax (e.g., horizontal viewpoint change in parallax). The focal planes may be scaled relative to each other for axial parallax from the new viewpoint. The lateral parallax adjusted MFPs for the right and left eyes may be received by the scale adjustment process for axial parallax 6216, 6218. The scaled MFP for the right eye may be received by the MFP rendering process 6220. Similarly, a scaled MFP for the left eye may be received by MFP rendering process 6222. The MFP is rendered and displayed for the new viewpoint / perspective for synthesized stereoscopic viewing by user 6228.Capturing a monoscopic view may use less bandwidth in transmission. Furthermore, when synthesizing two viewpoints in the receiver, these two viewpoints may be formed at a selected baseline orientation and amount of disparity to support personal preferences for stereoscopic and depth perception at any head tilt. Although some artifacts may be produced when synthesizing the views, these artifacts may be reduced by using redistribution and / or expansion of the focal plane.

[0338] FIG. 63 is a process diagram illustrating an example process for generating and summing a set of multiple focal planes for a two-dimensional display, according to some embodiments. FIG. 63 shows a motion parallax MFP generation and rendering process 6300 for generating and summing an MFP stack for a 2D display. An MFP, disparity, and disparity formation process 6306 receives a texture 6302 and a depth map 6304 for a monoscopic view. A disparity adjustment 6308 for orientation and amount (≧0%) may be received by the MFP, disparity, and disparity formation process 6306 to adjust the viewpoint separation of the MFP. The MFP may be generated for a position and orientation in space. A motion tracking process 6310 may send user motion data to the MFP, disparity, and disparity formation process 6306. The motion data may be represented as x(t), y(t), and z(t) data captured relative to a nominal viewpoint and orientation for the MFP anchored in space. In some embodiments, an inertial motion unit (IMU) 6316, e.g., with near-eye display (NED) glasses, and / or a Kinect-type unit 6318, e.g., with a flat screen, may be used to measure the motion data. In some embodiments, the MFP, mismatch, and parallax formation process 6306 may perform one or more of the processes including forming the MFP, adjusting the MFP for lateral mismatch, adjusting the MFP for lateral motion parallax (horizontal viewpoint change of parallax), and scaling for axial parallax. The focal planes may be scaled relative to each other for axial parallax from the new viewpoint. The MFPs are summed by the MFP summation process 6312. The summed MFPs are rendered and displayed by a 2D NED or flat screen 6314 for viewing by the user 6320. In some embodiments, the 2D display may exhibit a motion parallax effect that follows the user motion, although there is no support for natural accommodation. In some embodiments, a stereoscopic three-dimensional (S3D) display may be used to render and display the summed MFPs.

[0339] FIG. 64, FIG. 65, and FIG. 66 illustrate some embodiments for projecting and summing MFPs to a 2D display. FIG. 64 is a process diagram illustrating an example process for generating and summing a set of multifocal planes for a monocular 2D display, according to some embodiments. FIG. 64 shows a motion parallax MFP generation and summation process 6400. An MFP formation process 6406 receives a texture 6402 and a depth map 6404 for a monoscopic view. The MFP is sent to a lateral parallax adjustment process 6408. The adjusted MFP is sent to a lateral parallax adjustment process 6408. A motion tracking process 6410 may send user motion data to the lateral parallax adjustment process 6408 and to a scale adjustment process 6412 for axial parallax. The motion data may be represented as x(t) and y(t) data captured relative to a nominal viewpoint and orientation for an MFP anchored in space. In some embodiments, an inertial motion unit (IMU) 6418 and / or a Kinect type unit 6416 may be used to measure the motion data. A lateral parallax adjustment process 6408 may translate the MFP for lateral motion parallax (horizontal viewpoint change in parallax). The focal planes may be scaled relative to each other for axial parallax from the new viewpoint. The lateral parallax adjusted MFP may be received by a scale adjustment for axial parallax process 6412 for scaling the MFP. The scaled MFP may be received by an MFP summation process 6414. The MFPs are summed by the MFP summation process 6414. The summed MFP is rendered and displayed by a 2D NED or flat screen 6420 (monocular 2D display) for viewing by a user 6422.

[0340] FIG. 65 is a process diagram illustrating an example process for generating and summing a set of multiple focal planes for a true stereoscopic three-dimensional (S3D) display, according to some embodiments. In some embodiments, baseline changes may be supported along with limitations of true stereoscopic 3D input. The input stereoscopic viewpoint (e.g., baseline orientation) may impose some limitations on subsequent viewpoint changes (e.g., lateral or axial changes). FIG. 65 shows a motion parallax MFP generation and summation process 6500 using true stereoscopic input. A right eye MFP formation process 6510 receives a texture 6502 and a depth map 6504 for the right eye. A left eye MFP formation process 6512 receives a texture 6502 and a depth map 6508 for the left eye. In some embodiments, the MFPs for the right and left eyes are sent to a lateral mismatch adjustment process 6514. A mismatch adjustment 6516 for direction and amount (≧0%) may be received by the lateral mismatch adjustment process 6514 to adjust the viewpoint separation of the MFPs. The use of stereoscopic MFP input may limit reorientation and adjustment of the baseline MFP. MFPs may be generated for a position and orientation in space. The adjusted MFPs for the left and right eyes are sent to a lateral parallax adjustment process 6518. A motion tracking process 6520 may send user motion data to the lateral parallax adjustment process 6518 and to scale adjustment processes 6522, 6524 for axial parallax. The motion data may be represented as x(t) and y(t) data captured relative to a nominal viewpoint and orientation for the MFP anchored in space. In some embodiments, an inertial motion unit (IMU) 6530 and / or a Kinect type unit 6532 may be used to measure the motion data. The lateral parallax adjustment process 6518 may translate the MFP for lateral motion parallax (horizontal viewpoint change in parallax). The focal planes may be scaled to each other for axial parallax from the new viewpoint. The lateral disparity adjusted MFPs for the right and left eyes may be received by scale adjustment for axial disparity processes 6522, 6524. The scaled MFP for the right eye may be received by a sum MFP for right eye process 6526.The MFP for the right eye is summed by a MFP sum for right eye process 6526. Similarly, the scaled MFP for the left eye may be received by a MFP sum for left eye process 6528. The MFP for the left eye is summed by a MFP sum for left eye process 6528. The summed MFP for both the right and left eyes is sent to a 2D NED or flat screen 6534 (monocular 2D display) for rendering and display to the user 6536.

[0341] FIG. 66 is a process diagram illustrating an example process for generating and summing a set of multiple focal planes for a synthetic stereoscopic three-dimensional (S3D) display, according to some embodiments. FIG. 66 shows a motion parallax MFP generation and summation process 6600. An MFP formation process 6606 receives a texture 6602 and a depth map 6604 for a monoscopic view. The MFP is sent to a lateral mismatch formation process 6608. A mismatch adjustment 6610 for orientation and amount (≧0%) may be received by the lateral mismatch adjustment process 6608 to adjust the viewpoint separation of the MFP. The MFP may be generated for a position and orientation in space. The lateral mismatch formation process 6608 generates MFPs for the left and right eyes, which are sent to a lateral parallax adjustment process 6612. The motion tracking process 6614 may send user motion data to a lateral parallax adjustment process 6612 and to a scale adjustment process for axial parallax 6616, 6618. The motion data may be represented as x(t) and y(t) data captured relative to a nominal viewpoint and orientation for the MFP anchored in space. In some embodiments, an inertial motion unit (IMU) 6626 and / or a Kinect type unit 6628 may be used to measure the motion data. The lateral parallax adjustment process 6612 may translate the MFP for lateral motion parallax (horizontal viewpoint change in parallax). The focal planes may be scaled relative to each other for axial parallax from the new viewpoint. The lateral parallax adjusted MFPs for the right and left eyes may be received by the scale adjustment process for axial parallax 6616, 6618. The scaled MFP for the right eye may be received by the MFP summation process 6620 for the right eye. The MFP for the right eye is summed by a MFP sum for right eye process 6620. Similarly, the scaled MFP for the left eye may be received by a MFP sum for left eye process 6622. The MFP for the left eye is summed by a MFP sum for left eye process 6622. The summed MFP for both the right and left eyes is sent to the S3D NED or flat screen 6624 for rendering and display to the user 6630.Support for motion parallax effects with 2D display data may be provided by summing the MFPs to produce 2D display data that can be displayed by an external stereoscopic or monoscopic display.

[0342] In some embodiments using a monoscopic display, an external motion tracking sensor may be used to track user motion, and the user may not be wearing shutter glasses. In some embodiments using a stereoscopic three-dimensional (S3D) display, a sensor (e.g., an IMU) that tracks user motion may be embedded in the shutter glasses, etc. In some embodiments using an S3D display, some vergence accommodation (VAC) conflict may occur. This conflict is small when the viewing distance is long compared to the display size (e.g., see normal viewing distance to a stereoscopic TV). In some embodiments, displaying the oriented focal plane images may include combining (e.g., summing) the shifted focal plane images to generate a stereoscopic image pair.

[0343] FIG. 67 is a process diagram illustrating an example process for generating multiple focal planes with a front-end processing path and a display-end path, according to some embodiments. FIG. 67 shows a motion parallax MFP generation and summation process 6700 using true stereo input. In some embodiments of the front-end process 6702, the texture 6706 and depth map 6708 may be received by MFP formation 6714 using a depth blending process. The MFP may be generated using depth blending (weighting) to reduce the quantization effect of the MFP in the depth dimension as previously described. In some embodiments of the front-end process 6704, the texture 6710 and depth map 6712 may be received by MFP formation 6716 using a redistribution process. The MFP may be generated using MFP redistribution as previously described.

[0344] In some embodiments, the depth blended and / or redistributed MFP may be sent to a mismatch formation process 6718. A mismatch adjustment 6720 for orientation and amount (≧0%) may be received by the mismatch formation process 6718 to adjust the viewpoint separation of the MFP. The adjusted MFP may be sent to a lateral parallax formation process 6722. A motion tracking process 6724 may send user motion data to the lateral parallax formation process 6722 and to a scale adjustment process 6726 for axial parallax. The motion data may be represented as x(t) and y(t) data captured relative to a nominal viewpoint and orientation for the MFP anchored in space. In some embodiments, an inertial motion unit (IMU) 6730 and / or a Kinect type unit 6732 may be used to measure the motion data. The lateral parallax formation process 6722 may translate the MFP for lateral motion parallax (horizontal viewpoint change of parallax). A scale adjustment process 6726 for axial parallax may receive the translated MFP for scaling. The focal planes may be scaled relative to each other for axial parallax from the new viewpoint. In some embodiments, the front end may include a direct input of focal planes 6742 (e.g., without a depth map) formed using known methods, such as the method disclosed in Non-Patent Document 1, and one or more of the processing stages 6718, 6720, 6722, 6724, 6726 may be used as described above.

[0345] In some embodiments of display path 1, the scaled MFP may be received by an MFP rendering process 6728 for display to a user 6738. In some embodiments of display path 2, the scaled MFP may be received by an MFP summation process 6734 for summing the received MFPs. The summed MFP may be received by a 2D NED or flat screen display 6736 for rendering and displaying the content to a user 6740.

[0346] 68A-68J are illustrations showing a series of exemplary right and left images for axial movement between each set image pair (e.g., FIG. 68C and FIG. 68D) according to some embodiments. FIG. 68A-68J show a series of crossed eye stereograms during axial movement using the arrangement shown in FIG. 58 and the focal plane size adjustment process described above. FIG. 68A-68J are modified versions of the images and their depth maps found on Non-Patent Document 12. For the exemplary sequence of FIG. 68A-69J, three MFPs with linear depth blending are used. A unit size of 2 pixels is used for image scaling adjustment. The test image used is a flower with a resolution of 157×436 pixels. For low resolution focal plane images, users may find a unit size of 2 pixels to be reasonable. Higher resolution focal plane images may use a larger unit size.

[0347] After forming the monocular MFP stack, the stereoscopic disparity is synthesized by shifting the focal planes horizontally. In the exemplary sequence of crossed eye stereograms, the three planes from rear to front (rear, middle, and front) are shifted by 0, 2, and 4 pixels, respectively. The shifting is to the right for the left eye views 6810, 6830, 6850, 6870, 6890, and to the left for the right eye views 6800, 6820, 6840, 6860, 6880. The synthesized total disparity corresponds to 1.8% of the image width. After shifting for stereoscopic disparity, the focal planes are scaled to create axial motion parallax, as illustrated in FIG. 67. Viewing the sequence from top to bottom in FIG. 68A to FIG. 68J corresponds to moving away from the scene. All views in FIG. 68A to FIG. 68J are formed by shifting from one monoscopic MFP stack. Lateral parallax is not illustrated in this example.

[0348] Applying the exemplary procedure described above for scaling the three focal plane test images, the vertical resolution heights listed in Tables 1 and 2 are obtained for five axial movements (+2, +1, 0, -1, -2 steps) around the nominal viewing point d0. The actual scaling factors can be obtained by dividing the table values ​​by Vres.

[0349] [Table 1]

[0350] [Table 2]

[0351] Table 3 shows the axial position d i Figure 1 shows the vertical resolution adjustment for PFback. The horizontal resolution is adjusted to preserve the aspect ratio before the adjustment. The unit size of the scaling is 2 pixels (with the corresponding focal plane edges symmetrically shifted by 1 pixel). The size of both the left eye view (focal plane image) and the right eye view (focal plane image) is the same as at the nominal viewpoint d0 (157x436 pixels in this example). During movement, the size remains unchanged (157x436 pixels) for the farthest focal plane PFback, which is treated as being at infinity.

[0352] As shown in Table 3, the axial movement (position d i ), the scaling is the same for both the left eye focal plane (shifted for mismatch) and the right eye focal plane. In the original test image, Vres=157 pixels. After scaling, the vertical resolutions for both the left and right focal planes (FP1, FP2, FP3) for five different viewpoints are shown in Table 3. Position d0 is the nominal viewpoint.

[0353] [Table 3]

[0354] The actual scaling factor for focal plane size is V res = 157. By applying these scaling values ​​to the three focal planes of the test image, a sequence of crossed-eye stereograms for axial motion is obtained, shown in Figs. 68A-J. Viewing the sequence from top to bottom in Figs. 68A-J corresponds to moving away from the scene. To provide a reference for scaling, black rectangles are shown in Figs. 68A and 68I. Each row pair (e.g., Figs. 68C and 68D) corresponds to positions along the same viewing direction (or axis). Figs. 68A-J demonstrate the motion parallax effect.

[0355] The step size in the axial direction (the physical distances corresponding to each row of Figures 68A-68J, such as Figures 68C and 68D, which form a row showing one cross-eye stereoscopic view of the sequence) may be determined by the resolution and unit size used. This example shows axial disparity (based on focal plane scaling) of a stereo pair with a compounded disparity (based on shifting the focal plane by a chosen amount). Lateral disparity may be created by applying the same (or similar) transformation as for forming the (lateral) disparity. Any 3D movement that combines lateral and axial components may be determined by applying the transformation (or an approximation in some embodiments) used in this example.

[0356] When viewing the focal planes from a nominal position, they will all be seen within the same solid / spatial angle. Since the unadjusted focal plane resolution is the same, one pixel occupies the same solid angle, and the focal planes for the perceived MFP rendering (or projection, as shown in Figures 68A-68J) may be summed. When adjusting focal plane size for synthesized motion parallax, the resolution and size may be adjusted before summing. Pixels may be shifted by sub-pixel amounts relative to each other. To ensure the sum of corresponding pixels is accurate, image interpolation may be performed to obtain sub-pixel values.

[0357] In the sequences shown in Figures 68A-68J, the zero parallax setting for the stereogram, and the vanishing point of the axial motion parallax, is the farthest focal plane. The axial motion parallax can be expanded by adjusting the size of all focal planes. This situation corresponds to placing the zero parallax plane behind all of the focal planes and using a scaling factor derived for one lower focal plane, e.g., using a scaling factor of FP2 for FP1 as shown in Table 2. The result is a stronger motion parallax effect with smaller user movements.

[0358] FIG. 69 is a flow chart illustrating an example process according to some embodiments. Some embodiments of the process 6900 may include receiving 6902 an input image with depth information. The process 6900 may further include mapping 6904 the input image to a plurality of focal plane images. The process 6900 may further include orienting 6906 the plurality of focal plane images using head orientation information to provide stereo disparity between the left and right eyes. The process 6900 may further include displaying 6908 the oriented plurality of focal plane images. Some embodiments may include a device with one or more motion tracking sensors, a processor, and a non-transitory computer readable medium storing instructions that, when executed on the processor, operate to perform the process 6900. In some embodiments, the process 6900 may be performed by a device having a multi-focal plane (MFP) display.

[0359] FIG. 70 is a flow chart illustrating an example process according to some embodiments. Some embodiments of the process 7000 may include receiving 7002 a description of three-dimensional (3D) content. The process 7000 may further include receiving 7004 information from a tracker indicative of the viewer's movement relative to a real-world environment. The process 7000 may further include synthesizing motion parallax by altering 7006 a multi-focal plane formed using the 3D content, for example, including texture and depth information, in response to receiving the information indicative of the viewer's movement. The process 7000 may further include rendering 7008 an image on a multi-focal plane display using the altered multi-focal plane rendering. Some embodiments may include a device with one or more motion tracking sensors, a processor, and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform the process 7000. In some embodiments, the process 7000 may be performed by a device having a multi-focal plane (MFP) display.

[0360] Although methods and systems according to some embodiments have been discussed in the context of virtual reality (VR) and augmented reality (AR), some embodiments may apply to mixed reality (MR), augmented reality (AR), and virtual reality (VR) contexts as well. Also, while the term "head mounted display (HMD)" is used herein in accordance with some embodiments, some embodiments may apply to, for example, wearable devices (which may or may not be head mounted) that are capable of VR, AR, and / or MR in some embodiments.

[0361] In some embodiments, the term "mismatch vector" refers to the maximum separation (with direction) of corresponding pixels in a synthetic stereo pair formed, for example, by shifting and projecting the MFP. Due to the geometry, the maximum shift occurs to the focal plane closest to the viewer. The amount and direction of this maximum shift is called the "mismatch vector."

[0362] In the literature, the spatial separation of each corresponding pixel in the two images (left and right) may be represented by a brightness value. This is usually called the "mismatch image". In some embodiments of the present application, unlike the literature, the "mismatch image" is, for example, an MFP stack that is skewed by an amount (which may be indicated by a mismatch vector), projected, and summed to a viewing point.

[0363] In some embodiments, the 3D content includes, for example, a depth map and a texture image. In some embodiments, the texture image includes a projection of the 3D model and / or content to a selected viewpoint. In some embodiments, the rendering to the MFP display may be in monoscopic (e.g., a monoscopic MFP stack), true stereo (e.g., two monoscopic MFP stacks for right and left eyepoints correspondingly) or composite stereo (e.g., two MFP stacks composited with a shift) format.

[0364] An exemplary method according to some embodiments may include receiving an input image with depth information, mapping the input image to multiple focal plane images using the depth information, orienting the multiple focal plane images using head orientation information to provide stereoscopic disparity between the left and right eyes, and displaying the oriented multiple focal plane images.

[0365] In some embodiments of the example method, at least receiving the input image and displaying the oriented multiple focal plane images are performed by a multi-focal plane (MFP) display.

[0366] In some embodiments of the example method, orienting the multiple focal plane images may include rotating one or more of the multiple focal plane images.

[0367] In some embodiments of the example method, rotating one or more of the multiple focal plane images may include rotating a mismatch vector for one or more of the focal plane images relative to a vector parallel to the horizon.

[0368] In some embodiments of the example method, orienting the multiple focal plane images may include shifting one or more of the multiple focal plane images.

[0369] In some embodiments of the example method, shifting one or more of the multiple focal plane images may include scaling a disparity vector for the respective one or more focal plane images.

[0370] In some embodiments of the example method, shifting one or more of the multiple focal plane images may include rotating one or more of the multiple focal plane images relative to a fixed origin.

[0371] In some embodiments of the example method, shifting one or more of the multiple focal plane images includes moving one or more of the multiple focal plane images by an offset relative to a fixed origin.

[0372] Some embodiments of the example methods may further include processing the multiple focal plane images to address image mismatch caused by shifting the focal plane images (e.g., to reduce disocclusions or holes).

[0373] In some embodiments of the example method, processing the multiple focal plane images may include filtering one or more of the focal plane images.

[0374] In some embodiments of the example method, processing the multiple focal plane images includes determining depth blending weights for at least one of the multiple focal plane images and generating at least one of the multiple focal plane images using the respective depth blending weights.

[0375] In some embodiments of the example method, generating at least one of the plurality of focal plane images includes multiplying at least one of the plurality of textures with a respective depth blending weight.

[0376] In some embodiments of the example method, displaying the oriented multiple focal plane images may include combining the shifted focal plane images to generate a stereoscopic image pair.

[0377] Some embodiments of the example method may further include measuring motion tracking sensor readings of the viewer's position relative to the real-world environment to generate head orientation information.

[0378] Some embodiments of the example method may further include synthesizing motion parallax by altering one or more of the plurality of focal plane images in response to generating the head orientation information.

[0379] In some embodiments of the example method, the synthesized motion parallax may include scaling one or more of the multiple focal plane images relative to each other using head orientation information, which may indicate axial motion.

[0380] Some embodiments of the example method may further include filtering the input image for low frequency content and redistributing the low frequency content into multiple focal plane images.

[0381] Some embodiments of the example method may further include filtering the input image for high frequency content and decomposing the high frequency content into multiple focal plane images.

[0382] An exemplary apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform one of the exemplary methods described above.

[0383] An exemplary method according to some embodiments may include receiving a description of three-dimensional (3D) content, receiving information from a tracker indicative of a viewer's movement relative to a real-world environment, synthesizing motion parallax by altering multi-focal planes of the 3D content in response to receiving the information indicative of the viewer's movement, and rendering an image on a multi-focal plane display using the altered multi-focal plane rendering.

[0384] In some embodiments of the exemplary method, the synthesized motion parallax may include scaling the multi-focal planes relative to each other using information indicative of the viewer's motion, which may be axial.

[0385] An exemplary apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform one of the exemplary methods described above.

[0386] An exemplary method according to some embodiments may include determining a plurality of focal plane images to be mapped to an input image, orienting the plurality of focal plane images using head orientation information to provide stereoscopic disparity between the left and right eyes, and displaying the oriented plurality of focal plane images.

[0387] In some embodiments of the example method, determining the multiple focal plane images includes receiving the multiple focal plane images.

[0388] In some embodiments of the example method, determining the multiple focal plane images includes mapping the input image to the multiple focal plane images to generate the multiple focal plane images.

[0389] An exemplary apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform one of the exemplary methods described above.

[0390] An exemplary method according to some embodiments may include receiving three-dimensional (3D) video content, receiving a viewer orientation signal, determining image data for one or more focal planes of a display using the 3D video content and the viewer orientation signal, and rendering the one or more focal plane images on the display using the image data.

[0391] An exemplary apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform the exemplary methods described above.

[0392] An exemplary method according to some embodiments may include receiving a description of capabilities of a multi-focal plane (MFP) display, receiving three-dimensional (3D) video content, receiving a viewer orientation signal, calculating image data for one or more focal planes of the MFP display using the 3D video content and the viewer orientation signal, and rendering one or more MFP images on the MFP display using the calculated image data.

[0393] In some embodiments of the example method, the 3D video content may include one or more texture images and corresponding depth maps.

[0394] Some embodiments of the example method may further include filtering the depth map.

[0395] In some embodiments of the example method, calculating image data for the one or more focal planes may include extracting one or more images from the 3D video content, calculating depth blending weights for the one or more focal planes, generating one or more MFP weight maps by remapping the depth map by the depth blending weights, multiplying the one or more images by each MFP weight map to form a respective focal plane image, and performing a spatial shift for each focal plane image.

[0396] In some embodiments of the example method, the description of the capabilities of the MFP display may include the number and positions of focal planes.

[0397] In some embodiments of the example method, calculating image data for one or more focal planes may include calculating one or more MFP images for one or more focal planes of the MFP display using the 3D video content and shifting the one or more calculated MFP images.

[0398] In some embodiments of the example method, shifting the one or more calculated MFP images may include scaling a disparity vector for one or more of the MFP images.

[0399] In some embodiments of the example method, shifting the one or more calculated MFP images may include rotating a mismatch vector for one or more of the MFP images relative to a vector parallel to the horizon.

[0400] In some embodiments of the example method, shifting the one or more calculated MFP images may include translating one or more of the MFP images.

[0401] In some embodiments of the example method, calculating image data for one or more focal planes may include calculating one or more MFP images for one or more focal planes of the MFP display using the 3D video content, 3D warping the one or more calculated MFP images, and performing hole filling on the one or more calculated MFP images.

[0402] An exemplary method according to some embodiments may include receiving three-dimensional video content, calculating one or more multi-focal plane (MFP) images for a number and positions of multiple focal planes using the three-dimensional video content, receiving a viewer orientation signal, calculating one or more mismatch images for each view of a stereoscopic display using the one or more MFP images and the viewer orientation signal, and rendering the one or more mismatch images on the stereoscopic display.

[0403] In some embodiments of the example method, the three-dimensional video content may include one or more texture images and corresponding depth maps.

[0404] Some embodiments of the example method may further include filtering the depth map.

[0405] In some embodiments of the example method, calculating image data for the one or more focal planes may include extracting one or more images from the 3D video content, calculating depth blending weights for the one or more focal planes, generating one or more MFP weight maps by remapping the depth map by the depth blending weights, multiplying the one or more images by each MFP weight map to form a respective focal plane image, and performing a spatial shift for each focal plane image.

[0406] In some embodiments of the example method, rendering the one or more mismatched images on a stereoscopic display includes summing each spatially shifted focal plane to generate a stereoscopic image pair for each of the one or more mismatched images.

[0407] Some embodiments of the example method may further include receiving one or more pixel values ​​corresponding to one or more pixels in one of the multi-focal plane (MFP) images and adjusting the one or more pixel values ​​by a corresponding distance in the depth map.

[0408] In some embodiments of the example method, the description of the capabilities of the MFP display may include the number and positions of focal planes.

[0409] In some embodiments of the example method, the viewer orientation signal may indicate a viewer tilt, and calculating one or more disparity images for each view of the stereoscopic display may use a rotated disparity vector that corresponds to the indicated viewer tilt.

[0410] In some embodiments of the example method, calculating one or more MFP images for the number and positions of multiple focal planes may include calculating one or more MFP images for each focal plane position using 3D video content, 3D warping the one or more calculated MFP images, and performing hole filling on the one or more calculated MFP images.

[0411] An exemplary method according to some embodiments may include receiving three-dimensional video content, calculating one or more multi-focal plane (MFP) images for a number and positions of the multiple focal planes using the three-dimensional video content, measuring motion tracking sensor readings of a motion tracking sensor, generating a viewer orientation signal from the motion tracking sensor readings, calculating one or more mismatch images for each view of a shutter glasses display using the one or more MFP images and the viewer orientation signal, and rendering the one or more mismatch images on the shutter glasses display.

[0412] An exemplary device according to some embodiments may include one or more motion tracking sensors, a processor, and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform the processes of receiving three-dimensional video content, calculating one or more multi-focal plane (MFP) images for a number and position of the multi-focal planes using the three-dimensional video content, measuring motion tracking sensor readings of the motion tracking sensor, generating a viewer orientation signal from the motion tracking sensor readings, calculating one or more mismatch images for each view of a shutter glasses display using the one or more MFP images and the viewer orientation signal, and rendering the one or more mismatch images on the shutter glasses display.

[0413] An exemplary device according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform processes of receiving a description of the capabilities of a multi-focal plane (MFP) display, receiving three-dimensional video content, receiving a viewer orientation signal, calculating image data for one or more focal planes of the MFP display using the three-dimensional video content and the viewer orientation signal, and rendering one or more MFP images on the MFP display using the calculated image data.

[0414] An exemplary method according to some embodiments may include receiving three-dimensional (3D) video content, filtering the 3D video content into high frequency content and low frequency content, decomposing the high frequency content into one or more high frequency computed images, redistributing the low frequency content into the one or more high frequency computed images to generate one or more redistributed focal planes, and rendering one or more multi-focal plane images corresponding to the one or more redistributed focal planes.

[0415] In some embodiments of the example method, the 3D video content may include one or more texture images and corresponding depth maps.

[0416] In some embodiments of the example method, decomposing the high frequency content into one or more high frequency calculated images may include calculating image data for one or more focal planes of the MFP display using the high frequency content and decomposing the calculated image data into one or more high frequency calculated images.

[0417] In some embodiments of the example method, calculating image data for the one or more focal planes may include extracting one or more images from the 3D video content, calculating depth blending weights for the one or more focal planes, generating one or more MFP weight maps by remapping the depth map by the depth blending weights, multiplying the one or more images by each MFP weight map to form a respective focal plane image, and performing a spatial shift for each focal plane image.

[0418] In some embodiments of the example method, the depth blending weights may be a box filter.

[0419] In some embodiments of the example method, the depth blending weights may be a linear ramp.

[0420] In some embodiments of the example method, the depth blending weights may be polynomial functions.

[0421] In some embodiments of the example method, the depth blending weight may be a sinusoidal function.

[0422] In some embodiments of the example method, redelivering the low frequency content into one or more high frequency calculated images may include dividing pixel values ​​of the low frequency content by N and combining the divided low frequency content with each of the N high frequency calculated images to generate one or more redelivered focal planes, where N is a positive integer.

[0423] An exemplary method according to some embodiments may include receiving three-dimensional (3D) video content, calculating image data for N focal planes of an MFP display using the 3D video content, low-pass filtering the 3D video content to generate low frequency 3D video content, dividing pixel values ​​of the low frequency 3D video content by N, re-distributing the divided low frequency 3D video content to the N focal planes of the calculated image data to generate N re-distributed images, and rendering N sets of MFP images on the MFP display using the N re-distributed images, where N is a positive integer.

[0424] In some embodiments of the example method, the three-dimensional video content may include one or more texture images and corresponding depth maps.

[0425] In some embodiments of the example method, re-delivering the divided low frequency 3D video content may include low-pass filtering each of the N focal planes of the calculated image data, summing the divided low frequency video content with each of the N focal planes of the calculated image data to generate N sets of summed image data, and subtracting each of the N sets of low-pass filtered calculated image data from each of the summed image data to generate the N sets of re-delivered image data.

[0426] In some embodiments of the example method, calculating image data for the N focal planes may include extracting N images from the 3D video content, calculating depth blending weights for the N focal planes, generating N multi-focal plane (MFP) weight maps by remapping the depth map by the depth blending weights, multiplying the N images by respective MFP weight maps to form respective focal plane images, and performing a spatial shift for each of the N focal plane images, where N is a positive integer.

[0427] In some embodiments of the example method, the depth blending weights may be a box filter.

[0428] In some embodiments of the example method, the depth blending weights may be a linear ramp.

[0429] In some embodiments of the example method, the depth blending weights may be polynomial functions.

[0430] In some embodiments of the example method, the depth blending weight may be a sinusoidal function.

[0431] In some embodiments of the example method, redelivering the low frequency content to one or more high frequency focal planes may include dividing pixel values ​​of the low frequency content by N and combining the divided low frequency content with each of the N sets of high frequency calculated image data to generate one or more sets of redelivered image data, where N is a positive integer.

[0432] An exemplary device according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform the processes of receiving three-dimensional (3D) video content, filtering the 3D video content into high frequency content and low frequency content, calculating image data for one or more focal planes of an MFP display using the high frequency content, decomposing the calculated image data into N sets of high frequency calculated image data, dividing pixel values ​​of the low frequency content by N, redistributing the divided low frequency content into N high frequency calculated images, and rendering N sets of MFP images on the MFP display using the N redistributed images, where N is a positive integer.

[0433] An exemplary method according to some embodiments may include receiving three-dimensional (3D) video content, filtering the 3D video content into high frequency content and low frequency content, decomposing the high frequency content into one or more high frequency focal planes, redistributing the low frequency content to the one or more high frequency focal planes to generate one or more redistributed focal planes, and rendering one or more multi-focal plane images using the one or more redistributed focal planes.

[0434] An exemplary method according to some embodiments may include receiving an image represented as texture plus depth; forming a first decomposition into a plurality of N focal plane images based on depth plane values ​​of the image; creating a scaled low frequency representation by applying a low pass filter to the texture image and scaling pixel values ​​by dividing by N; and for each of the plurality of N focal plane images, calculating a plurality of re-distributed focal plane images, which may include calculating a low pass representation of each focal plane image, adding the scaled low frequency representation to each focal plane image to generate an adjusted focal plane image, and subtracting the calculated low pass representation of each focal plane image from the adjusted focal plane image to generate the re-distributed focal plane image.

[0435] In some embodiments of the example method, the multiple redistributed focal plane images may be rendered on a multi-focal plane display.

[0436] In some embodiments of the exemplary method, the multiple redistributed focal plane images may be used as input to generate a stereo pair projection on a stereoscopic display.

[0437] An exemplary method according to some embodiments may include receiving an image represented as a multi-focal plane stack of images composed of a plurality of N focal plane images based on depth plane values ​​of the image; creating a scaled low frequency representation by applying a low pass filter to the image and scaling pixel values ​​by dividing by N; and for each of the plurality of N focal plane images, calculating a plurality of re-distributed focal plane images, which may include calculating a low pass representation of each focal plane image, adding the scaled low frequency representation to each focal plane image to generate an adjusted focal plane image, and subtracting the calculated low pass representation of each focal plane image from the adjusted focal plane image to generate the re-distributed focal plane image.

[0438] In some embodiments of the example method, the multiple redistributed focal plane images may be rendered on a multi-focal plane display.

[0439] In some embodiments of the exemplary method, the multiple redistributed focal plane images may be used as input to generate a stereo pair projection on a stereoscopic display.

[0440] An exemplary method according to some embodiments may include calculating image data for one or more focal planes of an MFP display using three-dimensional video content and a viewer orientation signal, and rendering one or more MFP images on the MFP display using the calculated image data.

[0441] An exemplary device according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed on the processor, operate to perform a process of calculating image data for one or more focal planes of an MFP display using three-dimensional video content and a viewer orientation signal, and rendering one or more MFP images on the MFP display using the calculated image data.

[0442] An exemplary method according to some embodiments may include rendering to a multi-focal plane display and creating stereoscopic video in response to a user's head orientation.

[0443] An exemplary method according to some embodiments may include receiving 3D video content that includes a depth map.

[0444] An exemplary method according to some embodiments may include receiving 3D video content that comprises texture images.

[0445] An exemplary method according to some embodiments may include receiving an orientation signal from a viewer.

[0446] An exemplary method according to some embodiments may include receiving a description of multi-focal plane display capabilities (including the number and positions of focal planes).

[0447] An exemplary method according to some embodiments may include calculating image data for a plane of a multi-focal plane display using the orientation signal.

[0448] An exemplary method according to some embodiments may include calculating image data for a selected number and positions of multiple focal planes.

[0449] An exemplary method according to some embodiments may include computing a disparity image (or, for example, an image with disparity) for each view of a stereoscopic display using multiple focal planes and orientation signals.

[0450] An exemplary method according to some embodiments may include rendering a multi-focal plane / mismatch image on a display.

[0451] Table 4 shows the mapping of grayscale versions of the native images to the black and white line drawings of Figures 71-92.

[0452] [Table 4]

[0453] It should be noted that the various hardware elements of one or more of the described embodiments are referred to as "modules" that perform (i.e., implement, execute, etc.) various functions described herein with respect to the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more ASICs, one or more FPGAs, one or more memory devices) that is deemed suitable by one of ordinary skill in the art for a given implementation. It should be noted that each described module may also include instructions executable to perform one or more functions described as being performed by the respective module, which may take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., and may be stored in any suitable non-transitory computer-readable medium or media, such as those commonly referred to as RAM, ROM, etc.

[0454] Although features and elements have been described above in certain combinations, one skilled in the art will appreciate that each feature or element can be used alone or in any combination with other features and elements. The methods described herein can be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, ROM, RAM, registers, cache memory, semiconductor memory devices, internal hard disks and removable disks, magnetic media such as magneto-optical media, and optical media such as CD-ROM disks and DVDs. A processor in association with software can be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. generating a plurality of focal plane images using a description of three-dimensional (3D) content; receiving motion information, the motion information including information describing a user's motion with respect to a real-world environment; shifting one or more focal plane images of the plurality of focal plane images to generate lateral motion parallax in response to the motion information indicative of lateral movement of the user; scaling one or more focal plane images of the plurality of focal plane images to generate axial motion parallax in response to the motion information indicative of axial motion of the user; Rendering a view of 3D content using the shifted and scaled focal plane image; and A method for providing the above.

2. The method of claim 1 , wherein the movement information is received from a motion sensor attached to a head-worn device of the user.

3. The method of claim 1 , wherein the movement information is received from an external sensor.

4. The method of claim 1 , wherein the movement information includes information describing movement of the user's head.

5. 2. The method of claim 1 , wherein scaling one or more focal plane images of the plurality of focal plane images comprises scaling the one or more focal plane images to a larger size in response to motion information indicating that the user is axially approaching the one or more focal plane images.

6. 2. The method of claim 1 , wherein scaling one or more focal plane images of the plurality of focal plane images comprises scaling the one or more focal plane images to a smaller size in response to motion information indicating the user is moving axially away from the one or more focal plane images.

7. 2. The method of claim 1, wherein scaling one or more focal plane images of the plurality of focal plane images comprises scaling the one or more focal plane images using a scaling factor that varies based on a relative distance from a focal plane to a user.

8. The method of claim 1 , wherein at least one of the plurality of focal plane images has a zero parallax setting for axial motion of the user.

9. Rendering a view of the 3D content includes: generating a 2D image view of the 3D content from the shifted and scaled focal plane image; displaying the 2D image view using a 2D display; The method of claim 1 , comprising:

10. The method of claim 1 , wherein rendering a view of the 3D content includes displaying the shifted and scaled focal plane image using a multi-focal plane (MFP) display.

11. Rendering a view of the 3D content includes: generating a left-eye view and a right-eye view from the shifted and scaled focal plane image; displaying the left eye view and the right eye view using a stereoscopic display device; The method of claim 1 , comprising:

12. Producing the plurality of focal plane images includes: Receiving an input image having depth information; filtering the input image for low frequency content; redistributing the low frequency content from the filtered input image to the plurality of focal plane images; The method of claim 1 , comprising:

13. A processor; A memory for storing instructions; the instructions, when executed by the processor, generating a plurality of focal plane images using a description of three-dimensional (3D) content; receiving motion information, the motion information including information describing a user's motion with respect to a real-world environment; shifting one or more focal plane images of the plurality of focal plane images to generate lateral motion parallax in response to the motion information indicative of lateral movement of the user; scaling one or more focal plane images of the plurality of focal plane images to generate axial motion parallax in response to the motion information indicative of axial motion of the user; Rendering a view of 3D content using the shifted and scaled focal plane image; and An apparatus that causes an apparatus to execute the above.

14. The device of claim 13 , further comprising a motion sensor.

15. The device of claim 13 , wherein the device comprises a multi-focal plane (MFP) display configured to render a view of the 3D content.

16. Receiving a description of three-dimensional (3D) content; generating a plurality of focal plane images for the 3D content based on the description; receiving motion information from a motion tracking device indicative of a viewer's movement relative to a real-world environment; synthesizing motion parallax information by modifying one or more focal plane images of the plurality of focal plane images in response to receiving the motion information; generating a left-eye image and a right-eye image from the modified one or more focal plane images; displaying the left eye image and the right eye image using a stereoscopic display device; A method for providing the above.

17. 17. The method of claim 16, wherein synthesizing the motion parallax information includes shifting the one or more focal plane images to generate lateral motion parallax in response to the motion information indicative of lateral motion of the viewer.

18. 17. The method of claim 16, wherein synthesizing the motion parallax information includes scaling the one or more focal plane images to generate axial motion parallax in response to the motion information indicative of axial motion of the viewer.

19. The method of claim 16 , wherein the motion tracking device includes a motion sensor attached to a head-worn device of the viewer.

20. The method of claim 16 , wherein the motion tracking device includes a motion sensor external to the stereoscopic display device.

21. The method of claim 16 , wherein the motion tracking device includes a motion sensor attached to the stereoscopic display device.

22. The method of claim 16 , wherein the motion information is indicative of a head movement of the viewer.

23. The method of claim 16 , wherein the stereoscopic display device comprises a multi-focal plane (MFP) display.

Images