Volumetric immersive experience with multiple views

Abstract

A multiview input image covering a plurality of sampled views is received. A multiview layered image stack is generated from the multiview input image. A target view of a viewer for an image space depicted by the multiview input image is determined based on the user pose data. The target view is used to select a user pose selected sampled view from among the plurality of sampled views. The layered image for the user pose selected sampled view, along with an alpha map and a beta scale map for the user pose selected sampled view, are encoded into a video signal to cause a recipient device of the video signal to generate a display image for rendering on an image display.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to the following priority applications: U.S. Provisional Application No. 63 / 305,641, filed February 01, 2022 (Docket No. D21071USP1), and European Patent Application No. 22156127.7, filed February 10, 2022, the contents of each of which are incorporated herein by reference in their entirety.

[0002] technology The present disclosure relates generally to image processing operations. More particularly, certain embodiments of the present disclosure relate to video codecs. [Background technology]

[0003] As used herein, the term "dynamic range" (DR) may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, e.g., from darkest black (shadows) to brightest white (highlights). In this sense, DR relates to "scene-based" intensities. DR may also relate to the ability of a display device to fully or approximately render an intensity range of a particular width. In this sense, DR relates to "display-based" intensities. Unless a particular meaning is expressly specified as having a special significance at any point in the description herein, it should be presumed that the terms may be used in either sense, e.g., interchangeably.

[0004] As used herein, the term high dynamic range (HDR) refers to a DR width that spans about 14-15 orders of magnitude or more of HVS. In practice, the DR where humans can simultaneously perceive a wide range of intensity ranges may be somewhat truncated compared to HDR. As used herein, the terms enhanced dynamic range (EDR) or visual dynamic range (VDR), individually or interchangeably, refer to the DR perceivable in a scene or image by an HVS or observer, including eye movements, allowing for some light adaptation changes across the scene or image. As used herein, EDR may refer to a DR that spans 5-6 orders of magnitude. Although perhaps somewhat narrower in relation to true scene-based HDR, EDR still represents a wide DR width and may be referred to as HDR.

[0005] In practice, an image contains one or more color components / channels of a color space (e.g., luma Y and chroma Cb and Cr), each of which is represented with n bits of precision per pixel (e.g., n=8). Using non-linear luminance coding (e.g., gamma encoding), images with n≦8 (e.g., color 24-bit JPEG images) are considered standard dynamic range images, while images with n>8 are considered enhanced dynamic range images.

[0006] A reference electro-optical transfer function (EOTF) for a given display characterizes the relationship between the color values ​​of the input video signal (e.g., luminance represented in a codeword of the codeword representing the image) and the output screen color values ​​produced by the display (e.g., screen luminance represented in a display drive value of the display drive values ​​used to render the image). For example, “Video Signal Processing: A Reference for a Flat Panel Display,” IEEE Trans. Computer Vision, vol. 13, No. 10, 2003, which is incorporated herein by reference in its entirety, defines a reference EOTF for flat panel displays. Given a video stream, information about its EOTF can be embedded in the bitstream as (image) metadata. The term “metadata” in this specification relates to any auxiliary information transmitted as part of the encoded bitstream that assists a decoder to render a decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, as described herein. [Non-Patent Document 1] ITU Rec. ITU-R BT.1886 "Reference electro-optical transfer function for flat panel displays used in HDTV studio production" (March 2011)

[0007] As used herein, the term "PQ" refers to perceptual luminance amplitude quantization. HVS responds highly nonlinearly to increasing light levels. A human's ability to see a stimulus is affected by the luminance of the stimulus, the magnitude of the stimulus, the spatial frequencies that make up the stimulus, and the luminance level to which the eye is adapted at the particular moment the stimulus is viewed. In some embodiments, the perceptual quantization function maps linear input gray levels to output gray levels that better match contrast sensitivity thresholds in the human visual system. Exemplary PQ mapping functions are described in SMPTE, "Perceptual Quantization of Brightness and Brightness," which is incorporated herein by reference in its entirety. Here, for all luminance levels (e.g., stimulus levels), given a fixed stimulus size, the smallest visible contrast increment at that luminance level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to the HVS model). [Non-Patent Document 2] SMPTE ST 2084:2014 "High Dynamic Range EOTF of Mastering Reference Displays"

[0008] 302~1,000cd / m 2Displays supporting luminance of 1,000 or 1,000 nits represent low dynamic range (LDR), also called standard dynamic range (SDR), as opposed to EDR (or HDR). EDR content may be displayed on EDR displays supporting higher dynamic range (e.g., 1,000 nits to 5,000 nits or more). Such displays may be defined using an alternative EOTF that supports high luminance capabilities (e.g., 0 to 10,000 nits or more). Exemplary (e.g., HDR, Hybrid Log-Gamma or HLG) EOTFs are defined in SMPTE 2044 and IEEE 802.11b, IEEE 802.11b, IEEE 802.11b, IEEE 802.11c, IEEE, 802.11b ... [Non-Patent Document 3] Rec. ITU-R BT.2060, “Image parameter values ​​for high dynamic range television for use in production and international program exchange” (06 / 2017) [Non-Patent Document 4] ITU Rec. ITU-R BT.3040-2, “Parameter values ​​for ultra-high definition television systems for production and international program exchange” (October 2015)

[0009] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art by mere inclusion in this section. Similarly, it should not be assumed that issues identified with respect to one or more approaches were recognized in any prior art based on this section, unless otherwise indicated. [Brief description of the drawings]

[0010] The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference symbols refer to similar elements and in which: [Figure 1A] 1 illustrates an exemplary image process flow for generating a layered image stack. [Figure 1B] 1 illustrates an exemplary upstream device. [Figure 1C] 1 illustrates an exemplary downstream recipient device.

[0011] [Figure 2A] 13 illustrates an example set of user pose selected sampled views. [Figure 2B] 1 illustrates exemplary SDR image data and metadata in a layered image stack. [Figure 2C] 1 illustrates example HDR image data and metadata in a layered image stack.

[0012] [Figure 3A] 3 shows an example of image layers in a layered image. [Figure 3B] 3 shows an example of image layers in a layered image.

[0013] [Figure 4A] An exemplary process flow is shown. [Figure 4B] An exemplary process flow is shown.

[0014] [Diagram 5] 1 illustrates an exemplary hardware platform upon which a computer or computing device described herein may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Exemplary embodiments relating to a volumetric immersive experience with multiple views are described herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail in order to avoid unnecessarily obscuring, obscuring, or obfuscating the present invention.

[0016] Exemplary embodiments are described herein according to the following outline. 1. Overview 2. Representation and Delivery of Multi-View Image Data 3. Streaming of layer configuration images based on user pose selection 4. User pose selected sampled views 5. Exemplary Process Flow 6. Implementation Mechanism - Hardware Overview 7. Equivalents, Extensions, Substitutions and Miscellaneous

[0017] 1. Overview This overview presents a basic description of some aspects of exemplary embodiments of the present invention. It should be noted that this overview is not an extensive or exhaustive summary of aspects of the exemplary embodiments. Moreover, it should be noted that this overview is not intended to be understood as identifying any particularly important aspects or elements of the exemplary embodiments, nor as defining any scope of the exemplary embodiments in particular or the present invention in general. It should be understood that this overview merely presents some concepts related to the exemplary embodiments in a condensed and simplified form and is merely a conceptual prelude to the more detailed description of the exemplary embodiments that follows. It should be noted that although separate embodiments are discussed herein, any combination of the embodiments and / or partial embodiments discussed herein may be combined to form further embodiments.

[0018] The techniques described herein may be used to support volumetric immersive experiences using image data covering or representing multiple sampled views. An upstream device may receive as input a sequence of multi-view input images, each multi-view input image in the sequence covering multiple sampled views for a time point in the sequence of successive time points. The upstream device may generate a sequence of layered image stacks from the sequence of multi-view input images to support efficient video coding to support volumetric immersive experiences.

[0019] Each layered image stack in the sequence of layered image stacks can include, but is not limited to, a stack of layered images, alpha maps, beta scale maps, etc. Each such layered image stack may include multiple SDR layered images covering or representing multiple sampled views of a time point in the sequence of consecutive time points.

[0020] Each SDR layered image in the multiple SDR layered images in the layered image stack corresponds to or covers a respective sampled view in the multiple sampled views and includes one or more (e.g., 16, 32, etc.) SDR image layers (or one or more image pieces) in different depth sub-ranges within multiple mutually exclusive depth sub-ranges covering the entire depth range for each sampled view.

[0021] An alpha map, as described herein, includes alpha values ​​stored in a data frame or array (e.g., two-dimensional). The alpha values ​​can be used in an alpha compositing operation to combine multiple image layers of different depths or different depth subranges. For example, an alpha map for an SDR layered image of each sampled view includes alpha values ​​that can be used to perform an alpha compositing operation on the SDR image layers to generate an SDR non-layered (or single layer) image from the image layers of the SDR layered image as seen from each sampled view. As used herein, a non-layered (or single layer) image refers to an image that is not divided into multiple image layers.

[0022] The beta scale map described herein includes beta scaling data stored in a data frame or array (e.g., two-dimensional). The beta scaling data can be used in a scaling operation to integrate other image processing operations, such as a reshaping operation to convert an input image of a first dynamic range to an output image of a second dynamic range different from the first dynamic range. For example, the beta scale map for the SDR layered image of each sampled view includes beta scaling data that is used to select a scaling method and perform a scaling operation, such as applying scaling to the SDR image layer with operation parameters defined or specified in the beta scaling data, to generate one or more corresponding HDR image layers for each sampled view. Additionally, optionally, or alternatively, these HDR image layers can be alpha composited into an HDR non-layered (or single layer) image as seen from each sampled view using the same alpha value in the alpha map.

[0023] A viewer / user's real-time or near real-time viewing position and / or viewing direction relative to an image display can be monitored or tracked using real-time or near real-time user pose data collected while a display image derived from a layered image stack as described herein is being simultaneously rendered to the viewer / user. The user pose data can be generated as a result of applying machine learning (ML)-based face tracking, face detection and user pose analysis to images of the viewer / user captured in real-time or near real-time with a camera at a fixed spatial position relative to the image display while the viewer / user is viewing rendered image content on the image display.

[0024] A real-time or near real-time target view of a viewer / user for a given time instant (e.g., a new view not covered by any sampled views) may be determined based on the user pose data.

[0025] The upstream device may use the viewer / user's target view to select a subset of SDR layered images from the multiple SDR layered images in the layered image stack for a given time point, the subset covering the subset of sampled views. This subset may be referred to as a user pose selected set of sampled views. In some operational scenarios, the user pose selected set of sampled views may include the closest sampled views to the viewer / user's target view. Additionally, optionally, or alternatively, the user pose selected set of sampled views may include one or more reference sampling views, such as those corresponding to a center of symmetry or a furthest view, which may be used to provide reference or additional information for depth data generation or hole filling operations regarding newly unoccluded image details present in the closest sampling views.

[0026] A downstream recipient device may receive and decode the encoded video signals, such as the user pose selected layered images for a set of user pose selected sampled views, the alpha map and the beta scale map for the user pose selected layered images, etc. Based on the viewer / user's current view, the alpha values ​​in the alpha map may be adjusted from the user pose selected sampled views to the viewer / user's current view, resulting in adjusted alpha values ​​that constitute the adjusted alpha map. The viewer / user's current view may be the same as the target view used to select the user pose selected sampled views, or may be a different view from the target view (e.g., slightly shifted, etc.).

[0027] An SDR image for the current view may be generated or reconstructed from the SDR image layers of the user pose selected set of sampled views using an alpha compositing operation based on the adjusted alpha values ​​in the adjusted alpha map. These SDR images may be blended into a final SDR non-layered (or single layer) image and used to generate an SDR display image for rendering on an image display when the image display is operative to render SDR video content.

[0028] HDR image layers for the user pose selected set of sampled views may be generated or reconstructed from the SDR image layers of the user pose selected set of sampled views using a beta scaling operation. In addition, an HDR image for the current view may be generated or reconstructed from the HDR image layers of the user pose selected set of sampled views using an alpha compositing operation based on the adjusted alpha values ​​in the adjusted alpha map. These HDR images may be blended into a final HDR non-layered (or single layer) image and used to generate an HDR display image for rendering on the image display when the image display is operative to render HDR video content.

[0029] Exemplary embodiments described herein relate to encoding image content. A multiview input image is received. The multiview input image covers a plurality of sampled views for an image space depicted in the multiview input image. A multiview layered image stack of a plurality of layered images in a first dynamic range for the plurality of sampled views, a plurality of alpha maps for the plurality of layered images, and a plurality of beta scale maps for the plurality of layered images is generated from the multiview input image. A viewer's target view for the image space is determined based at least in part on a portion of user pose data generated from user pose tracking data collected while the viewer is viewing a rendered image on an image display. The viewer's target view is used to select a set of user pose-selected sampled views from among the plurality of sampled views depicted in the multiview input image. A set of layered images for the user pose selected set of sampled views in the multiple layered images of the multiview layered image stack, along with a set of alpha maps for the user pose selected set of sampled views in the multiple alpha maps of the multiview layered image stack, and a set of beta scale maps for the user pose selected set of sampled views in the multiple beta scale maps of the multiview layered image stack, are encoded into a video signal to cause a recipient device of the video signal to generate a display image from the set of layered images for rendering on an image display.

[0030] Exemplary embodiments described herein relate to decoding image content. A set of layered images of a first dynamic range for a user pose selected set of sampled views is decoded from a video signal. The user pose selected set of sampled views is selected based on user pose data from a plurality of sampled views covered by a multiview source image. The multiview source images are used to generate a corresponding multiview layered image stack. The corresponding multiview layered images are used to generate a set of layered images. A set of alpha maps for the user pose selected set of sampled views is decoded from a video signal. An alpha value in the set of alpha maps for the user pose selected set of sampled views is adjusted using a viewer's current view to generate an adjusted alpha value in the set of adjusted alpha maps for the current view. A display image is derived from the set of layered images and the set of adjusted alpha maps to be rendered on a target image display.

[0031] In some exemplary embodiments, the mechanisms described herein form part of a media processing system, including, but not limited to, a cloud-based server, a mobile device, a virtual reality system, an augmented reality system, a heads-up display device, a helmet-mounted display device, a cave-type system, a wall-sized display, a video game device, a display device, a media player, a media server, a media production system, a camera system, a home system, a communications device, a video processing system, a video codec system, a studio system, a streaming server, a cloud-based content services system, a handheld device, a gaming console, a television, a cinema display, a laptop computer, a netbook computer, a tablet computer, a cellular wireless telephone, an e-reader, a point-of-sale terminal, a desktop computer, a computer workstation, a computer server, a computer kiosk, or any of a variety of other types of terminals and media processing units.

[0032] Various modifications to the preferred embodiments and generic principles and features described herein will be readily apparent to those skilled in the art, and thus the present disclosure is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features described herein.

[0033] 2. Representation and Delivery of Multi-View Image Data FIG. 1A illustrates an exemplary image process flow for generating a layered image stack. The process flow can be implemented as part of an upstream image processing device, such as an encoder device or a video streaming server. Additionally, optionally, or alternatively, the process flow can be implemented in a separate or associated device, such as an image pre-processing device operating in conjunction with the encoder device or video streaming server. Some or all of the process flow can be implemented or performed using one or more of a computing processor, audio and video codecs, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other delivery formats, digital signal processors, graphics processing units or GPUs, and the like.

[0034] The SDR and HDR image content generator block 104 includes software, hardware, a combination of software and hardware, etc. configured to receive a series of (e.g., time-sequential, sequential, multi-view, etc.) input or source images 102. These input images (102) may be received from a video source or retrieved from a video data store. The input images (102) may be digitally captured (e.g., by a digital camera, etc.), generated by converting analog camera photographs captured on film to a digital format, computer-generated (e.g., using computer animation, image rendering, etc.), etc. The input images (102) may be images associated with one or more of a movie release, an archived media program, a media program library, video recordings / clips, a media program, a TV program, user-generated video content, etc.

[0035] The SDR and HDR image content generator block (104) may perform image content mapping operations on a sequence of input images (102) to generate a corresponding sequence of SDR images 106 depicting the same visual content as the input images (102), and a corresponding sequence of HDR images 108 depicting the same visual content as the sequence of input images (102). Exemplary image content mapping operations may include some or all of video editing operations, video conversion operations, color grading operations, dynamic range mapping operations, local and / or global reshaping operations, display management operations, video special effects operations, and the like.

[0036] Some or all of these operations may be performed automatically without human input (e.g., using a content mapping tool, a color grading toolkit, etc.). Additionally, optionally, or alternatively, some or all of these operations may be performed manually, automatically with human input, etc.

[0037] By way of example and not limitation, an HDR image (108), which may be of a relatively high dynamic range (or luminance range), may first be generated from an input image (102) using some or all of these image processing operations, including color grading operations, which may be performed partially automatically or fully automatically with human input. Local and / or global reshaping operations may be performed (e.g., automatically, without human input, etc.) on the HDR image (108) generated from the input image (102) to generate an SDR image (106), which may be of a relatively narrow dynamic range (or luminance range).

[0038] The layered image stack generator 110 includes software, hardware, a combination of software and hardware, etc. configured to receive as input an HDR image (108) and an SDR image (106) depicting the same visual content but having different dynamic ranges, and generate a corresponding sequence of HDR layered image stacks 114 depicting the same visual content, as well as a corresponding sequence of SDR layered image stacks 112 depicting the same visual content.

[0039] In some operating scenarios, an input image described herein - and / or a derived image, such as a corresponding HDR or SDR image depicting the same visual content as the input image - may be, for example, a multi-view image for a given time point in a plurality or series (e.g., successive, etc.) of time points over a time interval or duration covered by a sequence of input images (102). The multi-view input image includes image data for each sampled view in the plurality of sampled views. The image data for each such sampled view in the plurality of sampled views represents a single-view image for a plurality of single-view images in the multi-view image. The plurality of single-view images in the multi-view image may correspond (e.g., one-to-one, etc.) respectively to the plurality of sampled views represented in the multi-view image. The image data for each sampled view in the plurality of sampled views in the multi-view image may be represented as a pixel value in an image frame.

[0040] For example, the input image data for each sampled view of the multiple sampled views represented in the multiview input image may be represented as input pixel values ​​in an input image frame (or a single-view input image). The input image data for each such sampled view in the multiple sampled views covered by the multiview input image represents a single-view input image for the multiple single-view input images in the multiview input image. The multiple single-view input images in the multiview input image may correspond (e.g., one-to-one, etc.) respectively to the multiple sampled views represented in the multiview input image.

[0041] Similarly, HDR image data for each sampled view in the multiple sampled views represented in a corresponding multiview HDR image (e.g., one of the HDR images (108)), which may be the same as the multiple sampled views represented in the multiview input image used to directly or indirectly derive the multiview HDR image, may be represented as HDR pixel values ​​in an HDR image frame (or single-view HDR image). The HDR image data for each such sampled view in the multiple sampled views covered by the multiview HDR image represents a single-view HDR image for the multiple single-view HDR images in the multiview HDR image. The multiple single-view HDR images in the multiview HDR image may correspond (e.g., one-to-one, etc.) respectively to the multiple sampled views represented in the multiview HDR image.

[0042] The SDR image data for each sampled view in the multiple sampled views represented in a corresponding multiview SDR image (e.g., one of the SDR images (106)), which may be the same as the multiple sampled views represented in the multiview input image used to directly or indirectly derive the multiview SDR image, may be represented as SDR pixel values ​​in an SDR image frame (or single-view SDR image). The SDR image data for each such sampled view in the multiple sampled views covered by the multiview SDR image represents a single-view SDR image for the multiple single-view SDR images in the multiview SDR image. The multiple single-view SDR images in the multiview SDR image may correspond (e.g., one-to-one, etc.) respectively to the multiple sampled views represented in the multiview SDR image.

[0043] Single-view HDR images and single-view SDR images, both derived or generated directly or indirectly from the same single-view input image, may be of the same sampled view and of the same (e.g., planar, spherical, etc.) spatial dimensions and / or of the same spatial resolution (with one-to-one pixel correspondence). Additionally, optionally, or alternatively, single-view HDR images and single-view SDR images, both derived or generated directly or indirectly from the same single-view input image, may be of the same sampled view but of different spatial dimensions and / or of different spatial resolutions (with many-to-one pixel correspondence) (as determined by a downsampling or upsampling factor).

[0044] In some operating scenarios, the layered image stack generator (110) can turn a multi-view SDR image in a sequence of SDR images (106) into an SDR layered image stack in a sequence of SDR layered image stacks (112). The SDR layered image stack covers a plurality of sampled views covered in the multi-view SDR image. More specifically, the SDR layered image stack includes a plurality of single-view layered images, each of which covers a respective sampled view in the plurality of sampled views.

[0045] Each single-view layered image in the plurality of single-view layered images in the SDR layered image stack may be derived or generated from a respective single-view SDR image in the plurality of single-view SDR images in the multi-view SDR image. The single-view layered images may include image layer data in one or more image layers.

[0046] In some operating scenarios, instead of putting all image data for each single-view SDR image into a single image layer, the image data for each single-view SDR image may be split (e.g., physically, logically, using different buffers, using a buffering order, etc.) into multiple image data portions within multiple image layers - or multiple image layer data sets - respectively.

[0047] The image space depicted or represented in each single-view SDR image may be logically divided into multiple image subspaces (e.g., along a depth direction relative to the camera position / orientation). Different image data portions depicting image details / objects at different image subspaces of the image space, e.g., corresponding to different depths or different (mutually exclusive) depth subranges, may be divided into different image layers of the multiple image layers. Each image layer in the multiple image layers may represent a respective image subspace, e.g., corresponding to a respective depth or a respective depth subrange, within the multiple image subspaces divided from the image space depicted or represented in the respective single-view SDR image.

[0048] Each image data portion in the plurality of image data portions in a single-view SDR layered image - or each image layer data set in the plurality of image layer data sets - derived or generated from a respective single-view SDR image may represent an image piece in a respective image subspace in a plurality of image subspaces of the image space depicted or represented by the single-view SDR layered image or the respective single-view SDR image.

[0049] To split the image data in each single-view SDR image into multiple layers, multiple image layer data sets of a corresponding single-view SDR layered image, an alpha map can be generated by the layered image stack generator (110) to define or specify an alpha value (e.g., transparency value, weighting factor, alpha blending value, etc.) for each pixel in each single-view SDR image. These alpha values ​​can be used in an alpha compositing operation performed on multiple image layer data sets in the multiple image layers of the single-view SDR layered image to generate or recover the respective single-view SDR image resulting in the single-view SDR layered image. An exemplary alpha compositing operation can be to composite the image layer with the furthest depth to the image layer with the closest depth using an image compositing operation such as, for example, an "over" operator, based at least in part on the ordering of the image layers and the alpha values ​​indicating the opacity / transparency of the image layers. Additionally, optionally, or alternatively, the image layer data for the different image layers may be combined using alpha values ​​and weighting factors or blending values ​​defined in the alpha map to generate or restore pixel values ​​(e.g., YCbCr, RGB, etc.) of each single-view SDR image as a weighted or blended sum or average (e.g., normalized, etc.) of corresponding pixel values ​​in the multiple image layer data sets in the single-view SDR layered image.

[0050] A single-view HDR image in the multi-view HDR image may correspond to a respective single-view SDR image in the multi-view SDR image. For example, the single-view HDR image and the respective single-view SDR image may be generated or derived from the same single-view input image in the input image giving rise to the multi-view SDR image and the multi-view HDR image.

[0051] In some operating scenarios, instead of directly splitting a single-view HDR image into a corresponding HDR layered image, the system described herein (or the SDR and HDR image content generator (104) or layered image stack generator (110) therein) can generate a beta scale map including beta scaling data for scaling each single-view SDR image into a single-view HDR image. Thus, a single-view HDR image can be represented as a combination of each single-view SDR image and the beta scale map.

[0052] The single-view HDR image may be split into the same multiple image layers used to split the respective single-view SDR image by splitting the beta scale map into multiple image layers, thereby generating or deriving a single-view HDR layered image corresponding to the single-view HDR image.

[0053] As a result, an SDR layered image stack as described herein corresponds to a multiview input image and includes a plurality of single-view SDR layered images, each corresponding to a respective sampled view in a plurality of sampled views represented or covered in the multiview input image. Each single-view SDR layered image in the plurality of single-view SDR layered images in the SDR layered image stack includes a plurality of SDR image layer data sets (or a plurality of image pieces of a single-view SDR (non-layered or pre-layered) image) and an alpha map including alpha compositing related data for performing an alpha compositing operation on the plurality of image layer data sets to reconstruct the single-view SDR (non-layered or pre-layered) image.

[0054] Correspondingly, the HDR layered image stack described herein includes a plurality of single-view HDR layered images corresponding to the same multi-view input image and each corresponding to a respective sampled view in a plurality of sampled views represented or covered in the multi-view input image. Each single-view HDR layered image in the plurality of single-view HDR layered images in the HDR layered image stack includes a plurality of SDR image layer data sets (or a plurality of image pieces of a single-view SDR (non-layered or pre-layered) image), an alpha map, and a beta scale map. The alpha map includes alpha compositing related data for a single-view SDR image corresponding to or covering the same sampled view as the single-view SDR image. The beta scale map includes a plurality of sets of beta scaling related data each partitioned from the (pre-partitioned) beta scale map into the same plurality of image layers as the single-view SDR layered image generated or derived from the single-view SDR image. These multiple sets of beta-scaling related data can be used to perform a beta-scaling operation on the multiple SDR image layer data sets of the single-view SDR layered image to derive or generate a corresponding multiple HDR image layer data sets. The same alpha compositing operation can be performed on the multiple sets of HDR image layer data to restore a single-view HDR (non-layered or pre-layered) image using the same alpha map used to composite the single-view SDR layered images into a single-view SDR image.

[0055] Beta scaling may be used to incorporate or implement in place of other image processing operations, including, but not limited to, any, some, or all of the following: reshaping operations, content mapping with little or no human input, content mapping with human input, tone mapping, color space conversion, display mapping, PQ, non-PQ, linear or non-linear encoding, image blending, image mixing, linear image mapping, non-linear image mapping, application of EOTF, application of EETF, application of OETF, spatial or temporal downsampling, spatial or temporal upsampling, spatial or temporal resampling, chroma sampling format conversion, etc.

[0056] The beta scaling operations described herein may be implemented as simple scaling operations applying (e.g., linear, etc.) multiplications and / or additions. Additionally, optionally, or alternatively, the beta scaling operations described herein may be implemented in complex or non-linear scaling operations, including but not limited to LUT-based scaling. The beta scaling operations may be performed only once at run-time to achieve or generate the (equivalent) effect of other image processing operations for which the beta scaling operations are performed. As a result, relatively complex image processing operations permeating image processing chains / pipelines / frameworks may be avoided or significantly simplified under the beta scaling techniques as described herein.

[0057] The beta scaling described herein may support global mapping (e.g., global tone mapping, global reshaping, etc.), local mapping (e.g., local tone mapping, local reshaping, etc.), or a combination of global and local mapping.

[0058] Exemplary beta scaling operations can be found in U.S. patent application Ser. No. 63 / 305,626, entitled "BETA SCALE DYNAMIC DISPLAY MAPPING," by Ajit Ninan and Gregory Ward, filed on February 1, 2022 (Attorney Docket No. 60175-0480; D20114USP1), the entire contents of which are incorporated by reference herein as if fully set forth herein.

[0059] The layered image stack generator (110) may include a trained image layer prediction model implemented with one or more convolutional neural networks (CNNs).

[0060] The operating parameters for the CNN in the image layer prediction model can be optimized in a model training phase using training (e.g., non-layered SDR, pre-layered SDR, etc.) images and ground truth represented by training image layers for those training images, each segmented from the training images using a training depth image. The image layer prediction model can use the training images as inputs to the CNN to generate predicted image layers (e.g., corresponding predicted alpha masks, etc.) from the training images. A prediction error or cost can be calculated as a difference or distance based on an error or cost function between the predicted image layer and the ground truth represented by the training image layer for the training images, and back-propagated to modify or optimize the operating parameters of the CNN, such as the weights or biases of the CNN. The CNN can be trained using multiple training images into a trained image layer prediction model with optimized operating parameters (e.g., final, etc.).

[0061] The layered image stack generator (110) may be configured or downloaded with optimized operating parameters for the trained image layer prediction model CNN that can receive a single-view image, such as a single-view SDR image, as input, generate features of the same type used in the training phase, and use those features to generate or derive SDR image layers (e.g., corresponding alpha masks, etc.) from the single-view SDR image.

[0062] 3. Streaming of layer configuration images based on user pose selection 1B shows an exemplary upstream device such as a video streaming server 100 comprising a multi-view layered image stack receiver 116, a user posture monitor 118, a posture selected layered image encoder 120, etc. Some or all of the components of the video streaming server (100) may be implemented by one or more devices, modules, units, etc. in the form of software, hardware, a combination of software and hardware, etc.

[0063] In some operating scenarios, the video streaming server (100) may include or implement processing blocks of the process flow as shown in Figure 1 A. Additionally, optionally, or alternatively, the video streaming server (100) may operate in conjunction with a separate upstream device that includes or implements processing blocks of the process flow as shown in Figure 1 A.

[0064] The multi-view layered image stack receiver (116) comprises software, hardware, a combination of software and hardware, etc. configured to receive SDR and HDR layered image stacks (e.g., 112, 114, etc.) from an internal or external layered image stack source.

[0065] The (multi-view) SDR and HDR layered image stacks (112, 114) include a sequence of pairs of SDR and HDR layered image stacks depicting a visual scene in an image space (e.g., a space depicted in three dimensions or 3D). Each pair of SDR and HDR layered image stacks in the sequence includes an SDR layered image stack and an HDR layered image stack corresponding to the SDR layered image stack. Both the SDR layered image stack and the HDR layered image stack depict the same visual content, but may have different dynamic ranges (or luminance ranges) for or at corresponding points in time in a time interval or duration covered or represented by the (multi-view) SDR and HDR layered image stacks (112, 114).

[0066] In some operating scenarios, an SDR layered image stack may include SDR layered images covering multiple sampled views as well as respective alpha maps for the SDR layered images that may be used to composite the SDR layered images into an (original, non-layered, pre-layered, single-layer) SDR image. Similarly, an HDR layered image stack may include HDR layered images covering multiple sampled views as well as respective alpha maps for HDR layered images that may be used to composite the HDR layered images into an (original, non-layered, pre-layered, single-layer) HDR layered image (which may be, but is not limited to, the same as for the SDR layered images).

[0067] In some operating scenarios, the SDR layered image stack may include SDR layered images covering multiple sampled views, as well as respective alpha maps for the SDR layered images that may be used to composite the SDR layered images into the (original, non-layered, pre-layered, single-layer) SDR image. However, the HDR layered image stack may not include either HDR or SDR layered images. The HDR layered image stack may simply include references to the SDR layered images, alpha maps already included in the SDR layered image stack in the same pair, and a beta scale map used to perform a beta scaling operation on the SDR pixel or codeword values ​​in the SDR layered image. These may be converted into the (original, non-layered, pre-layered, single-layer) HDR layered images, respectively, using the same alpha map used to composite the SDR layered images into the (original, non-layered, pre-layered, single-layer) SDR image.

[0068] In some operating scenarios, the HDR layered image stack may include HDR layered images covering multiple sampled views, as well as respective alpha maps for the HDR layered images that may be used to composite the HDR layered images into the (original, non-layered, pre-layered, single-layer) HDR image. However, the SDR layered image stack may not include HDR layered images or SDR layered images. The SDR layered image stack may simply include references to HDR layered images, alpha maps already included in the HDR layered image stack for the same pair, and a beta scale map used to perform a beta scaling operation on the HDR pixel or codeword values ​​in the HDR layered image into the corresponding SDR layered image. These can be converted into the (original, non-layered, pre-layered, single-layer) SDR image, respectively, using the same alpha map used to composite the HDR layered image into the (original, non-layered, pre-layered, single-layer) SDR image.

[0069] The multiple sampled views represented in a multiview image or corresponding multiview layered image stack as described herein may correspond to viewpoints or camera positions spatially arranged or distributed in a viewing surface or volume. By way of example and not limitation, the multiple sampled views may correspond to different viewpoints or camera positions spatially arranged or distributed as vertices of a grid in a two-dimensional plane.

[0070] The depth or disparity data can be generated (e.g., by a CNN implementing layered image (or image layer) generation or prediction, by the layered image stack generator (110), etc.) using pixel correspondences between different images from different sampled views. For example, the depth or disparity data may be obtained as a solution to a problem of minimizing a cost function defined based on the intensity / chromaticity difference of pixels from different images in different sampled views. Additionally, optionally, or alternatively, the depth or disparity data can be obtained using camera geometry information or camera settings (e.g., zoom factor, etc.). The depth or disparity data can be used by the layered image stack generator (110) to split the multiview input image (102) into multiview SDR and HDR layered image stacks (112, 114) and generate an alpha map (e.g., used in an alpha compositing operation that converts a layered image to a non-layered image) included in the multiview SDR and HDR layered image stacks (112, 114).

[0071] The user pose monitor (118) comprises software, hardware, a combination of software and hardware, etc. configured to receive viewer rendered environment user pose data 124 in real time or near real time from a video client device operated by the viewer. The viewer rendered environment pose data (124) may be collected or generated in real time or near real time by the video client device using any combination of user pose tracking methods including, but not limited to, machine learning or ML-based face detection, gaze tracking, viewport tracking, POV tracking, viewer position tracking, face tracking, etc.

[0072] The viewer's rendering environment user pose data (124) may include, for example, a real-time or near real-time representation of some or all of: a user pose image of the viewer captured by a camera positioned in front of and facing the viewer; a face mesh formed by a plurality of vertices positioned over the viewer's face depicted in the detected face image portion in the captured image; coordinates and position information of a plurality of vertices in the face mesh positioned over the viewer's face depicted in the detected face image portion in the captured image; and the position and / or orientation of particular features or positions within the viewer's face depicted in the detected face image portion in the captured image, such as the viewer's pupil position, the viewer's face orientation, the midpoint of the viewer's interpupillary distance, etc.

[0073] The user pose monitor (118) can use the viewer rendering environment user pose data (124) to monitor, establish, determine, and / or generate viewer user poses--representing the viewer's (e.g., logical, virtual, represented, mapped, etc.) position or orientation in an image space or visual scene depicted in the SDR and HDR layered image stacks (112, 114)--for multiple time points across a time interval / duration of an AR, VR, or volumetric video application. In a video application, a display image is derived by the video client device from the SDR and HDR layered image stacks (106) and rendered at multiple time points in a viewer's viewport provided on an image display operating in conjunction with the video client device.

[0074] The pose-selected layered image encoder (120) comprises software, hardware, a combination of software and hardware, etc. configured to receive a sequence of (real-time or near real-time) user poses of a viewer for a plurality of time points, dynamically or adaptively select a sequence of layered images within a sequence of SDR and HDR layered image stacks (106) for a plurality of time points using the sequence of user poses, and encode the sequence of (pose-selected) layered images—along with a sequence of alpha maps, a sequence of beta scale maps, etc. corresponding to the sequence of (pose-selected) layered images—into a (e.g., 8-bit, backwards compatible, multi-layer, etc.) video signal 122. The sequence of alpha maps, beta scale maps, etc. may be encoded as ancillary data, such as image metadata, carried in a signal layer separate from the base layer used to encode the sequence of (pose-selected) layered images in the video signal (122). In various operating scenarios, the sequence of (pose-selected) layered images may include one of only SDR layered images, only HDR layered images, a combination of SDR and HDR layered images, etc.

[0075] A sequence of pose-selected layered images may cover or correspond to a sequence of user-pose-selected sampled views (for or at multiple time points) that are close to or adjacent to a sequence of target views as determined or represented by a sequence of user poses of the viewer. A denser set of user-pose-selected sampled views may be used to capture relatively more view-dependent effects around novel or synthetic views represented by the viewer / user's target views. A less dense set of user-pose-selected sampled views may be used to capture relatively less view-dependent effects, such as diffuse image details around novel or synthetic views.

[0076] Each target view in the sequence of target views may be determined by a position and / or orientation (mapped, projected, or represented) of the viewer as indicated in a user pose of the viewer in a sequence of user poses of the viewer at each time point among the multiple time points. For example, the viewer's pose may be mapped or represented in the image space of a pair of SDR and HDR layered image stacks in a sequence of pairs of SDR and HDR layered image stacks at each time point.

[0077] Each such target view may be used to identify or determine a respective set of user pose selected sampled views for or at a respective time instant in a sequence of sets of user pose selected sampled views. Each set of user pose selected sampled views for a target view may include sampled views that are near or adjacent to that target view (e.g., the single closest, the two closest, the three closest, the four closest, etc.).

[0078] As used herein, the video content in a video signal (or stream) described herein may include, but is not necessarily limited to, any of audiovisual programs, movies, video programs, TV broadcasts, computer games, augmented reality (AR) content, virtual reality (VR) content, automotive entertainment content, etc.

[0079] As used herein, a "video streaming server" may refer to one or more upstream devices that prepare and stream video content to one or more video streaming clients, such as video decoders, for rendering at least a portion of the video content on one or more displays. The displays on which the video content is rendered may be part of or operate in conjunction with one or more video streaming clients.

[0080] Exemplary video streaming servers may include, but are not necessarily limited to, any of the following: a cloud-based video streaming server located remotely from the video streaming client, a local video streaming server connected to the video streaming client through a local wired or wireless network, a VR device, an AR device, an automobile entertainment device, a digital media device, a digital media receiver, a set-top box, a gaming console (e.g., Xbox), a general-purpose personal computer, a tablet, a dedicated digital media receiver such as an Apple TV or Roku box, and the like.

[0081] The video streaming server (100) may be used to support AR applications, VR applications, 360° video applications, volumetric video applications, real-time video applications, near real-time video applications, non-real-time omnidirectional video applications, automotive entertainment, helmet mounted display applications, heads up display applications, gaming, 2D display applications, 3D display applications, multi-view display applications, and the like.

[0082] 1C illustrates an exemplary downstream recipient device, such as a video client device 150, including a pose-selected layered image receiver 152, a user pose tracker 154, a pose variation image renderer 156, an image display 158, etc. Some or all of the components of the video client device (150) may be implemented by one or more devices, modules, units, etc. in the form of software, hardware, a combination of software and hardware, etc.

[0083] Exemplary video client devices described herein may include, but are not necessarily limited to, any of a large screen image display, a home entertainment system, a set-top box and / or an audiovisual device operating in conjunction with an image display, a mobile computing device held by a user / viewer (e.g., in a spatially stable or changing relationship with the user / viewer's eyes), a wearable device that includes or operates in conjunction with an image display, a computing device that includes or operates in conjunction with a head-mounted or head-up display, and the like.

[0084] The user pose tracker (154) comprises software, hardware, a combination of software and hardware, etc. configured to operate in conjunction with one or more user pose tracking sensors (e.g., cameras, depth of field sensors, motion sensors, position sensors, eye trackers, etc.) to collect real-time or near real-time user pose tracking data in association with a viewer (or user) operating in conjunction with the video client device (150).

[0085] In some operational scenarios, the user pose tracker (154) may perform image processing operations, computer vision operations, and / or incorporate ML tools to generate viewer rendered environment user pose data 124 from real-time or near real-time user pose tracking data collected by the user pose tracking sensor.

[0086] For example, the user pose tracker (154) may include, deploy, and / or implement one or more CNNs used to detect the viewer's face in a user pose tracked image acquired by a camera at a spatially fixed position relative to the image display (158), logically impose a face mesh on the viewer's detected face in the image, determine coordinates of the vertices of the face mesh, and determine the position or orientation of the viewer's face or a midpoint along the viewer's interpupillary line. Some or all of the output from those CNNs may be included in the viewer's rendered environment user pose data (124).

[0087] The user pose tracking data and / or viewer rendering environment user pose data derived therefrom (124) may include static and / or dynamic data related to the image display (158) and / or results of analysis performed based at least in part on data related to the image display (158).

[0088] For example, the spatial size / dimensions of the image display (158) and the spatial relationship between the camera used to acquire the user pose tracking images and the image display (158) may be included as part of the user pose tracking data and / or the viewer's rendered environment user pose data (124).

[0089] Additionally, optionally, or alternatively, the actual spatial size / dimensions and spatial location of particular display screen portions within the image display (158) used to render the display image derived from the layered image received from the video streaming server (100) may be included as part of the user pose tracking data and / or viewer rendering environment user pose data (124).

[0090] Additionally, optionally, or alternatively, any zoom factor used to render the display image on the image display (158) or portion(s) thereof may be included as part of the user pose tracking data and / or viewer rendering environment user pose data (124).

[0091] The static and / or dynamic data related to the image display (158) may be used -- by the video streaming server (100) alone, by the video streaming client device (150) alone, or by a combination of the server device (100) and the client device (150) -- to determine a viewer's position and / or orientation relative to an image space or visual scene depicted by a displayed image rendered on the image display (158).

[0092] The video client device (150) can transmit viewer rendered environment user pose data (124) to the video streaming server (100). The viewer rendered environment user pose data (124) can be sampled, generated, and / or measured on a relatively fine time scale (e.g., every 1 millisecond, every 5 milliseconds, etc.). The viewer rendered environment user pose data (124) can be used—by the video streaming server (100) alone, the video streaming client device (150) alone, or a combination of the server device (100) and the client device (150)—to establish / determine the viewer's position and / or orientation relative to the image space or visual scene depicted in the displayed image at a given time resolution (e.g., every 1 millisecond, every 5 milliseconds, etc.).

[0093] The pose-selected layered image receiver (152) comprises software, hardware, a combination of software and hardware, etc., configured to receive and decode a (e.g., real-time, near real-time, etc.) video signal (122) for a series of (e.g., continuous, sequential, etc.) time points within a time interval or duration of an AR, VR, or immersive video application into a series of (pose-selected) layered images. In addition, the pose-selected layered image receiver (152) derives from the video signal (122) a sequence of alpha maps for each of a plurality of time points corresponding to the sequence of (pose-selected) layered images, a sequence of beta scale maps for each of a plurality of time points corresponding to the sequence of (pose-selected) layered images, etc.

[0094] A particular (pose-selected) layered image in the sequence of (pose-selected) layered images for or at a particular time point in the series of time points—along with a particular alpha map and a particular beta scale map corresponding to the particular (pose-selected) layered image—may cover a particular set of user-pose-selected sampled views selected by the video streaming server (100)—from a plurality of sampled views (e.g., corresponding to cameras located at vertices of a planar grid, etc.) represented in a particular pair of SDR and HDR layered image stacks for or at that particular time point—based, at least in part, on a particular target view that is mapped to, represented, and / or indicated by a particular portion of user-pose data for or at that particular time point.

[0095] The pose-variant image renderer (156) comprises software, hardware, a combination of software and hardware, etc., configured to receive the decoded sequence of (pose-selected) layer-composed images, perform client-side image processing operations to generate (e.g., consecutive, sequential, etc.) a sequence of display images from the decoded sequence of (pose-selected) layer-composed images, and render the sequence of display images on the image display (158) for or at multiple time points.

[0096] Client-side image processing operations performed by the video client device (150) or a posture variation image renderer (156) therein may include adjusting alpha values ​​in an alpha map of the sampled views to generate adjusted alpha values ​​that constitute an adjusted alpha map for a real-time or near real-time viewpoint of the viewer (as indicated by the viewer's real-time or near real-time position and / or orientation).

[0097] By way of example and not limitation, a pixel in a first image layer of a sampled view represented in a received (pose-selected) layered image may be at a first depth, which may be a relatively large depth relative to a virtual or real camera located at an origin or reference position of the sampled view, where the pixel is unoccluded or visible in the sampled view.

[0098] However, at a shifted perspective from the sampled view, such as a viewer's real-time or near real-time perspective, pixels of a first image layer at a first depth may be occluded or invisible due to the presence of image details (or groups of pixels) of a second image layer (of the same received (pose selected) layer composed image) at a second depth that is narrower or smaller than the first depth. This shifted perspective may represent a new or synthetic view that is not covered / represented in any sampled view represented in the received (pose selected) layer composed image, or that is not covered / represented in any sampled view represented in the original or input images used to derive the SDR and HDR layer composed image stacks from which the received (pose selected) layer composed image was selected.

[0099] In response to determining (e.g., through ray tracing from or with reference to the shifted viewpoint, through ray space interpolation, etc.) that a previously unoccluded or visible pixel for the sampled view is obstructed or located behind image detail (in another image layer of the received (pose selected) layered image) at or for a closer depth for the shifted viewpoint, the pose-varying image renderer (156) adjusts alpha values ​​in a corresponding alpha map for the (pose selected) layered image to generate adjusted alpha values ​​for opacity, transparency, weighting factors, etc. for associated pixels of those image layers in the received (pose selected) layered image at or for the shifted viewpoint.

[0100] The adjusted alpha values—e.g., transparency, opacity, etc.—in the adjusted alpha map can indicate newly occluded regions (or holes) for a composite image generated from the (pose-selected) layered image at the shifted viewpoint. The composite image represents a warped image at or for the shifted viewpoint compared to the non-layered image represented by the (pose-selected) layered image at or for the sampled view. Pixel values ​​of the composite image can be generated or derived as a weighted or blended sum or average (e.g., normalized, etc.) of pixel values ​​in different image layers of the (pose-selected) layered image using alpha compositing operations (e.g., performed in order from furthest depth image layer to closest depth image layer, etc.) on different image layers in the (pose-selected) layered image at the shifted viewpoint based at least in part on the adjusted alpha values ​​in the adjusted alpha map.

[0101] Image warping operations, represented by alpha map adjustment and alpha compositing operations, may be performed on each of some or all of the (pose-selected) layered images for each of some or all of the other sampled views within the same set of user-pose-selected sampled views of which the sampled view is a part.

[0102] In some operating scenarios, newly occluded regions of a composite image at a shifted perspective derived from the (pose-selected) layer composition image for a sampled view may be filled in, blended, and / or complemented by other composite images at shifted perspectives derived from the (pose-selected) layer composition images for other sampled views within the same set of user-pose-selected sampled views (for a given time point among multiple time points) of which the sampled view is a part.

[0103] For purposes of example only, a sequence of (pose-selected) layered images for a sequence of time points decoded from a video signal (122) may represent a sequence of (pose-selected) SDR layered images for a sequence of time points.

[0104] In an operational scenario in which SDR view images are rendered on an image display (158), the video client device (150), or a pose-varying image renderer (156) therein, can render a sequence of (final or blended or hole-filled) composite images generated from the sequence of (pose-selected) layered images as a sequence of view images on the image display (158).

[0105] In an operational scenario in which an HDR display image is rendered on an image display (158), instead of performing an alpha compositing operation based on the target viewpoint adjusted alpha values ​​in the adjusted alpha map directly on the received (pose selected) SDR layered images in the decoded sequence of (pose selected) SDR layered images for the sequence of time points, the video client device (150), or an attitude-variant image renderer (156) therein, may first perform a beta scaling operation on some or all of the (SDR) image layers in the received (pose selected) SDR layered images to generate some or all of the corresponding (HDR) image layers not encoded in the video signal (122). The video client device (150), or an attitude-variant image renderer (156) therein, may then perform an alpha compositing operation based on the target viewpoint adjusted alpha values ​​in the adjusted alpha map directly on the (HDR) image layers generated from the beta scaling operation to generate the HDR composite image. The video client device (150), or a pose-varying image renderer (156) therein, can directly render the sequence of (final or blended or hole-filled) HDR composite images—generated from the sequence of (pose-selected) SDR layered images through beta-scaling and alpha-compositing operations—as a sequence of display images on the image display (158). Additionally, optionally, or alternatively, the video client device (150), or a pose-varying image renderer (156) therein, can perform additional image processing operations, such as display management or DM operations (e.g., based on DM image metadata received with the video signal (122)), on the sequence of HDR composite images to generate a sequence of display images for rendering on the image display (158).

[0106] It should be noted that in some operational scenarios, the video signals described herein may be encoded with an attitude selected HDR layered image (instead of an SDR layered image), an alpha map and a beta scale map corresponding to the attitude selected HDR layered image, etc. Similar operations described as being performed on a received attitude selected SDR layered image may be performed by a video client device to generate a display image that is rendered on an image display operating in conjunction with the video client device.

[0107] 4. User pose selected sampled views 2A illustrates an exemplary set of user pose-selected sampled views selected for a given time point among a plurality of time points or at a given time point based in part or in whole on a target view, denoted as "t", from a plurality of sampled views represented or covered by a layered image stack as described herein (e.g., SDR, HDR, etc.). The target view may be a new view not represented or covered by any sampled view in the plurality of sampled views, representing a shifted viewpoint from these sampled views.

[0108] A light field of a 3D image space or visual scene depicted in the layered image stack for or at a given time point is captured and / or discretized based on multiple layered images in the layered image stack, each covering multiple sampled views.

[0109] Without loss of generality, as shown in FIG. 2A, the multiple sampled views in a layered image stack can be represented as a discrete distribution of points (or vertices) in a uniform grid. Each point in the discrete distribution represents a corresponding sampled view and includes a combination of a corresponding view location and a corresponding view direction. The view locations covered by the multiple sampled views can be distributed across a 2D viewing area, a 3D viewing volume, etc., which can be up to an entire venue in a multi-view video experience (e.g., for a VR experience, for an AR experience, etc.). The view directions covered by the multiple sampled views can cover one or more solid angles, which can be up to an entire sphere.

[0110] It should be noted that in various embodiments, the multiple sampled views in a layered image stack may or may not be represented in a uniform grid as shown in FIG. 2A. For example, the multiple sampled views may be represented by a discrete distribution of points in a non-uniform grid, such as, but not necessarily limited to, a spherical discrete distribution. Additionally, optionally, or alternatively, the view locations covered by the multiple sampled views may or may not be spatially uniformly distributed in the spatial viewing surface or volume. For example, one or more (e.g., central, near-center, etc.) spatial regions may have a denser distribution of view locations than other (e.g., peripheral, non-center) spatial regions in the spatial viewing surface or volume. Additionally, optionally, or alternatively, the view directions covered by the multiple sampled views may or may not be spatially uniformly distributed in solid angle(s). For example, one or more (central, near-central, etc.) subdivisions of the solid angle(s) may have a denser distribution of view directions than other (e.g., peripheral, non-central) subdivisions of the solid angle.

[0111] A target view at a given time “t” may be determined as a combination of a particular spatial location (or view position) and a particular spatial orientation (or view direction) of the detected face of the viewer / user at the given time.

[0112] The target view "t", or a view position and / or view direction therein, may be used to select or identify a set of user pose selected sampled views from among a plurality of sampled views in the layered image stack. The user pose selected sampled views in the set of user pose selected sampled views may be selected based on one or more selection factors, such as one or more of the proximity of the view position of the user pose selected sampled view to the view position of the target view, the proximity of the view direction of the user pose selected sampled view to the view direction of the target view, weighted or unweighted combinations of the above, etc.

[0113] In one example, the user pose selected sampled view may represent the closest sampled view, such as those denoted as "v1", "v2", "v3", "v4", etc., compared to or relative to a target view "t". In another example, the user pose selected sampled view may represent the closest sampled view (e.g., "v1", "v2", "v3", "v4", etc.) compared to or relative to a target view "t" and one or more non-closest sampled views, such as those denoted as "v5". The one or more non-closest sampled views may be those located at a symmetric center of the multiple sampled views, those located furthest from the symmetric center, etc. These non-closest sampled views may be used to fill holes or provide newly unoccluded (e.g., diffuse) image details that may be missing or incomplete in the closest sampled views.

[0114] An upstream device (e.g., 100 in FIG. 1B ) may obtain / access image data and image metadata for a set of user-pose selected sampled views. The image data and image metadata may include, but are not necessarily limited to, posture selected (or target view selected) layered images corresponding to the set of user-pose selected sampled views selected based on a target view “t”, alpha maps for these layered images, beta scale maps for these layered images, etc. A video signal (e.g., 122 in FIG. 1B and FIG. 1C ) may be transmitted / or delivered by the upstream device to a downstream recipient device (e.g., 150 in FIG. 1C ) that encodes the image data and image metadata as posture selected image data or metadata for a given time point of a plurality of time points.

[0115] In response to receiving or decoding the posture selected image data or metadata for a given point in time from the video signal, the downstream recipient device may use the posture selected image data or metadata to perform image warping operations (e.g., adjusting alpha values ​​of the sampled views to generate an adjusted map for the viewer's current viewpoint, an alpha compositing operation based at least in part on the adjusted alpha values ​​in the adjusted alpha map to generate a non-layered image for the current viewpoint, image blending from non-layered images generated for two or more sampled views, etc.) and / or beta scaling operations (e.g., enhancing or increasing dynamic range or luminance range from SDR to HDR, etc.) to generate or derive a corresponding display image for rendering on an image display (e.g., 158 of FIG. 1C ) operating in conjunction with the downstream recipient device.

[0116] In some operating scenarios, the spatial position of the viewer / user as represented in the image space or visual scene depicted in the layered image stack -- or the view position of the target view "t" -- may not be located within or co-located with the surface (e.g., the 2D plane in which the grid is located) formed by the view positions of multiple sampled views (e.g., three or more).

[0117] For example, a viewer / user may perform head movements to move spatially closer to or farther away from a stationary (relative to the Earth coordinate system) image display (e.g., 158 in FIG. 1C). Similarly, a viewer / user may perform hand movements to move a mobile phone including an image display (158) closer to or farther away from the viewer / user's eyes. Thus, the view position of a viewer / user's target view may or may not be located or collocated within the same surface formed by the view positions of sampled views in the multiple sampled views.

[0118] A downstream recipient device (or an upstream device) may scale image data (e.g., layered images or different image pieces at different depths or depth subranges, etc.) and / or image metadata (e.g., alpha map, beta scale map, etc.) for the sampled view according to the spatial difference between the target view and the sampled view. For example, when a viewer / user is mapped to a target view that is closer to the image display than the sampled view, a zoom-in or enlargement operation may be performed on the image data and / or image metadata in consideration of, or based at least in part on, the closer view position. Conversely, when a viewer / user is mapped to a target view that is further away from the image display than the sampled view, a zoom-out or shrink-out operation may be performed on the image data and / or image metadata in consideration of, or based at least in part on, the further away view position.

[0119] 2B illustrates an example set of SDR image data and metadata in a layer composed image stack for a sample view in a plurality of sampled views represented or covered by the layer composed image stack. The layer composed image stack includes a plurality of sets of SDR image data and metadata for the plurality of sampled views. Each set of SDR image data and metadata in the plurality of sets of SDR image data and metadata corresponds to or is for a respective sampled view in the plurality of sampled views.

[0120] Each sampled view in the plurality of sampled views may be specified or defined with a view position and a view direction. As shown in FIG. 2B, a sampled view may be specified by a camera or sampled view position 202 and a camera or sampled view direction 206. The set of SDR image data and metadata may include, but is not necessarily limited to, an SDR layered image 208. The SDR image metadata in the set of SDR image data and metadata may include, but is not necessarily limited to, an alpha map 210. The SDR layered image (208) may be alpha composited into an SDR non-layered (e.g., single layer, etc.) image (e.g., in a compositing order from furthest image layer to closest image layer, using an image compositing operation such as an "over" operator, etc.). The SDR non-layered image or the SDR layered image covers a field of view 204 as seen by a camera or reference viewer / user located at a camera or sampled view position (202) along a camera or sampled view direction (206).

[0121] 2C illustrates an example set of HDR image data and metadata in a layer composed image stack for a sample view in a plurality of sampled views represented or covered by the layer composed image stack. The layer composed image stack includes a plurality of sets of HDR image data and metadata for the plurality of sampled views. Each set of HDR image data and metadata in the plurality of sets of HDR image data and metadata corresponds to or is for a respective sampled view in the plurality of sampled views.

[0122] As described above, each sampled view in the multiple sampled views may be specified or defined using a view position and view direction, such as a camera or sampled view position (202) and a camera or sampled view direction (206), as shown in Figures 2B and 2C.

[0123] In some operating scenarios, the set of HDR image data and metadata may lack actual HDR pixel or codeword values ​​or HDR image layers. Instead, the HDR image metadata in the set of HDR image data and metadata may include a first reference or pointer to an SDR layered composition image (208) for the same sampled view and a second reference or pointer to an alpha map (210) corresponding to the SDR layered composition image (208) for the same sampled view. In addition, the HDR image metadata in the set of HDR image data and metadata may include, but is not limited to, a beta scale map 212 for the same sampled view. The beta scale map 212 may be used to perform a scaling operation on the SDR layered composition image (208) to generate an HDR layered composition image of a different (e.g., relatively higher) dynamic range than the SDR layered composition image (208). The HDR layered image may be further alpha composited into an HDR non-layered (e.g., single layer) image (e.g., in a compositing order from the furthest image layer to the closest image layer, using an image compositing operation such as the "over" operator, etc.).

[0124] The HDR non-layered image or the HDR layered image covers the same field of view (204) as covered by the corresponding SDR non-layered image or the corresponding SDR layered image, as seen by a camera or a reference viewer / user positioned at a camera or sampled view position (202) along a camera or sampled view direction (206).

[0125] 3A illustrates example SDR image layers within an SDR layered image (208) having different depths or depth sub-ranges along the depth 302 dimension as viewed from a camera or sampled view position (202) along a camera or sampled view direction (206). As illustrated, the SDR layered image (208) includes multiple SDR image layers including, but not necessarily limited to, a first SDR image layer 304-1 at a first depth value or depth sub-range along the depth (302) dimension, a second SDR image layer 304-2 at a second depth value or depth sub-range along the depth (302) dimension, etc. The second depth value or depth sub-range may be different from the first depth value or depth sub-range.

[0126] The alpha values ​​in the alpha map (210) for the SDR layered image (208) of the sampled view may be used to construct an SDR non-layered (or single layer) image from the SDR layered image (208). Although the image details depicted in the pixels in these image layers are visible or unoccluded in the sampled view represented by the camera or sampled view position (202) and the camera or sampled view direction (206), some of these image details may be partially or fully occluded from a different perspective, such as a target view represented by a viewer / user's view position and direction. Thus, the alpha values ​​in the alpha map (210) for the SDR layered image (208) of the sampled view may be adjusted (e.g., using ray tracing, ray space interpolation, etc.) to reflect the new opacity or transparency of any, some, or all of the pixels in the target view. The adjusted values ​​in the adjusted alpha map may be used to construct an SDR non-layered (or single layer) image from the SDR layered image (208). However, there may be holes or newly occluded image details, which can be filled or blended with the constructed SDR non-layered (or single layer) images for the other sampled views.

[0127] FIG. 3B illustrates an exemplary HDR image layer in an HDR layered image (e.g., 308). The HDR image layers may be constructed by applying a beta-scaling operation to the corresponding SDR image layers as pointed to by the image metadata for the sampled views, and may be of the same different depth or depth sub-range along the depth (302) dimension as the corresponding SDR image layers, as seen from the camera or sampled view position (202) along the camera or sampled view direction (206). As illustrated, the HDR layered image (308) includes multiple (constructed) HDR image layers, including a first HDR image layer 306-1 at a first depth value or depth sub-range along the depth (302) dimension, a second HDR image layer 306-2 at a second depth value or depth sub-range along the depth (302) dimension, and so on. A first HDR image layer (306-1) at a first depth value or depth sub-range may be constructed by applying a scaling operation based on first beta scaling data in the beta scale map 212 for the sampled view. A second HDR image layer (306-2) at a second depth value or depth sub-range may be constructed by applying a scaling operation based on second beta scaling data in the beta scale map (212) for the sampled view.

[0128] For an SDR layered image (208) for a sampled view, the same alpha values ​​in the alpha map (210) pointed to by the image metadata for the sampled view may be used to construct an HDR non-layered (or single layer) image for the sampled view from the HDR layered image (308). In addition, the same adjusted alpha values ​​in the adjusted alpha map for the target view may be used to construct an HDR non-layered (or single layer) image for the target view from the HDR layered image (308), which may be filled or blended with the constructed HDR non-layered (or single layer) images for the other sampled views.

[0129] 5. Exemplary Process Flow 4A illustrates an example process flow according to an example embodiment of the present invention. In some example embodiments, one or more computing devices or components may perform this process flow. In block 402, an image processing device (e.g., an upstream device, an encoder device, a transcoder, a media streaming server, etc.) receives a multi-view input image, which covers multiple sampled views into an image space represented in the multi-view input image.

[0130] In block 404, the image processing device generates from the multi-view input image a multi-view layered image stack of a plurality of layered images in a first dynamic range for a plurality of sampled views, a plurality of alpha maps for the plurality of layered images, and a plurality of beta scale maps for the plurality of layered images.

[0131] In block 406, the image processing device determines a target view of the viewer with respect to the image space, the target view being determined based at least in part on a user pose data portion generated from user pose tracking data collected while the viewer is viewing the rendered image on the image display.

[0132] In block 408, the image processing device uses the viewer's target view to select a set of user pose selected sampled views from among the multiple sampled views represented in the multiview input image.

[0133] In block 410, the image processing device encodes a set of layered images for the user pose selected set of sampled views within the multiple layered images of the multiview layered image stack, along with a set of alpha maps for the user pose selected set of sampled views within the multiple alpha maps of the multiview layered image stack, and a set of beta scale maps for the user pose selected set of sampled views within the multiple beta scale maps of the multiview layered image stack, into a video signal to cause a recipient device of the video signal to generate a display image from the set of layered images for rendering on an image display.

[0134] In one embodiment, the set of beta scale maps can be used to apply a scaling operation to the set of layer composition images to generate a set of scaled layer composition images of a second dynamic range for the user pose-selected set of sampled views, the second dynamic range being different from the first dynamic range.

[0135] In one embodiment, the display image represents one of a standard dynamic range image, a high dynamic range image, a display-mapped image optimized for rendering on a target image display, or the like.

[0136] In one embodiment, the multi-view input image includes a plurality of single-view input images for a plurality of sampled views; the plurality of single-view images in a first dynamic range are generated from the plurality of single-view input images used to generate the plurality of layer composition images; and each single-view image in the first dynamic range in the plurality of single-view images in the first dynamic range corresponds to a respective sampled view in the plurality of sampled views and is divided into a respective layer composition image for each sampled view in the plurality of layer composition images.

[0137] In an embodiment, a plurality of single-view input images for a plurality of sampled views are used to generate a second plurality of single-view images of different dynamic ranges for the plurality of sampled views; the second plurality of single-view images of different dynamic ranges include second single-view images of different dynamic ranges for each sampled view; the multiple beta scale maps include respective beta scale maps for each sampled view; and the respective beta scale maps include beta scale data used to perform a beta scaling operation on the single view image of the first dynamic range to generate beta scaled images of different dynamic ranges that approximate the second single view images of different dynamic ranges.

[0138] In one embodiment, the beta scaling operation includes one of simple scaling with a scaling factor or applying one or more codeword mapping relationships to map codewords of the single-view image of the first dynamic range to generate corresponding codewords of a beta-scaled image of a different dynamic range.

[0139] In some embodiments, a beta scaling operation is performed in place of one or more of a global tone mapping, a local tone mapping, a display mapping operation, a color space conversion, a linear mapping, a non-linear mapping, or the like.

[0140] In one embodiment, a set of layered images for a user pose selected set of sampled views is encoded in a base layer of a video signal.

[0141] In one embodiment, the set of alpha maps and the set of beta scale maps for the user pose selected set of sampled views are carried in the video signal as image metadata in a data container separate from the set of layer composed images.

[0142] In one embodiment, the multiple layer composition images include layer composition images for sampled views in the multiple sampled views, where the layer composition images include different image layers at different depth subranges from a view position of the sampled views.

[0143] 4B illustrates an exemplary process flow according to an exemplary embodiment of the present invention. In some exemplary embodiments, one or more computing devices or components may perform this process flow. In block 452, the recipient device decodes from the video signal a set of first dynamic range layered images for a user pose selected set of sampled views, the user pose selected set of sampled views being selected based on the user pose data from a plurality of sampled views covered by the multiview source image, the multiview source images being used to generate a corresponding multiview layered image stack; the corresponding multiview layered images being used to generate the set of layered images.

[0144] In block 454, the recipient device decodes from the video signal a set of alpha maps for the user pose selected set of sampled views.

[0145] In block 456, the recipient device adjusts alpha values ​​in the set of alpha maps for the user pose-selected set of sampled views using the viewer's current view to generate adjusted alpha values ​​in the set of adjusted alpha maps for the current view.

[0146] In block 458, the recipient device causes a display image derived from the set of layered images and the set of adjusted alpha maps to be rendered on the target image display.

[0147] In one embodiment, a set of beta scale maps for a user pose selected set of sampled views are decoded from the video signal; display images are of a second dynamic range different from the first dynamic range; and the display images are generated from the set of beta scale maps, the set of layer composition images, and the set of adjusted alpha maps.

[0148] In one embodiment, the user pose selected set of sampled views includes two or more sampled views; and the display image is generated by performing an image blending operation on two or more intermediate images generated for the current view from the set of layered images and the set of adjusted alpha maps.

[0149] In various exemplary embodiments, an apparatus, system, device, or one or more other computing devices performs any or part of the foregoing methods as described. In one embodiment, a non-transitory computer-readable storage medium stores software instructions that, when executed by one or more processors, cause execution of the methods described herein.

[0150] Although separate embodiments are discussed herein, it should be noted that any combination of the embodiments and / or sub-embodiments discussed herein may be combined to form further embodiments.

[0151] 6. Implementation Mechanism - Hardware Overview According to an embodiment, the techniques described herein are implemented by one or more special purpose computing devices. The special purpose computing devices may be fixedly configured to perform the techniques, or may include digital electronic devices such as one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) permanently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques according to program instructions in firmware, memory, other storage, or combinations. Such special purpose computing devices may also combine custom fixed configuration logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other devices incorporating fixed configuration and / or program logic to implement these techniques.

[0152] 5 is a block diagram illustrating a computer system 500 in which an exemplary embodiment of the invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled to bus 502 for processing information. Hardware processor 504 may be, for example, a general-purpose microprocessor.

[0153] Computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 502 for storing information and instructions to be executed by processor 504. Main memory 506 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions, when stored in a non-transitory storage medium accessible to processor 504, render computer system 500 into a special purpose machine customized to perform the operations specified in the instructions.

[0154] Computer system 500 further includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504 .

[0155] A storage device 510, such as a magnetic or optical disk or semiconductor RAM, is provided and coupled to bus 502 for storing information and instructions.

[0156] Computer system 500 may be coupled via bus 502 to a display 512, such as a liquid crystal display, for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to bus 502 for communicating information and command selections to processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to processor 504 and for controlling cursor movement on display 512. The input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), allowing the device to specify a position in a plane.

[0157] Computer system 500 may use customized fixed logic, one or more ASICs or FPGAs, firmware and / or program logic that in combination with the computer system configures or programs computer system 500 as a special purpose machine to implement the techniques described herein. According to one embodiment, the techniques are performed by computer system 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 510. Execution of the sequences of instructions contained in main memory 506 causes processor 504 to perform the process steps described herein. In alternative embodiments, fixed circuitry may be used in place of or in combination with software instructions.

[0158] The term "storage media" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tape or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROMs and EPROMs, flash EPROMs, NVRAMs, any other memory chips or cartridges.

[0159] Storage media is distinct from, but may be used in conjunction with, transmission media, which participate in transferring information between storage media. For example, transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0160] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 500 can receive the data on the telephone line and convert the data to an infrared signal using an infrared transmitter. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on the bus 502. The bus 502 carries the data to main memory 506, from which the processor 504 retrieves and executes the instructions. The instructions received by main memory 506 may optionally be stored on storage device 510 either before or after execution by processor 504.

[0161] Computer system 500 also includes a communication interface 518 coupled to bus 502. The communication interface 518 provides a two-way data communication coupling to a network link 520 that is connected to a local network 522. For example, the communication interface 518 may be an Integrated Services Digital Network (ISDN) card, cable modem, satellite modem, or modem to provide a data communication connection to a corresponding type of telephone line. As another example, the communication interface 518 may be a Local Area Network (LAN) card to provide a data communication connection to a compatible LAN. A wireless link may also be implemented. In any such implementation, the communication interface 518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0162] Network link 520 typically provides data communication through one or more networks to other data devices. For example, network link 520 may provide a connection through local network 522 to a host computer 524 or to data equipment operated by an Internet Service Provider (ISP) 526. ISP 526 provides data communication services through the world-wide packet data communication network now commonly referred to as the "Internet" 528. Local network 522 and Internet 528 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 520 and through communication interface 518, which carry the digital data to and from computer system 500, are exemplary forms of transmission media.

[0163] Computer system 500 can send messages and receive data, including program code, through the network(s), network link 520 and communication interface 518. In the Internet example, a server 530 might transmit a requested code for an application program through Internet 528, ISP 526, local network 522 and communication interface 518.

[0164] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510, or other non-volatile storage for later execution.

[0165] 7. Equivalents, Extensions, Substitutions, etc. In the foregoing specification, exemplary embodiments of the present disclosure have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indication of what is and is intended by the applicant to be the invention is the specific form in which the claims of the patent granted on this application, including any subsequent amendments thereto, are patented. The definitions, if any, expressly set forth herein for the terms contained in such claims govern the meaning of such terms used in the claims. Thus, no limitations, elements, characteristics, features, advantages or attributes not expressly set forth in the claims should limit the scope of such claims in any manner. Thus, the specification and drawings are to be regarded in an illustrative and not restrictive sense.

[0166] Various aspects of the present invention can be understood from the following enumerated example embodiments (EEE). [EEE1] receiving a multiview input image, the multiview input image covering a plurality of sampled views into an image space represented in the multiview input image; generating from the multi-view input image a multi-view layered image stack of a plurality of layered images in a first dynamic range for the plurality of sampled views, a plurality of alpha maps for the plurality of layered images, and a plurality of beta scale maps for the plurality of layered images; determining a target view of a viewer with respect to the image space, the target view being determined based at least in part on a user pose data portion generated from user pose tracking data collected while the viewer is viewing the rendered image on an image display; selecting a set of user pose-selected sampled views from among the plurality of sampled views represented in the multiview input image using the target view of a viewer; encoding a set of layered images for the set of user pose selected sampled views in the plurality of layered images of the multiview layered image stack, along with a set of alpha maps for the set of user pose selected sampled views in the plurality of alpha maps of the multiview layered image stack and a set of beta scale maps for the set of user pose selected sampled views in the plurality of beta scale maps of the multiview layered image stack, into a video signal to cause a recipient device of the video signal to generate a display image from the set of layered images for rendering on the image display. method. [EEE2] The method of claim 1, wherein the set of beta scale maps can be used to apply a scaling operation to the set of layer composition images to generate a set of scaled layer composition images of a second dynamic range for the set of user pose selected sampled views, the second dynamic range being different from the first dynamic range. [EEE3] 3. The method of claim 1 or 2, wherein the display image represents one of a standard dynamic range image, a high dynamic range image, or a display-mapped image optimized for rendering on a target image display. [EEE4] The method of any one of EEE1 to 3, wherein the multi-view input image includes a plurality of single-view input images for the plurality of sampled views; the plurality of single-view images in the first dynamic range are generated from the plurality of single-view input images used to generate the plurality of layer composition images; and each single-view image in the first dynamic range in the plurality of single-view images in the first dynamic range corresponds to a respective sampled view in the plurality of sampled views and is divided into a respective layer composition image for each sampled view in the plurality of layer composition images. [EEE5] The method of claim 4, wherein the multiple single view input images for the multiple sampled views are used to generate a second multiple single view images of a different dynamic range for the multiple sampled views; the second multiple single view images of the different dynamic range include a second single view image of the different dynamic range for each sampled view; the multiple beta scale maps include a respective beta scale map for each sampled view; and the respective beta scale maps include beta scale data used to perform a beta scaling operation on the single view image of the first dynamic range to generate a beta scaled image of the different dynamic range that approximates the second single view image of the different dynamic range. [EEE6] The method according to EEE5, wherein the beta scaling operation includes one of simple scaling with a scaling factor or applying one or more codeword mapping relations to map codewords of the single-view image of the first dynamic range to generate corresponding codewords of the beta-scaled image of the different dynamic range. [EEE7] 7. The method of claim 5 or 6, wherein the beta scaling operation is performed instead of one or more of a global tone mapping, a local tone mapping, a display mapping operation, a color space conversion, a linear mapping, or a non-linear mapping. [EEE8] The method of any one of EEE1 to 7, wherein the set of layer composed images for the set of user pose selected sampled views is encoded in a base layer of the video signal. [EEE9] The method of any one of EEE1 to 8, wherein the set of alpha maps and the set of beta scale maps for the set of user pose selected sampled views are carried in the video signal as image metadata in a data container separate from the set of layer composed images. [EEE10] The method of any one of EEE1 to EEE9, wherein the plurality of layer composition images include layer composition images for sampled views in the plurality of sampled views, the layer composition images including different image layers at different depth sub-ranges from a view position of the sampled views. [EEE11] decoding from the video signal a set of first dynamic range layered images for a set of user pose selected sampled views, the set of user pose selected sampled views being selected based on user pose data from a plurality of sampled views covered by a multiview source image, the multiview source images being used to generate a corresponding multiview layered image stack; and the corresponding multiview layered images being used to generate the set of layered images; decoding a set of alpha maps for said set of user pose selected sampled views from said video signal; adjusting alpha values ​​in the set of alpha maps for the set of user pose selected sampled views using a viewer's current view to generate adjusted alpha values ​​in a set of adjusted alpha maps for the current view; and rendering on a target image display a display image derived from said set of layered images and said set of adjusted alpha maps. method. [EEE12] The method of claim 8, wherein a set of beta scale maps for the set of user pose selected sampled views is decoded from the video signal; the display images are of a second dynamic range different from the first dynamic range; and the display images are generated from the set of beta scale maps, the set of layer composition images, and the set of adjusted alpha maps. [EEE13] The method of any one of claims 1 to 5, wherein the set of user pose selected sampled views includes two or more sampled views; and the display image is generated by performing an image blending operation on two or more intermediate images generated for a current view from the set of layered images and the set of adjusted alpha maps. [EEE14] Apparatus for carrying out the method according to any one of EEE1 to EEE13. [EEE15] A non-transitory computer-readable storage medium storing software instructions that, when executed by one or more processors, cause performance of the methods described in any one of EEE1 to EEE13.

Claims

1. receiving a multiview input image, the multiview input image covering a plurality of sampled views into an image space represented in the multiview input image; generating from the multiview input image a multiview layered image stack of a plurality of layered images in a first dynamic range for the plurality of sampled views, a plurality of alpha maps for the plurality of layered images, and a plurality of beta scale maps for the plurality of layered images; determining a target view of the viewer with respect to the image space, the target view being determined at least in part based on a user pose data portion generated from user pose tracking data collected while the viewer is viewing the rendered image on an image display; using the target view of a viewer to select a set of user pose selected sampled views from among the plurality of sampled views represented in the multiview input image; encoding a set of layered images for the set of user pose selected sampled views in the plurality of layered images of the multiview layered image stack, together with a set of alpha maps for the set of user pose selected sampled views in the plurality of alpha maps of the multiview layered image stack and a set of beta scale maps for the set of user pose selected sampled views in the plurality of beta scale maps of the multiview layered image stack, into a video signal to cause a recipient device of the video signal to generate a display image from the set of layered images for rendering on the image display. method.

2. 2. The method of claim 1, wherein the set of beta scale maps can be used to apply a scaling operation to the set of layer composition images to generate a set of scaled layer composition images of a second dynamic range for the set of user pose selected sampled views, the second dynamic range being different from the first dynamic range.

3. The method of claim 1 , wherein the display image represents one of a standard dynamic range image, a high dynamic range image, or a display-mapped image optimized for rendering on a target image display.

4. 2. The method of claim 1 , wherein the multi-view input image includes a plurality of single-view input images for the plurality of sampled views; the plurality of single-view images in the first dynamic range are generated from the plurality of single-view input images used to generate the plurality of layer composition images; and each single-view image in the first dynamic range in the plurality of single-view images in the first dynamic range corresponds to a respective sampled view in the plurality of sampled views and is divided into a respective layer composition image for each sampled view in the plurality of layer composition images.

5. 5. The method of claim 4, wherein the multiple single-view input images for the multiple sampled views are used to generate a second multiple single-view images of a different dynamic range for the multiple sampled views; the second multiple single-view images of the different dynamic range include a second single-view image of the different dynamic range for each sampled view; the multiple beta scale maps include a respective beta scale map for each sampled view; and the respective beta scale maps include beta scale data used to perform a beta scaling operation on the single-view image of the first dynamic range to generate a beta scaled image of the different dynamic range that approximates the second single-view image of the different dynamic range.

6. 6. The method of claim 5, wherein the beta scaling operation includes one of: simple scaling with a scaling factor or applying one or more code word mapping relations to map code words of the single-view image in the first dynamic range to generate corresponding code words of the beta-scaled image in the different dynamic range.

7. The method of claim 5 , wherein the beta scaling operation is performed in place of one or more of a global tone mapping, a local tone mapping, a display mapping operation, a color space conversion, a linear mapping, or a non-linear mapping.

8. The method of claim 1 , wherein the set of layered images for the set of user pose selected sampled views are encoded in a base layer of the video signal.

9. 2. The method of claim 1 , wherein the set of alpha maps and the set of beta scale maps for the set of user pose selected sampled views are carried in the video signal as image metadata in a data container separate from the set of layer composed images.

10. 2. The method of claim 1, wherein the plurality of layer composition images includes layer composition images for sampled views in the plurality of sampled views, the layer composition images including different image layers at different depth subranges from a view position of the sampled views.

11. decoding from the video signal a set of first dynamic range layered images for a set of user pose selected sampled views selected based on user pose data from a plurality of sampled views covered by a multiview source image, the multiview source images used to generate a corresponding multiview layered image stack; and the corresponding multiview layered images used to generate the set of layered images; decoding a set of alpha maps for said set of user pose selected sampled views from said video signal; adjusting alpha values ​​in the set of alpha maps for the set of user pose selected sampled views using a viewer's current view to generate adjusted alpha values ​​in a set of adjusted alpha maps for the current view; and causing a display image derived from said set of layered images and said set of adjusted alpha maps to be rendered on a target image display; a set of beta scale maps for the set of user pose selected sampled views are decoded from the video signal; the display image is of a second dynamic range different from the first dynamic range; the display image is generated from the set of beta scale maps, the set of layer composition images, and the set of adjusted alpha maps. method.

12. 11. The method of claim 10, wherein the set of user pose selected sampled views includes two or more sampled views; and the display image is generated by performing an image blending operation on two or more intermediate images generated for a current view from the set of layer composition images and the set of adjusted alpha maps.

13. Apparatus for carrying out the method according to any one of claims 1 to 12.

14. A non-transitory computer readable storage medium storing software instructions which, when executed by one or more processors, cause the performance of a method according to any one of claims 1 to 12.

Images