Support for multiple target display types

Abstract

A sequence of base layer images for the base reference image display and a set of one or more sequences of beta scale maps corresponding to the one or more non-base reference image displays are generated. In the set of one or more sequences of beta scale maps, a subset of the one or more sequences of beta scale maps is determined based at least in part on a display capability of the target image display. The sequence of base layer images, along with the subset of the one or more sequences of beta scale maps, are encoded into a video signal such that a recipient device of the video signal generates a sequence of display images from the sequence of base layer images and the subset of the one or more sequences of beta scale maps for rendering on an image display.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to the following priority applications: U.S. Provisional Application No. 63 / 305,644, filed February 01, 2022, and European Application No. 22156124.4, filed February 10, 2022.

[0002] technology The present disclosure relates generally to image processing operations. More particularly, certain embodiments of the present disclosure relate to video codecs. [Background technology]

[0003] As used herein, the term "dynamic range" (DR) may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, e.g., from darkest black (shadows) to brightest white (highlights). In this sense, DR relates to "scene-based" intensities. DR may also relate to the ability of a display device to fully or approximately render an intensity range of a particular width. In this sense, DR relates to "display-based" intensities. Unless a particular meaning is expressly specified as having a special significance at any point in the description herein, it should be presumed that the terms may be used in either sense, e.g., interchangeably.

[0004] As used herein, the term high dynamic range (HDR) refers to a DR width that spans about 14-15 orders of magnitude or more of HVS. In practice, the DR where humans can simultaneously perceive a wide range of intensity ranges may be somewhat truncated compared to HDR. As used herein, the terms enhanced dynamic range (EDR) or visual dynamic range (VDR), individually or interchangeably, refer to the DR perceivable in a scene or image by an HVS or observer, including eye movements, allowing for some light adaptation changes across the scene or image. As used herein, EDR may refer to a DR that spans 5-6 orders of magnitude. Although perhaps somewhat narrower in relation to true scene-based HDR, EDR still represents a wide DR width and may be referred to as HDR.

[0005] In practice, an image contains one or more color components / channels of a color space (e.g., luma Y and chroma Cb and Cr), each of which is represented with n bits of precision per pixel (e.g., n=8). Using non-linear luminance coding (e.g., gamma encoding), images with n≦8 (e.g., color 24-bit JPEG images) are considered standard dynamic range images, while images with n>8 are considered enhanced dynamic range images.

[0006] A reference electro-optical transfer function (EOTF) for a given display characterizes the relationship between the color values ​​of the input video signal (e.g., luminance represented in a codeword of the codeword representing the image) and the output screen color values ​​produced by the display (e.g., screen luminance represented in a display drive value of the display drive values ​​used to render the image). For example, “Video Signal Processing: A Reference for a Flat Panel Display,” IEEE Trans. Computer Vision, vol. 13, No. 10, 2003, which is incorporated herein by reference in its entirety, defines a reference EOTF for flat panel displays. Given a video stream, information about its EOTF can be embedded in the bitstream as (image) metadata. The term “metadata” in this specification relates to any auxiliary information transmitted as part of the encoded bitstream that assists a decoder to render a decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, as described herein. [Non-Patent Document 1] ITU Rec. ITU-R BT.1886 "Reference electro-optical transfer function for flat panel displays used in HDTV studio production" (March 2011)

[0007] As used herein, the term "PQ" refers to perceptual luminance amplitude quantization. HVS responds highly nonlinearly to increasing light levels. A human's ability to see a stimulus is affected by the luminance of the stimulus, the magnitude of the stimulus, the spatial frequencies that make up the stimulus, and the luminance level to which the eye is adapted at the particular moment the stimulus is viewed. In some embodiments, the perceptual quantization function maps linear input gray levels to output gray levels that better match contrast sensitivity thresholds in the human visual system. Exemplary PQ mapping functions are described in SMPTE, "Perceptual Quantization of Brightness and Brightness," which is incorporated herein by reference in its entirety. Here, for all luminance levels (e.g., stimulus levels), given a fixed stimulus size, the smallest visible contrast increment at that luminance level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to the HVS model). [Non-Patent Document 2] SMPTE ST 2084:2014 "High Dynamic Range EOTF of Mastering Reference Displays"

[0008] 302~1,000cd / m 2Displays supporting luminance of 1,000 or 1,000 nits represent low dynamic range (LDR), also called standard dynamic range (SDR), as opposed to EDR (or HDR). EDR content may be displayed on EDR displays supporting higher dynamic range (e.g., 1,000 nits to 5,000 nits or more). Such displays may be defined using an alternative EOTF that supports higher luminance capabilities (e.g., 0 to 10,000 nits or more). Exemplary (e.g., HDR, Hybrid Log-Gamma or HLG) EOTFs are defined in SMPTE 2044 and IEEE 802.11b, IEEE 802.11b, IEEE 802.11b, IEEE 802.11c, IEEE, 802.11b ... [Non-Patent Document 3] Rec. ITU-R BT.2060, “Image parameter values ​​for high dynamic range television for use in production and international program exchange” (06 / 2017) [Non-Patent Document 4] ITU Rec. ITU-R BT.3040-2, “Parameter values ​​for ultra-high definition television systems for production and international program exchange” (October 2015)

[0009] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art by mere inclusion in this section. Similarly, it should not be assumed that issues identified with respect to one or more approaches were recognized in any prior art based on this section, unless otherwise indicated. [Brief description of the drawings]

[0010] The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference symbols refer to similar elements and in which:

[0011] [Figure 1] FIG. 1 illustrates an exemplary process flow for generating base layer images and beta scale maps.

[0012] [Figure 2A] An exemplary beta scaling and image data server is shown. [Figure 2B] 1 illustrates an exemplary downstream recipient device.

[0013] [Figure 3A] 1 illustrates an exemplary operating scenario in which beta scale maps and base layer images are sent in a video bitstream by an upstream device to a downstream recipient device and received by the downstream recipient device. [Figure 3B] 1 illustrates an exemplary operating scenario in which beta scale maps and base layer images are sent in a video bitstream by an upstream device to a downstream recipient device and received by the downstream recipient device. [Figure 3C]1 illustrates an exemplary operating scenario in which beta scale maps and base layer images are sent in a video bitstream by an upstream device to a downstream recipient device and received by the downstream recipient device. [Figure 3D] 1 shows an exemplary control curve associated with the interpolation or extrapolation operation of beta scaling data.

[0014] [Figure 4A] An exemplary process flow is shown. [Figure 4B] An exemplary process flow is shown.

[0015] [Diagram 5] 1 illustrates an exemplary hardware platform upon which a computer or computing device described herein may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Exemplary embodiments relating to supporting multiple target display types are described herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail in order to avoid unnecessarily obscuring, obscuring, or obfuscating the present invention.

[0017] Exemplary embodiments are described herein according to the following summary. 1. General Overview 2. Beta Scaling and Base Layer Image Data 3. Upstream and downstream devices 4. Example Operational Scenarios 5. Exemplary Process Flow 6. Implementation Mechanism - Hardware Overview 7. Equivalents, Extensions, Substitutions, and Miscellaneous

[0018] 1. General Overview This overview provides a basic description of some aspects of exemplary embodiments of the present invention. It should be noted that this overview is not an extensive or exhaustive summary of aspects of the exemplary embodiments. Moreover, it should be noted that this overview is not intended to be understood as identifying any particularly important aspects or elements of the exemplary embodiments, nor as defining any scope of the exemplary embodiments in particular or the present invention in general. It should be understood that this overview merely presents some concepts related to the exemplary embodiments in a condensed and simplified form and is merely a conceptual prelude to the more detailed description of the exemplary embodiments that follows. It should be noted that although separate embodiments are discussed herein, any combination of the embodiments and / or partial embodiments discussed herein may be combined to form further embodiments.

[0019] The techniques described herein may be used to support optimized rendering of image data for a wide variety of image displays. An upstream device may receive as input a sequence of source or input images (e.g., relatively high quality, consecutive, incremental, etc.) and generate from the sequence of source or input images a corresponding sequence of base layer images (e.g., relatively low quality, consecutive, incremental, etc.) and one or more sequences of beta scaling maps (e.g., indicating beta scaling factors, beta scaling methods, etc.). The sequence of base layer images may be used to support optimized image rendering for a base reference (e.g., SDR, etc.) image display. The one or more sequences of beta scaling maps may be used in combination with the sequence of base layer images to support optimized image rendering for one or more non-base reference (e.g., HDR, etc.) image displays having capabilities different from those of the base reference image display. Each sequence of beta scaling maps in the one or more sequences of beta scaling maps may be used in combination with the sequence of base layer images to support optimized image rendering for a respective non-base reference image display in one or more non-base reference (e.g., HDR, etc.) image displays.

[0020] As used herein, a beta scale map refers to an image that includes beta scaling data for a corresponding base layer image of a base reference color grade optimized for image rendering operations with a base reference image display. The beta scaling data in the beta scale map includes a beta scaling method selector and / or a beta scaling factor to be used to scale a corresponding base layer image of a base reference color grade to a non-base layer image of a non-base reference color grade optimized for image rendering operations with a non-base reference image display in one or more non-base reference image displays. The beta scale map may be of the same or coarser spatial resolution compared to the corresponding base layer image.

[0021] The beta scaling data in the beta scale map may be used in a scaling operation that aggregates other image processing operations, including, but not limited to, reshaping and / or display operations that convert a corresponding base layer image of a base reference color grade (e.g., a relatively narrow dynamic range, a relatively small color gamut, etc.) into a non-base layer image of a non-base reference color grade that is different from the base reference color grade (e.g., a relatively high dynamic range, a relatively wide color gamut, etc.).

[0022] In addition to supporting image rendering optimized for the display capabilities of the base and non-base reference image displays, the sequence of base layer images and the one or more sequences of beta scale maps can be used to support target image displays that differ from all said base and non-base reference image displays.

[0023] In one example, a sequence of beta scale maps in one or more sequences of beta scale maps can be sent, directly or indirectly, from an upstream device along with a sequence of base layer images to a downstream recipient device operating with a target image display that is not one of said base or non-base reference image displays. The sequence of beta scale maps can be used to generate a target display-specific sequence of beta scale maps, e.g., via interpolation or extrapolation. A target display-specific sequence of images with a target display-specific color grade optimized for image rendering with the display capabilities of the target image display can be generated using the target display-specific sequence of beta scale maps in combination with the base layer image sequence. The upstream device can select a sequence of beta scale maps from all available sequences of beta scale maps based at least in part on the information of the display capabilities of the target image display. For example, the sequence of beta scale maps can correspond to a color grade that is closest to a target display-specific color grade optimized for image rendering with the display capabilities of the target image display.

[0024] In another example, two sequences of beta scale maps, e.g., selected from among sequences of beta scale maps, may be sent directly or indirectly from an upstream device along with a sequence of base layer images to a downstream recipient device operating with a target image display that is neither the base reference image display nor the non-base reference image display. The two sequences of beta scale maps may be used to generate a target display-specific sequence of beta scale maps, e.g., via interpolation or extrapolation. The upstream device may select the two sequences of beta scale maps from all available sequences of beta scale maps based at least in part on information of the display capabilities of the target image display. For example, the two sequences of beta scale maps may correspond to two color grades that are closest to a target display-specific color grade optimized for image rendering using the display capabilities of the target image display.

[0025] The total number of sequences of beta scale maps generated by the upstream device can be adjusted or selected to increase or decrease the approximation error. For example, more sequences of beta scale maps can be generated (e.g., in advance, simultaneously with image streaming, etc.) by the upstream device to reduce the approximation error in the interpolation / extrapolation operation. Additionally, optionally, or alternatively, the interpolation / extrapolation operation used to generate the target display-specific sequence of beta scale maps from the received sequence of beta scale maps of the reference image display can be performed linearly or nonlinearly. In some operating scenarios, a control curve (e.g., linear, nonlinear, etc.) can be used to facilitate the interpolation / extrapolation operation from the received sequence of beta scale maps to the target display-specific sequence of beta scale maps.

[0026] Exemplary embodiments described herein relate to encoding image content. A sequence of base layer images for a base reference image display and a set of one or more sequences of beta scale maps corresponding to one or more non-base reference image displays are generated. In the set of one or more sequences of beta scale maps, a subset of the one or more sequences of beta scale maps is determined based at least in part on a display capability of a target image display. The sequence of base layer images, together with the subset of the one or more sequences of beta scale maps, are encoded into a video signal, causing a recipient device of the video signal to generate a sequence of display images for rendering on an image display from the sequence of base layer images and the subset of the one or more sequences of beta scale maps.

[0027] Exemplary embodiments described herein relate to decoding image content. A sequence of base layer images is decoded from a video signal. The sequence of base layer images is optimized for image rendering on a base reference image display. A subset of one or more sequences of beta scale maps is decoded from the video signal. The subset of one or more sequences of beta scale maps is determined from a set of one or more sequences of beta scale maps corresponding to one or more non-base reference image displays based on a display capability of a target image display. A sequence of display images is derived from the sequence of base layer images and the subset of one or more sequences of beta scale maps and rendered on the target image display.

[0028] In some exemplary embodiments, the mechanisms described herein form part of a media processing system including, but not limited to, a cloud-based server, a mobile device, a virtual reality system, an augmented reality system, a heads-up display device, a helmet-mounted display device, a cave-type system, a wall-sized display, a video game device, a display device, a media player, a media server, a media production system, a camera system, a home system, a communications device, a video processing system, a video codec system, a studio system, a streaming server, a cloud-based content services system, a handheld device, a gaming console, a television, a cinema display, a laptop computer, a netbook computer, a tablet computer, a cellular wireless telephone, an e-reader, a point-of-sale terminal, a desktop computer, a computer workstation, a computer server, a computer kiosk, or any of a variety of other types of terminals and media processing units.

[0029] Various modifications to the preferred embodiments and generic principles and features described herein will be readily apparent to those skilled in the art, and thus the present disclosure is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features described herein.

[0030] 2. Beta Scaling and Base Layer Image Data FIG. 1 illustrates an exemplary process flow for generating a base layer image and a beta scale map. The process flow may be implemented as part of an upstream image processing device, such as a video encoder device, a video streaming server, and / or a beta scaling and image data server. Additionally, optionally, or alternatively, the process flow may be implemented in a separate or associated device, such as an image pre-processing device operating in conjunction with the video encoder device, the video streaming server, and / or a beta scaling and image data server. Some or all of the process flow may be implemented or performed using one or more of a computing processor, audio and video codecs, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other distribution formats, a digital signal processor, a graphics processing unit or GPU, and the like.

[0031] The beta scale map and base layer image generator 104 comprises software, hardware, a combination of software and hardware, etc. configured to receive input or source images 102 (e.g., time-sequential, sequential, multi-view, etc.). These input images (102) may be received from a video source or retrieved from a video data store. The input images (102) may be digitally captured (e.g., by a digital camera, etc.), generated by converting analog camera photographs captured on film to a digital format, computer-generated (e.g., using computer animation, image rendering, etc.), etc. The input images (102) may be images associated with one or more of a movie release, an archived media program, a media program library, a video recording / clips, a media program, a TV program, user-generated video content, etc.

[0032] In some operational scenarios, the beta scale map and base layer image generator (104) may perform an image content mapping operation on a sequence of input images (102) to generate a corresponding sequence of base layer images 106 that depict the same visual content as the input images (102), as well as one or more sequences of non-base layer images that depict the same visual content as the sequence of input images (102). Exemplary image content mapping operations may include some or all of video editing operations, video conversion operations, color grading operations, dynamic range mapping operations, local and / or global reshaping operations, display management operations, video special effects operations, etc.

[0033] Some or all of these operations may be performed automatically without human input (e.g., using a content mapping tool, a color grading toolkit, etc.). Additionally, optionally, or alternatively, some or all of these operations may be performed manually, automatically with human input, etc.

[0034] By way of example and not limitation, a sequence of non-base layer images of a non-base reference color grade (e.g., non-base dynamic or luminance range, non-base color gamut, etc.) that differs from the base reference color grade of the sequence of base layer images (106) may include a particular sequence of non-base layer images of a particular non-base reference color grade. The particular sequence of non-base layer images may be initially generated from an input image (102) using some or all of these image processing operations, including color grading operations that are performed fully automatically or partially automatically with human input.

[0035] Local and / or global reshaping operations may then be performed (e.g., automatically, without human input, etc.) on the particular sequence of non-base layer images generated from the input image (102) to generate a sequence of base layer images (106), which may be of a relatively narrow dynamic range (or luminance range). Additionally, optionally, or alternatively, one or more sequences of non-base layer images may be generated from the particular sequence of non-base layer images and / or the sequence of base layer images (106) using additional local and / or global reshaping operations, additional content mapping operations, additional display management operations, etc.

[0036] The sequence of base layer images may be of a base reference color grade optimized for image rendering using base reference image display 112-0. The one or more sequences of non-base layer images, including the particular sequence of non-base layer images and any additionally generated sequences of non-base layer images, may be of one or more non-base reference color grades each optimized for image rendering using one or more different sets of display capabilities of one or more non-base reference image displays 112-1 through 112-N, where N represents a positive non-zero integer.

[0037] The beta scale map and base layer image generator (104) can use the sequence of base layer images and the one or more sequences of non-base layer images to generate one or more sequences of beta scale maps 108-1 through 108-N, where the one or more sequences of beta scale maps (108-1 through 108-N) correspond to the one or more sequences of non-base layer images, respectively.

[0038] Each sequence of beta scale maps in the one or more sequences of beta scale maps (108-1 to 108-N) corresponds to a respective sequence of non-base layer images in the one or more sequences of non-base layer images and is generated from the respective sequence of non-base layer images and the sequence of base layer images.

[0039] Each sequence of beta scale maps in the one or more sequences of beta scale maps (108-1 through 108-N) includes beta scaling data used to scale the sequence of base layer images to a respective sequence of non-base layer images in the one or more sequences of non-base layer images. For example, the sequence of base layer images may cover a sequence of time points (e.g., consecutive, sequential, etc.) during the duration of a media program. Each base layer image in the sequence of base layer images may be rendered for a respective time point in the sequence of time points. Similarly, each of the sequence of beta scale maps and the sequence of non-base layer images generated from the sequence of base layer images and the sequence of beta scale maps covers the same sequence of time points. The base layer image for each time point may be scaled based on the beta scaling data in the respective beta scale map in the sequence of beta scale maps to generate a respective non-base layer image in the sequence of non-base layer images for each time point.

[0040] Beta scaling may be incorporated in place of or used to implement any, partial, or full of other image processing operations, including, but not limited to, reshaping operations, content mapping with little or no human input, content mapping with human input, tone mapping, color space conversion, display mapping, PQ, non-PQ, linear or non-linear encoding, image blending, image mixing, linear image mapping, non-linear image mapping, application of EOTF, application of EETF, application of OETF, spatial or temporal downsampling, spatial or temporal upsampling, spatial or temporal resampling, chroma sampling format conversion, etc.

[0041] The beta scaling operations described herein may be implemented as simple scaling operations applying (e.g., linear, etc.) multiplications and / or additions. Additionally, optionally, or alternatively, the beta scaling operations described herein may be implemented in complex or non-linear scaling operations, including but not limited to LUT-based scaling. The beta scaling operations may be performed only once at run-time to achieve or generate the (equivalent) effect of other image processing operations for which the beta scaling operations are performed. As a result, relatively complex image processing operations permeating image processing chains / pipelines / frameworks may be avoided or significantly simplified under the beta scaling techniques as described herein.

[0042] The beta scaling described herein may support global mapping (e.g., global tone mapping, global reshaping, etc.), local mapping (e.g., local tone mapping, local reshaping, etc.), or a combination of global and local mapping.

[0043] Exemplary beta scaling operations can be found in U.S. patent application Ser. No. 63 / 305,626, entitled "BETA SCALE DYNAMIC DISPLAY MAPPING," by Ajit Ninan and Gregory Ward, filed on February 1, 2022 (Attorney Docket No. 60175-0480; D20114USP1), the entire contents of which are incorporated by reference herein as if fully set forth herein.

[0044] 3. Upstream and downstream devices 2A illustrates an exemplary beta scaling and image data server 200, such as a video encoder device, a video streaming server, or an upstream device, including a beta scale map and base layer image receiver 202, a beta scaling and image data encoder 204, etc. Some or all of the components of the beta scaling and image data server (200) may be implemented by one or more devices, modules, units, etc., such as software, hardware, a combination of software and hardware, etc.

[0045] In some operating scenarios, the beta scaling and image data server (200) may include or implement processing blocks of a process flow such as that shown in Figure 1. Additionally, optionally, or alternatively, the beta scaling and image data server (200) may operate in conjunction with a separate upstream device that includes or implements processing blocks of a process flow such as that shown in Figure 1.

[0046] The beta scale map and base layer image receiver (202) includes software, hardware, a combination of software and hardware, etc., configured to receive or access SDR and HDR layered image stacks, a sequence of base layer images (e.g., 106 in FIG. 1 ), and one or more sequences of beta scale maps 108 (e.g., 108-1, 108-2, ... 108-N in FIG. 1 ) originating from an internal or external beta scale map and base layer image source.

[0047] The sequence of base layer images (106) describes a 2D or 3D visual scene for a sequence of time points covering a duration (e.g., a media program). The one or more sequences of beta scale maps (108) specify or define one or more sequences of scaling operations and / or scaling operation parameters to be performed or executed on the sequence of base layer images to obtain or generate one or more sequences of non-base layer images, each of which describes the same 2D or 3D visual scene. More specifically, each sequence of beta scale maps in the one or more sequences of beta scale maps (108) specifies or defines a sequence of scaling operations and / or scaling operation parameters—of one or more sequences of scaling operations and / or scaling operation parameters—to be performed or executed on the sequence of base layer images to obtain or generate a respective sequence of non-base layer images in the one or more sequences of non-base layer images.

[0048] Each of the sequence of base layer images and the one or more sequences of non-base layer images represents a base reference color grade and one or more non-base reference color grades, respectively. As used herein, a color grade may refer to a (e.g., released, professionally color graded, color graded by a user or video professional, reconstructed, predicted, etc.) version of a video image. These color grades include a set or sequence of corresponding images depicting the same visual scene or semantic content, but may differ in one or more of different dynamic range levels, different color space sizes, different EOTFs, different color space types, etc. These color grades may be optimized for image rendering using different sets of display capabilities of a base reference image display (e.g., 112-0, etc., of FIG. 1) and one or more non-base reference image displays (e.g., 112-1, ... 112-N, etc., of FIG. 1), respectively.

[0049] The beta scaling and image data server (200), or a beta scaling and image data encoder 204 therein, may generate a video bitstream 206 encoded at a non-base color grade (e.g., requested, pre-configured, pre-generated, etc.) of the video content depicted in the sequence of base layer images.

[0050] In one example, the non-basic color grade or video bitstream (206) may be generated by the beta scaling and image data encoder (204) in response to the server (200) receiving a request (e.g., in the form of one or more request messages) from a downstream recipient device or video streaming client.

[0051] In another example, a non-basic color grade or video bitstream (206) may be generated or pre-generated by the beta scaling and image data encoder (204) without the server (200) receiving any request from a downstream recipient device or video streaming client; the generated or pre-generated color grade or video bitstream (206) can then be accessed, downloaded, streamed, or otherwise provided directly or indirectly to a downstream recipient device or video streaming client.

[0052] In some operating scenarios, the beta scaling and image data server (200) may receive, for example, in a request for streaming video content to a downstream recipient device or video streaming client, specific display identification information that identifies the particular image display on which the requested video content is to be rendered.

[0053] The particular display identification information may be used by the beta scaling and image data server (200) to determine whether the particular image display is the same as or a closest match to the base reference image display (112-0) with respect to the particular display capabilities of the particular image display and the base reference display capabilities of the base reference image display.

[0054] In that case, the beta scaling and image data server (200) or a separate upstream device operating in conjunction with the beta scaling and image data server (200) may simply stream a video bitstream (206) that encodes the sequence of base layer images. The video bitstream (206) may not include any sequence of beta scale maps.

[0055] Meanwhile, in response to determining that a particular image display differs from the base reference image display (112-0) with respect to their respective sets of display capabilities, the beta scaling and image data server (200) or a separate upstream device operating in conjunction with the beta scaling and image data server (200) can determine, identify or select one or two closest matching reference image displays from among the one or more non-base reference image displays and proceed to stream a video bitstream (206) in which is encoded a sequence of base layer images and one or more sequences of beta scale maps corresponding to the one or two closest matching reference image displays.

[0056] As used herein, the video content in a video signal (or stream) described herein may include, but is not necessarily limited to, any of audiovisual programs, movies, video programs, TV broadcasts, computer games, augmented reality (AR) content, virtual reality (VR) content, automotive entertainment content, etc.

[0057] As used herein, a "beta scaling and image data server" may refer to one or more upstream devices, video encoders, and / or video streaming servers that prepare and stream video content to one or more video streaming clients, such as video decoders, for rendering at least a portion of the video content on one or more displays. The displays on which the video content is rendered may be part of or operate in conjunction with one or more video streaming clients.

[0058] Exemplary beta scaling and image data servers may include, but are not necessarily limited to, any of the following: a cloud-based beta scaling and image data server located remotely from the video streaming client, a video streaming server connected to the video streaming client through a local wired or wireless network, a VR device, an AR device, an automotive entertainment device, a digital media device, a digital media receiver, a set-top box, a gaming console (e.g., an Xbox), a general-purpose personal computer, a tablet, a dedicated digital media receiver such as an Apple TV or Roku box, and the like.

[0059] The beta scaling and image data server (200) may be used to support AR applications, VR applications, 360° video applications, stereoscopic video applications, real-time video applications, near real-time video applications, non-real-time omnidirectional video applications, automotive entertainment, helmet mounted display applications, heads up display applications, gaming, 2D display applications, 3D display applications, multi-view display applications, and the like.

[0060] 2B illustrates an exemplary downstream recipient device, such as a video client device 250, including a beta scaling and image data recipient 252, a target display beta scale map generator 254, a target display image renderer 256, a target image display 258, etc. Some or all of the components of the video client device (150) may be implemented by one or more devices, modules, units, etc., in the form of software, hardware, a combination of software and hardware, etc.

[0061] Exemplary video client devices described herein may include, but are not necessarily limited to, any of a large screen image display, a home entertainment system, a set-top box and / or an audiovisual device operating in conjunction with an image display, a mobile computing device held by a user / viewer (e.g., in a spatially stable or changing relationship with the user / viewer's eyes), a wearable device that includes or operates in conjunction with an image display, a computing device that includes or operates in conjunction with a head-mounted or head-up display, and the like.

[0062] The video streaming client (250) or the beta scaling and image data receiver (252) can send a request to an upstream device to receive a video bitstream (e.g., 206) in which the video content and the beta scaling data have been encoded. The video content can be adapted based on the beta scaling data to make the video content target-display-specific for image rendering on the target image display (258). In some operating scenarios, the request sent by the video streaming client (250) to the upstream device can include specific display identification information that identifies the (specific) target image display (258) on which the requested video content will be rendered and / or identifies a specific set of display capabilities of the (specific) target image display (258).

[0063] In response to receiving the video bitstream (206), the beta scaling and image data receiver (252) can decode the (e.g., real-time, near real-time, etc.) video bitstream (206) into a sequence of base layer images for a sequence of (e.g., continuous, sequential, etc.) points in time within a time interval or duration of an AR, VR, or immersive video application. The sequence of base layer images represents a base reference color grade optimized for image rendering using a base reference image display (e.g., 112-0 of FIG. 1).

[0064] In addition, the beta scaling and image data receiver (252) retrieves from the video bitstream (206) one or two sequences of beta scale maps corresponding to the sequence of base layer images. The one or two sequences of beta scale maps can be used to adapt or scale the sequence of base layer images to one or two sequences of non-base layer images representing one or two color grades optimized for the one or two non-base reference image displays, respectively. In some operating scenarios, the one or two non-base reference image displays can be selected from a larger set of non-base reference image displays based on specific display identification information provided to the upstream device in a request by the video streaming client (250). More specifically, each of the one or two sequences of beta scale maps can be used to adapt or scale the sequence of base layer images to a respective sequence of non-base layer images in the one or two sequences of non-base layer images of one or two color grades for image rendering on a respective non-base reference image display of the one or two non-base reference image displays.

[0065] The target display beta scale map generator (254) includes software, hardware, a combination of software and hardware, etc. configured to perform interpolation and / or extrapolation operations to convert one or two sequences of received beta scale maps into a sequence of beta scale maps specific to the target display based at least in part on comparing a particular display capability of the target image display (258) with some or all of the display capabilities of the base reference image display (112-1) and one or more non-base reference image displays.

[0066] The target display image renderer (256) includes software, hardware, a combination of software and hardware, etc., configured to receive the decoded sequence of base layer images, perform a beta scaling operation using the sequence of target display specific beta scale maps to generate (e.g., consecutively, sequentially, etc.) a sequence of display images from the decoded sequence of base layer images, and render the sequence of display images on the target image display (258) across or at multiple time points. For example, each target display specific beta scale map in the sequence of target display specific beta scale maps can be used by the target display image renderer (256) to perform a beta scaling operation on a respective base layer image in the decoded sequence of base layer images to generate a respective display image in the sequence of display images.

[0067] 4. Example Operational Scenarios The base reference color grade and one or more non-base reference color grades generated from one or more sets or sequences of beta scale maps in combination with the base reference color grade may support optimized image rendering for only a (suitable) subset of image displays out of the wide variety of image displays for which optimized image rendering may be supported under the techniques described herein.

[0068] In one example, a base reference color grade may be optimized for image rendering on an SDR image display with a relatively low dynamic range (e.g., having a peak luminance of 100 nits, 400 nits, etc.), and one or more non-base reference color grades may be optimized for image rendering on an HDR image display with a relatively high dynamic range (e.g., having a peak luminance of 1000 nits, 4000 nits, 10,000+ nits, etc.).

[0069] Under the techniques described herein, a target display-specific color grade in an intermediate dynamic range (e.g., having a peak luminance of 800 nits, 2000 nits, etc.) between a standard dynamic range (or a relatively low dynamic range) and a high dynamic range (or a relatively high dynamic range) may be generated for optimized image rendering on a mid-range image display between an SDR image display and an HDR image display using a set or sequence of interpolated or extrapolated beta scale maps derived from one or more sets or sequences of beta scale maps in combination with a base reference color grade. Additionally, optionally, or alternatively, a target display-specific color grade in a dynamic range not between the relatively low dynamic range and the relatively high dynamic range (e.g., a mobile phone supported dynamic range, such as lower than the relatively low dynamic range, higher than the relatively high dynamic range, etc.) may be generated for optimized image rendering on a corresponding image display using a set or sequence of interpolated or extrapolated beta scale maps derived from one or more sets or sequences of beta scale maps in combination with a base reference color grade. Thus, the techniques described herein can be used to extend optimized image rendering from only a small number of basic and non-basic reference image displays of reference display capabilities with respect to reference dynamic ranges such as SDR and HDR to a wide range of non-reference (target) image displays of non-reference display capabilities such as intermediate, lower, or higher dynamic ranges with respect to SDR and HDR.

[0070] In another example, the base reference color grade may be optimized for image rendering on a first image display with a relatively narrow color gamut (e.g., Rec. 709 color space), while one or more non-base reference color grades may be optimized for image rendering on a second image display with a relatively wide color gamut (e.g., Rec. 2020).

[0071] Under the techniques described herein, a target display-specific color grade of a gamut intermediate between a relatively narrow gamut and a relatively wide gamut (e.g., DCI-P3, smaller than Rec. 2020, etc.) may be generated using a set or sequence of interpolated or extrapolated beta scale maps derived from one or more sets or sequences of beta scale maps in combination with a base reference color grade for optimized image rendering on a mid-range image display between a first image display and a second image display. Additionally, optionally or alternatively, a target display-specific color grade of a gamut not intermediate between a relatively narrow gamut and a relatively wide gamut (e.g., smaller than Rec. 709, higher than Rec. 2020, etc.) may be generated using a set or sequence of interpolated or extrapolated beta scale maps derived from one or more sets or sequences of beta scale maps in combination with a base reference color grade for optimized image rendering on a corresponding image display. Thus, the techniques described herein can be used to extend optimized image rendering from only a small number of basic and non-basic reference image displays with reference display capabilities for reference gamuts such as Rec.709 and Rec.2020 to a wide range of non-reference (target) image displays with non-reference display capabilities such as intermediate, smaller, or wider gamuts relative to Rec.709 and Rec.2020.

[0072] FIG. 3A illustrates an exemplary operating scenario in which two sequences (or sets) of beta scale maps 108-i and 108-(i+1) may be sent by an upstream device to a downstream recipient device and received by the downstream recipient device in a video bitstream that encodes a sequence of base layer images representing a base reference color grade.

[0073] A first sequence of beta scale maps (108-i) can be used in combination with the sequence of base layer images to generate a first non-base reference color grade to support optimized image rendering for a first non-base reference image display 112-i. A second sequence of beta scale maps (108-(i+1)) can be used in combination with the sequence of base layer images to generate a second non-base reference color grade to support optimized image rendering for a second non-base reference image display 112-(i+1).

[0074] The target image display 304-1 may be of a set of target display capabilities that are different from or fall between those of the reference display capabilities of the first non-basic reference image display (112-i) and the second non-basic reference image display (112-(i+1)). As a result, none of the basic reference color grade, the first non-basic reference color grade, and the second non-basic reference color grade may support image rendering optimized for the target image display (304-1).

[0075] A sequence of target display-specific interpolated beta scale maps 302-1 may be generated from a combination of the first sequence of beta scale maps (108-i) and the second sequence of beta scale maps (108-(i+1)) and used in combination with the sequence of base layer images to generate a target display-specific sequence of images for optimized image rendering (e.g., within an approximation error or tolerance) on the target image display (304-1).

[0076] Each target display-specific interpolated beta scale map in the sequence of target display-specific interpolated beta scale maps (302-1) may be generated from a combination of a first respective beta scale map in the first sequence of beta scale maps (108-i) and a second respective beta scale map in the second sequence of beta scale maps (108-(i+1)). One or more of a wide variety of interpolation methods may be used to interpolate the beta scaling data in the first respective beta scale map and the second respective beta scale map into the target display-specific interpolated beta scale map. Exemplary interpolation methods described herein may include, but are not necessarily limited to, any of linear interpolation, piecewise constant interpolation, polynomial interpolation, spline interpolation, function approximation based interpolation, interpolation based at least in part on a control function, and the like.

[0077] By way of example and not limitation, corresponding display capability values, such as peak luminance or brightness levels, of the target image display (304-1), the first non-base reference image display (112-i), and the second non-base reference image display (112-(i+1)) may be used to generate input operating parameters for the interpolation. Some or all of the differences in peak luminance or brightness levels between each pair of all possible pairs of the target image display (304-1), the first non-base reference image display (112-i), and the second non-base reference image display (112-(i+1)) may be generated.

[0078] In a first example, where the same scaling method is specified in each of the first and second (decoded or received) beta scale maps (e.g., for pixels or pixel blocks in the respective base layer images, for background or foreground in the respective base layer images, etc.), some or all of the differences generated from the peak luminance or luminance levels of the target and reference image displays may be used as input operating parameters to calculate or derive interpolation-related ratios or weights, etc. These interpolation-related ratios or weights may be used to linearly or non-linearly interpolate the (pre-interpolated) scaling values ​​specified in the respective first and second beta scale maps into interpolated scaling values ​​in the target display-specific interpolated beta scale map for that scaling method.

[0079] In a second example where the same scaling method is specified in the first and second respective (decoded or received) beta scale maps (e.g., for pixels or pixel blocks in the respective base layer images, for background or foreground in the respective base layer images, etc.), some or all of the differences generated from the peak luminance or brightness levels of the target and reference image displays may be used as input operating parameters to calculate or derive interpolation-related ratios or weights, etc. These interpolation-related ratios or weights may be used to linearly or nonlinearly interpolate the (pre-interpolated) function indices or lookup table indices specified in the first respective beta scale map and the second respective beta scale map into interpolated function indices or lookup table indices in the target display-specific interpolated beta scale map. The interpolated function indices or lookup table indices may be used to look up or generate target display-specific scaling values ​​in a control function or lookup table for that scaling method.

[0080] In a third example where different scaling methods are specified in the first and second respective (decoded or received) beta scale maps (e.g., for pixels or pixel blocks in the respective base layer images, for background or foreground in the respective base layer images, etc.), the scaling data for the different scaling methods may first be converted to scaling values ​​in a common base scaling method (e.g., a scaling method using simple arithmetic operations such as division / subtraction). Some or all of the differences generated from the peak luminance or luminance levels of the target image display and the reference image display may be used as input operating parameters to calculate or derive interpolation-related ratios or weights, etc. These interpolation-related ratios or weights may be used to linearly or nonlinearly interpolate the converted scaling values ​​derived from the scaling data specified in the first respective beta scale map and the second respective beta scale map into interpolated scaling values ​​in the target display-specific interpolated beta scale map for the common base scaling method.

[0081] FIG. 3B shows an exemplary operating scenario in which a single sequence (or set) of beta scale maps 108-1 may be sent by an upstream device to a downstream recipient device and received by the downstream recipient device in a video bitstream that encodes a sequence of base layer images representing a base reference color grade.

[0082] The sequence of beta scale maps (108-1) may be used in combination with the sequence of base layer images to generate a non-base reference color grade to support optimized image rendering for the non-base reference image display 112-1.

[0083] The target image display 304-2 may be a set of target display capabilities that are different from or fall between the reference display capabilities of the base reference image display (112-0) and the non-base reference image display (112-1). As a result, neither the base reference color grade nor the non-base reference color grade may support image rendering optimized for the target image display (304-2).

[0084] A sequence of target display-specific interpolated beta scale maps 302-2 may be generated from the sequence of beta scale maps (108-1) and used in combination with the sequence of base layer images to generate a target display-specific sequence of images for optimized image rendering (e.g., within approximation error or tolerance) on the target image display (304-2).

[0085] Each target display-specific interpolated beta scale map in the sequence of target display-specific interpolated beta scale maps (302-2) may be generated from a respective beta scale map in the sequence of beta scale maps (108-1). One or more of a wide variety of interpolation methods may be used to interpolate the beta scaling data in each beta scale map into the target display-specific interpolated beta scale map. Exemplary interpolation methods described herein may include, but are not necessarily limited to, any of linear interpolation, piecewise constant interpolation, polynomial interpolation, spline interpolation, function approximation based interpolation, interpolation based at least in part on a control function, and the like.

[0086] By way of example and not limitation, corresponding display capability values, such as peak luminance or brightness levels, of the target image display (304-2), the base reference image display (112-0), and the non-base reference image display (112-1) may be used to generate input operating parameters for the interpolation. Some or all differences in peak luminance or brightness levels between each pair among all possible pairs of the target image display (304-2), the base reference image display (112-0), and the non-base reference image display (112-1) may be generated.

[0087] In a first example, where a scaling method is specified in each (decoded or received) beta scale map (e.g., for pixels or pixel blocks in each base layer image, for background or foreground in each base layer image, etc.), default scaling values ​​for the scaling method, corresponding to identity or no scaling operation, may be generated for the base layer image. Some or all of the generated differences from the peak luminance or luminance levels of the target image display and the reference image display may be used as input operation parameters to calculate or derive interpolation-related ratios or weights, etc. These interpolation-related ratios or weights may be used to linearly or nonlinearly interpolate between the (pre-interpolation) scaling values ​​specified in each beta scale map and the default scaling values ​​corresponding to identity or no scaling operation for that scaling method into interpolated scaling values ​​in the target display-specific interpolated beta scale map for that scaling method.

[0088] In a second example where a scaling method is specified in each (decoded or received) beta scale map (e.g., for pixels or pixel blocks in each base layer image, for background or foreground in each base layer image, etc.), default function indices or lookup table indices for that scaling method, corresponding to an identity or no-scaling operation, may be generated for the base layer image. Some or all of the differences generated from the peak luminance or luminance levels of the target image display and the reference image display may be used as input operation parameters to calculate or derive interpolation-related ratios or weights, etc. These interpolation-related ratios or weights may be used to linearly or nonlinearly interpolate between the (pre-interpolation) function indices or lookup table indices specified in each beta scale map and the default function indices or lookup table indices corresponding to the identity or no-scaling operation for that scaling method into interpolated function indices or lookup table indices in the target display-specific interpolated beta scale map. The interpolated function index or lookup table index may be used to look up or generate a target display specific scaling value in a control function or lookup table for that scaling method.

[0089] FIG. 3C shows an exemplary operating scenario in which a single sequence (or set) of beta scale maps 108-1 may be sent by an upstream device to a downstream recipient device and received by the downstream recipient device in a video bitstream that encodes a sequence of base layer images representing a base reference color grade.

[0090] The sequence of beta scale maps (108-1) may be used in combination with the sequence of base layer images to generate a non-base reference color grade to support optimized image rendering for the non-base reference image display 112-1.

[0091] The target image display 304-3 may be a set of target display capabilities that are different from, and not necessarily between, the reference display capabilities of the base reference image display (112-0) and the non-base reference image display (112-1). As a result, neither the base reference color grade nor the non-base reference color grade may support image rendering optimized for the target image display (304-3).

[0092] A target display-specific sequence of extrapolated beta scale maps 302-3 may be generated from the sequence of beta scale maps (108-1) and used in combination with the sequence of base layer images to generate a target display-specific sequence of images for optimized image rendering (e.g., within an approximate error or tolerance) on the target image display (304-3).

[0093] Each target display specific extrapolated beta scale map in the sequence of target display specific extrapolated beta scale maps (302-3) may be generated from a respective beta scale map in the sequence of beta scale maps (108-1). One or more of a wide variety of extrapolation methods may be used to extrapolate the beta scaling data in each beta scale map into a target display specific extrapolated beta scale map. Exemplary extrapolation methods described herein may include, but are not necessarily limited to, any of linear extrapolation, conic extrapolation, polynomial extrapolation, French curve extrapolation, geometric extrapolation, extrapolation based at least in part on a control function, and the like.

[0094] By way of example and not limitation, corresponding display capability values, such as peak luminance or brightness levels, of the target image display (304-2), the base reference image display (112-0), and the non-base reference image display (112-1) may be used to generate input operating parameters for extrapolation. Some or all differences in peak luminance or brightness levels between each pair of all possible pairs of the target image display (304-2), the base reference image display (112-0), and the non-base reference image display (112-1) may be generated.

[0095] In a first example, some or all of the differences generated from the peak luminance or brightness levels of the target image display and the reference image display may be used as input operating parameters to calculate or derive extrapolation-related ratios or weights, etc. These extrapolation-related ratios or weights may linearly or non-linearly extrapolate the (pre-extrapolation) scaling values ​​specified in the respective beta scale maps to extrapolated scaling values ​​in a target display-specific extrapolated beta scale map for that scaling method.

[0096] In a second example, some or all of the differences generated from the peak luminance or brightness levels of the target image display and the reference image display may be used as input operating parameters to calculate or derive extrapolation-related ratios or weights, etc. These extrapolation-related ratios or weights may be used to linearly or non-linearly extrapolate the (pre-extrapolation) function indices or lookup table indices specified in the respective beta scale maps to extrapolated function indices or lookup table indices in the target display-specific extrapolated beta scale map. The extrapolated function indices or lookup table indices may be used to look up or generate target display-specific scaling values ​​in a control function or lookup table for that scaling method.

[0097] For purposes of example only, it has been described that display capability values ​​of an image display, such as peak luminance or brightness level, can be used to derive the interpolated or extrapolated operating parameters. It should be noted that in other embodiments, one or more other display capabilities can be used to derive the interpolated or extrapolated operating parameters in addition to or instead of the peak luminance or brightness level. For example, a mean or average luminance or brightness level can be used in part or in whole to derive the interpolated or extrapolated operating parameters as described herein.

[0098] For purposes of example only, it has been described that the display capability values ​​of the image display can be used to derive the interpolated or extrapolated operating parameters. It should be noted that in other embodiments, one or more non-display capability values ​​can be used in addition to or instead of the display capability values ​​to derive the interpolated or extrapolated operating parameters. For example, a (content-dependent) mean or average luminance or brightness level in the base layer or non-base layer reference images can be calculated for the entire image frame or a portion thereof. The mean or average luminance or brightness levels derived from the base layer reference image and the non-base layer reference image can be used in part or in whole to derive the interpolated or extrapolated operating parameters as described herein.

[0099] Each target display-specific interpolated or extrapolated beta scale map in the sequence of target display-specific interpolated or extrapolated beta scale maps (e.g., 302-1, 302-2, 302-3, etc.) may be used to perform a scaling operation on a respective base layer image in the sequence of base layer images to generate a target display-specific image in the sequence of target display-specific images.

[0100] Each target display-specific interpolated or extrapolated beta scale map in the sequence of target display-specific interpolated or extrapolated beta scale maps (302-1) may include or specify a scaling method selector and / or a scaling factor to be applied to pixel values ​​in pixels, pixel blocks, foreground areas, background areas, etc. of a respective base layer image in the sequence of base layer images. A scaling method selector as described herein may be used by a recipient device to determine, identify, or select a particular scaling method (e.g., a particular mapping curve, a particular one-dimensional or 1D LUT, a particular 3D LUT, a particular single-piece or multi-piece polynomial, a particular multi-channel mapping matrix, etc.) to be used for one of a particular pixel in a respective base layer image, a particular pixel block in a base layer image, a background or foreground in a base layer image, an entire base layer image, an entire scene including the base layer image, a picture / group or GOP of images including the base layer image, etc., from among a plurality of scaling methods. The scaling factors described herein may be used by a recipient device to apply a scaling operation (e.g., multiplication of a scaling factor in a non-logarithmic representation or domain, addition of a scaling factor in a logarithmic representation or domain, a linear mapping using both multiplication and addition or offset of a scaling factor, etc.) to a codeword(s) (e.g., luma, chroma, RGB, YCbCr, etc.) at a pixel(s) corresponding to one of: a particular pixel in a respective base layer image, a particular block of pixels in a respective base layer image, background or foreground in a respective base layer image, an entire respective base layer image, an entire scene including the respective base layer image, a picture / image group or GOP including the respective base layer image, etc.

[0101] Additionally, optionally, or alternatively, in some operating scenarios, the beta scale data map can be used to apply a nonlinear scaling method to the scaling of the codewords in the respective base layer images. For example, a scaling method selector can be indicated in the beta scale data map and used to select a nonlinear scaling method, such as a nonlinear function (e.g., a sigmoid function, a power function, etc.), for a spatial region in the respective base layer image. Further, a scaling factor corresponding to the scaling method selector can be indicated in the beta scale data map and used as a control parameter (or input parameter) in the nonlinear function to determine or set a particular curve or shape, instead of being used to directly linearly multiply with the pre-scaled codeword to derive the scaled codeword.

[0102] Additionally, optionally, or alternatively, in some operating scenarios, the beta scale data map can be used to apply different scaling methods to different value subranges in the entire codeword value range. The entire codeword value range may represent a particular dimension (e.g., color space component, color space channel, etc.) in a multi-dimensional (e.g., three-dimensional, three color channels / components, etc.) codeword space. The codeword space may include all available codewords in all channels / components of a color space (e.g., RGB color space, YCbCr color space, etc.). As an example, the entire codeword value range may be represented by a particular dynamic range, such as a particular HDR or a particular SDR. Pixels with luma or luminance codeword values ​​within a first subrange of a dynamic range, such as a bright range, in each base layer image may be scaled or mapped using a first scaling method as shown in the beta scale data set that applies a single first reshape mapping curve to these luma or luminance codewords. Pixels with luma or luminance codeword values ​​in a second sub-range of the dynamic range, such as the mid-tone range, in each base layer image may be scaled or mapped using a second scaling method as shown in the beta scale data set, which applies different scaling factors to different luma or luminance codewords among the luma or luminance codewords in the mid-tone range in different local image portions.

[0103] The techniques described herein may operate on a video signal or bitstream that is encoded with a sequence of base layer images and a single sequence of beta scale maps. Additionally, optionally, or alternatively, some or all of these techniques operate on a video signal or bitstream that is encoded with two or more sequences of base layer images and beta scale maps.

[0104] In some operating scenarios, two or more sequences of beta scale maps for each of two or more non-basic reference image displays are available for selection by an upstream device. The target image display may represent an intermediate image display of a display capability very different (e.g., intermediate dynamic range, etc.) from all the base and non-basic reference image displays of the lowest and highest display capabilities. To minimize approximation errors in the sequence of interpolated (or extrapolated) beta scale maps to be generated for the target image display, two (closest) sequences of beta scale maps corresponding to the two reference image displays with display capabilities closest to that of the target image display may be selected by the upstream device - for example, from the two or more sequences of beta scale maps corresponding to the two or more reference image displays - and included, together with the sequence of base layer images, in a video signal or bitstream from the upstream device to a downstream receiving device operating with the target image display. The interpolation or extrapolation operation used to generate the sequence of interpolated (or extrapolated) beta scale maps for the target image display may apply a linear or nonlinear scaling function with per-pixel or per-pixel block spatial resolution to the two closest sequences of beta scale maps to generate the sequence of interpolated (or extrapolated) beta scale maps for the target image display.

[0105] Additionally, optionally, or alternatively, a single sequence of beta scale maps may be provided by an upstream device to a downstream recipient device. The interpolation or extrapolation operation used to generate the sequence of interpolated (or extrapolated) beta scale maps for the target image display may apply a linear or nonlinear scaling function with per-pixel or per-pixel block spatial resolution to the single sequence of beta scale maps to generate the sequence of interpolated (or extrapolated) beta scale maps for the target image display.

[0106] The scaling functions described herein can generate target display-specific images optimized for image rendering on a target image display with approximation error. The size or amount of the approximation error can be controlled depending on how many sequences of beta scale maps, supporting a corresponding number of non-base reference image displays, are available for selection by an upstream device. To improve accuracy and minimize approximation error, beta scale maps incorporating different local reshaping operations for different pixels or different pixel blocks or different image areas can be used instead of or in addition to a global reshaping operation applied to every pixel or every pixel block or every image area in the entire base layer image.

[0107] The approximation error can be minimized if there is a sequence of beta scale maps available for selection by the upstream device for every target image display (e.g., a specific sequence for a specific iPhone model, a specific sequence for a specific image display model, etc.) The approximation error can be increased if there is only a single sequence of beta scale maps (along with an SDR or low dynamic range image in the base layer), for example, for an HDR target display.

[0108] As mentioned above, the interpolation or extrapolation can be performed linearly or nonlinearly based on a control curve, for example as shown in FIG. 3D. The control curve as shown in FIG. 3D can have different slopes or scaling factors in different value subranges of the input operating parameters for the interpolation or extrapolation. For example, the values ​​of the input operating parameters can be calculated or derived from the display capabilities of the target image display and used to look up the slopes or scaling factors in the control curve. The looked up slopes or scaling factors can be used to interpolate or extrapolate one or more received or decoded sequences of beta scale maps corresponding to one or more reference image displays having display capabilities closest to the display capabilities of the target image display for the purpose of generating a sequence of target display-specific interpolated or extrapolated beta scale maps for the target image display.

[0109] 5. Exemplary Process Flow 4A illustrates an exemplary process flow according to an exemplary embodiment of the present invention. In some exemplary embodiments, one or more computing devices or components may perform this process flow. In block 402, an image processing device (e.g., an upstream device, an encoder device, a transcoder, a media streaming server, etc.) generates a sequence of base layer images for a base reference image display and a set of one or more sequences of beta scale maps corresponding to one or more non-base reference image displays.

[0110] At block 404, the image processing device determines a subset of the one or more sequences of beta scale maps in the set of one or more sequences of beta scale maps based at least in part on the display capabilities of the target image display.

[0111] In block 406, the image processing device encodes the sequence of base layer images along with the subset of one or more sequences of beta scale maps into a video signal to cause a recipient device of the video signal to generate a sequence of display images from the sequence of base layer images and the subset of one or more sequences of beta scale maps for rendering on an image display.

[0112] In one embodiment, the sequence of base layer images is of a first dynamic range, and the subset of the one or more sequences of beta scale maps includes a sequence of beta scale maps used to apply a scaling operation to the sequence of base layer images to generate a sequence of non-base layer images of a second dynamic range, the second dynamic range being different from the first dynamic range.

[0113] In one embodiment, the subset of the one or more sequences of beta scale maps includes a second sequence of beta scale maps used to apply a second scaling operation to the sequence of base layer images to generate a second sequence of non-base layer images in a third dynamic range, the third dynamic range being different from the first dynamic range and the second dynamic range.

[0114] In one embodiment, each display image in the sequence of display images represents one of a standard dynamic range image, a high dynamic range image, an intermediate dynamic range image between the standard dynamic range and the high dynamic range, an image with a dynamic range lower than the standard dynamic range, an image with a dynamic range higher than the high dynamic range, a display mapped image optimized for rendering on a target image display, etc.

[0115] In one embodiment, the subset of one or more sequences of beta scale maps is selected from a plurality of sequences of beta scale maps based at least in part on comparing a display capability of the target image display with a corresponding display capability of one or more non-base reference image displays.

[0116] In one embodiment, the scaling operation includes one or more of: simple scaling with a scaling factor, applying one or more codeword mapping relationships to map codewords of the sequence of base layer images to generate corresponding codewords of a corresponding sequence of beta-scaled images, etc.

[0117] In some embodiments, a scaling operation is performed in place of one or more of a global tone mapping, a local tone mapping, a display mapping operation, a color space conversion, a linear mapping, a non-linear mapping, or the like.

[0118] In one embodiment, a sequence of base layer images is encoded in the base layer of a video signal.

[0119] In one embodiment, a common sequence of source images is used to generate both a sequence of base layer images for a base reference image display and a set of one or more sequences of beta scale maps corresponding to one or more non-base reference image displays.

[0120] 4B illustrates an exemplary process flow according to an exemplary embodiment of the present invention. In some exemplary embodiments, one or more computing devices or components may perform this process flow. At block 452, the recipient device decodes a sequence of base layer images from the video signal. The sequence of base layer images is optimized for image rendering on a base reference image display.

[0121] At block 454, the recipient device decodes from the video signal a subset of one or more sequences of beta scale maps, the subset of one or more sequences of beta scale maps being determined from a set of one or more sequences of beta scale maps corresponding to the one or more non-base reference image displays based on a display capability of the target image display.

[0122] At block 454, the recipient device causes a sequence of display images derived from the sequence of base layer images and a subset of the one or more sequences of beta scale maps to be rendered on the target image display.

[0123] In one embodiment, the subset of one or more sequences of beta scale maps includes a single sequence of beta scale maps, or one of two sequences of beta scale maps.

[0124] In one embodiment, a sequence of display images is generated from a sequence of extrapolated beta scale maps in combination with a sequence of base layer images; the sequence of extrapolated beta scale maps is generated from a subset of the one or more sequences of beta scale maps through an extrapolation operation using operational parameters determined at least in part based on the display capabilities of the one or more non-base reference image displays and the display capabilities of the target image display.

[0125] In one embodiment, a sequence of display images is generated from a sequence of interpolated beta scale maps in combination with a sequence of base layer images; the sequence of interpolated beta scale maps is generated from a subset of the one or more sequences of beta scale maps through an interpolation operation using operational parameters determined at least in part based on the display capabilities of the one or more non-base reference image displays and the display capabilities of the target image display.

[0126] In one embodiment, the user pose selected set of sampled views includes two or more sampled views; and the display image is generated by performing an image blending operation on two or more intermediate images generated for the current view from the set of layered images and the set of adjusted alpha maps.

[0127] In various exemplary embodiments, an apparatus, system, device, or one or more other computing devices performs any or part of the foregoing methods as described. In one embodiment, a non-transitory computer-readable storage medium stores software instructions that, when executed by one or more processors, cause execution of the methods described herein.

[0128] Although separate embodiments are discussed herein, it should be noted that any combination of the embodiments and / or sub-embodiments discussed herein may be combined to form further embodiments.

[0129] 6. Implementation Mechanism - Hardware Overview According to an embodiment, the techniques described herein are implemented by one or more special purpose computing devices. The special purpose computing devices may be fixedly configured to perform the techniques, or may include digital electronic devices such as one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) permanently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques according to program instructions in firmware, memory, other storage, or combinations. Such special purpose computing devices may also combine custom fixed configuration logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other devices incorporating fixed configuration and / or program logic to implement these techniques.

[0130] 5 is a block diagram illustrating a computer system 500 in which an exemplary embodiment of the invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled to bus 502 for processing information. Hardware processor 504 may be, for example, a general-purpose microprocessor.

[0131] Computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 502 for storing information and instructions to be executed by processor 504. Main memory 506 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions, when stored in a non-transitory storage medium accessible to processor 504, render computer system 500 into a special purpose machine customized to perform the operations specified in the instructions.

[0132] Computer system 500 further includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504 .

[0133] A storage device 510, such as a magnetic or optical disk or semiconductor RAM, is provided and coupled to bus 502 for storing information and instructions.

[0134] Computer system 500 may be coupled via bus 502 to a display 512, such as a liquid crystal display, for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to bus 502 for communicating information and command selections to processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to processor 504 and for controlling cursor movement on display 512. The input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), allowing the device to specify a position in a plane.

[0135] Computer system 500 may use customized fixed logic, one or more ASICs or FPGAs, firmware and / or program logic that in combination with the computer system configures or programs computer system 500 as a special purpose machine to implement the techniques described herein. According to one embodiment, the techniques are performed by computer system 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 510. Execution of the sequences of instructions contained in main memory 506 causes processor 504 to perform the process steps described herein. In alternative embodiments, fixed circuitry may be used in place of or in combination with software instructions.

[0136] The term "storage media" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tape or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROMs and EPROMs, flash EPROMs, NVRAMs, any other memory chips or cartridges.

[0137] Storage media is distinct from, but may be used in conjunction with, transmission media, which participate in transferring information between storage media. For example, transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0138] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 500 can receive the data on the telephone line and convert the data to an infrared signal using an infrared transmitter. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on the bus 502. The bus 502 carries the data to main memory 506, from which the processor 504 retrieves and executes the instructions. The instructions received by main memory 506 may optionally be stored on storage device 510 either before or after execution by processor 504.

[0139] Computer system 500 also includes a communication interface 518 coupled to bus 502. The communication interface 518 provides a two-way data communication coupling to a network link 520 that is connected to a local network 522. For example, the communication interface 518 may be an Integrated Services Digital Network (ISDN) card, cable modem, satellite modem, or modem to provide a data communication connection to a corresponding type of telephone line. As another example, the communication interface 518 may be a Local Area Network (LAN) card to provide a data communication connection to a compatible LAN. A wireless link may also be implemented. In any such implementation, the communication interface 518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0140] Network link 520 typically provides data communication through one or more networks to other data devices. For example, network link 520 may provide a connection through local network 522 to a host computer 524 or to data equipment operated by an Internet Service Provider (ISP) 526. ISP 526 provides data communication services through the world-wide packet data communication network now commonly referred to as the "Internet" 528. Local network 522 and Internet 528 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 520 and through communication interface 518, which carry the digital data to and from computer system 500, are exemplary forms of transmission media.

[0141] Computer system 500 can send messages and receive data, including program code, through the network(s), network link 520 and communication interface 518. In the Internet example, a server 530 might transmit a requested code for an application program through Internet 528, ISP 526, local network 522 and communication interface 518.

[0142] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510, or other non-volatile storage for later execution.

[0143] 7. Equivalents, Extensions, Substitutions, etc. In the foregoing specification, exemplary embodiments of the present disclosure have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indication of what is and is intended by the applicant to be the invention is the specific form in which the claims of the patent granted on this application, including any subsequent amendments thereto, are patented. The definitions, if any, expressly set forth herein for the terms contained in such claims govern the meaning of such terms used in the claims. Thus, no limitations, elements, characteristics, features, advantages or attributes not expressly set forth in the claims should limit the scope of such claims in any manner. Thus, the specification and drawings are to be regarded in an illustrative and not restrictive sense.

[0144] Various aspects of the present invention can be understood from the following enumerated example embodiments (EEE). [EEE1] generating a sequence of base layer images for the base reference image display and a set of one or more sequences of beta scale maps corresponding to one or more non-base reference image displays; determining a subset of one or more sequences of beta scale maps in the set of one or more sequences of beta scale maps based at least in part on a display capability of a target image display; encoding the sequence of base layer images together with the subset of one or more sequences of beta scale maps into a video signal to cause a recipient device of the video signal to generate a sequence of display images from the sequence of base layer images and the subset of one or more sequences of beta scale maps for rendering on the image display. method. [EEE2] The method of claim 1, wherein the sequence of base layer images is of a first dynamic range, and the subset of one or more sequences of beta scale maps includes a sequence of beta scale maps used to apply a scaling operation on the sequence of base layer images to generate a sequence of non-base layer images of a second dynamic range, the second dynamic range being different from the first dynamic range. [EEE3] The method of claim 8, wherein the subset of one or more sequences of beta scale maps includes a second sequence of beta scale maps used to apply a second scaling operation to the sequence of base layer images to generate a second sequence of non-base layer images of a third dynamic range, the third dynamic range being different from the first dynamic range and the second dynamic range. [EEE4] A method according to any one of claims 1 to 3, wherein each display image in the sequence of display images represents one of a standard dynamic range image, a high dynamic range image, an image with an intermediate dynamic range between the standard dynamic range and the high dynamic range, an image with a dynamic range lower than the standard dynamic range, an image with a dynamic range higher than the high dynamic range, or a display-mapped image optimized for rendering on a target image display. [EEE5] The method of any one of EEE1 to 4, wherein the subset of one or more sequences of beta scale maps is selected from a plurality of sequences of beta scale maps based at least in part on comparing the display capabilities of a target image display with corresponding display capabilities of the one or more non-base reference image displays. [EEE6] The method of any one of EEE1 to 5, wherein the scaling operation includes one or more of: simple scaling with a scaling factor, or applying one or more codeword mapping relations to map codewords of the sequence of base layer images to generate corresponding codewords of a corresponding sequence of beta-scaled images. [EEE7] The method of any one of EEE1 to 6, wherein the scaling operation is performed instead of one or more of a global tone mapping, a local tone mapping, a display mapping operation, a color space conversion, a linear mapping, or a non-linear mapping. [EEE8] 8. A method according to any one of claims 1 to 7, wherein said sequence of base layer images is encoded in a base layer of the video signal. [EEE9] A method according to any one of EEE1 to 8, wherein a common sequence of source images is used to generate both the sequence of base layer images for the base reference image display and the set of one or more sequences of beta scale maps corresponding to the one or more non-base reference image displays. [EEE10] decoding a sequence of base layer images from a video signal, said sequence of base layer images being optimized for image rendering on a base reference image display; decoding a subset of one or more sequences of beta scale maps from the video signal, the subset of one or more sequences of beta scale maps being determined from a set of one or more sequences of beta scale maps corresponding to one or more non-base reference image displays based on a display capability of a target image display; causing the target image display to render a sequence of display images derived from the sequence of base layer images and the subset of one or more sequences of beta scale maps. method. [EEE11] The method of claim 8, wherein the subset of the one or more sequences of beta scale maps comprises one of a single sequence of beta scale maps, or two sequences of beta scale maps. [EEE12] The method of any one of EEE10 to 11, wherein the sequence of display images is generated from a sequence of extrapolated beta scale maps in combination with the sequence of base layer images; and the sequence of extrapolated beta scale maps is generated from the subset of one or more sequences of beta scale maps through an extrapolation operation using operational parameters determined based at least in part on a display capability of the one or more non-base reference image displays and the display capability of the target image display. [EEE13] The method of any one of EEE10 to 11, wherein the sequence of display images is generated from a sequence of interpolated beta scale maps in combination with the sequence of base layer images; and the sequence of interpolated beta scale maps is generated from the subset of one or more sequences of beta scale maps through an interpolation operation using operational parameters determined based at least in part on a display capability of the one or more non-base reference image displays and the display capability of the target image display. [EEE14] Apparatus for carrying out the method according to any one of EEE1 to EEE13. [EEE15] A non-transitory computer-readable storage medium storing software instructions that, when executed by one or more processors, cause performance of the methods described in any one of EEE1 to EEE13.

Claims

1. generating a sequence of base layer images for the base reference image display and a set of one or more sequences of beta scale maps corresponding to one or more non-base reference image displays; determining a subset of one or more sequences of beta scale maps in the set of one or more sequences of beta scale maps based at least in part on a display capability of a target image display; encoding the sequence of base layer images together with the subset of one or more sequences of beta scale maps into a video signal to cause a recipient device of the video signal to generate a sequence of display images from the sequence of base layer images and the subset of one or more sequences of beta scale maps for rendering on the target image display; the beta scale maps correspond to images including beta scaling data for corresponding base layer images of a base reference color grade optimized for image rendering operations using the base reference image display, and the subset of the one or more sequences of beta scale maps is a target display-specific sequence of the beta scale maps for the target image display. method.

2. 2. The method of claim 1, wherein the sequence of base layer images is of a first dynamic range, and the subset of one or more sequences of beta scale maps includes a sequence of beta scale maps used to apply a scaling operation to the sequence of base layer images to generate a sequence of non-base layer images of a second dynamic range, the second dynamic range being different from the first dynamic range.

3. 3. The method of claim 2, wherein the subset of one or more sequences of beta scale maps includes a second sequence of beta scale maps used to apply a second scaling operation to the sequence of base layer images to generate a second sequence of non-base layer images in a third dynamic range, the third dynamic range being different from the first dynamic range and the second dynamic range.

4. 2. The method of claim 1 , wherein each display image in the sequence of display images represents one of a standard dynamic range image, a high dynamic range image, an intermediate dynamic range image between the standard dynamic range and the high dynamic range, an image with a dynamic range lower than the standard dynamic range, an image with a dynamic range higher than the high dynamic range, or a display-mapped image optimized for rendering on the target image display.

5. 2. The method of claim 1, wherein the subset of one or more sequences of beta scale maps is selected from a plurality of sequences of beta scale maps based at least in part on comparing the display capabilities of a target image display with corresponding display capabilities of the one or more non-base reference image displays.

6. 2. The method of claim 1, wherein the scaling operation comprises one or more of: simple scaling with a scaling factor, or applying one or more codeword mapping relations to map codewords of the sequence of base layer images to generate corresponding codewords of a corresponding sequence of beta-scaled images.

7. The method of claim 1 , wherein the scaling operation is performed in place of one or more of a global tone mapping, a local tone mapping, a display mapping operation, a color space conversion, a linear mapping, or a non-linear mapping.

8. The method of claim 1 , wherein the sequence of base layer images is encoded in a base layer of the video signal.

9. 2. The method of claim 1 , wherein a common sequence of source images is used to generate both the sequence of base layer images for the base reference image display and the set of one or more sequences of beta scale maps corresponding to the one or more non-base reference image displays.

10. decoding a sequence of base layer images from a video signal, said sequence of base layer images being optimized for image rendering on a base reference image display; decoding a subset of one or more sequences of beta scale maps from the video signal, the subset of one or more sequences of beta scale maps being determined from a set of one or more sequences of beta scale maps corresponding to one or more non-base reference image displays based on a display capability of a target image display; causing the target image display to render a sequence of display images derived from the sequence of base layer images and the subset of one or more sequences of beta scale maps; the beta scale maps correspond to images including beta scaling data for corresponding base layer images of a base reference color grade optimized for image rendering operations using the base reference image display, and the subset of the one or more sequences of beta scale maps is a target display-specific sequence of the beta scale maps for the target image display. method.

11. 11. The method of claim 10, wherein the subset of the one or more sequences of beta scale maps includes one of a single sequence of beta scale maps or two sequences of beta scale maps.

12. 11. The method of claim 10, wherein the sequence of display images is generated from a sequence of extrapolated beta scale maps in combination with the sequence of base layer images; and the sequence of extrapolated beta scale maps is generated from the subset of one or more sequences of beta scale maps through an extrapolation operation using operational parameters determined based at least in part on the display capabilities of the one or more non-base reference image displays and the display capabilities of the target image display.

13. 11. The method of claim 10, wherein the sequence of display images is generated from a sequence of interpolated beta scale maps in combination with the sequence of base layer images; and the sequence of interpolated beta scale maps is generated from the subset of one or more sequences of beta scale maps through an interpolation operation using operational parameters determined at least in part based on the display capabilities of the one or more non-base reference image displays and the display capabilities of the target image display.

14. Apparatus for carrying out the method according to any one of claims 1 to 13.

15. A non-transitory computer readable storage medium storing software instructions which, when executed by one or more processors, cause the performance of a method according to any one of claims 1 to 13.

Images