Delivery of high dynamic range images across playback systems of various capabilities
The method encodes HDR content into an HDR bitstream with primary and auxiliary metadata packets, addressing the challenge of delivering HDR content to devices with varying processing capabilities, ensuring consistent image quality across a wide range of playback devices.
Patent Information
- Application Number
- JP2022581412
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-07-01
- Filing Date
- 2021-06-30
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-06-30
AI Technical Summary
Current HDR systems primarily target high-end playback devices, leaving low-end devices such as those in mobile systems unable to benefit from HDR content due to processing limitations.
A method for encoding HDR content into an HDR bitstream that includes mapping HDR images to specified color volumes, generating primary and auxiliary metadata packets based on playback device capabilities, and combining these to create a bitstream that can be decoded and processed by a wide range of devices, from low to high performance, using either dynamic or static processing based on device capabilities.
Enables delivery of HDR content to diverse playback devices, ensuring consistent image quality across varying levels of processing power, from minimal to high, by optimizing base layers and adding auxiliary metadata packets to simplify processing on lower-capacity devices.
Smart Images

Figure 0007577769000001 
Figure 0007577769000002 
Figure 0007577769000003
Abstract
Description
[Technical field]
[0001]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 046,796, filed July 1, 2020, and European Patent Application No. 20183398.5, filed July 1, 2020, both of which are incorporated herein by reference in their entireties. [Background technology]
[0002]
[0002] 1. Field The present invention relates to high dynamic range images. More particularly, one embodiment of the present invention relates to the delivery of high dynamic range image content to playback devices having different levels of processing power.
[0003] 2. Description of Related Applications As used herein, the term "dynamic range" (DR) may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, e.g., from darkest gray (black) to brightest white (highlight). In this sense, DR relates to "scene-referred" intensity. DR may also relate to the ability of a display device to properly or approximately render a particular width of intensity range. In this sense, DR relates to "display-referred" intensity. Unless a particular meaning is explicitly specified somewhere in the description of this case to have a particular meaning, it should be inferred that the term may be used in either sense, i.e., interchangeably.
[0004]
[0004] As used herein, the term "high dynamic range (HDR)" refers to a DR width that spans approximately 14-15 times the size of the human visual system (HVS). In practice, DR, where humans can simultaneously perceive a wide range in intensity range, may be somewhat truncated in relation to HDR. An HDR playback system is a system that facilitates the display of HDR images. In video display systems, HDR video playback systems are typically characterized by the display of video images with improved color range and improved contrast when compared to standard video playback systems.
[0005] Currently, most consumer desktop displays have a resolution of 200 to 300 cd / m 2 Most consumer HDTVs support brightness in the range of 300 to 500 nits, with newer models supporting 1000 nits (cd / m 2 ). Such conventional displays represent a lower dynamic range, also referred to as standard dynamic range (SDR) in relation to HDR. As the availability of HDR content increases due to advances in both capture devices (e.g., cameras) and HDR displays (e.g., the PRM-4200 Professional Reference Monitor from Dolby Laboratories), HDR content may be color graded and displayed on HDR displays that support a higher dynamic range (e.g., 1,000 nits to 5,000 nits or more). In general, the methods of the present disclosure relate to any dynamic range higher than SDR.
[0006]
[0006] WO 2006 / 050305 A1 discloses a method and system for mastering and distributing enhanced color space content (target color space) for different display devices having display capabilities beyond that of the CRT color space. For each target color space, a content creator establishes a base or reference color space and enhanced color space data. The enhanced color space data is stored as metadata and transmitted over an enhanced color channel separately from the base / reference color space. Both the base / reference data and the metadata are encoded before transmission and decoded at the consumer side by a display device having either a separate decoder or an integrated decoder. Auxiliary data related to the target color space, such as brightness, luminance, contrast and other display settings, can be transmitted and decoded over the enhanced color channel to control the target color space settings, including the display settings. The color adjustment metadata can be included in the auxiliary data or maintained separately from the auxiliary data.
[0007]
[0007] US 2017 / 064334 A1 discloses a method for encoding digital video, the method comprising the steps of receiving a high dynamic range (HDR) master of the video, a reference standard dynamic range (SDR) master of the video, and target SDR display properties at an encoder, finding a color volume transformation to convert HDR values from the HDR master to SDR values that, when converted for display on a target SDR display, are substantially similar to SDR values in the reference SDR master, converting the HDR values from the HDR master to SDR values using the color volume transformation, generating a metadata item identifying the color volume transformation for a decoder, and encoding the SDR values into a bitstream.
[0008]
[0008] US 2018 / 077453 A1 discloses a device and a method for transmitting / receiving a broadcast signal including video data and high dynamic range information. The broadcast signal transmission method may include generating video data, generating a broadcast signal including the generated video data and picture quality enhancement metadata, and transmitting the generated broadcast signal.
[0009]
[0009] EP 3 367 684 A1 discloses a method and a device for reconstructing an HDR image by applying a reconstruction process to an SDR image, the content of which is similar to that of the HDR image, but the dynamic range of the luminance values of the SDR image is smaller than the dynamic range of the luminance values of the HDR image, the reconstruction process requiring parameters obtained from the bitstream, the method further comprising the steps of determining whether all necessary parameters are available from the bitstream and of restoring missing or corrupted parameters from additional data, characterised in that the reconstruction process further takes the restored parameters into account. Summary of the Invention
[0010]
[0010] The invention is defined by the independent claims. The dependent claims relate to optional features of some embodiments of the invention. An embodiment of the invention is based on a method for delivering high dynamic range (HDR) content to a playback device for displaying images based on the HDR content, the method comprising: encoding the HDR content into an HDR bitstream, the encoding of the HDR content comprising: the method includes the steps of: mapping an HDR image in the HDR content to one or more designated color volume base layers; generating primary metadata packets based on the HDR content; generating one or more auxiliary metadata packets, the information in the auxiliary metadata packets being based on capabilities of a playback device; encoding the one or more designated color volume base layers into one or more designated coded base layers; combining the one or more designated coded base layers, the one or more primary metadata packets, and the one or more auxiliary metadata packets to generate an HDR bitstream, the method including the steps of: decoding the HDR bitstream, the decoding the HDR bitstream including: selecting information from the one or more primary metadata packets and the one or more auxiliary metadata packets based on capabilities of a target playback device; and processing the one or more coded base layers in the HDR bitstream based on the selected information to generate information for display on the target playback device.
[0011]
[0011] An embodiment of the present invention is based on a method for encoding HDR content into an HDR bitstream, the method including: mapping HDR images in the HDR content to one or more specified color volumes, electro-optical transfer function (EOTF), and peak luminance base layers; generating one or more primary metadata packets based on the HDR content; generating one or more auxiliary metadata packets, where the information in the auxiliary metadata packets is based on capabilities of one or more playback devices; encoding the one or more specified color volumes, EOTF, and peak luminance base layers into one or more specified base layers; and combining the one or more specified base layers, the one or more primary metadata packets, and the one or more auxiliary metadata packets to generate an HDR bitstream.
[0012]
[0012] An embodiment of the present invention is based on a method for decoding an HDR bitstream, the HDR bitstream including one or more coded base layers, one or more primary metadata packets, and one or more auxiliary metadata packets, the method including: selecting information from the one or more primary metadata packets and the one or more auxiliary metadata packets based on capabilities of a target playback device; and processing the one or more coded base layers in the HDR bitstream based on the selected information to generate an image for display on the target playback device. [Brief description of the drawings]
[0013]
[0013] Embodiments of the present invention are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like reference numbers indicate similar elements. [Figure 1]
[0014] FIG. 1 shows a block diagram of an example implementation of an HDR bitstream encoding process. [Diagram 2]
[0015] FIG. 2 shows a block diagram of an example process for generating an image from an HDR bitstream using a playback process. [Diagram 3]
[0016] Figure 3 shows a block diagram of an encoder that uses both the addition of auxiliary metadata packets and base layer optimization. [Figure 4]
[0017] Figure 4 shows a block diagram of a low-complexity playback process for generating images from an HDR bitstream including auxiliary metadata packets. [Figure 5A]
[0018] FIG. 5A shows an example of a decision tree for the application of playback processing in a playback device. [Figure 5B] FIG. 5B shows an example of a decision tree for application of a playback process in a playback device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014]
[0019] Current HDR systems are primarily targeted towards high-end playback devices. However, lower-end playback devices, such as those found in mobile systems, can also benefit from HDR content. Low-end playback systems are typically characterized by minimal or no processing due to limitations of the processors within the system. Embodiments of the present invention provide HDR content to a range of playback devices, ranging from very low performance to very high performance.
[0015]
[0020] As used herein, "metadata" refers to data that characterizes or references the bitstream, the resulting image, and / or the devices used in the process of converting the bitstream into the resulting image. As used herein, "primary metadata" refers to data that describes the characteristics of an HDR image as well as composer data. As used herein, "auxiliary metadata" refers to data that describes the display mapping data for the image as well as the capabilities of the playback device.
[0016]
[0021] FIG. 1 shows a block diagram of an exemplary implementation of the encoding process. In FIG. 1, an HDR image 121 and some "HDR image metadata" 123 are input to an encoder 130. The encoder 130 includes a first pre-processing step 131 and a second encoding step 135. In an embodiment, the encoding step 135 may include a video compression step known in the art (e.g., AVC, HEVC, AV1, etc.). The pre-processing step 131 performs a mapping 133 on the selected color volume to generate a base layer image 136, and at the same time generates 132 composer metadata 139 to enable a playback device to undo the base layer mapping and reconstruct the original HDR image. Typically, the base layer mapping is selected to enable compatibility with a particular playback device, e.g., an "SDR", "HDR10" or "HLG" device. The composer metadata 139, in addition to any HDR image metadata 123, is combined to create a primary metadata packet 138. The encoding step generates a coded base layer picture 137 from the base layer picture 136. The combination of the coded base layer picture 137 and the primary metadata packets 138 comprises a bitstream 140 (e.g., a Dolby Vision® bitstream).
[0017]
[0022] 2 shows a block diagram of an exemplary process for generating an image from a bitstream 240 using a reproduction process 260. As shown in FIG. 2, in the reproduction process 260, a decoder and composer step 242 first decodes the coded base layer image 237 and then applies the composer metadata from the primary metadata packet 238 to reconstruct the original HDR image 221. Then, a display mapping step 245 maps the HDR image 221 to a target display 250 under the guidance of the HDR image metadata 223. The decoder and composer process 242 corresponds to a decoding process compatible with the encoding process 135 shown in FIG. 1.
[0018]
[0023] 2 may require a significant amount of processing power, first, in applying the decoder and composer process 242, and second, in applying the display mapping process 245, both of which may vary from frame to frame. The above discussion of Dolby Vision and Dolby Vision bitstreams provides an overview of the encoding and decoding processes for HDR images. Those skilled in the art will appreciate that other processes may be used to encode and decode HDR images for display on a target display.
[0019]
[0025] According to an embodiment of the present invention, the playback complexity can be simplified by: 1) adding auxiliary metadata packets, and 2) optimizing the base layer image to map to the target display class. The addition of auxiliary metadata packets can be selected depending on the processing capabilities of various playback devices. At least one of these auxiliary metadata packets only contains static composer (allowing the playback device to bypass the composer) and display mapping (DM) metadata (describing the color volume and signal encoding of the base layer). Additional auxiliary metadata packets can be selected based on the playback device capabilities to add an increased level of functionality towards a full metadata packet. The key to optimizing the base layer is that most of the dynamic high quality mapping is performed during encoding (132), so that less secondary mapping (245) is required by the playback device. Thus, the secondary mapping can reduce the complexity and therefore run on a larger number of less capable playback devices. In one embodiment, the secondary mapping can be static.
[0020]
[0026] FIG. 3 shows a block diagram of an exemplary encoder 300 using both the addition of auxiliary metadata packets and base layer optimization. In FIG. 3, an HDR image 321 and HDR image metadata 323 are input to the encoder 300. The encoder 300 includes a first pre-processing step 331 and a second encoding step 335, which may include compression or other processing, to generate a coded optimized base layer image 337 for a bitstream 340. The pre-processing step 331 may perform an optimized mapping 333 to an optimal color volume to generate an optimized base layer image 336. At the same time, the pre-processing step 332 generates a primary metadata packet 338 in a manner as described above with respect to FIG. 1. The pre-processing step 332 also generates one or more auxiliary metadata packets 339. The encoding step 335 generates a coded optimized base layer image 337 from the optimized base layer image 336. The encoding step 335 can generate one or more versions of the content from the same original HDR image, in a manner described in more detail below. The combination of the coded optimized base layer image 337, primary metadata packets 338, and auxiliary data packets 339 comprises a bitstream 340 configured for playback on a diverse range of playback devices.
[0021]
[0027] A playback device receiving a bitstream with auxiliary data packets may be configured to select between primary metadata packets or auxiliary metadata packets based on the processing capabilities of the device. These capabilities may be due to hardware limitations of the device or may be dynamically selected based on the availability of processing power or battery power. A very capable playback device may follow the operation as shown in FIG. 1, while a less complex playback device may take advantage of auxiliary metadata packets to simplify the playback process (see, for example, FIG. 4).
[0022]
[0028] FIG. 4 shows a block diagram of an exemplary low-complexity playback process for generating an image from a bitstream 440 including primary and auxiliary metadata packets 431. As shown in FIG. 4, the playback process has a decoder step 442 that decodes the coded optimized base layer image 437 into an optimized base layer image 436. A display mapping step 445 then maps the optimized base layer image 436 to a target display 480 under the guidance of the auxiliary metadata 439. In the low-complexity embodiment shown in FIG. 4, a composer (as shown in FIG. 2) is not required to reconstruct the HDR image. Rather, the optimized base layer image 436 is mapped to an output 480 based solely on the auxiliary metadata packets 439. In one embodiment, this auxiliary metadata packet remains static for the duration of the content, and thus requires minimal dynamic processing.
[0023]
[0029] In some implementations (e.g., Dolby Vision), there may be multiple versions of the same content, encoded with different encoding settings. For example, an encoder can be configured to generate two or more bitstreams intended for different playback devices. The differences between these versions are due to base layer compatibility.
[0024]
[0030] For example, there may be a "most compatible" base layer with color volume corresponding to SDR (Standard Dynamic Range). This base layer can be viewed on some legacy SDR playback device without HDR capabilities. In contrast, an HDR playback device can apply both the "composer" and the "mapping" to optimally display the image, as illustrated by the process shown in Figure 2. In accordance with the present invention, an HDR playback device can alternatively use auxiliary metadata packets to directly map the SDR signal to the target device capabilities (e.g., Figure 4).
[0025]
[0031] There may also be a "least compatible" base layer with a color volume corresponding to the original HDR image. This base layer is not directly visible to an SDR playback device. HDR playback devices can be configured to process the base layer together with the composer and then perform display mapping according to Figure 2.
[0026]
[0032] There may be an "intermediate compatible" base layer that is between the extremes of the "most compatible" and "least compatible" base layers. The color volume of this intermediate base layer can be set. In a preferred embodiment, the color volume is set to the average of the color volumes of the target playback device. In one embodiment, the color volume of the base layer corresponds to P3 primaries and 200-400 nits peak luminance. Legacy (non-HDR) playback devices will be able to correctly and optimally display the decoded image if the playback device characteristics match the base layer characteristics. Fully capable playback devices can use the primary metadata packets to apply a composer and mapping process to render the device optimally (e.g., FIG. 2), and limited capability playback devices can use the auxiliary metadata packets to decode and apply a limited display mapping process to achieve an image quality between these two (e.g., FIG. 4).
[0027]
[0033] In embodiments according to the invention, the bitstream may include both primary and auxiliary metadata. The choice of compatibility is up to the user. In general, "maximum compatibility" is capable of reaching the most devices, while "minimum compatibility" has the best image quality for compatible devices (i.e., the original HDR image deviates the most from the SDR image). An embodiment of the invention using configurable base layer and auxiliary metadata provides image quality that may fall between the maximum compatibility and minimum compatibility configurations. Note that a playback system 260 as shown in FIG. 2 may have slight quality differences in reconstructed HDR images using "maximum compatibility", "intermediate compatibility", or "minimum compatibility" content types, depending on the color volume and peak luminance values of the playback device.
[0028]
[0034] Encoder embodiments according to the present invention may be configured to generate one or more versions of content from the same original HDR image, including a "maximally compatible" content version; an "intermediate compatible" content version; and a "minimally compatible" content version. The "maximally compatible" content is derived by automatically mapping from HDR to a compatible base layer SDR. It also includes playback device metadata to reconstruct the original HDR and perform the HDR to target display mapping. The base layer color volume can be 100 nits, R709, to maximize compatibility with playback devices expecting SDR content. The "intermediate compatible" content is derived by automatically mapping from HDR to a non-compatible base layer (referred to herein as a "medium dynamic range" base layer, or MDR). The base layer color volume is configurable. In one embodiment, the color volume is set to the average of the color volume of the target playback device. In another embodiment, the base layer color volume corresponds to P3 primaries and 200 nits peak luminance. The "minimum compatible" version is essentially the HDR image above.
[0029]
[0035] Note that the encoder may also be configured to create a non-HDR version of the content entirely without any metadata: this latter version is typically derived independently from the HDR image, using a different processing flow.
[0030]
[0036] According to an embodiment of the present invention, a bitstream includes both a primary metadata packet and one or more auxiliary metadata packets. The primary metadata packet includes: composer metadata, which when applied by the composer to a base layer causes the composer to reconstruct an HDR version of the original as close as possible; and display mapping metadata corresponding to the reconstructed HDR image, which when applied by the display mapping process to the reconstructed HDR image causes the display mapping process to map the reconstructed HDR to the capabilities of a target device. The auxiliary metadata packet can include: an identifier indicating the processing capabilities of the playback device; a signal to bypass the composer; and display mapping metadata corresponding to a mapped base layer, which when applied by the display mapping process to the base layer causes the display mapping process to map the base layer to the capabilities of a target device.
[0031]
[0037] Various content versions including associated metadata can be encoded using standard codecs, which can include adaptive bitrate ladders, and made available for distribution.
[0032]
[0038] As mentioned above, a wide range of playback devices can be fed with the same bitstream. The final picture quality depends on the capabilities of the playback device and the availability of different content versions. An embodiment of the present invention provides consistency in playback picture quality across such a wide range of playback devices.
[0033]
[0039] According to an embodiment of the present invention, a method for delivering HDR content to a playback device that displays an image based on the HDR content is provided. Examples of delivering content include streaming, video on demand, broadcast, digital file transmission, and the like. Examples of playback devices include computers, televisions, tablets, smartphones, projectors, and the like. The method can include encoding the HDR content into an HDR bitstream. The encoding can include mapping an HDR image in the HDR bitstream to a specified color volume base layer, which can then be encoded into the coded base layer. For example, the HDR image can be mapped and coded into an SDR base layer and / or an MDR base layer. The encoding can also include generating primary and / or auxiliary metadata into corresponding metadata packets. The primary metadata can be based on the HDR content, and the auxiliary metadata can be based on the base layer. The coded base layer and both the primary and auxiliary metadata are embedded into the bitstream. The method can also include decoding the HDR bitstream, which can include selecting information from among the primary metadata packets or the auxiliary metadata packets, or a combination thereof.
[0034]
[0040] 5A and 5B are example decision trees for applying playback processing according to an embodiment of the present invention. Figures 5A and 5B show how image quality is improved depending on playback capabilities and content type. Note that, without limitation, the decision tree shown in Figure 5A only addresses the situation where there is a single asset being delivered, and Figure 5B only addresses the delivery of two assets: a maximally compatible bitstream and an intermediate compatible bitstream.
[0035]
[0041] As used herein, "base SDR" refers to an SDR base layer that has no metadata associated with it or that has a minimal set of metadata (e.g., third party metadata where only basic functionality is supported by the device).
[0036]
[0042] As used herein, "Primary Metadata" (or PRIME) refers to metadata that utilizes dynamic processing (e.g., dynamic multivariate multiple regression and dynamic display management) on a display device.
[0037]
[0043] As used herein, "auxiliary metadata" (or AUX) refers to metadata that supports static processing in a decoder and is selected when dynamic processing is not used by the display device.
[0038]
[0044] As used herein, "asset" refers to a base layer for which both PRIME and AUX metadata are provided. An "SDR asset" refers to an asset whose base layer is maximum compatibility. An "MDR asset" refers to an asset whose base layer is intermediate compatibility. A bitstream may contain multiple assets whose base layers share the same metadata. An example of an SDR asset is DOLBY® Profile P9.2. An example of an MDR asset is DOLBY® Profile P32.
[0039]
[0045] In some embodiments, the base layer of the asset is derived from an HDR workflow. In some embodiments, the base layer of the asset is derived separately from an HDR workflow.
[0046] An example use case for assets could be a user generated content (UGC) provider that only offers SDR assets in their bitstream, which offers the greatest reach (most compatible with devices) at a low bitrate cost. Another example use case for assets could be a premium content provider that offers both SDR and MDR assets in their bitstream, which offers the greatest reach (by including the most compatible assets) while offering increased quality for high-end devices (by including intermediate compatibility assets).
[0040]
[0047] FIG. 5A shows an exemplary flow chart of a system according to one embodiment. A determination 511 is made whether the playback device supports a compatible application for bitstream processing (e.g., whether the device can play Dolby Vision® bitstreams). A "compatible application" is an application in the playback device that is specifically designed for playback of the encoded bitstreams described herein. If not, the basic SDR base layer is decoded (547) and rendered (555) to generate an image as an SDR image. Whatever basic metadata information is available is used for processing. If the application is supported (511), a second determination is made (521) whether an MDR asset is present in the bitstream.
[0041]
[0048] If an MDR asset is present (521), a determination is made whether the device supports dynamic processing (531). Then, if dynamic processing is supported, high power decode and render operations 551 are performed on the MDR asset and the PRIME metadata 542 to generate an image along with dynamic display mapping (e.g., Dynamic Display Management - Dynamic DM) and dynamic composer predictions (e.g., Dynamic Multivariate Multiple Regression - Dynamic MMR) 551 (see, e.g., FIG. 2 for high power processing).
[0042]
[0049] If an MDR asset is present (521) and the device does not support dynamic processing (531), a static decode and render process 553 is performed on the MDR asset using AUX metadata 541 to generate an image (e.g., see FIG. 4 for low power processing).
[0043]
[0050] If MDR assets are not present (or are not supported by the device) (521) and the device supports dynamic processing (533), the SDR assets can be used along with the PRIME metadata to generate a dynamically processed SDR image.
[0044]
[0051] If no MDR assets are present (521) and the device does not support dynamic processing (533), the SDR assets along with the AUX metadata can be used in static processing (554) to generate an SDR image.
[0045]
[0052] In this way a wide range of devices can be used while still providing ever improving image quality for those devices that support it.
[0046]
[0053] 5B shows an exemplary flow chart of a system according to another embodiment. The flow chart is similar to that shown in FIG. 5A, with a decision to check whether an SDR asset exists (513, 323). If the device does not support a compatible application 511 (511) but an SDR asset exists (513), the SDR asset can be rendered as an SDR.
[0047]
[0054] If the device supports the application (511), but no MDR assets are present (521), an SDR asset is present (523), and the device supports dynamic processing (533), the SDR BL and PRIME metadata can be used to generate an SDR image rendered with dynamic MMR and dynamic DM (552).
[0048]
[0055] If the device supports the application (511), but no MDR assets are present (521), an SDR asset is present (523), and the device does not support dynamic processing (533), then an SDR image can be generated using static DM using the SDR BL and AUX metadata (554).
[0049]
[0056] If the device supports the application (511) but no MDR assets are present (521) and no SDR assets are present (523), then whatever basic metadata is available can be decoded and rendered as SDR using Basic SDR (547, 555).
[0050]
[0057] 5A and 5B, no decision branch is shown for minimally compatible bitstreams (e.g., HDR assets). The decision branch for minimally compatible bitstreams branches further to the left (i.e., should generally be of higher quality) and follows a similar logic, except that minimally compatible bitstreams must also support dynamic playability.
[0051]
[0058] In Figures 5A and 5B, the decision to only use primary (PRIME) or auxiliary (AUX) metadata packets is binary. However, the playback device may also be configured to choose to use additional auxiliary metadata packets that it can support in combination with the primary metadata. For example, the playback device may only have capabilities for static composer and static display mapping, but the device may have capabilities for dynamic resizing, frame rate letterbox padding / cropping, or other features. The processing decision may also be based on values in a configuration file, which may be specified by the processing capabilities of the playback device. The processing decision may also be based on other factors, such as the power level of the playback device; other demands on the processing capabilities of the playback device; user preferences; and additional content metadata.
[0052]
[0059] In practice, playback devices may not use all the assets provided, and there is still value in allowing MDR and SDR assets to coexist. First, the MDR assets may not be sufficient to ensure full compatibility with the target devices. However, users with higher-end and mid-range devices will expect better quality than that provided by SDR assets alone. Since there is less mapping by the composer between the MDR BL and HDR images, the inclusion of MDR assets provides better quality, even for mid-range devices that only support static mappings. Note that MDR assets have fewer constraints, since the base layer is not expected to closely match the SDR deliverables.
[0053]
[0060] In the example of premium content that has both SDR and MDR assets, there are at least three possibilities: 1. The device has a compatible application and supports HDR and dynamic processing -> MDR assets are used along with dynamic processing (PRIME is used); 2. The device has a compatible application and supports HDR but does not support dynamic processing -> MDR assets are used without dynamic processing (AUX is used); or 3. The device does not have a compatible application (e.g. a competitor's device) -> SDR assets are used with universal (basic) metadata.
[0054]
[0061] On the device side, for our example user with an HDR compatible device that supports dynamic processing along with a compatible application, there are at least six possible outcomes: 1. A basic SDR stream is available (e.g. user generated content not encoded in a PRIME / AUX compatible way) -> SDR+basic metadata is used; 2. Old technology SDR assets are available (PRIME only) -> SDR+PRIMI is used; 3. New technology SDR assets are available (PRIME and AUX) -> SDR+PRIMI will be used; 4. New technology SDR assets are available (PRIME and AUX) but the device is in low power mode -> SDR+PRIMI is used; 5. New technology MDR assets are available (e.g. premium content) -> MDR+PRIMI is used; or 6. New technology MDR assets are available (e.g. premium content) but device is in low power mode -> MDR+AUX will be used.
[0055]
[0062] Exemplary embodiments relating to delivery of high dynamic range images are described herein with reference to Dolby Vision or a Dolby Vision bitstream as an example of an HDR bitstream. Those skilled in the art will appreciate that HDR image data may be included in an HDR bitstream that is different from a Dolby Vision bitstream. Embodiments of the invention can be configured to operate with and support other HDR bitstreams.
[0056]
[0063] The embodiments of the present invention may be implemented using computer systems, systems of electronic circuits and components, integrated circuit (IC) devices such as microcontrollers, field programmable gate arrays (FPGAs), other configurable or programmable logic devices (PLDs), discrete time or digital signal processors (DSPs), application specific ICs (ASICs), and / or apparatuses including one or more of these systems, devices, or components. The computers and / or ICs may execute, control, or perform instructions related to block-based reshaping of images having extended dynamic range as described herein. The computers and / or ICs may calculate any of a variety of parameters or values related to delivery of high dynamic range images as described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.
[0057]
[0064] Certain implementations of the present invention include a computer processor executing software instructions that cause the processor to perform the methods of the present invention. For example, one or more processors in a display, encoder, set-top box, transcoder, etc., may perform the methods of high dynamic range image delivery as described above by executing software instructions in a program memory accessible to the processor. The present invention may be provided in the form of a program product. The program product may include any non-transitory medium carrying a set of computer readable signals that include instructions that, when executed by a data processor, cause the data processor to perform the methods of the present invention. A program product according to the present invention may be in any of a wide variety of forms. The program product may include physical media such as, for example, magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD-ROMs, DVDs, electronic data storage media including ROMs, flash RAM, etc. The computer readable signals in the program product may optionally be compressed or encrypted.
[0058]
[0065] Where a component (e.g., a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, references to such components (including references to "means") should be interpreted to include any components (e.g., functional equivalents) that perform the function of the described component as equivalents of such components, including components that are not structurally equivalent to the disclosed structures that perform that function in the illustratively described embodiments of the invention.
[0059]
[0066] As described herein, embodiments of the invention may relate to one or more of the exemplary embodiments enumerated below. Thus, the present invention may be embodied in any form described herein, including, but not limited to, the enumerated exemplary embodiments (EEE) that describe partial structures, features, and functions of certain parts of the invention:
[0067] EEE1. 1. A method for delivering high dynamic range (HDR) content to a playback device for displaying an image based on the HDR content, comprising: encoding the HDR content into an HDR bitstream, the encoding of the HDR content comprising: Mapping an HDR image in the HDR content onto a specified color volume base layer; generating a primary metadata packet based on the HDR content; generating an auxiliary metadata packet, the information in the auxiliary metadata packet being based on capabilities of the playback device; encoding the designated color volume base layer into a coded base layer; and combining the coded base layer, the primary metadata packets, and the auxiliary metadata packets to generate an HDR bitstream; The method includes decoding the HDR bitstream, the decoding of the HDR bitstream including: selecting information from primary metadata packets and auxiliary metadata packets based on capabilities of a playback device; and processing a coded base layer in the HDR bitstream based on the selected information to generate information for display on a target playback device.
[0060]
[0068] EEE2. In the method according to enumerated exemplary embodiment 1, the primary metadata packet includes: composer metadata and display mapping metadata.
[0061]
[0069] EEE3. In the method according to enumerated exemplary embodiment 1 or 2, the auxiliary metadata packet includes: an identifier describing the processing capabilities of the playback device; display mapping data; and a signal for bypassing construction operations when decoding the HDR bitstream.
[0062]
[0070] EEE4. In the method according to any one of enumerated exemplary embodiments 1-3, the HDR bitstream includes a designated coded base layer, and the designated coded base layer includes: Maximum compatibility base layer including standard dynamic range (SDR) color volume; A minimal compatible base layer containing a color volume that corresponds to the color volume of the HDR image; and Intermediate compatibility base layer containing color volumes compatible with playback devices; It is one of the following.
[0063]
[0071] EEE5. In the method according to enumerated exemplary embodiment 4, the step of processing the coded base layer comprises: determining a processing capability of a playback device, the processing capability of the playback device being determined to support a processing capability selected from: SDR processing, maximum compatibility base layer processing, or intermediate compatibility base layer processing; and if the playback device only supports SDR processing, the HDR bitstream will be rendered as an image with SDR color volume; If the playback device supports maximum compatibility base layer processing, determining whether the target playback device supports dynamic processing, rendering the HDR bitstream with static display mapping if the playback device does not support dynamic processing, and decoding and rendering the HDR bitstream with dynamic display mapping if the target playback device supports dynamic processing; and If the playback device supports intermediate compatibility base layer processing, determine whether the target playback device supports dynamic processing, render the HDR bitstream with static display mapping if the target playback device does not support dynamic processing, and decode and render the HDR bitstream with dynamic display mapping if the target playback device supports dynamic processing.
[0064]
[0072] EEE6. An apparatus configured to implement the method according to any one of enumerated example embodiments 1-5, comprising a processor.
[0065]
[0073] EEE7. A non-transitory computer-readable storage medium having stored thereon computer-readable instructions for performing, by one or more processors, a method according to any one of enumerated exemplary embodiments 1-5.
[0066]
[0074] EEE8. 1. A method for encoding high dynamic range (HDR) content into an HDR bitstream, comprising: Mapping an HDR image in the HDR content onto a specified color volume base layer; generating a primary metadata packet based on the HDR content; generating an auxiliary metadata packet, the information in the auxiliary metadata packet being based on capabilities of the playback device; encoding the designated color volume base layer into the designated coded base layer; and combining the coded base layer, the primary metadata packets, and the auxiliary metadata packets to generate an HDR bitstream.
[0067]
[0075] EEE9. In the method according to enumerated exemplary embodiment 8, the primary metadata packets include: composer metadata; and display mapping metadata, and the auxiliary metadata packets include: an identifier indicating the processing capabilities of the playback device; the display mapping metadata; and a signal to bypass the construction operation when decoding an HDR bitstream.
[0068]
[0076] EEE10. In the method according to enumerated exemplary embodiment 8 or 9, the HDR bitstream includes a designated coded base layer, and the designated coded base layer is: Maximum compatibility base layer including standard dynamic range (SDR) color volume; A minimal compatible base layer containing a color volume that corresponds to the color volume of the HDR image; and an intermediate compatibility base layer that includes a color volume compatible with the playback device;
[0069]
[0077] EEE11. An apparatus configured to perform the method according to any one of enumerated exemplary embodiments 8-10, comprising a processor.
[0070]
[0078] EEE12. A non-transitory computer-readable storage medium having stored thereon computer-readable instructions for performing, by one or more processors, a method according to any one of the enumerated exemplary embodiments 8-10.
[0071]
[0079] EEE13. 1. A method for decoding an HDR bitstream, the HDR bitstream including a coded base layer, a primary metadata packet, and an auxiliary metadata packet, the method comprising: selecting information from the primary metadata packets and the auxiliary metadata packets based on the capabilities of the target playback device; and The method includes processing the coded base layer in the HDR bitstream based on the selected information to generate information for display on a playback device.
[0072]
[0080] EEE14. In the method according to enumerated exemplary embodiment 13, the step of processing the coded base layer comprises: decoding the coded base layer into a decoded base layer image; and mapping the decoded base layer image to a playback display based on the data in the auxiliary metadata packet.
[0073]
[0081] EEE15. In the method according to enumerated exemplary embodiment 13 or 14, the step of processing the coded base layer comprises: determining a processing capability of the playback device, the processing capability of the playback device being determined to support a processing capability selected from among: SDR processing; maximum compatibility base layer processing; or intermediate compatibility base layer processing; and if the playback device only supports SDR processing, the HDR bitstream will be rendered as an image with SDR color volume; If the playback device supports maximum compatibility base layer processing, determining whether the target playback device supports dynamic processing, rendering the HDR bitstream with static display mapping if the playback device does not support dynamic processing, and decoding and rendering the HDR bitstream with dynamic display mapping if the playback device supports dynamic processing; and If the playback device supports intermediate compatibility base layer processing, the playback device determines whether it supports dynamic processing, renders the HDR bitstream with static display mapping if the playback device does not support dynamic processing, and decodes and renders the HDR bitstream with dynamic display mapping if the playback device supports dynamic processing.
[0074]
[0082] EEE16. An apparatus configured to perform the method according to any one of enumerated exemplary embodiments 13-15, comprising a processor.
[0075]
[0083] EEE17. A non-transitory computer-readable storage medium having stored thereon computer-readable instructions for performing, by one or more processors, a method according to any one of enumerated exemplary embodiments 13-15.
[0076]
[0084] Thus, exemplary embodiments relating to the delivery of high dynamic range images have been described. In this specification, embodiments of the invention are described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indication of what is, and what is intended by applicant to be, the invention is the set of claims issuing from this application in the specific form in which such claims are issued. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of the terms used in such claims. Thus, no limitations, elements, properties, features, advantages or attributes not expressly recited in a claim should in any manner limit the scope of such claims. Thus, the specification and drawings are to be interpreted in an illustrative and not restrictive sense.
Claims
1. 1. A method for decoding high dynamic range (HDR) content in a playback device to display an image based on the HDR content, comprising: (a) receiving an HDR bitstream in which HDR content is encoded, the HDR bitstream comprising: (1) a coded base layer that is coded from a designated color volume base layer to which an HDR image of the HDR bitstream is mapped; (2) A primary metadata packet, information of the primary metadata packet being based on the HDR content, the primary metadata packet comprising: (i) composer metadata that, when applied by a composer to the specified color volume base layer, causes the composer to reconstruct the original HDR content; and (ii) a primary metadata packet including display mapping metadata corresponding to a reconstructed HDR content for dynamic display mapping, the display mapping metadata, when applied to the reconstructed HDR content by a dynamic display mapping process, causing the dynamic display mapping process to map the reconstructed HDR content to the capabilities of the playback device; (3) An auxiliary metadata packet, comprising: (i) static composer and display mapping metadata corresponding to the specified color volume base layer for static display mapping, the static composer and display mapping metadata, when applied to the specified color volume base layer by a static display mapping process, causing the static display mapping process to map the specified color volume base layer to the capabilities of the playback device; (ii) an identifier indicating the processing capabilities of the playback device required to apply the static composer and display mapping metadata; (iii) an auxiliary metadata packet including a signal to bypass a construction operation when decoding the HDR bitstream; (4) Additional auxiliary metadata packets based on the capabilities of the playback device to add increasing levels of functionality; and (b) decoding the HDR bitstream, the decoding comprising: (1) using the additional auxiliary metadata packets in combination with the primary metadata packets if the playback device determines that the required processing capabilities of the primary metadata packets are met and that the playback device supports one or more features defined by the additional auxiliary metadata packets; and (2) processing the coded base layer in the HDR bitstream based on metadata selected from the primary metadata packets, the auxiliary metadata packets, and the additional auxiliary metadata packets based on capabilities of the playback device to generate information for display on the playback device; The method includes:
2. 2. The method of claim 1, wherein the coded base layer comprises: Maximum compatibility base layer including standard dynamic range (SDR) color volume; a minimally compatible base layer that includes a color volume corresponding to the HDR image; and an intermediate compatibility base layer including a color volume corresponding to said playback device; The method is any one of the following:
3. 3. The method according to claim 1 or 2, wherein the step of processing the coded base layer comprises: decoding the coded base layer into a decoded base layer image; and mapping the decoded base layer image to a playback device based on data in the auxiliary metadata packets; A method comprising:
4. 3. The method of claim 2, wherein the step of processing the coded base layer comprises: determining a processing capability of the playback device, the processing capability of the playback device being determined to support a processing capability selected from: SDR processing, maximum compatibility base layer processing, or intermediate compatibility base layer processing; and if the playback device supports only SDR processing, the HDR bitstream is rendered as an image having an SDR color volume; If the playback device supports maximum compatibility base layer processing, determine whether the playback device supports dynamic processing, render the HDR bitstream with static display mapping if the playback device does not support dynamic processing, and decode and render the HDR bitstream with dynamic display mapping if the playback device supports dynamic processing; and 2. The method of claim 1, wherein, when the playback device supports intermediate compatibility base layer processing, the playback device determines whether the playback device supports dynamic processing, and if the playback device does not support dynamic processing, the playback device renders the HDR bitstream with static display mapping, and if the playback device supports dynamic processing, the playback device decodes and renders the HDR bitstream with dynamic display mapping.
5. 5. The method according to claim 1, further comprising the steps of: before receiving the HDR bitstream, encoding a high dynamic range (HDR) content into an HDR bitstream, comprising: (a) mapping an HDR image in the HDR content onto a specified color volume base layer; (b) generating a primary metadata packet based on the HDR content, the primary metadata packet comprising: (i) composer metadata that, when applied by a composer to the specified color volume base layer, causes the composer to reconstruct the original HDR content; and (ii) display mapping metadata corresponding to the reconfigured HDR content for dynamic display mapping, the display mapping metadata, when applied to the reconfigured HDR content by a dynamic display mapping process, causing the dynamic display mapping process to map the reconfigured HDR content to the capabilities of the playback device; (c) generating an auxiliary metadata packet, the auxiliary metadata packet comprising: (i) static composer and display mapping metadata corresponding to the specified color volume base layer for static display mapping, the static composer and display mapping metadata, when applied to the specified color volume base layer by a static display mapping process, causing the static display mapping process to map the specified color volume base layer to the capabilities of the playback device; (ii) an identifier indicating the processing capabilities of the playback device required to apply the static composer and display mapping metadata; and (iii) signaling to bypass construction operations when decoding the HDR bitstream; (d) generating additional auxiliary metadata packets based on the capabilities of the playback device to add incremental levels of functionality; (e) encoding the designated color volume base layer into a coded base layer; and (f) combining the coded base layer, the primary metadata packets, the auxiliary metadata packets, and the additional auxiliary metadata packets to generate the HDR bitstream; The method includes:
6. A decoding device comprising a processor configured to perform the method according to any one of claims 1 to 4.
7. A system comprising a decoding device according to claim 6 and an encoding device configured to perform the encoding step according to claim 5.
8. A computer readable storage medium having stored thereon computer executable instructions for carrying out, using one or more processors, the method of any one of claims 1-5.
Citation Information
Patent Citations
Segment-Based Reconstruction for High Dynamic Range Video Coding
JP2020507958A
Method and apparatus for processing a medium dynamic range video signal in SL-HDR2 format
JP2021530884A
Segment-based reshaping for coding high dynamic range video
US20190272643A1
Transmission device, transmission method, reception device, and reception method
US20200154143A1
Method and apparatus for processing a medium dynamic range video signal in SL-HDR2 format
US20210266608A1
