Method of encoding signal, method of decoding signal, computer program, computer readable storage medium, and data processing apparatus

Abstract

To provide a method for encoding and decoding a signal.SOLUTION: There is provided an encoding system, where an SDR signal 900 uses a first signal element coding format and an HDR signal 960 uses a second signal element coding format. The SDR signal 900 is encoded by a first encoder 1103 to generate a first encoded stream 910. The first encoded stream 910 is decoded to generate a lower decoded signal. The lower decoded signal is processed by at least converting from the SDR signal element coding format to the HDR signal element coding format and up-sampling, to produce a processed signal. The processed signal and the HDR signal 960 are processed by a second encoder 1107 to generate a second encoded stream 920.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to a method for processing a signal, in particular a video signal, including but not limited to acquiring, deriving, outputting, receiving and reconstructing data to increase the dynamic color range in a video signal. [Background technology]

[0002] Advances in technology have shown that there are many different techniques and methods for processing signals, particularly for compression, storage, and transmission. For example, over the years, many methods have been developed for encoding image and video signals so that the information is compressed. This has the advantage of reducing the storage and bandwidth requirements of any transmission path, whether over terrestrial over-the-air broadcast, cable broadcast, satellite broadcast, or the Internet, or other data networks. As technology advances, more sophisticated methods of compression and transmission have been developed. High quality signals are desired, for example signals having high definition video (manifested as an increased number of pixels per frame or an increased number of frames per second). As a result, there are many existing signal formats, and many types of signal encoders and decoders that may or may not be able to use these formats. This is a problem for content creators and distributors who want to ensure content is available to the widest audience. As a result, creators and distributors must provide content in many different encoding formats, encoded using standard encoding techniques such as, for example, MPEG-2, MPEG-4 Part 10, High Efficiency Video Coding (HEVC), etc. The distribution of coded signals is also being driven by standardized and non-standardized broadcast and streaming technologies. Overall, this creates a complex landscape that increases costs for content distributors and makes the adoption of newer and better technologies more difficult due to lack of compatibility with prevalent older technologies (e.g., older decoders and set-top boxes).

[0003] High Dynamic Range (HDR) video has shown the potential to transmit high quality video by adding another dimension of quality improvement: apart from increasing resolution (i.e., more pixels per frame) and increasing motion fidelity (i.e., more frames per second), operators can also increase the dynamic range (i.e., a wider range of brightness, tones, and therefore more vivid contrast and color). The wide availability of HDR-capable displays makes HDR video even more relevant.

[0004] High dynamic range (HDR) typically refers to a higher brightness and / or color gamut than standard dynamic range (SDR), the latter of which uses a traditional gamma curve to represent the color gamut. HDR is typically operated by changing the values ​​of the brightest and darkest colors and by using different color planes (typically extended from the standard color planes to wide color gamut planes).

[0005] There are various HDR implementations in the current market that typically require 10-bit (or higher) representation of color components (e.g., luminance Y and chroma Cb and Cr) to have a sufficiently large range of utility. One of these HDR implementations is HDR 100, which uses statistical metadata in the video signal to adjust the luminance and color gamut displayed in the video stream. Thus, a single HDR setting is applied throughout the video sequence. Different implementations that use dynamic metadata have various commercial names such as HDR10+ or ​​Dolby™ Vision. With dynamic metadata, the HDR settings can be changed on a frame-by-frame basis, thus providing greater granularity and adaptation to scene changes.

[0006] However, currently, the transmission of HDR video requires decoders capable of decoding native 10-bit video formats such as HEVC. This often requires duplication of video workflows, since most operators need to continue to serve a large portion of their customer base that only owns older equipment capable of decoding 8-bit AVC / H.264 video. Similarly, it also prevents operators from using HDR with 8-bit formats such as VP9. Furthermore, the tone mapping required by HDR is not always the best fit to optimize compression efficiency. More information on this perspective can be found by referring to the following papers: A. Banitalebi-Dehkordi, M. Azimi, MT Pourazad and P. Nasiopoulos, "Compression of high dynamic range video using the HEVC and H.264 / AVC standards," 10th International Conference on Heterogeneous Networking for Quality, Reliability, Security and Robustness, Rhodes, 2014, pp. 8-12.

[0007] There is a general problem of incompatibility between newer coding standards, especially HDR-related standards or technologies, and older decoding equipment that cannot be easily replaced or upgraded. There is also a general problem of limited bandwidth available for signal transmission. There is also a general problem of processing and storage overhead for content distributors in creating and maintaining many versions of the same content encoded in different ways, which problem is exacerbated by the introduction of several competing and widely incompatible technologies for encoding HDR-type signals (i.e. signals that have the potential for a wider luminance range and contrast ratio and / or contain information representing a wider color gamut). Summary of the Invention [Problem to be solved by the invention]

[0008] The present disclosure is directed to solving one or more of the problems set forth above. [Means for solving the problem]

[0009] In this disclosure, we describe how HDR type information is combined in a hierarchical coding scheme to solve one or more of the problems mentioned above. In particular, we aim to provide an encoding and decoding scheme that allows hierarchical signals to be coded, transmitted and decoded in such a way that allows HDR type signals to be presented to a user via an HDR compatible display device and allows HDR non-equivalent signals to be presented to a user when one or more decoder technologies or associated display devices are unable to process HDR type information. In particular, we aim to provide a more elegant and efficient method. Also, we aim to provide a composite coded video stream that consists of all information required for HDR type decoding and SDR type encoding, if possible, and to provide flexibility as to whether the information required for HDR type decoding is input to the decoder or whether the information required for HDR type decoding is used in the decoder (e.g., the information can be ignored or discarded).

[0010] According to the appended claims there is provided a method, a computer program, a computer readable storage medium or data carrier, and a data processing apparatus.

[0011] A first aspect of the present invention discloses a method for encoding a signal, comprising receiving a first signal and a second signal. The first and second signals have different versions of a common content. The first signal uses a first signal element coding format and the second signal uses a second signal element coding format. The first signal is encoded by a sub-encoding module that generates a sub-encoded signal. The sub-encoded signal is decoded to generate a sub-decoded signal. The sub-decoded signal is processed and upsampled by at least converting from the first signal element coding format to the second signal element coding format to generate a processed signal. In this way, one or more of signal improvement, potential bandwidth reduction, and backward compatibility are provided.

[0012] The first signal element coding format may represent a first color space, and the second signal element coding format may represent a second color space.

[0013] The first signal element coding format may represent a standard dynamic range (SDR) and the second signal element coding format may represent a high dynamic range (HDR), where the HDR format may have a higher brightness and / or a wider color gamut than the SDR format.

[0014] Optionally, the first signal element coding format may have a first bit-depth and the second signal element coding format may have a second bit-depth, where the first bit-depth may be less than the second bit-depth.

[0015] Optionally, the first signal element coding format may be of a first level of quality and the second signal element coding format may be of a second level of quality, where the second level of quality may be higher than the first level of quality.

[0016] The higher-level encoded signal may be an enhancement signal consisting of residual data representing the difference between the second signal as a reference signal and the processed signal as a reconstructed version of the reference signal.

[0017] The step of processing the lower-order decoded signal may include upsampling the lower-order decoded signal to generate an upsampled signal, and processing the upsampled signal and the first signal in an intermediate encoder to generate an intermediate encoded signal.

[0018] Optionally, the intermediate coded signal may comprise an enhancement signal comprising residual data representing the difference between the first signal as a reference signal and the upsampled signal as a reconstructed version of the reference signal.

[0019] Optionally, the intermediate encoded signal may be decoded and processed by an upsampled version of the lower decoded signal to output an intermediate reconstructed signal.

[0020] The step of processing the upsampled signal by converting at least the first signal element coding format signal to a second signal element coding format signal may include converting the intermediate reconstructed signal from the first color space to the second color space.

[0021] The step of processing the lower-order decoded signal may include converting the lower-order decoded signal from a first signal element coding format to a second signal element coding format, and upsampling an output of the conversion to generate a processed signal.

[0022] The above-mentioned method comprises downsampling the first signal before encoding it to generate a sub-encoded signal by a sub-encoding module, such that the encoding process is performed on fewer pixels and can therefore be performed more quickly.

[0023] The first signal may comprise a successive number of frames, and the second signal may comprise a corresponding successive number of frames, each frame being encoded by a lower encoder, thus reducing the need for frame counting and signaling, and additionally greatly reducing the need to manage intra-coded reference frames.

[0024] The step of converting from the first signal element coding format to the second signal element coding format may be non-linear.

[0025] The step of converting from the first signal element coding format to the second signal element coding format may be non-predictive. Thus, the conversion from the first signal element coding format to the second signal element coding format may be performed in a stand-alone manner without the need for the calculation of additional prediction information, which facilitates modular exchange of conversion functions more easily. Also, thus, the computational load at the decoder may be reduced and more backward compatibility may be achieved.

[0026] Also provided is a method of decoding a signal, the method comprising decoding a lower-level encoded signal by a lower-level decoding module to generate a lower-level decoded signal in a first signal element coding format. The lower-level decoded signal is processed by at least converting from the first signal element coding format to a second signal element coding format and upsampling to generate a higher-level reconstructed signal. The higher-level encoded signal is decoded by the higher-level decoding module to generate a higher-level decoded signal. The higher-level reconstructed signal and the higher-level decoded signal are processed by the higher-level decoding module to generate a higher-level combined decoded signal that reconstructs the original signal in the second signal element coding format. In this way, one or more of signal improvement, potential bandwidth reduction, and backward compatibility are provided.

[0027] Optionally, the first signal element coding format may represent a first color space and the second signal element coding format may represent a second color space.

[0028] Optionally, the lower decoded signal may have a standard dynamic range (SDR) and the higher decoded signal may have a high dynamic range (HDR), where HDR format may have higher brightness and a wider color gamut than SDR format.

[0029] Optionally, the lower-level decoded signal may have a first bit-depth and the higher-level reconstructed signal may have a second bit-depth, where the first bit-depth may be less than the second bit-depth.

[0030] Optionally, the first signal element coding format may have a first level of quality and the second signal element coding format may have a second level of quality, where the second level of quality may be higher than the first level of quality.

[0031] Optionally, the higher-level decoded signal may be an enhancement signal comprising residual data for adjusting the higher-level reconstructed signal to reconstruct the original signal in a second signal element coding format.

[0032] The intermediate encoded signal may be decoded by an intermediate decoding module to generate an intermediate decoded signal, where processing the lower-order decoded signal includes upsampling the lower-order decoded signal to generate an upsampled signal, and processing the upsampled signal and the intermediate decoded signal in the intermediate decoder to generate an intermediate reconstructed signal.

[0033] Optionally, the intermediate decoded signal may comprise residual data for adjusting the upsampled signal to reconstruct the original signal in the first signal element coding format, thus allowing inaccuracies in the codec type to be corrected.

[0034] Optionally, the step of processing the lower-decoded signal may comprise converting the intermediate reconstructed signal from a first color space to a second color space.

[0035] Optionally, the method of decoding may include detecting whether the connected display is capable of displaying the second signal element coding format and, if so, outputting an intermediate reconstructed signal, thus providing backward compatibility.

[0036] The upper combined decoded signal may be converted back to the first signal element coding format if the display is only capable of displaying the first signal element coding format, thus providing backward compatibility.

[0037] The lower-order encoded signal may comprise all frames of the upper-order combined decoded signal, where each frame of the lower-order encoded signal may be decoded by the lower-order decoding module and provide a base version of the upper-order combined decoded signal, and the upper-order encoded signal may comprise enhancement information for each frame of the base version.

[0038] Optionally, the converting from the first signal element coding format to the second signal element coding format may be non-linear.

[0039] Optionally, the step of converting from the first signal element coding format to the second signal element coding format may be non-predictive. Thus, the coding format conversion does not require the calculation of additional prediction information and may be performed in a stand-alone manner, which may facilitate a simpler modular exchange of conversion functions. Also, thus, the computational load at the decoder may be reduced and more backward compatibility may be achieved.

[0040] A computer program is provided which is adapted to carry out the signal encoding and decoding method described above.

[0041] A computer readable storage medium or data carrier is provided containing a computer program adapted to carry out the signal encoding and decoding methods described above.

[0042] A data processing apparatus is provided having a processor and a memory adapted to perform the signal encoding and decoding methods described above.

[0043] The present invention will now be described, by way of example only, with reference to the accompanying drawings in which: [Brief description of the drawings]

[0044] [Figure 1] FIG. 1 shows an exemplary system for encoding and decoding a signal. [Diagram 2] FIG. 2 shows an example of a decoding system according to the first embodiment. [Diagram 3] FIG. 3 shows an example of a decoder according to a first modification of the first embodiment. [Figure 4] FIG. 4 shows an example of a decoder according to a second modification of the first embodiment. [Diagram 5] FIG. 5 shows an example of a decoder according to a third modification of the first embodiment. [Figure 6] FIG. 6 illustrates an example of a computer system. [Figure 7] FIG. 7 shows an example of an encoding system according to the second embodiment. [Figure 8] FIG. 8 shows an example of a decoding system and method according to the second embodiment. [Figure 9] FIG. 9 shows an example of an encoding system according to the third embodiment. [Figure 10] FIG. 10 shows an example of a decoding system according to the third embodiment. [Figure 11] FIG. 11 shows an example of an encoding system according to the first embodiment. [Figure 12]FIG. 12 shows an example of an encoding system according to the fourth embodiment. [Figure 13] FIG. 13 shows an example of an encoding system according to a first variant of the fourth embodiment. [Figure 14] FIG. 14 shows an example of a decoding system corresponding to the first modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] To assist the reader of this specification with respect to the terms used, some definitions are provided below.

[0046] As non-limiting examples, the signal may be an image, an audio signal, a multi-channel audio signal, a video signal, a multi-view video signal (e.g., 3D video), a volumetric signal (e.g., medical imaging, scientific imaging, holographic imaging, etc.), a volumetric video signal, or a signal having four or more dimensions.

[0047] Broadly speaking, video is defined as the recording, playback or broadcast of moving images. A video signal consists of a sequence of individual frames, each of which may be individually compressed through encoding / decoding methods.

[0048] Encoding is the process involved in converting information from one format to another to allow efficient transmission from one source to another or storage of that information. The information to be encoded may be raw video, which often contains large amounts of data that cannot be successfully transmitted in the original format. Thus, when raw video is encoded or decoded, the amount of space required for transmission of the raw video content is reduced. Decoding is the reverse process of encoding and involves converting (decompressing) the encoded information back to the original pre-transmission information. Encoding and decoding may be lossy or lossless.

[0049] For video signals, especially those related to HDR video, which are consumer displays with limited color volume (i.e., not providing the peak brightness / contrast and color gamut required by the relevant HDR standard), SMPTE defines metadata to describe the scene as it appears on the mastering display. SMPTE ST 2086 "Mastering Display Colour Volume Metadata Supporting High Luminance and Wide Colour Gamut Images" describes static data such as MaxFALL (Maximum Frame Average Light Level) and MaxCLL (Maximum Content Light Level). SMPTE ST 2094 "Content-Dependent Metadata for Color Volume Transformation of High Luminance and Wide Colour Gamut Images" has dynamic metadata that changes from scene to scene. This includes Colour Volume Reconstruction Information (CVRI) as defined in ST 2094-10 (Dolby Vision format), ST 2094-30 (Technicolor format), SMPTE ST 2094-20 (Philips format) and Colour Remapping Information (CRI), and HDR10+ ST 2094-40 (Samsung format). The auxiliary data may include metadata as described herein, and additional enhancement information such as residual information may be part of any auxiliary data.

[0050] Video content may be transmitted at a specified level (or levels) of quality, which may be defined, for example, by a particular spatial resolution or frame rate or peak signal-to-noise ratio (PSNR). The level of quality may be limited by the constraints of the transmission channel carrying the video content. As an example, existing decoding hardware may decode a signal up to a given resolution and / or frame rate (a first quality level). One or more additional levels of quality may be reconfigured using computer decoding methods. Legacy decoding devices that cannot be updated to perform decoding of levels of quality higher than the first level of quality simply decode the signal at the first level of quality and ignore the additional enhancement data.

[0051] The bit depth of a signal relates to the number of bits used to communicate a particular value. In video, bit depth (often referred to as color depth) relates to the color components (e.g., luminance Y and chroma Cb and Cr) that represent pixels in an image, and defines, among other things, the number of luminance levels and colors recorded in the image. Images with a bit depth of 8 bits or less are considered to be images with standard dynamic range, while images with bit signals greater than 8 bits are considered to be images with high or enhanced dynamic range.

[0052] Image resolution may be considered a measure of how much information is available. In essence, video resolution is a basic measure of how much information is visible in a particular image. Spatial video resolution is typically defined by the number of pixels that can be displayed in each dimension on a display device, usually given as the width of the display device relative to the height of the display device. For example, a Full HD signal is represented as 1920×1080 pixels, or more simply as 1080p. A UHD signal may typically have a frame resolution of 3840×2160 pixels, usually simply referred to as 4K resolution. A UHD signal may also have an 8K resolution (e.g., 7680×4320 pixels). Similarly, there may be additional temporal resolutions in a video signal, e.g., a video signal may be available at various frame rates, such as 25, 30, 50, 60, 100, and 120 frames per second.

[0053] The process of upsampling involves converting a lower resolution version of a signal or sequence of signals into a higher resolution version of the signal to enhance the quality of the signal. Upsampling produces an approximation of a sequence obtained by sampling the signal at a higher rate and is typically performed on the encoded data by a decoder. Upsampling can be performed spatially (i.e., with respect to pixel resolution) or temporally (i.e., with respect to frame rate).

[0054] Conversely, downsampling produces an approximation of the sequence obtained by sampling the signal at a lower rate, resulting in a lower resolution version of the signal.

[0055] A color space may be defined as a mathematical representation of a set of colors that allows a particular color to be reproduced. The two most commonly used color spaces for digital image and video display are RGB (Red / Green / Blue) and YCrCb (Luminance / Red Chroma / Blue Chroma). A color space may be more broadly defined as defined by one or more of the ITU-R Recommendations BT709, 2020, 2100, 2390, which give details of the color definition. It includes specifications of how image elements such as pixels are represented in the video signal and in some cases covers or defines parameters of the following properties of the signal. The properties of the signal include the sampling pattern (e.g., 4:4:4 to 4:2:2), bit depth, dynamic range of the luminance component, color gamut, electro-optical transfer function model used, minimum luminance level (e.g., black level), and maximum luminance level (e.g., white level).

[0056] It should be noted that the following examples are performed for a signal having video content that includes frames, with any processing being performed for each frame in each video segment available. In some examples, some frames may be processed differently, but processing all frames the same is advantageous as it reduces the need for frame counting and signaling, and significantly reduces the need for intra-coding reference frame management.

[0057] FIG. 1 shows an exemplary system for encoding and decoding a signal as described in this application. In particular, an original signal 100 is encoded by an encoder 1100. A data stream 1150 is generated by the encoder and transmitted to decoders 1200-0 to 1200-N. The data stream may be split and a part of the data stream 1150 is sent to the decoders. Each decoder decodes a respective part of the data stream to obtain a reconstructed signal at level #0 to a reconstructed signal at level #N. The final signal is obtained by processing and combining different signals reconstructed at different levels to obtain one signal. Different decoders receive different parts of the data stream and can reconstruct the signal at different levels of quality. The decoders may decode and output different levels of quality.

[0058] Fig. 2 shows an example of a decoding system as described in the present application. In particular, a data stream 1150 is received by a number of decoders 1200-0 and 1200-1. The data stream 1150 may be formed from a standard-compliant data stream (e.g., compliant with standard and non-standard codecs such as AVC, HEVC, VVC, etc.) and an auxiliary data stream. The first sub-stream is provided to a first decoder 1200-1, which splits the data stream into a data stream 220, which is provided to a standard decoder 2230 to generate a low-quality video 240. The low-quality video 240 is processed together with the data stream 210 by an enhancement decoder 270 to generate an output video SDR 280, i.e. a video output having SDR. In this case, the output video SDR corresponds either to the low-quality video 240 after processing (e.g., upsampling) but without any additional information added by the data stream 210 (e.g., if the decoder 1200-1 cannot decode the data stream 210), or to the low-quality video 240 after processing it with additional information derived from the data stream 210 (composite stream), after which the composite stream is downsampled to SDR resolution to accommodate a TV or a display that can only display SDR video. The second sub-stream is fed to a second decoder 1200-0, which splits the data stream into data streams 220, which are fed to the standard decoder 230 to generate the low-quality video 240. The low-quality video 240 is processed together with the data stream 210 by the enhancement decoder 250 to generate the output video HDR 260.

[0059] In Figures 1 and 2, a data stream is received at the decoding modules 1200-0, 1200-N. The data stream may be a stream of data conforming to a standard such as a protocol standard (e.g., DASH, HLS, etc.) and / or a coding standard (MPEG standard such as AVC, HEVC, VVC, etc., or other codec formats such as VP9, ​​AV1, VC-6, etc.). The data stream is then split into two (or more) separate streams by a de-muxing module. The first stream is sent to a decoding module that decodes according to a first decoding scheme, such as a standard-based scheme (e.g., AVC, HEVC, VVC, VC-6, etc.) or a non-standard-based scheme (e.g., VP9, ​​AV1, etc.). The output of the first decoding module (e.g., the first decoded video signal) is sent to a second decoding module that decodes according to a second decoding scheme. The second decoding module may also receive an auxiliary data stream that, when processed by the output of the first decoding module, produces an enhanced decoded video signal.

[0060] In one example, the first decoding scheme is the same as the second decoding scheme.

[0061] In another example, the first decoding scheme is different from the second decoding scheme. The second decoding scheme may also include operations such as upsampling, residual inverse quantization, residual transformation, adding residual from a motion compensation frame buffer, post-processing, etc. The residual information is, for example, a difference signal generated by comparing a reference signal with a reconstructed version of the reference signal, as will be apparent from the following description.

[0062] In one example, the auxiliary data stream also includes some metadata (static or dynamic) necessary to reconstruct the HDR signal. This metadata may be processed by the second decoding module together with the output of the first decoding module and other data in the auxiliary data stream to generate an enhanced HDR encoded video signal. The metadata may include range information to identify the range of the HDR information, and may include information to ensure that the HDR signal is properly presented on a display in a title-by-title manner, or in a scene-by-scene or frame-by-frame manner. The auxiliary data stream may also include residual information.

[0063] In one example, the first decoding module reconstructs a low bit depth (e.g., 8-bit) signal, and the second decoding module converts the signal into a high bit depth (e.g., 10-bit) signal conforming to HDR. In one example, the second decoding module can detect whether the decoding device is unable to display a high bit depth (e.g., 10-bit) HDR image, in which case it encodes and displays a low bit depth (e.g., 8-bit) signal.

[0064] In some examples, the second decoding module combines the processing required to adjust the range of values ​​with other processing required to reconstruct the output signal from the first (lower) level of quality. In one example, an upsampling operation is combined with the range adjustment to avoid touching the memory buffer multiple times, thereby reducing both the number of memory accesses and the required memory bandwidth. For example, the remaining processing includes upsampling and / or adding residual data to convert the upsampled lower level of quality to a higher level of quality.

[0065] In some examples, the first decoding module reconstructs a low-resolution representation of the signal. HDR adjustments are then applied to the low-resolution reconstruction of the signal by a second decoding module. Following the adjustments, a second encoding module further processes the signal and remaining auxiliary data to obtain a high-resolution and / or high-quality signal.

[0066] In some examples, an encoder receives a high-bit-depth (e.g., 10-bit) HDR source signal. The encoder downsamples the source signal and adjusts the downsampled representation to obtain a low-bit-depth (e.g., 8-bit) SDR signal. The SDR signal is processed by a first codec to obtain a first encoded data set. The decoded output of the first codec is converted to a high-bit-depth HDR signal, which is then processed by a second encoder module to generate a second encoded data set including additional auxiliary data to be used to reconstruct a full-resolution reconstructed representation of the original source signal.

[0067] In some examples, the encoder processes the high bit depth HDR signal before or after the downsampling operation. For example, the high bit depth HDR signal may be represented in a color space as defined in ITU-R Recommendation BT.2020, ITU-R Recommendation BT.2100, etc., which are incorporated herein by reference. By way of background, Rec.2100 includes two transfer function definitions that may be used for HDR signals: Perceptual Quantizer (PQ), previously standardized as SMPTE ST 2084 (which is incorporated herein by reference), and Hybrid Log-Gamma (HLG), previously standardized as ARIB STD-B67 (which is incorporated herein by reference). The PQ scheme with a color bit depth of 10 bits has also been called HDR10 (https: / / en.wikipedia.org / wiki / Rec._2020 - cite_note-HDRCompatibleCEA2015-49). Similarly, an HLG scheme with a color bit depth of 10 bits has been called HLG10.

[0068] The HDR signal may be processed before or after downsampling to convert it to a different color space, such as defined in ITU-R Recommendation BT.709, which is incorporated herein by reference. For example, this conversion may be performed such that the resulting signal is a low bit depth (e.g., 8-bit) SDR signal. This signal is then used by the first encoding module as described above.

[0069] Conversely, the decoded output of the first codec is converted from a first color space (e.g., BT.709) to a second color space (e.g., BT.2020 or BT.2100) before or after upsampling to generate a high bit depth HDR signal (e.g., 10 bits), which can then be processed by a second encoder module as described above.

[0070] In some examples, the first decoding module receives the first encoded signal and reconstructs a signal of a low resolution representation, for example a low bit depth (e.g., 8 bits) SDR signal. The SDR signal may be represented in a first color space (e.g., BT.709) and converted to a second color space (e.g., BT.2020 or BT.2100) before or after an upsampling operation to a high resolution representation, thus generating a HDR signal at a high resolution, for example a high bit depth (e.g., 10 bits). The second encoded signal is decoded by the second decoding module to generate an adjustment signal or a residual signal. The HDR signal may then be further processed by the second decoding module to generate an adjusted HDR signal using the adjustment signal. Following this adjustment, the second encoding module further processes the signal and other auxiliary data to obtain a higher resolution and / or higher quality signal.

[0071] Conversion can be performed using methods well known in the art, such as those contained in ITU-R Recommendation BT.2407 and ITU-R Recommendation BT.2087, which are incorporated herein by reference. There are also look-up tables (LUTs) and / or other well-known methods that can be used to support conversion between color spaces (e.g., Report ITU-R BT.2408-0 or HLG Look-Up Table Licensing by the BBR R&D, both of which are incorporated herein by reference).

[0072] In another example, the source signal is adjusted and converted to a lower bit depth SDR signal, for example by performing a color transformation prior to the procedure by a downsampling process.

[0073] Fig. 3 shows an example of a decoder according to the first embodiment. The data stream 210 is split by the demux 300 into three different streams: HDR stream or range data 310, a first set of enhancement data 330, and a second set of enhancement data 360. The HDR range data 310 is fed to the HDR adjustment module 320 and combined with the low quality video 240 to generate a first output stream. The first output stream is fed to the first enhancer 340, which processes it together with the first set of enhancement data 330 to generate a second output stream. The second output stream is finally fed to the upsampler 350, where the upsampled stream is processed by the second enhancer 370 with the second set of enhancement data 360 to output the video HDR output 260. It should be noted that if the first set of enhancement data 330 is not present, the first output stream is fed directly to the upsampler 350, and the remaining steps remain unchanged.

[0074] Fig. 4 shows an example of a decoder according to the second embodiment. The second embodiment may be seen as an alternative embodiment of the first embodiment. The data stream 210 is split by the demux 300 into three different streams: the HDR stream 310, the first set of enhancement data 330 and the second set of enhancement data 360. The first set of enhancement data 330 is fed to the first enhancer 400, which combines it with the low quality video 240 to generate a first output stream. This first output stream is fed to the upsampler 410 to generate an upsampled version, which is then fed together with the HDR stream data 310 to the HDR adjustment module 420 to generate a second output stream. The second output stream is processed by the second enhancer 370 with the second set of enhancement data 360 to output the video HDR output 260. It should be noted that if the first set of enhancement data 330 were not present, the low quality video 240 would be fed directly to the upsampler 410, with the remaining steps remaining unchanged.

[0075] Fig. 5 shows an example of a decoder according to a third embodiment. The third embodiment may be implemented as an alternative to one or more of the first and second embodiments. The data stream 210 is split by the demux 300 into three different streams: the HDR stream 310, the first set of enhancement data 330 and the second set of enhancement data 360. The first set of enhancement data 330 is provided to a first enhancer 500, which combines it with the low quality video 240 to generate a first output stream. This first output stream is provided to an upsampler 510, which processes it together with the HDR range data 310 to generate a second output stream. The second output stream is processed by a second enhancer 520 with the second set of enhancement data 360 to output a video output 280. In this case, upsampling and HDR adjustment are combined into one operation.

[0076] In the embodiment shown in Figures 3, 4 and 5, if one or more of the HDR range data 310, the first set of enhancement data 330, and the second set of enhancement data 360 are missing from the data stream 210, the corresponding enhancement or adjustment operation using this data is also omitted. In this case, if a portion of the demuxed data stream is not present, then the receiving processing device (e.g., one of the enhancer or HDR adjustment modules) may be configured to pass the received video stream. In some cases, for example, if the display device is not capable of rendering HDR video, the enhancement decoder may be configured to output SDR video by omitting the HDR adjustment operation. Different video outputs may be configured by selectively controlling the operation of the components of the enhancement decoder 250.

[0077] FIG. 6 illustrates an example of a computer system 600 as described herein. The computer system 600 is shown including multiple processors on which the decoders described above may be implemented. The computer system includes a system bus 611, a computer readable storage medium 612, one or more processors 613, an I / O interface 614, and a communication interface 615. FIG. 6 illustrates the computer system 600 communicatively connected to an input device 670 and a storage device 680 communicatively connected to the I / O interface 614, and communicatively connected to one or more networks 690 via the communication interface 615. The computer readable storage medium 612 stores a signal processor application 640-1, which executes at least the enhancement decoder described above. The signal processor application 640-1 may be executed by one or more processors 613, with one or more signal processors processing 640-2. It is particularly advantageous to be able to distribute the decoding process across multiple processors to enable parallel processing of the elements. The encoding and decoding systems and methods described herein allow for the efficient use of parallel processing.

[0078] FIG. 7 shows an example of an encoding system and method as described in the present application. An input signal 700, for example a high bit depth (for example 10 or 12 bits, or even 16 bits or more) HDR type signal, is received at the encoding system. The input signal is then converted from a first color space (for example BT.2020 or BT.2100) to a second color space (for example BT.709) via a first color conversion module 701. The first color space and the second color space are different. During the conversion, the bit depth of the signal may be reduced from the original high bit depth to a low bit depth (for example 8 bits) SDR signal. The SDR signal is then processed by a downsampler 702 to pass from high resolution to low resolution. The output signal is then processed by a first encoder 703. In one example, the first encoder 703 is a well-known, standards-based encoding module, such as based on the MPEG standard, and in one example, a widely used encoding standard with which a significant portion of end users are compatible, providing broad backward compatibility of the technology.

[0079] In other examples, the conversion and downsampling may be performed in the reverse order, i.e., first downsampling and then conversion. In other examples, no downsampling operation is performed and thus the converted signal is passed directly to the first encoder 703. In this most basic case, where no downsampling operation is performed, there is no change in resolution.

[0080] The output of the first encoder 703 is a first encoded stream 710. This may be thought of as a base layer output that is universally decodable as described above, and may provide the user with a representation of the input signal in a different color space (and optionally at a lower resolution) than the input signal 700. The different color space may be one or any number of different formats, or may be defined by one or any other constraints on the signal. In one example, the color space may change from an RGB color space to a YUV color space, or from an HDR type color space to an SDR type color space.

[0081] The output of the first encoder 703 is further processed by a first decoder 704, and the output encoded signal is then passed to a second color conversion module 705, which converts the encoded signal back from the second color space to the first color space. The second color conversion module 705 may use an inverse process to that used by the first color space conversion module 701. The advantage in utilizing an inverse process or otherwise, without any information flowing from the first color conversion module 701 to the second color conversion module 705, is that no metadata or other instructions on how to perform the color space conversion 705 are required. As will become apparent, the high level encoding process provides a way to correct for any inaccuracies in the color space conversion. Thus, there is no need to attempt to accurately or perfectly predict the signal in the first color space because of the high level encoding.

[0082] Additionally, during the conversion performed by the second color conversion module 705, the bit depth of the signal may be increased from a lower bit depth (e.g., 8 bits) to a higher bit depth (e.g., 10 bits, 12 bits, or more) to generate an HDR type signal. The HDR type signal, or simply the color converted signal, is then upsampled by an upsampler 706, and the generated upsampled HDR type signal is processed together with the original input signal by a second encoder 707 to generate a second encoded stream 720.

[0083] The processing performed by the second encoder 707 may generate a residual signal representing the difference between the input signal 700 as a reference signal and an upsampled HDR type signal as a reconstruction of the original signal 700. The residual signal may be used to adjust the corresponding upsampled HDR type signal at the decoder to reconstruct the original input signal 700 or an approximation thereof at the decoder. In this case, the residual signal constitutes any artifacts or other signal inconsistencies introduced by the first encoder 703 and the first decoder 704. In other words, the residual signal corrects the inaccuracies of the codec type. Additionally, the residual signal corrects any artifacts or signal inconsistencies introduced by the color space conversion performed by both the first and second color space conversion modules 701 and 705. Finally, when used, the residual signal corrects any inconsistencies introduced by the downsampler 702 and the upsampler 706. This allows the design decisions to be made to use simpler or more efficient versions of those mentioned above. The residual signal may be encoded using known techniques.

[0084] In one example, the transform and upsampling can be performed in the reverse order, i.e. first the upsampling and then the transform. In another example, no upsampling operation is performed and thus the transformed signal is passed directly to the second encoder 707, which in this case generates a residual signal as before, but this time without a change in resolution.

[0085] As can be seen from FIG. 7, the encoder and associated encoding method generate a first encoded stream 710, which may be encoded using a standard-compliant or other encoding means, that represents a signal at a first or base level of quality. The encoder and associated encoding method also generate a second encoded stream 720, or an enhancement stream, that may have residual information for adjusting or correcting a color-transformed version of the signal at the base level of quality derived from the first encoded stream 710 at a decoder. Resolution changes may be present in addition to color changes. All these features individually and / or collectively result in a composite signal that allows color space conversion, such as but not limited to HDR to SDR conversion, to be encoded and decoded in addition to a widely codable version of the encoded input signal, thus providing one or more of signal improvement, potential bandwidth reduction, and backward compatibility. The architecture and structure may provide efficiencies over the prior art.

[0086] FIG. 8 shows an example of a decoding system and method as described herein. A first decoder 801 receives a first encoded stream 710 and encodes it to generate an encoded signal 820 in a second color space, which may be a signal of SDR type with a low bit depth (e.g., 8 bits). The encoded signal 820 is then passed to a color conversion module 802 for converting the encoded signal back from the second color space to the first color space. During the conversion, the bit depth of the signal is increased from a low bit depth (e.g., 8 bits) to a high bit depth (e.g., 10 bits, 12 bits or more) to generate an HDR type signal that is a reconstruction of the original HDR type signal at a lower resolution. The converted signal (first color space and / or HDR type) is then upsampled by an upsampler 803, and the generated upsampled signal is processed by a second decoder 804 together with the second encoded stream 720 to generate the first encoded signal 810 (e.g., a signal in the first color space and / or a compensated HDR type). Here, the second encoded stream 720 may include residual information that is combined (e.g., summed) with the upsampled HDR type signal to reconstruct the original input signal 700 (e.g., as received at the encoder) or an approximation thereof. Alternatively, in other examples, the color conversion and upsampling can be performed in the reverse order, i.e., upsampling first, followed by color conversion. In other examples, the upsampling operation 803 is not performed, and thus the transformed signal is passed directly to the second decoder 804. This occurs in the base case when there is no resolution change.

[0087] The decoding system of FIG. 8 generates a first decoded signal 820 in a second color space, which may be an SDR type signal. The first decoded signal 820 can be used for output to a display, particularly advantageous for displays that cannot take advantage of the information available when the first decoded signal 820 is converted to the second color space. The decoding system of FIG. 8 also generates a final decoded signal 810 in a first color space, which may be an HDR type signal, which is corrected by a residual signal in the second encoded stream 720. The final decoded signal 810 can be used for output to a display that is compatible with the first color space. The final decoded signal 810 can be an additional level of quality when compared to the first decoded signal 820 (e.g., enhanced color and / or resolution). The decoding system may determine what capabilities any connected displays have and may therefore choose whether to create a final decoded signal 810, whether to use the initial decoded signal 820, and / or whether to convert the final decoded signal back to a compatible format.

[0088] Figure 9 shows an example of an encoding system as described herein. The example of Figure 9 may be seen as a variant of the example of Figure 7. Features in common with the disclosure of Figure 7 will not be described in detail for the sake of brevity. In particular, Figure 9 shows an example of three layer encoding, and where there are two separate input signals, which may be of the same resolution but differ in color space and / or SDR / HDR characteristics.

[0089] In this example, an SDR input signal 900, for example a low bit depth (e.g. 8-bit) SDR signal, is received at the encoding system. It can be either an original signal (e.g. obtained directly from a video source such as a camera or video feed) or an HDR signal (e.g. HDR input signal 960) that has been converted from an HDR color space (e.g. BT.2020 or BT.2100) to an SDR color space (e.g. BT.709). The SDR signal 900 is processed by a downsampler 902 to pass from high resolution to low resolution. The output signal is then processed by a first encoder 903. The output of the first encoder 903 is a first encoded stream 910. The output of the first encoder 903 is further decoded by a first decoder 904, and the output decoded signal is then upsampled by an upsampler 906, and the resulting upsampled SDR signal is processed by a second encoder 907 together with the original SDR input signal 900 to generate a second encoded stream 920. As described with reference to FIG. 7, the second encoded stream 920 may have a residual window to correct any discrepancies in the downsampling / upsampling processes 902 and 906 and the encoding / decoding processes 903 and 904.

[0090] The second encoded stream 920 and the first encoded stream 910 are then passed to an SDR decoder 930, which may be similar to the SDR decoder shown in the rectangular area of ​​Fig. 10. The SDR decoder 930 decodes the first encoded stream 910 and upsamples it to create a reconstructed version of the SDR input signal 900. The second encoded stream 920, if there is residual information in it, is then used to refine or adjust the reconstructed version of the SDR input signal 900 to the SDR input signal or a signal that is close to it. The color space of the previous signal is the same as that of the SDR input signal 900.

[0091] The output of the SDR decoder 930 is then passed to a color conversion module 940 to convert the decoded signal from the SDR color space to the HDR color space. During the conversion, the bit depth of the signal may be increased from a low bit depth (e.g., 8 bits) to a high bit depth (e.g., 10 bits or 12 bits) to generate an HDR signal (e.g., with a larger dynamic range of luminance and / or a wider color gamut). The generated HDR signal is then fed to a third encoder 950 along with an HDR input signal 960 to generate a second encoded stream 970. This third encoded stream 970 may contain additional decoding information that can be used to reconstruct the HDR signal at the decoder.

[0092] The third encoded stream 970 may have a residual signal that primarily corrects any inconsistencies in the color conversion process. The residual signal represents the difference between the HDR input signal 960 as a reference signal and the generated HDR signal. The residual signal of the third encoded stream 970 may correct any problems resulting from the downsampling / upsampling processes 902 and 906 and the encoding / decoding processes 903 and 904 that have not yet been resolved by the residual of the second encoded stream 920. The residual signal may be used to adjust the corresponding color space converted HDR signal in the decoder to reconstruct the HDR input signal 960 or an approximation thereof. For example, the residual signal generated by the third encoder 950 may correct the difference between the output of the SDR decoder 930 and the HDR input signal 960 following conversion to HDR format.

[0093] Fig. 10 shows an example of a decoding system. The decoding system of Fig. 10 is designed to receive the encoded stream output by the encoder of Fig. 9. A first decoder 1001 receives the first encoded stream 910 and decodes it to generate a first decoded signal. The first decoded signal is then upsampled by an upsampler 1003, and the generated upsampled signal is processed by a second decoder 1004 together with a second encoded stream 920, for example using a residual signal therein, to generate an SDR encoded signal 1010. The SDR encoded signal 1010 is then passed to a color conversion module 1002 to convert the decoded signal from an SDR color space to an HDR color space. During the conversion, the bit depth of the signal may also be increased from a low bit depth to a high bit depth, generating an HDR reconstructed signal. The HDR reconstructed signal is then passed to a third decoder 1020 together with a third encoded stream 970, for example using a residual signal therein, to generate an HDR encoded signal 1030. For example, the residual signal in the second encoded stream 970 may be combined (eg, added) with the output of the color transformation module 1002 .

[0094] The encoding and decoding systems of Figures 9 and 10 provide a flexible and efficient technique for encoding and decoding signals, particularly video signals available in two resolutions and two color spaces, non-exclusively. Two separate signals are input to the encoder, one having a higher or more sophisticated color space than the other. For example, one signal may be in a high dynamic range color space (e.g., having a high brightness and contrast ratio and / or a wide color gamut) and the other signal may be in a non-high dynamic range or standard dynamic range. Multiple encoded streams associated with the two separate input signals are available to the decoder.

[0095] Additionally, the encoding and decoding system also provides a downsampled version of the signal in a lower or less refined color space, for example, a downsampled version of the SDR signal at a low bit depth. This signal is encoded by the first encoder 903 and forms a base layer for the signal output from the encoder system. The base layer is at a lower resolution than the input signal 900, 960. Thus, a more useful encoding and decoding system is provided that ensures backward compatibility so that older decoders can decode the first encoded stream 910 (e.g., the base layer) at a lower resolution. For example, the first encoded stream 910 may be at standard resolution, while the second encoded stream 920 and the third encoded stream 970 may be at HD or 1080p resolution. Alternatively, the first encoded stream 910 may be at HD or 1080p resolution, and the second encoded stream 920 and the third encoded stream 970 may be at UHD or 4K resolution. A decoder capable of handling higher resolutions can utilize the second encoded stream 920, and in particular the residual information contained therein, to upsample and correct the upsampled representation at higher resolution, for example UHD or 4K resolution, and output it as an SDR decoded signal 1010. In some cases, the SDR decoded signal 1010 may be viewable by a display device capable of displaying higher resolution, but not adapted to display video data in the first or HDR color space. The upsampling 1030 and correction in the second decoder 1004 may be performed by software, which may allow a software update to be provided for existing older decoders to apply this technique. Of course, the upsampled and corrected rendition may be used as a basis for further signal refinement provided by the color space conversion 1002 and the third encoded stream 970, provided the decoder has this capability and the corrected display has the corresponding capability to display this signal.As can be seen, a content provider may supply content for one or more signals, and the encoding and decoding system may appropriately distribute the first, second and third encoded streams necessary to reproduce the signal in three formats on various decoder / display combinations. If the second and third encoded streams 920 and 970 are residual signals, then this may be efficiently encoded (if they represent low density or unstructured data compared to the high density or structured data output by the first encoder 903). For example, the residual may be near zero mean and may have many zero values ​​(in regions where the two signals being compared do not differ). This form of the signal may be effectively compressed.

[0096] FIG. 11 shows an example of an encoding system described herein. An SDR input signal 900, for example a low-bit signal (e.g., 8-bit) SDR signal, is received at the encoding system. This can be either an original signal (e.g., obtained directly from a video source such as a camera or video feed) or an HDR input signal (e.g., HDR input signal 960) that has been converted from an HDR color space (e.g., BT.2020 or BT.2100) to an SDR color space (e.g., BT.709). The SDR input signal 900 is then processed by a first encoder 1103. The output of the first encoder 1103 is a first encoded stream 910. The output of the first encoder 1103 is further decoded by a first decoder 1104, and the output decoded signal is then passed to a color conversion module 1105 for back conversion from the SDR color space to the HDR color space. During the conversion, the bit depth of the signal may be increased from a low bit depth to a high bit depth, generating an HDR signal. The HDR signal is then upsampled by an upsampler 1106 and the resulting upsampled signal is processed together with the HDR original input signal 960 by a second encoder 1107 to generate a second encoded stream 920.

[0097] In one example, the transform and upsampling may be performed in the reverse order, i.e., first upsampling and then transform. In another example, no upsampling operation is performed and the transformed signal is then passed directly to the second encoder 1107. In some cases, it is beneficial to perform the color transform before the upsampling, since the color transform can be performed more quickly at lower resolutions. If a residual signal is generated by the second encoder 1107, it compensates for any differences between the HDR input signal 960 and the output of the upsampler 1106.

[0098] The first encoded stream 910 and the second encoded stream 920 are then sent to a decoder as shown in FIG. 8 (e.g., instead of the corresponding first encoded stream 710 and second encoded stream 720 shown in FIG. 11) and decoded to generate a decoded SDR signal 820 and a decoded HDR signal 810.

[0099] FIG. 12 shows an example of an encoding system as described herein. An input signal 1000, for example a high bit depth (e.g., 10-bit or 12-bit) HDR signal, is received at the encoding system. The input signal is then processed by a downsampler 1002 to pass from high resolution to low resolution. The downsampled signal is then converted to a tone mapped converted downsampled signal using a tone mapping converter 1003. The tone mapping converter 1003 derives a tone map 1016 for the input signal using conventional and known tone mapping operators and algorithms (for background purposes only, the reader may refer to US Patent Application Nos. 20150078661, 20170070719 or US Patent No. 9437171, which are incorporated herein by reference). The conversion may include adjusting various parameters related to downsampling, such as, for example and without limitation, luminance and chroma values, brightness, contrast, and color, gamma logic, chroma decimation, scaling, chroma noise reduction, etc. In the conversion process, the tone mapping converter 1003 may further convert the downsampled signal from a high bit depth (e.g., 10-bit or 12-bit) HDR signal to a low bit depth (e.g., 8-bit) SDR signal. For example, this is most beneficial when the first encoder 1004 can only process 8-bit signals and not signals with a bit depth of 10 bits or more. The tone map 1016 may be transmitted (e.g., as metadata) to a decoding system, for example, compressed or uncompressed. The tone mapped downsampled signal is then processed by the first encoder 1004. In one example, the conversion and downsampling may be performed in the reverse order, i.e., downsampling first and then conversion. In another example, no downsampling operation is performed and thus the transformed signal is passed directly to the first encoder 1004 .

[0100] The output of the first encoder 1004, i.e. the first encoded stream 1012, is further decoded by a first decoder 1005 to generate a first decoded signal. Optionally, the first decoded signal is processed by a second encoder 1006 together with the tone mapped downsampled signal to generate a second encoded stream 1014. Optionally, the second encoded stream is decoded by a second decoder 1007 and added to the first decoded signal to generate a second encoded signal (not shown). The second decoded signal (or the first decoded signal, if a second encoder and decoder are not used) is then processed by a tone mapping inverse converter 1008, which generates a tone map adjusted signal using the tones generated by the tone mapping converter 1003. The tone mapping inverse converter 1008 may operate in an inverse manner to the tone mapping converter, by re-adjusting the signal as indicated by the tone map using conventional and / or known inverse tone mapping operators and algorithms (the reader may refer to the above-mentioned published patent documents only as background). If the second decoded signal is a low bit-depth (e.g., 8-bit) SDR signal, the tone mapping inverse converter may further convert the second decoded signal from a low bit-depth (e.g., 8-bit) SDR signal to a high bit-depth (e.g., 10-bit or 12-bit) HDR signal. The signal is then upsampled by the upsampler 1009, and the generated upsampled signal is processed by the third encoder 1010 together with the original input signal to generate the third encoded signal 1020. In one example, the conversion and upsampling may be performed in the reverse order, i.e., first upsampling and then conversion. In another example, no upsampling operation is performed, and thus the converted signal is passed directly to the third encoder 1010.

[0101] The optional second encoder 1006 may operate to correct the difference between the output of the first decoder and the tone mapped downsampled signal. This may be performed to derive the difference (e.g., a residual) between the output of the first decoder and the tone mapped downsampled signal and to encode the residual in an efficient manner. The encoded residual is then inversely processed by the optional second decoder 1007 to decode it and add it to the first decoded signal to generate a second decoded signal that is effectively a corrected version of the first decoded signal.

[0102] Similarly, the third encoder 1010 may operate to correct the difference between the original input signal 1000 and the upsampled HDR signal, e.g., the output of the upsampler 1009. This may be performed, for example, to derive the difference (e.g., a residual) between the original input signal and the upsampled HDR signal and to encode the residual in an efficient manner. The encoded residual is then inversely processed by a decoding system and added to the upsampled HDR signal.

[0103] The optional second encoder / decoder and the third encoder / decoder may, for example, operate in a manner as disclosed in patent documents WO2014 / 170819, WO2013 / 011466, WO2013 / 011492 or WO2017 / 089839, which are incorporated herein by reference.

[0104] By deriving the tone map on the low resolution signal (e.g. after downsampling), the processing of the tone map is significantly reduced as it is performed on fewer pixels and can therefore be performed more quickly. Also, the size of the tone map is reduced compared to the tone map at full resolution, which can then be compressed to give greater efficiency. Furthermore, if an optional second encoder / decoder is used in the process, any artifacts (e.g. color banding, blocks, etc.) introduced by the first encoder / decoder are corrected at the low resolution, thus allowing for more efficient correction of the artifacts compared to artifacts corrected at the high resolution. Furthermore, the second encoder further enhances the signal by adding more residual to bring the encoded signal as close as possible to the HDR input signal.

[0105] FIG. 13 illustrates an example of an encoding system as described herein. An input signal 1000, for example a high bit depth (e.g. 10-bit or 12-bit) HDR signal, is received at the encoding system. The input signal is then processed by a downsampler 1002 to pass from high resolution to low resolution. The downsampled signal may then be converted to a tone mapped converted downsampled signal using a tone mapping converter 1003. The tone mapping converter 1003 derives a tone map 1016 of the input signal using conventional and / or known tone mapping operators and algorithms (e.g. as described above). The conversion may include adjusting various parameters associated with the downsampled signal, such as, for example, by way of non-limiting example, luminance and chroma values, brightness, contrast, color, gamma logic, chroma decimation, scaling, chroma noise reduction, etc. In the process of conversion, the tone map converter 1003 may further convert the downsampled signal from a high bit depth (e.g., 10-bit or 12-bit) HDR signal to a low bit depth (e.g., 8-bit) SDR signal. For example, this may be most beneficial when the first encoder 1004 can only process 8-bit signals and not 10-bit or higher bit depth signals. The tone map 1016 may be transmitted (e.g., as metadata) to the decoding system, either compressed or uncompressed. The tone mapped downsampled signal is then processed by the first encoder 1004. In one example, the conversion and downsampling may be performed in the reverse order, i.e., downsampling first and then conversion. In another example, no downsampling operation is performed and thus the converted signal is passed directly to the first encoder 1004. The output of the first encoder 1004, i.e., the first encoded stream 1020, is further decoded by the first decoder 1005 to generate a first decoded signal.The first decoded signal is then processed by a tone mapping inverse converter 1008 using the tone map 1016 generated by the tone map converter 1003 to generate a tone mapped adjusted signal. The tone mapping inverse converter 1008 may operate in an inverse manner to the tone mapping converter by readjusting the signal as indicated by the tone map using conventional and / or known inverse tone mapping operators and algorithms (such as those described above). If the second decoded signal is a low bit depth (e.g., 8 bit) SDR signal, the tone mapping inverse converter may further convert the second decoded signal from a low bit depth (e.g., 8 bit) SDR signal to a high bit depth (e.g., 10 bit or 12 bit) HDR signal. Optionally, the tone mapped corrected signal is processed by a second encoder 1006 together with the downsampled signal to generate a second encoded signal 1014. Optionally, the second encoded stream is decoded by a second decoder 1007 and added to the first decoded signal to generate a second decoded signal (not shown). The second decoded signal (or the first decoded signal, if a second encoder and decoder are not used) is then upsampled by upsampler 1009, and the resulting upsampled signal is processed together with the original input signal by third encoder 1010 to generate third encoded signal stream 1020. In one example, the transform and upsampling may be performed in the reverse order, i.e., first upsampling, then transform. In another example, no upsampling operation is performed, and thus the transformed signal is passed directly to third encoder 1010.

[0106] The optional second encoder 1006 may operate to correct the difference between the output of the first decoder and the downsampled signal. This may be performed, for example, to derive the difference (e.g., a residual) between the output of the first decoder and the downsampled signal, and to encode the residual in an efficient manner. The encoded residual is then inversely processed by the optional second decoder 1007 to decode it and add it to the first decoded signal to generate a second decoded signal that is an effectively corrected version of the first decoded signal.

[0107] Similarly, the third encoder 1010 may operate to correct the difference between the original input signal and the upsampled HDR signal. This may be performed, for example, to derive the difference (e.g., a residual) between the original signal and the upsampled HDR signal, and to encode the residual in an efficient manner. The encoded residual may then be inversely processed by a decoding system and added to the upsampled HDR signal.

[0108] The optional second and third encoders / decoders may, for example, operate in a manner similar to that described in the above-referenced patent documents.

[0109] By deriving the tone map for a low resolution signal (e.g. after downsampling), the processing of the tone map is greatly reduced since it is performed for a smaller number of pixels and therefore it can be performed more quickly. Also, the size of the tone map is reduced compared to the tone map at full resolution, which gives great efficiency if it is subsequently compressed. Furthermore, if an optional second encoder / decoder is used for processing, any artifacts (e.g. color banding, blocks, etc.) introduced by the first encoder / decoder and / or the tone mapping process can be corrected at the low resolution, thus allowing for more efficient correction of the artifacts compared to the artifacts corrected at the high resolution. Furthermore, the third encoder further enhances the signal by adding additional residuals to bring the encoded signal as close as possible to the HDR input signal.

[0110] FIG. 14 shows an example of a decoding system as described herein. A first decoder 1101 receives a first encoded stream 1010 and decodes it to generate a first decoded signal 1120. The first decoded signal 1120 is then processed by a tone mapping inverse converter 1108, which uses the received tone map metadata 1016 to generate a tone map adjusted signal. The tone mapping inverse converter 1108 may operate in an inverse manner to the tone mapping converter by re-adjusting the signal as indicated by the tone map using conventional and / or known inverse tone mapping operators and algorithms (e.g., as described above). If the first decoded signal is a low bit depth (e.g., 8 bit) SDR signal, the tone mapping inverse converter may further convert the first decoded signal from a low bit depth (e.g., 8 bit) SDR signal to a high bit depth (e.g., 10 bit or 12 bit) HDR signal. Optionally, a second decoder 1201 decodes the second encoded stream 1014 to generate a decoded first residual stream. The tone mapped adjusted signal is then optionally processed by the second decoder 1201 together with the decoded first residual stream to generate a second encoded signal. The second decoded signal (e.g., the tone mapped adjusted signal if the second decoded stream 1014 is not received and therefore the second decoder 1201 is not used) is then upsampled by the upsampler 1103 to generate a tone mapped compensated upsampled signal. In one example, the tone mapped transformation and upsampling can be performed in the reverse order, i.e., first upsampling and then transformation. In another example, no upsampling operation is performed and thus the tone mapped signal is passed directly to the upsampler 1103. The third decoder 1104 decodes the third encoded stream 1020 to generate a decoded second residual stream. The tone mapped compensated upsampled signal is further processed by the third decoder 1104 together with the decoded second residual stream to generate a third decoded signal. The third decoded signal may be an HDR signal.

[0111] By processing the tone map on the low resolution signal (e.g., before upsampling), the processing of the tone map is greatly reduced since it is performed on fewer pixels, and therefore it can be performed more quickly. Also, the size of the tone map is reduced compared to a tone map at full resolution, making decompression more efficient. Furthermore, if an optional second decoder is used for processing, any artifacts introduced by the first decoder and / or the tone mapping process (e.g., color banding, blocks, etc.) can be corrected at the low resolution, thus allowing for more efficient correction of the artifacts compared to artifacts corrected at the high resolution. Furthermore, the third encoding stream further enhances the signal by adding more residual to bring the encoded signal as close as possible to the HDR input signal.

[0112] The examples of Figures 12 to 14 may be applied, for example, when dynamic tone mapping is performed where tone mapping parameters change based on the frame being processed. The examples may be easily applied, for example, to successive frames where each frame is processed individually, and they may be efficiently and easily implemented in encoder and decoder systems.

[0113] The examples described herein allow efficient conversion of 10-bit or 12-bit HDR video to a dataset that includes 8-bit low-resolution video, a tone map for local reconstruction of 10-bit or 12-bit HDR information, and correction metadata to enable reconstruction of a full-resolution full-quality image. Performing the tone map conversion at low resolution in a hierarchical based coding method provides a unique set of benefits that allow the processing power required for this operation to be minimized (both in encoding and decoding, which is particularly important for live video and low power decoder devices). For example, the availability of correction metadata allows correction of any banding that may be introduced in the 10-bit to 8-bit to 10-bit conversion process. Finally, the 10-bit or 12-bit to 8-bit conversion can be calibrated to either give a viewable backward compatible 8-bit standard dynamic range (SDR) low-resolution video or maximize compression efficiency by intelligently distributing the local image histogram for maximum compression.

[0114] In the transformations discussed in the above examples, so-called blind transformations may be used in the decoder or the encoder may simulate the decoder's operation. This means that the transformations may be performed in a standalone manner that facilitates modular exchange of transformation functions. This applies to upsampling, downsampling, color space transformations, HDR-SDR transformations and vice versa, signal element coding format changes, etc. For example, different transformation methods may be applied to different frames or different video streams. This may involve signaling between the encoder and the decoder to coordinate the encoding and decoding. However, no changes are required in the system architecture or the processing flow to incorporate these changes. This is possible due to the use of residual signals that are independent processes, for example, they correct the difference between two comparison signals (reference signal and reconstructed signal), but the reconstructed signal may be generated in different ways. This example can also be performed in parallel on small parts of the video data, for example coding units or blocks of 2x2 or 4x4 pixels. This is because the execution of the processing flow does not introduce spatial dependencies within parts of a frame of video data. This allows for efficient parallel processing of different frame portions, for example embodiments may be applied in parallel to different frame portions prior to creating a final output to generate an output signal at one or more quality levels.

[0115] The example described here provides an easily implemented simple scheme that can handle different color representations, which may provide different levels of color quality. In one example, the individual transformation components or modules do not try to predict the signal, but operate in a stand-alone fashion, and an enhancement signal using the residual can provide the appropriate adjustment. This approach also allows for a more modular approach to these transformation components and modules, without having to think about modifying the transport stream for metadata, while modifying them as needed.

[0116] Several types of color space transformations are possible. In particular, non-linear color space transformations can be used.

[0117] Additional benefits may be achieved by using the residual information (e.g., only the changes to the signal) to encode color space, HDR or signal element coding format information when transforming a relatively less sophisticated signal into a relatively more sophisticated signal.

[0118] In all the above examples, the first encoding module / encoder (and the corresponding first decoding module / decoder) may correspond to any type of coding scheme, such as a standard MPEG codec (e.g., AVC, HEVC, etc.), a non-standard codec (e.g., VP9, ​​AV1, etc.), or a hierarchical codec such as PERSEUS (trademark).

[0119] The above detailed description is given as a non-limiting example, and any permutation or combination is included in the present description. [Prior art documents] [Patent documents]

[0120] [Patent Document 1] U.S. Patent Application No. 20150078661 [Patent Document 2] U.S. Patent Application No. 20170070719 [Patent Document 3] U.S. Patent No. 9,437,171 [Patent Document 4] International Publication No. WO2014 / 170819 [Patent Document 5] International Publication No. WO2013 / 011466 [Patent Document 6] International Publication No. WO2013 / 011492 [Patent Document 7] International Publication No. WO2017 / 089839 [Non-patent literature]

[0121] [Non-Patent Document 1] A. Banitalebi-Dehkordi, M. Azimi, M. T. Pourazad and P. Nasiopoulos, "Compression of high dynamic range video using the HEVC and H.264 / AVC standards," 10th International Conference on Heterogeneous Networking for Quality, Reliability, Security and Robustness, Rhodes, 2014, pp. 8-12.

Claims

1. 1. A method for encoding a signal, comprising the steps of: receiving a first signal and a second signal, the first signal and the second signal having different versions of a common content, the first signal using a first signal element coding format and the second signal using a second signal element coding format; encoding the first signal by a first encoding module to generate a first encoded signal; decoding the first encoded signal to generate a first decoded signal; processing the first decoded signal by at least converting from the first signal element coding format to the second signal element coding format to generate a processed signal, the processing of the first decoded signal comprising: upsampling the first decoded signal to generate an upsampled signal, and processing the upsampled signal and the first signal in an intermediate encoder to generate an intermediate coded signal; combining and decoding the first encoded signal and the intermediate encoded signal to generate an intermediate decoded signal; converting the intermediate decoded signal from the first signal element coding format to the second signal element coding format to generate the processed signal; and processing the processed signal and the second signal by a second encoding module to generate a second encoded signal; A method for providing the above.

2. 2. The method of claim 1, wherein the first signal element coding format represents a first color space and the second signal element coding format represents a second color space.

3. 3. The method according to claim 1 or 2, characterized in that the first signal element coding format represents a standard dynamic range (SDR) and the second signal element coding format represents a high dynamic range (HDR), the HDR format having a higher brightness and / or a wider color gamut than the SDR format.

4. 4. The method according to claim 1, wherein the first signal element coding format has a first bit-depth and the second signal element coding format has a second bit-depth, the first bit-depth being smaller than the second bit-depth.

5. 5. The method according to claim 1, wherein the first signal element coding format is of a first level of quality and the second signal element coding format is of a second level of quality, the second level of quality being higher than the first level of quality.

6. 6. The method according to claim 1, wherein the second encoded signal is an enhancement signal having residual data representing the difference between the second signal as a reference signal and the processed signal as a reconstructed version of the reference signal.

7. 3. The method of claim 2, wherein the intermediate coded signal comprises an enhancement signal comprising residual data representing the difference between the first signal as a reference signal and the upsampled signal as a reconstructed version of the reference signal.

8. 8. The method of claim 7, wherein processing the upsampled signal by at least converting from the first signal element coding format to the second signal element coding format comprises converting the intermediate decoded signal from the first color space to the second color space.

9. 9. The method of claim 1, wherein processing the first decoded signal comprises converting the first decoded signal from the first signal element coding format to the second signal element coding format, and upsampling an output of the conversion to generate the processed signal.

10. 10. The method according to claim 1 , comprising downsampling the first signal prior to encoding by the first encoding module to generate the first encoded signal.

11. 11. The method of claim 10, wherein the first signal comprises a plurality of consecutive frames and the second signal comprises a corresponding plurality of consecutive frames, each frame being encoded by the first encoding module.

12. 12. The method according to any one of claims 1 to 11, wherein the step of converting from the first signal element coding format to the second signal element coding format is non-linear.

13. 13. A method according to any one of claims 1 to 12, characterized in that the step of converting from the first signal element coding format to the second signal element coding format is non-predictive.

14. 1. A method of decoding a signal, comprising the steps of: decoding the first encoded signal by a first decoding module to generate a first decoded signal in a first signal element coding format; decoding the intermediate encoded signal by an intermediate decoding module to generate an intermediate decoded signal; upsampling the first decoded signal to generate an upsampled signal; combining and processing the upsampled signal and the intermediate decoded signal in the intermediate decoding module to generate an intermediate reconstructed signal; processing the intermediate reconstructed signal by at least converting from the first signal element coding format to a second signal element coding format to generate a second reconstructed signal; decoding the second encoded signal by a second decoding module to generate a second decoded signal; processing the second reconstructed signal and the second decoded signal by the second decoding module to generate a second combined decoded signal that is a reconstruction of the original signal in the second signal element coding format; A method for providing the above.

15. 15. The method of claim 14, wherein the first signal element coding format represents a first color space and the second signal element coding format represents a second color space.

16. 16. The method of claim 14 or 15, wherein the first decoded signal has a standard dynamic range (SDR) and the second combined decoded signal has a high dynamic range (HDR), the HDR format having a higher brightness and / or a wider color gamut than the SDR format.

17. 17. The method according to claim 14, wherein the first decoded signal has a first bit-depth and the second reconstructed signal has a second bit-depth, the first bit-depth being smaller than the second bit-depth.

18. 18. The method according to claim 14, wherein the first signal element coding format is of a first level of quality and the second signal element coding format is of a second level of quality, the second level of quality being higher than the first level of quality.

19. 19. The method according to any one of claims 14 to 18, characterized in that the second decoded signal is an enhancement signal comprising residual data for adjusting the second reconstructed signal to reconstruct an original signal in the second signal element coding format.

20. 20. The method according to any one of claims 14 to 19, characterized in that the intermediate decoded signal is an enhancement signal comprising residual data for adjusting the upsampled signal to reconstruct an original signal in the first signal element coding format.

21. 21. The method according to claim 14 or 20, wherein processing the first decoded signal comprises converting the intermediate reconstructed signal from a first color space to a second color space.

22. 15. The method of claim 14, further comprising the step of detecting whether a connected display is capable of displaying the second signal element coding format, and if so, outputting the intermediate reconstructed signal.

23. 16. The method of claim 15, comprising selecting, if a display is capable of displaying only the first signal element coding format, for converting the second combined decoded signal back to the first signal element coding format.

24. 24. The method of claim 14, wherein the first encoded signal comprises all frames of the second synthesized decoded signal, each frame in the first encoded signal being decoded by the first decoding module to provide a base version of the second synthesized decoded signal, and the second encoded signal comprises enhancement information for each frame of the base version.

25. 25. A method according to any one of claims 14 to 24, wherein the step of converting from the first signal element coding format to the second signal element coding format is non-linear.

26. 26. A method according to any one of claims 14 to 25, characterized in that the step of converting from the first signal element coding format to the second signal element coding format is non-predictive.

27. A computer program adapted to carry out the method according to any one of claims 1 to 13.

28. A computer program adapted to carry out the method according to any one of claims 14 to 26.

29. 28. A computer readable storage medium carrying a computer program according to claim 27.

30. 29. A computer readable storage medium carrying a computer program according to claim 28.

31. A data processing apparatus having a processor and a memory adapted to carry out a method according to any one of claims 1 to 13.

32. 27. A data processing apparatus having a processor and a memory adapted to carry out a method according to any one of claims 14 to 26.

Images