Signal processing with overlay regions

Abstract

The first signal (801) is processed by at least sending the first signal (801) to an encoder (804) to generate an encoded signal (805). A second signal (808) is acquired. The second signal (808) includes an overlay region (810). The overlay region (810) of the second signal (808) includes overlay content. At least the second signal (808) is processed to generate residual data (815). The residual data (815) is encoded to generate encoded residual data. At least the encoded residual data is output.

Description

Technical Field

[0001] The present invention relates to methods for processing signals, such as video, images, hyperspectral images, audio, point clouds, 3DoF / 6DoF and volumetric signals as non-limiting examples. Processing data may include, but is not limited to, acquiring, deriving, encoding, outputting, receiving and reconstructing signals in the context of a hierarchical (layer-based) coding format (wherein signals are decoded in layers with subsequently higher quality levels), utilizing and combining subsequent layers ("steps") of reconstructed data. Signals of different layers may be encoded in different coding formats (e.g., by way of non-limiting example, conventional single-layer DCT-based codecs, ISO / IEC MPEG-5 Part 2 Low Complexity Enhancement Video Coding SMPTE VC-6 2117, etc.) by different elementary streams, which may or may not be multiplexed in a single bitstream. Background Art

[0002] Hybrid backward-compatible decoding techniques have been previously proposed in, for example, WO 2014 / 170819 and WO 2018 / 046940, the contents of which are incorporated herein by reference.

[0003] Processing residuals in video coding has been previously proposed, for example in WO 2020 / 188229, the contents of which are incorporated herein by reference.

[0004] The contents of WO 2020 / 074896, WO 2020 / 074897, WO 2020 / 089583 and WO 2020 / 074898 are also incorporated herein by reference.

[0005] A low complexity enhancement video coding system, referred to as the “LCEVC standard” or “LCEVC”, has been previously described, for example in WO 2020 / 188273 and in ISO / IEC 23094-2 (draft first published in January 2020), the contents of which are incorporated herein by reference.

[0006] Technological advances mean that there are many different technologies and methods for processing signals, especially for compression, storage and transmission. For example, over the years, many methods of encoding picture and video signals to compress information have been developed. The benefit of this is to reduce the storage requirements and bandwidth requirements of any transmission path, whether via air terrestrial broadcast, cable broadcast, satellite broadcast, or via the Internet or other data networks. As technology advances, more complex compression and transmission methods have been developed. It is expected that the signal quality will be improved. For example, a signal with increased video resolution (expressed as an increase in pixels per frame or an increase in the number of frames per second) is required. Therefore, there are many signal formats, and many types of signal encoders and decoders may or may not be able to use these formats. This poses a problem for content creators and distributors because they want to ensure that their content is available to the widest audience. Therefore, creators and distributors must provide their content in many different encoding formats, for example, encoded using standard encoding techniques such as MPEG-2, MPEG-4 Part 10, High Efficiency Video Coding (HEVC). In addition, standard and non-standard broadcast and streaming technologies also facilitate the distribution of encoded signals. Altogether, this creates a complex situation that increases costs for content distributors and makes the adoption of newer and better technologies more difficult due to the lack of compatibility of older technologies that are still widely available (such as, for example, legacy decoders and set-top boxes).

[0007] High dynamic range (HDR) video has demonstrated the potential to deliver higher quality video by adding a quality improvement dimension, that is, in addition to increased resolution (i.e., more pixels per frame) and improved motion fidelity (i.e., more frames per second), the operator can also increase the dynamic range (i.e., a greater range of brightness and more tones, and therefore more vivid contrast and color). The widespread adoption of HDR-enabled displays makes HDR video increasingly important.

[0008] HDR generally refers to a higher brightness and / or color range than standard dynamic range (SDR), which uses a traditional gamma curve to represent the color range. HDR generally works by changing the values ​​of the brightest and darkest colors and by using different color planes, usually extended from the standard color plane to a wider gamut plane.

[0009] There are various HDR implementations on the market today, which usually require a 10-bit (or even higher bit depth) representation of color components (such as luminance Y and chrominance Cb and Cr) in order to have a sufficiently large range of available values. One such HDR implementation is HDR10, which uses static metadata in the video signal to adjust the range of brightness and color represented in the video stream. In this way, a single HDR setting is applied to the entire video sequence. Different implementations that use dynamic metadata have different commercial names, such as HDR10+ or ​​HDR10+. Vision. Through dynamic metadata, HDR settings are changed on a frame-by-frame basis, allowing for greater granularity and adaptation to changes in the scene.

[0010] However, the transmission of HDR video currently requires decoders capable of decoding native 10-bit video formats such as HEVC. This typically requires duplication of video workflows, as most operators need to continue serving a large portion of their customer base that only has legacy devices capable of decoding 8-bit AVC / H.264 video. Likewise, it also prevents operators from using HDR with 8-bit formats such as VP9. Furthermore, the tone mapping required by HDR is not necessarily best suited to optimize compression efficiency. For more information on this, we refer to the following paper: A. Banitelebi-Dehkordi, M. Azimi, MT Pourazad, and P. Nasiopoulos, “Compression of high dynamic range video using the HEVC and H.264 / AVC standards,” 10th International Conference on Heterogeneous Networking for Quality, Reliability, Security and Robustness, Rhodes, 2014, pp. 8-12.

[0011] There is a general problem that newer encoding standards, and particularly HDR-related standards or technologies, are incompatible with legacy decoding equipment that cannot be easily replaced or upgraded. There is also a general problem of limited bandwidth availability for transmitting signals. There is also a general problem of processing and storage overhead for content distributors in creating and maintaining multiple versions of the same content encoded in different ways. This problem has been exacerbated by the introduction of several competing and widely incompatible technologies for encoding HDR-type signals (i.e., those with the potential for a greater brightness range and contrast and / or those containing information representing a wider color gamut).

[0012] There is also a need to encode a "hybrid" image having portions of the content that are HDR and portions of the content that are SDR. Such above issues hinder implementation of such arrangements, such as attempting to encode a single image having portions that are SDR and portions that are HDR. For example, the above issues lead to the overall problem of how to efficiently encode an image that includes HDR content (e.g., an image) superimposed on an SDR image. Equally, the above issues lead to the overall problem of how to efficiently encode an image that includes an SDR image superimposed with HDR content.

[0013] More generally, there is currently a problem of efficiently encoding mixed image, video and other signals with different types of content, of which mixed signals with SDR and HDR content is one example. Summary of the invention

[0014] Various aspects and variations of the invention are set out in the following claims.

[0015] Certain non-claimed aspects are further described in the detailed description which follows.

[0016] Other features and advantages will become apparent from the following description, given by way of example only, with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A high level schematic diagram of an example hierarchical encoding and decoding process is shown;

[0018] Figure 2 A high-level schematic diagram of an example hierarchical deconstruction process is shown;

[0019] Figure 3 An alternative high-level schematic diagram of an example hierarchical deconstruction process is shown;

[0020] Figure 4 A high level schematic diagram illustrating an example encoding process suitable for encoding a residual of a hierarchical output is shown;

[0021] Figure 5 Shows the application of Figure 4 A high level diagram of an example hierarchical decoding process for decoding each output level of ;

[0022] Figure 6 A high level schematic diagram illustrating an example encoding process of a hierarchical encoding technique;

[0023] Figure 7 Shows the application of Figure 6 A high-level diagram of an example decoding process for decoding the output of

[0024] Figure 8 A high level schematic diagram illustrating another example encoding process of a hierarchical encoding technique is shown;

[0025] Fig. 9 A high level schematic diagram illustrating another example encoding process of a hierarchical encoding technique is shown;

[0026] Fig.10 A high level schematic diagram illustrating another example encoding process of a hierarchical encoding technique is shown;

[0027] Fig.11 A high level schematic diagram illustrating another example encoding process of a hierarchical encoding technique is shown;

[0028] Fig.12 A high level schematic diagram illustrating another example encoding process of a hierarchical encoding technique is shown;

[0029] Fig.13 A high level schematic diagram illustrating another example encoding process of a hierarchical encoding technique is shown;

[0030] Fig.14 A high level schematic diagram illustrating another example encoding process of a hierarchical encoding technique; and

[0031] Fig.15 A block diagram showing an example of an apparatus according to an embodiment is shown. DETAILED DESCRIPTION

[0032] Certain examples described herein relate to methods for encoding and decoding signals. Processing data may include, but is not limited to, acquiring, exporting, outputting, receiving, and reconstructing data. This example relates to the control of signal processing operations performed at an encoder and / or decoder. These may include optional signal processing operations to provide an enhanced output signal. For a video signal, the enhanced output signal may include a so-called "super-resolution" signal, such as a signal with improved detail resolution compared to a reference signal. The reference signal may include an encoding of a video sequence at a first resolution and the enhanced output signal may include a decoded version of the video sequence at a second resolution, the second resolution being higher than the first resolution. The first resolution may include an original resolution for a video sequence, such as a resolution at which the video sequence is acquired for encoding.

[0033] introduce

[0034] The examples described herein relate to signal processing. A signal may be viewed as a sequence of samples (i.e., a two-dimensional image, a video frame, a video field, a sound frame, etc.). In the specification, the term "image", "picture" or "plane" (meaning the broadest meaning of "hyperplane", i.e., an array of elements with arbitrary dimensions and a given sampling grid) will generally be used to identify a digital reproduction of a signal sample along a sequence of samples, wherein each plane has a given resolution for each of its dimensions (e.g., X and Y) and comprises a set of plane elements (or "signal elements", "elements" or "pixels" or display elements for two-dimensional images, commonly referred to as "pixels", display elements for volumetric images, etc.), which are characterized by one or more "signal element values", "element values", "values" or "settings" (e.g., by way of non-limiting example, color settings in a suitable color space, settings indicating density levels, settings indicating temperature levels, settings indicating audio pitch, settings indicating amplitude, settings indicating depth, settings indicating alpha channel transparency levels, etc.). Each plane element is identified by a set of suitable coordinates indicating the integer position of the element in the sampling grid of the image. The signal dimensions may contain only spatial dimensions (eg, in the case of images) or only temporal dimensions (eg, in the case of a signal evolving over time, such as a video signal).

[0035] For example, the signal may be an image, an audio signal, a multi-channel audio signal, a telemetry signal, a video signal, a 3DoF / 6DoF video signal, a volumetric signal (e.g., medical imaging, scientific imaging, holographic imaging, etc.), a volumetric video signal, or even a signal in more than four dimensions.

[0036] For simplicity, the examples described herein generally refer to signals displayed as a 2D set plane (e.g., a 2D image in a suitable color space), such as a video signal. The terms "frame" or "field" will be used interchangeably with the term "image" to indicate a temporal sample of a video signal. Any concepts and methods described for a video signal consisting of frames (progressive video signals) can also be easily applied to a video signal consisting of fields (interlaced video signals), and vice versa. Although the embodiments described herein focus on images and video signals, those skilled in the art can easily understand that the same concepts and methods are also applicable to any other type of multi-dimensional signal (e.g., audio signals, volumetric signals, stereoscopic video signals, 3DoF / 6DoF video signals, plenoptic signals, point clouds, etc.).

[0037] Certain layer-based hierarchical formats described herein use varying amounts of correction (e.g., also in the form of "residual data", or simply "residual") to generate a signal reconstruction that is most similar to (or even lossless reconstruction of) the original signal at a given quality level. The amount of correction can be based on the fidelity of the predicted reproduction at a given quality level.

[0038] To achieve high fidelity reconstruction, the encoding method may upsample a lower resolution reconstruction of the signal to the next higher resolution reconstruction of the signal. In some cases, different signals may be best processed by different methods, ie, the same method may not be optimal for all signals.

[0039] Furthermore, it has been determined that nonlinear methods may be more efficient than more traditional linear kernels (particularly separable kernels), but at the expense of increased processing power requirements. In most cases, due to processing power limitations, various sizes of linear upsampling kernels have been used to date (e.g., bilinear, bicubic, multi-lobe Lanczos, etc.), but more recently even more sophisticated nonlinear techniques, such as the use of convolutional neural networks in VC-6, have been shown to produce higher quality preliminary reconstructions, thereby reducing the entropy of the residual data added for a high-fidelity final reconstruction.

[0040] In formats such as LCEVC, the coefficients of the upsampling kernel to be used can be sent to the decoder before LCEVC's non-linear addition of the "prediction residual". At the same time, it is proposed to extend the coding standard's ability to embed coded stream reconstruction metadata that is ignored by uninformed decoders but processed by decoders capable of decoding said user data.

[0041] Examples of layer-based hierarchical coding schemes or formats

[0042] In a preferred example, the encoder or decoder is part of a layer-based hierarchical coding scheme or format. Examples of layer-based hierarchical coding schemes include LCEVC: MPEG-5 Part 2 LCEVC ("Low Complexity Enhanced Video Coding") and VC-6: SMPTE VC-6ST-2117, the former described in PCT / GB2020 / 050695 disclosed as WO 2020 / 188273 (and related standard documents), and the latter described in PCT / GB2018 / 053552 disclosed as WO 2019 / 111010 (and related standard documents), all of which are incorporated herein by reference. However, the concepts shown herein are not necessarily limited to these specific hierarchical coding schemes.

[0043] Figures 1 to 7 An overview of different exemplary layer-based hierarchical coding formats is provided. These are described below. Figure 7 is provided as a context for adding further signal processing operations. Figures 1 to 5 provides an example similar to the implementation of SMPTE VC-6ST-2117, while Figure 6 and Figure 7Examples similar to the implementation of MPEG-5 Part 2 LCEVC are provided. It can be seen that both sets of examples utilize common underlying operations (e.g., downsampling, upsampling, and residual generation) and can share modular implementation techniques.

[0044] Figure 1 A hierarchical encoding scheme is shown very generally. Data to be encoded 101 is retrieved by a hierarchical encoder 102 which outputs encoded data 103. The encoded data 103 is then received by a hierarchical decoder 104 which decodes the data and outputs decoded data 105.

[0045] Typically, the hierarchical coding scheme used in the examples herein creates a base or core layer, which is a representation of the original data at a lower quality layer, and one or more residual layers, which can be used to reconstruct the original data at a higher quality layer using a decoded version of the base layer data. In general, the term "residual" as used herein refers to the difference between the values ​​of a reference array or reference frame and the actual array or frame of data. The array can be a one-dimensional or two-dimensional array representing a coding unit. For example, a coding unit can be a 2x2 or 4x4 group of residual values ​​corresponding to a similarly sized area of ​​an input video frame.

[0046] It should be noted that the generalized example is agnostic to the nature of the input signal. Reference to "residual data" as used herein refers to data derived from a set of residuals, such as a set of residuals themselves or the output of a set of data processing operations performed on a set of residuals. Throughout this specification, in general, a set of residuals comprises a plurality of residuals or residual elements, each of which corresponds to a signal element, i.e., an element of the signal or original data.

[0047] In specific examples, the data may be an image or a video. In these examples, the set of residuals corresponds to an image or a frame of the video, wherein each residual is associated with a pixel of the signal, the pixel being a signal element.

[0048] The methods described herein can be applied to so-called data planes reflecting different color components of a video signal. For example, these methods can be applied to different planes of YUV or RGB data reflecting different color channels. Different color channels can be processed in parallel. The components of each stream can be arranged in any logical order.

[0049] A hierarchical coding scheme will now be described in which the concepts of the present invention can be deployed. Figures 2 to 5and generally corresponds to the above-mentioned VC-6. In this type of coding technique, residual data is used to gradually increase the quality level. In the proposed technique, the core layer represents the image at a first resolution, and the subsequent layers in the hierarchical structure are the residual data or adjustment layers required for the decoding side to reconstruct the image at a higher resolution. Each layer or level can be referred to as a step index, so that the residual data is the data required to correct the low-quality information present in the lower step indices. Each layer or step index in this hierarchical technique, especially each residual layer, is typically a relatively sparse data group with many zero-valued elements. When a step index is mentioned, it refers to all steps or component groups of that level, for example, all subgroups produced in the transformation step performed at that quality level.

[0050] In this particular hierarchical manner, the described data structure eliminates any requirement or dependency on previous or subsequent quality layers. The quality layers can be encoded and decoded individually without reference to any other layer. Thus, in contrast to many other known hierarchical coding schemes, where the lowest quality layer needs to be decoded in order to decode any higher quality layer, the described method does not require decoding of any other layer. However, the principles of exchanging information described below may also be applicable to other hierarchical coding schemes.

[0051] like Figure 2 As shown in , the encoded data represents a set of layers or levels, generally referred to herein as step indices. The base or core level represents the original data frame 210, albeit at the lowest quality level or resolution, and subsequent residual data levels can be combined with data at the core step indices to reconstruct the original image at progressively higher resolutions.

[0052] To create the core step index, the input data frame 210 may be downsampled using a number of downsampling operations 201 corresponding to the number of levels or step indices to be used in the hierarchical encoding operation. The required downsampling operations 201 are one less than the number of levels in the hierarchy. In all the examples described herein, there are 4 levels or step indices of output coded data and corresponding 3 downsampling operations, but it will of course be appreciated that these are for illustration only. Where n indicates the number of levels, the number of downsamplers is n-1. The core level R 1-n is the output of the third downsampling operation. As mentioned above, the core level R 1-n A representation of the input data frame corresponding to the lowest quality level.

[0053] To distinguish the downsampling operations 201, each operation will be referenced by the order in which the operation is performed on the input data 210 or by the data represented by its output. 1-n It can also be called a core downsampler because its output generates a core step index or step1-n , that is, the indices of all rungs at this level are 1-n. Therefore, in this example, the first downsampling operation 201 -1 Corresponding to R -1 Downsampler, second downsampling operation 201 -2 Corresponding to R -2 Downsampler and third downsampling operation 201 1-n Corresponds to the core or R -3 Downsampler.

[0054] like Figure 2 As shown in the figure, it indicates the core quality level R 1-n The data undergoes an upsampling operation 202 1-n , here referred to as the core upsampler. The second downsampling operation 201 -2 Output (R -2 The output of the downsampler, ie the input of the core downsampler) and the core upsampler 202 1-n The difference between the output of 203 -2 is output as the first residual data R -2 This first residual data R -2 Correspondingly, the core level R -3 and the error between the signal used to create the level. Since the signal itself has undergone two downsampling operations in this example, the first residual data R -2 is an adjustment layer that can be used to reconstruct the original signal at a higher quality level than the core quality level, but a lower quality level than the input data frame 210 .

[0055] Figure 2 and Figure 3 It is conceptually shown how to create variations of residual data representing higher quality levels.

[0056] exist Figure 2 In the second downsampling operation 201 -2 (or R -2 Downsampler, which is used to create the first residual data R -2 The output of the signal is upsampled 202 -2 And the second downsampling operation 201 -2 (or R -2 Downsampler, i.e. R -1 The difference between the output of the downsampler) and the input 203 -1 Create the first residual data R with -2 The calculation is performed in almost the same way. This difference is correspondingly the second residual data R -1 and represents an adjustment layer that can be used to reconstruct the original signal at a higher quality level using data from lower layers.

[0057] However, in Figure 3 In a variant of -2 (or R -2 The output of the downsampler) is related to the first residual data R -2 Combine or add 304 -2 To recreate the core upsampler 202 1-n In this variant, the upsampling 202 -2 It is this recreated data that is used instead of the downsampled data. Similarly, the upsampled data is combined with the input of the second downsampling operation (or R -2 Downsampler, i.e. R -1 The output of the downsampler) is compared 203 -1 , to create the second residual data R -1 .

[0058] Figure 2 and Figure 3 The change between the implementations of results in slight changes in the residual data between the two implementations. Figure 2 Benefit from greater parallelization potential.

[0059] Repeat the process or cycle to create the third residual R 0 .exist Figure 2 and Figure 3 In the example, the output residual data R 0 (ie, the third residual data) corresponds to the highest level and is used at the decoder to reconstruct the input data frame. At this level, the difference operation is based on the same input data frame as the input of the first downsampling operation.

[0060] Figure 4 An example encoding process 401 is shown for encoding each of the levels or step indices of data to produce a set of encoded data steps with step indices. This encoding process is merely an example of a suitable encoding process for encoding each of the levels, but it should be understood that any suitable encoding process may be used. The input to the process is from Figure 2 or Figure 3 The output is residual data of a corresponding level, and the output is a set of levels of encoded residual data, which together hierarchically represent the encoded data.

[0061] In the first step, a transform 402 is performed. The transform may be a directional decomposition transform or a wavelet or discrete cosine transform as described in WO 2013 / 171173. If a directional decomposition transform is used, a set of four components (also referred to as transform coefficients) may be output. When the step index is referred to, it is collectively referred to as all directions (A, H, V, D), i.e., 4 steps. The component groups are then quantized 403 before entropy coding. In this example, the entropy coding operation 404 is coupled to a sparseness step 405, which exploits the sparsity of the residual data to reduce the total data size and involves mapping the data elements to an ordered quadtree. Such coupling of entropy coding and sparseness is further described in WO2019 / 111004, but the precise details of such a process are irrelevant to an understanding of the present invention. Each residual array can be considered as a step.

[0062] The process set forth above corresponds to an encoding process suitable for encoding data for reconstruction according to SMPTE ST 2117, VC-6 multiplane picture format. VC-6 is a flexible, multi-resolution, intra-frame only bitstream format capable of compressing any ordered set of integer element grids, each of which has an independent size, but also designed for picture compression. It uses a data agnostic technique for compression and is capable of compressing low or high bit depth pictures. The header of the bitstream may contain various metadata about the picture.

[0063] As will be appreciated, each step or step index may be implemented using a separate encoder or encoding operation. Similarly, the encoding module may be divided into steps of downsampling and comparison to produce residual data, and then encoding the residual, or alternatively, each of the steps of the step may be implemented in a combined encoding module. Thus, the process may be implemented, for example, using 4 encoders, one encoder for each step index, 1 encoder and multiple encoding modules operating in parallel or serially, or one encoder repeating operations on different data groups.

[0064] An example of reconstructing an original data frame that has been encoded using the exemplary process described above is given below. This reconstruction process may be referred to as pyramidal reconstruction. Advantageously, the method provides an efficient technique for reconstructing an image encoded in a received data set that may be received via a data stream, for example, by separately decoding different component groups corresponding to different image sizes or resolution levels, and combining image details from one decoded component group with scaled-up decoded image data from a lower resolution component group. Thus, by performing the process on two or more component groups, structures or details therein may be reconstructed to obtain progressively higher resolutions or a greater number of pixels without the need to receive all or complete image details of the highest resolution component group. Instead, the method facilitates the gradual addition of increasingly high resolution details while reconstructing images from lower resolution component groups in a staged manner.

[0065] Furthermore, individually decoding each component group facilitates parallel processing of the received component groups, thereby increasing reconstruction speed and efficiency in implementations where multiple processes are available.

[0066] Each resolution level corresponds to a quality level or step index. This is a collective term associated with a plane (in this example a grid representation of integer-valued elements) describing all new inputs or groups of components received and the output reconstructed images of the index m cycle. For example, the reconstructed image with step index zero is the output of the last cycle of pyramidal reconstruction.

[0067] The cone reconstruction may be a process of reconstructing an inverted cone starting from an initial step index and using a cycle of new residuals to derive higher step indices until a maximum quality at a step index of zero, the quality being zero. A cycle may be considered a step in such a cone reconstruction, the step being identified by an index m. The step typically comprises upsampling the output data from a possible previous step, e.g., amplifying the decoded first component group, and taking new residual data as further input in order to obtain output data to be upsampled in a possible subsequent step. In the case where only the first and second component groups are received, the number of step indices will be two, and there will be no possible subsequent steps. However, in examples where the number of component groups or step indices is three or more, the output data may be upsampled step by step in the following steps.

[0068] The first component group generally corresponds to an initial step index, which may be represented by step index 1-N, where N is the number of step indices in the plane.

[0069] Typically, the upscaling of the decoded first component group includes applying an upsampler to the output of the decoding process of the initial step index. In an example, this involves making the resolution of the reconstructed picture output from the decoding of the initial step index component group consistent with the resolution of the second component group corresponding to 2-N. Typically, the output of the upscaling from the lower step index component group corresponds to a predicted image at a higher step index resolution. Due to the lower resolution initial step index image and the upsampling process, the predicted image typically corresponds to a smooth or blurred picture.

[0070] Higher resolution detail is added to the predicted picture from the step index above, which provides a combined reconstructed image set. Advantageously, where the received component groups in one or more of the higher step index component groups include residual image data or data indicative of pixel value differences between the scaled up predicted picture and the original, uncompressed or pre-encoded image, the amount of received data required to reconstruct an image or data set of a given resolution or quality may be significantly less than the amount of data or data rate required to receive an image of the same quality using other techniques. Thus, according to the method, data rate requirements are reduced by combining low detail image data received at a lower resolution with progressively more detailed image data received at increasingly higher resolutions.

[0071] Typically, the group of encoded data includes one or more additional component groups, wherein each of the one or more additional component groups corresponds to a higher image resolution than the second component group, and wherein each of the one or more additional component groups corresponds to a progressively higher image resolution, the method comprising: for each of the one or more additional component groups, decoding the component group to obtain a decoded group, the method further comprising: for each of the one or more additional component groups, in ascending order of corresponding image resolution: scaling up the reconstruction group with the highest corresponding image resolution so that the corresponding image resolution of the reconstruction group is increased to be equal to the corresponding image resolution of the additional component groups, and combining the reconstruction group with the additional component groups to produce an additional reconstruction group.

[0072] In this way, the method may involve obtaining a reconstructed picture output for a given component group level or step index, scaling the reconstructed group, and combining it with the decoded output of the above component group or step index to produce a new, higher resolution reconstructed picture. It should be understood that this can be repeated for progressively higher step indices based on the total number of component groups in the received group.

[0073] In a typical example, each of the component groups corresponds to progressively higher image resolutions, wherein each progressively higher image resolution corresponds to four times the number of pixels in the corresponding image. Thus, typically, the image size corresponding to a given component group is four times the size or number of pixels, or twice the height and width, of the image corresponding to the following component group (i.e., the component group having a step index one less than the step index in question). For example, a received set of component groups in which each corresponding image has a linear size twice that of the following image size may facilitate a simpler upscaling operation.

[0074] In the example shown, the number of other component groups is two. Therefore, the total number of component groups in the received group is four. This corresponds to an initial step index of step 3.

[0075] The first component group may correspond to image data and the second component group and any other component groups may correspond to residual image data. As described above, where the lowest step index (i.e., the first component group) contains a low resolution or downsampled version of the image being transmitted, the method provides a particularly advantageous reduction in data rate requirements for a given image size. In this way, in each reconstruction cycle, starting with a low resolution image, the image is scaled up to produce a high resolution yet smooth version, and then the image is improved by adding the difference between the scaled up predicted image and the actual image to be transmitted at that resolution, and this additional improvement may be repeated for each cycle. Therefore, each component that is set to a higher than initial step index need only contain residual data in order to reintroduce information that may have been lost when the original image was downsampled to the lowest step index.

[0076] The method provides a way to obtain image data when receiving a set containing data that has been compressed, for example, by decomposition, quantization, entropy coding and sparseness, which image data may be residual data. The sparseness step is particularly advantageous when used in conjunction with a set of sparse original or pre-transmitted data, which may generally correspond to residual image data. The residual may be the difference between an element of a first image and an element of a second image, which elements are generally located at the same position. Such residual image data may generally have a high degree of sparsity. This can be considered to correspond to an image in which detail areas are sparsely distributed in areas where detail is minimal, negligible or non-existent. Such sparse data can be described as an array of data in which the data is organized in at least a two-dimensional structure (e.g., a grid), and in which most of the data so organized is zero (logically or digitally) or is considered to be below a certain threshold. Residual data is just an example. In addition, metadata may be sparse and therefore be reduced in size to a great extent by this process. Sending data that has been sparsely transmitted allows a significant reduction in the required data rate to be achieved by omitting sending such sparse areas and reintroducing them at appropriate locations within the received byte group at the decoder.

[0077] Typically, the entropy decoding, dequantization and directional synthesis transformation steps are performed according to parameters defined by an encoder or node transmitting the received encoded data groups. For each step index or component group, these steps are used to decode the image data to obtain groups that can be combined with different step indices according to the techniques disclosed above, while allowing the transmission of each level group in a data efficient manner.

[0078] A method for reconstructing a coded data set according to the method disclosed above may also be provided, wherein decoding of each of the first component group and the second component group is performed according to the method disclosed above. Thus, the advantageous decoding method disclosed herein may be used for each component group or step index in a received image data set and reconstructed accordingly.

[0079] refer to Figure 5 , now a decoding example is described. A set of encoded data 501 is received, wherein the set includes four step indices, each step index includes four steps: steps from the highest resolution or quality level 0 Steps to the initial step -3 .step -3 The image data carried by the component group corresponds to the image data, and the other component groups contain the residual data of the transmitted image. Although each level can output data that can be regarded as a residual, the initial ladder level (i.e., ladder -3 ) effectively corresponds to the actual reconstructed image. At stage 503, each of the component groups is processed in parallel to decode the encoded group.

[0080] Refer to the initial step index or core step index, and step each component group -3 To the steps 0 Perform the following decoding steps.

[0081] At step 507, the component groups are de-sparsed. De-sparsed may be an optional step that is not performed in other layer-based hierarchical formats. In this example, de-sparsed results in the reconstruction of a sparse two-dimensional array from the encoded byte groups received at each rung. The zero values ​​grouped at positions that were not received within the two-dimensional array (because they are omitted from the transmitted byte groups in order to reduce the amount of data transmitted) are refilled by this process. The non-zero values ​​in the array retain their correct values ​​and positions in the reconstructed two-dimensional array, and the de-sparsed step refills the transmitted zero values ​​at the appropriate positions or at the positions therebetween.

[0082] At step 509, a range decoder is applied to the de-sparsened group of each rung to replace the coded symbols within the array with pixel values, the configuration parameters of the range decoder corresponding to the parameters used to encode the transmission data before transmission. The coded symbols in the received group are replaced with pixel values ​​based on an approximation of the distribution of pixel values ​​of the image. Using an approximation of the distribution, i.e., the relative frequency of each value of all pixel values ​​in the image, rather than the true distribution, allows the amount of data required to decode the set to be reduced, since the range decoder requires the distribution information to perform this step. As described in the present disclosure, the steps of de-sparseness and range decoding are interdependent, rather than sequential. This is indicated by the loop formed by the arrows in the flow chart.

[0083] The array of values ​​is dequantized at step 511. The process is performed again according to the parameters used to quantize the decomposed image before transmission.

[0084] After dequantization, the group is transformed at step 513 by a synthetic transform which comprises applying an inverse directional decomposition operation to the dequantized array. This results in the directional filtering being inverted, according to a group of operators including average, horizontal, vertical and diagonal operators, so that the resulting array is stepped. -3 Image data and ladder -2 To the steps 0 The residual data.

[0085] Stage 505 shows several loops involved in reconstructing using the output of the composite transform for each of the step component groups 501. Stage 515 indicates the reconstructed image data for the initial step output from decoder 503. In the example, the reconstructed picture 515 has a resolution of 64x64. At 516, the reconstructed picture is upsampled to increase its constituent pixel count by a factor of four, thereby producing a predicted picture 517 having a resolution of 128x128. At stage 520, the predicted picture 517 is added to the output from the step component groups 501. -2 The decoded residual 518 of the output of the decoder. The two 128x128 size images are added to generate a 128x128 size reconstructed image, which contains the reconstructed image from the ladder. -2 The higher resolution details of the residuals are enhanced from the smoothed image details of the initial steps. If the desired output resolution is corresponding to the steps -2 The resulting reconstructed image 519 can be output or displayed if the resolution is 256x256. In this example, the reconstructed image 519 is used for further looping. At step 512, the reconstructed image 519 is upsampled in the same manner as step 516 to produce a predicted image 524 of size 256x256. Then, at step 528, it is compared with the decoded ladder. -1 The outputs 526 are combined to produce a 256x256 reconstructed image 527, which is an upscaled version of the prediction 519 enhanced with the higher resolution details of the residual 526. At 530, the process is repeated a final time and the reconstructed image 527 is upscaled to a 512x512 resolution for use with the ladder at stage 532. 0 The residuals are combined to obtain a 512x512 reconstructed image 531.

[0086] Figure 6 and Figure 7 Other hierarchical coding techniques that can utilize the principles of the present invention are shown. This technique is a flexible, adaptable, efficient and computationally inexpensive coding format that combines different video coding formats, base codecs (e.g., AVC, HEVC or any other current or future codecs) with at least two enhancement levels of the coded data.

[0087] The general structure of the coding scheme uses a downsampled source signal encoded with a base codec, adds a first level of correction data to the decoded output of the base codec to produce a corrected picture, and then adds another level of enhancement data to the upsampled version of the corrected picture. Therefore, the stream is considered to be a base stream and an enhancement stream, which can be further multiplexed or otherwise combined to generate a coded data stream. In some cases, the base stream and the enhancement stream can be transmitted separately. As described herein, references to coded data can refer to an enhancement stream or a combination of a base stream and an enhancement stream. The base stream can be decoded by a hardware decoder, and the enhancement stream can be suitable for software processing implementation with appropriate power consumption. This general coding structure creates multiple degrees of freedom that allow great flexibility and adaptability to many situations, making the coding format suitable for many use cases, including OTT transmission, live streaming, live ultra-high-definition UHD broadcasting, and so on. Although the decoded output of the base codec is not intended for viewing, it is a fully decoded video at a lower resolution, making the output compatible with existing decoders and can also be used as a lower resolution output if deemed appropriate.

[0088] In some examples, each or both enhancement streams may be encapsulated into one or more enhancement bitstreams using a set of Network Abstraction Layer Units (NALUs). A NALU is intended to encapsulate an enhancement bitstream so that the enhancement is applied to the correct base reconstructed frame. A NALU may, for example, contain a reference index to a NALU containing the base decoder reconstructed frame bitstream to which the enhancement must be applied. In this way, the enhancement may be synchronized to the base stream and the frames of each bitstream combined to produce a decoded output video (i.e., the residual of each frame of the enhancement level is combined with the frames of the base decoded stream). A group of pictures may represent multiple NALUs.

[0089] Returning to the initial process described above, where a base stream is provided along with two levels (or sub-levels) of enhancements within an enhancement stream, an example of a generalized encoding process is depicted in Figure 6 , a block diagram of . An input video 600 at an initial resolution is processed to generate various coded streams 601, 602, 603. A first coded stream (coded base stream) is produced by feeding a downsampled version of the input video to a base codec (e.g., AVC, HEVC, or any other codec). The coded base stream may be referred to as a base layer or base level. A second coded stream (coded layer 1 stream) is produced by processing a residual obtained by taking the difference between a reconstructed base codec video and a downsampled version of the input video. A third coded stream (coded layer 2 stream) is produced by processing a residual obtained by taking the difference between an upsampled version of a corrected version of the reconstructed base coded video and the input video. In some cases, Figure 6The components of may provide a general low-complexity encoder. In some cases, the enhancement stream may be generated by an encoding process that forms part of the low-complexity encoder, and the low-complexity encoder may be configured to control an independent base encoder and decoder (e.g., encapsulated in a base codec). In other cases, the base encoder and decoder may be provided as part of the low-complexity encoder. In one case, Figure 6 The low-complexity encoder may be viewed as a form of wrapper for the base codec, where the functionality of the base codec may be hidden from the entity implementing the low-complexity encoder.

[0090] The downsampling operation illustrated by the downsampling component 605 can be applied to the input video to produce a downsampled video to be encoded by the base encoder 613 of the base codec. Downsampling can be performed in both vertical and horizontal directions, or alternatively only in the horizontal direction. The base encoder 613 and the base decoder 614 can be implemented by the base codec (e.g., as different functions of a common codec). The base codec and / or one or more of the base encoder 613 and the base decoder 614 may include appropriately configured electronic circuitry (e.g., a hardware encoder / decoder) and / or computer program code executed by a processor.

[0091] Each enhancement stream encoding process may not necessarily include an upsampling step. Figure 6 In , the first enhancement stream is conceptually the correction stream, while the second enhancement stream is upsampled to provide the enhancement level.

[0092] Referring to the process of generating an enhancement stream in more detail, to generate an encoded layer 1 stream, the encoded base stream is decoded by the base decoder 614 (i.e., a decoding operation is applied to the encoded base stream to generate a decoded base stream). The decoding may be performed by a decoding function or mode of the base codec. The difference between the decoded base stream and the downsampled input video is then created at the layer 1 comparator 610 (i.e., a subtraction operation is applied to the downsampled input video and the decoded base stream to generate a first set of residuals). The output of the comparator 610 may be referred to as a first set of residuals, such as a surface or frame of residual data, wherein the residual values ​​are determined for each image element at the resolution of the output of the base encoder 613, the base decoder 614, and the downsampling block 605.

[0093] The difference is then encoded by the first encoder 615 (ie, the level 1 encoder) to generate the encoded level 1 stream 602 (ie, the encoding operation is applied to the first set of residuals to generate the first enhancement stream).

[0094] As described above, the enhancement stream may include a first enhancement level 602 and a second enhancement level 603. The first enhancement level 602 may be considered a corrected stream, e.g., a stream that provides a correction level to a base encoded / decoded video signal at a lower resolution than the input video 600. The second enhancement level 603 may be considered to convert the corrected stream into another enhancement level of the original input video 600, e.g., it applies the enhancement level or correction to a signal reconstructed from the corrected stream.

[0095] exist Figure 6 In the example of , a second enhancement layer 603 is created by encoding another set of residuals. Another set of residuals is generated by a layer 2 comparator 619. The layer 2 comparator 619 determines the difference between the upsampled version of the decoded layer 1 stream (e.g., the output of the upsampling component 617) and the input video 600. The input of the upsampling component 617 is generated by applying the first decoder (i.e., the layer 1 decoder) to the output of the first encoder 615. This generates a set of decoded layer 1 residuals. These residuals are then combined with the output of the base decoder 614 at the summing component 620. This effectively applies the layer 1 residuals to the output of the base decoder 614. It allows the losses in the layer 1 encoding and decoding process to be corrected by the layer 2 residuals. The output of the summing component 620 can be regarded as an analog signal representing the output of applying the layer 1 processing to the encoded base stream 601 and the encoded layer 1 stream 602 at the decoder.

[0096] As previously described, the upsampled stream is compared to the input video, which produces another set of residuals (i.e., a difference operation is applied to the upsampled reconstructed stream to generate another set of residuals). The other set of residuals is then encoded by the second encoder 621 (i.e., the layer 2 encoder) into an encoded layer 2 enhancement stream (i.e., the encoding operation is then applied to the other set of residuals to generate another encoded enhancement stream).

[0097] Therefore, if Figure 6 As shown in and described above, the output of the encoding process is a base stream 601 and one or more enhancement streams 602, 603, which preferably include a first enhancement layer and another enhancement layer. The three streams 601, 602 and 603 can be combined with or without additional information such as control headers to generate a combined stream representing the video coding framework structure of the input video 600. It should be noted that Figure 6The components shown in the figure can operate on blocks or coding units of data, which correspond to, for example, 2x2 or 4x4 portions of a frame at a particular resolution level. The components operate without any inter-block dependencies, so they can be applied to multiple blocks or coding units within a frame in parallel. This is different from comparative video coding schemes, in which there are dependencies (e.g., spatial dependencies or temporal dependencies) between blocks. The dependencies of comparative video coding schemes limit the level of parallelism and require much higher complexity.

[0098] exist Figure 7 The corresponding generalized decoding process is depicted in the block diagram of . It is said that Figure 7 Can show corresponding to Figure 6 The low complexity decoder receives the three streams 601, 602, 603 generated by the low complexity encoder together with a header 704 containing further decoding information. The coded elementary streams 601, 701 are decoded by a basic decoder 710 corresponding to the basic codec used in the low complexity encoder. The coded layer 1 streams 602, 702 are received by a first decoder 711 (i.e., a layer 1 decoder) which decodes the encoded elementary streams 601, 701 by a basic decoder 710 corresponding to the basic codec used in the low complexity encoder. Figure 6 The first set of residuals encoded by the first encoder 615 is decoded. At the first summing component 712, the output of the base decoder 710 is combined with the decoded residual obtained from the first decoder 711. The combined video, which may be referred to as the layer 1 reconstructed video signal, is upsampled by an upsampling component 713. The encoded layer 2 stream 703 is received by a second decoder 714 (i.e., a layer 2 decoder). The second decoder 714 decodes the encoded layer 2 stream 703 as received by Figure 6 The second group of residuals encoded by the second encoder 621 are decoded. Although the header 704 is Figure 7 714, but it may also be used by the first decoder 711 as well as the base decoder 710. The output of the second decoder 714 is a second set of decoded residuals. These may be at a higher resolution than the first set of residuals and the input to the upsampling component 713. At a second summing component 715, the second set of residuals from the second decoder 714 is combined with the output of the upsampling component 713 (i.e., the upsampled reconstructed level 1 signal) to reconstruct the decoded video 750.

[0099] According to the low complexity encoder, Figure 7 The low complexity decoder of the video signal can operate in parallel on different blocks or coding units of a given frame of the video signal. In addition, the decoding performed by two or more of the base decoder 710, the first decoder 711 and the second decoder 714 can be performed in parallel. This is possible because there is no inter-block dependency.

[0100] During the decoding process, the decoder may parse the header 704 (which may contain global configuration information, picture or frame configuration information, and data block configuration information) and configure the low-complexity decoder based on those headers. In order to recreate the input video, the low-complexity decoder may decode each of the base stream, the first enhancement stream, and the other or second enhancement stream. The frames of the streams may be synchronized and then combined to derive the decoded video 750. Depending on the configuration of the low-complexity encoder and decoder, the decoded video 750 may be a lossy or lossless reconstruction of the original input video 600. In many cases, the decoded video 750 may be a lossy reconstruction of the original input video 600, where the loss has a reduced or minimal impact on the perception of the decoded video 750.

[0101] exist Figure 6 and Figure 7 In each of the above, the level 2 and level 1 encoding operations may include the steps of transform, quantization, and entropy encoding (e.g., in that order). These steps may be performed in a manner similar to Figure 4 and Figure 5 The encoding operation may also include residual scalaring, weighting and filtering. Similarly, in the decoding stage, the residual may be passed through an entropy decoder, a dequantizer and an inverse transform module (e.g., in that order). Any suitable encoding and corresponding decoding operations may be used. However, preferably, the level 2 and level 1 encoding steps may be performed in software (e.g., as performed by one or more central or graphics processing units in the encoding device).

[0102] Transforms as described herein may use directional decomposition transforms, such as Hadamard-based transforms. Both may include a small kernel or matrix applied to the flattened coding units of the residual (i.e., 2x2 or 4x4 residual blocks). More details about the transforms can be found in, for example, patent applications PCT / EP2013 / 059847 (published as WO 2013 / 171173) or PCT / GB2017 / 052632 (published as WO 2018 / 046941), which are incorporated herein by reference. The encoder may choose between different transforms to be used, for example, between the sizes of the kernels to be applied.

[0103] The transform may transform the residual information into four surfaces. For example, the transform may produce the following components or transform coefficients: average, vertical, horizontal, and diagonal. A particular surface may include all values ​​of a particular component, for example, a first surface may include all average values, a second surface may include all vertical values, and so on. As mentioned earlier in this disclosure, these components output by the transform may be employed in these embodiments as coefficients to be quantized according to the described methods. A quantization scheme may be used to create the residual signal as a quantity so that a particular variable may take only specific discrete values. In this example, entropy coding may include run-length encoding (RLE) and then the encoded output may be processed using a Huffman encoder. In some cases, when entropy coding is required, only one of these schemes may be used.

[0104] In summary, the methods and apparatus herein are based on an overall approach that is built upon existing encoding and / or decoding algorithms (such as MPEG standards, such as AVC / H.264, HEVC / H.265, etc.; and non-standard algorithms, such as VP9, ​​AV1, etc.), which are used as a baseline for enhancement layers for different encoding and / or decoding methods, respectively. The idea behind the overall approach of the example is to encode / decode video frames in a hierarchical manner, in contrast to using a block-based approach used in the MPEG family of algorithms. Encoding frames in a hierarchical manner includes generating a residual for a full frame, and then generating a residual for a decimated frame, and so on.

[0105] As described above, these processes can be applied in parallel to the coding units or blocks of the color components of a frame because there are no inter-block dependencies. The encoding of each color component within a set of color components can also be performed in parallel (e.g., such that the operation is repeated according to (number of frames) * (number of color components) * (number of coding units per frame)). It should also be noted that different color components can have different numbers of coding units per frame, for example, the luma (e.g., Y) component can be processed at a higher resolution than a set of chroma (e.g., U or V) components when human vision can detect a greater change in illumination than a change in color.

[0106] Therefore, as shown and described above, the output of the decoding process is (optional) base reconstruction, and the original signal reconstruction at a higher level. This example is particularly suitable for creating encoded and decoded videos at different frame resolutions. For example, the input signal can be an HD video signal including a frame at a 1920x1080 resolution. In some cases, both base reconstruction and layer 2 reconstruction can be used by a display device. For example, in the case of network services, layer 2 streams may be more disturbed than layer 1 streams and base streams (because it may include up to 4 times the amount of data, where downsampling divides the size by 2 in each direction). In this case, when business occurs, the display device can recover to display the base reconstruction, and the layer 2 stream is interrupted (for example, when the layer 2 reconstruction is unavailable), and then recover to display the layer 2 reconstruction when network conditions improve. Similar methods can be applied when the decoding device is subject to resource constraints, such as the set-top box performing system updates may have an operating base decoder to output the base reconstruction, but may not have the processing capacity to calculate the layer 2 reconstruction.

[0107] The encoding device also enables a video distributor to distribute video to a set of heterogeneous devices; those devices with only a base decoder view the base reconstruction, while those devices with an enhancement layer can view the higher quality layer 2 reconstruction. In the contrast case, two complete video streams at separate resolutions are required to serve the two sets of devices. Since the layer 2 and layer 1 enhancement streams encode residual data, the layer 2 and layer 1 enhancement streams can be encoded more efficiently, for example, the distribution of residual data is generally mostly around 0 (i.e., there is no difference) and generally takes a small range of values ​​around 0. This is especially true after quantization. In contrast, the complete video streams at different resolutions will have different distributions with non-zero means or medians, which require higher bit rates for transmission to the decoder. In the examples described herein, the residual is encoded by an encoding pipeline. This may include transform, quantization, and entropy encoding operations. It may also include residual scalability, weighting, and filtering. The residual is then transmitted to the decoder, for example as L-1 and L-2 enhancement streams, which may be combined with the base stream as a hybrid stream (or transmitted separately). In one case, a bit rate is set for a mixed data stream comprising a base stream and two enhancement streams, and then different adaptive bit rates are applied to individual streams based on the data being processed to meet the set bit rate (e.g., high quality video perceived with low levels of artifacts can be constructed by adaptively assigning bit rates to different individual streams (even at a frame-by-frame level) so that constrained data can be used by the most perceptually impactful individual streams, which can change as image data changes).

[0108] The group of residuals as described herein can be considered as sparse data, e.g., in many cases there is no difference for a given pixel or region, and the resulting residual value is zero. When looking at the distribution of the residuals, a lot of probability mass is assigned to small residual values ​​located close to zero, e.g., for certain video values ​​of -2, -1, 0, 1, 2, etc., which occur most frequently. In some cases, the distribution of residual values ​​is symmetric or approximately symmetric about 0. In some test video cases, it was found that the distribution of residual values ​​was shaped similar to a logarithmic or exponential distribution about 0 (e.g., symmetrically or approximately symmetrically). The exact distribution of residual values ​​may depend on the content of the input video stream.

[0109] The residual can itself be processed as a two-dimensional image, such as a delta image of the difference. In this way, it can be seen that the sparsity of the data is related to features such as "dots", small "lines", "edges", "corners", etc. that are visible in the residual image. It has been found that these features are usually not completely correlated (e.g., spatially and / or temporally). The feature has characteristics that are different from the characteristics of the image data from which it is derived (e.g., the pixel characteristics of the original video signal).

[0110] Because the characteristics of the residual are different from the characteristics of the image data from which the residual is derived, it is generally not possible to apply standard encoding methods, such as those found in the traditional Moving Picture Experts Group (MPEG) encoding and decoding standards. For example, many contrasting schemes use large transforms (e.g., transforms of larger pixel areas in normal video frames). Due to the characteristics of the residual, such as described above, it would be extremely inefficient to use these contrasting large transforms on the residual image. For example, it would be very difficult to encode a small point in the residual image using a large block designed for an area of ​​the normal image.

[0111] Some of the examples described herein address these issues by using smaller and simpler transform kernels (e.g., 2x2 or 4x4 kernels as presented herein) instead. The transforms described herein can be applied using Hadamard matrices (e.g., 4x4 matrices for flattening 2x2 coding blocks, or 16x16 matrices for flattening 4x4 coding blocks). This moves in a different direction from the contrasting video coding methods. Applying these new methods to residual blocks generates compression efficiency. For example, some transforms generate uncorrelated transform coefficients (e.g., in space) that can be effectively compressed. Although correlations between transform coefficients can be exploited, such as for lines in residual images, these can result in coding complexity that is difficult to implement on traditional and low-resource devices, and often generates other complex artifacts that need to be corrected. Preprocessing the residual by setting certain residual values ​​to 0 (i.e., not forwarding these residual values ​​for processing) can provide a controllable and flexible way to manage bit rate and stream bandwidth, as well as resource usage.

[0112] Examples of signal processing methods

[0113] In an example described in more detail below. The first and second signals are processed. The second signal includes superimposed content. The first signal does not include the superimposed content. The first signal can be processed and output without the superimposed content. The second signal can be processed to generate residual data. The residual data can be combined with the first signal (and / or a processed version of the first signal) to generate a second signal with superimposed content. In this way, the superimposed content can be selectively available to one group of recipients but not to another group of recipients, but both groups of recipients use the first signal as the base signal.

[0114] Specific examples will now be described in the context of SDR and HDR content, it being understood that the examples described herein are applicable to other types of content.

[0115] For example, the overlay content may be a (SDR) advertisement. One group of consumers (e.g., those without a subscription) will see this overlay, while another group of consumers (e.g., premium users with a subscription) will not see this overlay.

[0116] Another example is regional and / or geographic content, such as content that differs in different countries. For example, a content provider may wish to encode a base layer, and then overlay a UK-specific overlay in the UK, and a Germany-specific overlay in Germany.

[0117] In another example, the overlay content includes (different) timer content. For example, there are multiple time zones in the United States. A broadcaster may use different overlays to display the local time in the time zone where the content is being consumed, using different overlays for different time zones.

[0118] Thus, in an example embodiment, obtaining the second signal (i.e., the signal including the overlay area) includes determining one or more of the following: attributes of the user (e.g., the user's subscription level, the user's location, the user's time zone, the user's marketing preferences, etc.); and attributes of an entity associated with the method of signal processing (e.g., an entity performing the method, an entity outputting encoded residual data, an entity transmitting the signal, an entity responsible for creating the content on which the signal is based, a distributor of the signal, a broadcaster associated with the signal, etc.) and selecting the second signal based on the determination. In this way, an overlay associated with one or more of the user or the entity associated with the method of signal processing can be selected. In such an example, the user can be considered to be the final consumer of the signal, for example, in an embodiment where the signal is a video signal, the user is considered to be a viewer of the video.

[0119] HDR content may be referred to as an HDR overlay. In addition, an HDR overlay may be referred to as an HDR image.

[0120] The expression 'superimposed on' (or similar) is used to describe content 'superimposed on top of other content'. For example, the expression could be used to describe a first image being superimposed on a second image. This could be equivalent to the second image being superimposed on the first image. In these examples, the first image would be in the foreground, which is why the language 'superimposed on' is used.

[0121] The terms 'portion' or 'region' may be used to describe a first image that is a portion of another image. In this sense, portion means that the first image is smaller than the entire other image. In other words, portion means a percentage that is less than 100%.

[0122] According to the understood definition of the term 'reproduction', the term 'reproduction' has the meaning of version.

[0123] The embodiments described herein provide various advantages.

[0124] For example, as mentioned above, not all video content is produced in HDR. For example, for various reasons, a video provider (e.g., a content provider such as a streaming service) may not want to provide all given content as HDR. One reason may be concerns about end device compatibility. Another reason may be that there may not be enough bandwidth to carry HDR for all content.

[0125] However, it would be very advantageous if, for example, an HDR overlay could be overlaid onto SDR content. Embodiments of the present disclosure provide a solution to this problem.

[0126] Accordingly, embodiments of the present disclosure provide several advantages.

[0127] For example, the overlay may be a static (e.g., an image or logo) ad or a dynamic ad (e.g., a video ad, GIF, or the like). There are many advantages to using HDR for such an overlay. For example, the ad will stand out relative to the SDR content and therefore attract the viewer's attention in a more efficient manner. Another advantage is that an HDR ad may have a wider color gamut and therefore be able to mimic a logo more accurately than an SDR ad due to the increased color gamut.

[0128] In a further example, the overlay may be a timer (e.g., a countdown or ticking clock). Having such an overlay in HDR would advantageously mean that the timer stands out compared to the main SDR content of the video (due to the wider brightness and / or color gamut).

[0129] In another example, the HDR overlay relates to news content, such as a news film or news channel.

[0130] The examples described herein can generate residuals indicating the difference between the SDR content and the HDR overlay. These residuals are then encoded by a hierarchical encoder, such as using the residual processing described in the LCEVC standard, to obtain an enhancement layer. The SDR content is encoded by a base encoder. The decoder is configured to decode the base encoded SDR content and the hierarchical encoded enhancement layer. The decoder is configured to combine the decoded SDR content and the decoded enhancement layer to generate an image including the SDR content overlaid with the HDR content.

[0131] The residual may be generated in a number of different ways. The residual may correspond to a representation of an HDR overlay. Such a representation may be obtained by processing only the HDR overlay, for example by quantizing the HDR overlay.

[0132] The residual can be generated by processing the reproduction of HDR content in conjunction with the reproduction of SDR content. The SDR content can be processed to obtain the reproduction of the SDR content, for example, the original SDR image can be encoded and decoded and / or reduced and enlarged to obtain the reproduction of the SDR content. The difference between the reproduction of the SDR content and the reproduction of the HDR content can be calculated. The reproduction of the HDR content can be obtained in a variety of ways. For example, the reproduction of the HDR content can be obtained by superimposing the HDR content on a representation of the SDR content. The representation can be the same as the original signal of the SDR content used to obtain the reproduction of the SDR content. Alternatively, the representation can be a higher resolution version of the original signal of the SDR content used to obtain the reproduction of the SDR content. Regardless of how it is obtained, the reproduction of the HDR content is compared with the reproduction of the SDR content to generate a residual.

[0133] The residual is encoded using a hierarchical based coding method such as LCEVC. Details of how LCEVC handles the residual can be found in WO 2020 / 188273.

[0134] Figure 8 An example signal processing method is presented in the form of an example encoding process 800. The example encoding process 800 may encode an image including HDR content (eg, an image) superimposed on an SDR image.

[0135] A first signal 801 is processed. In this example, the first signal 801 includes a series of component signals 802. The first signal 801 includes a non-overlapping region 803. The term "non-overlapping region" is used to refer to a region that is not an overlapping region. A signal having a non-overlapping region may or may not have an overlapping region.

[0136] The first signal 801 is processed by at least sending the first signal 801 to an encoder 804. In this example, the encoder 804 includes a base code encoding module and is therefore generally referred to as a "base encoder". The base encoder 804 encodes the first signal 801 to generate an encoded signal 805. In this example, the encoded signal 805 includes an encoded base bit stream 805.

[0137] In this example, the first signal 801 is downsampled by a downsampler 806 before being sent to the base encoder 804. The output of the downsampler 806 is a downsampled signal 807. However, in other examples, the first signal 801 is not downsampled before being encoded by the base encoder 804.

[0138] Therefore, it should be understood that processing the first signal 801 by at least sending the first signal to the base encoder 804 covers (i) sending the first signal 801 itself to the base encoder 804 to be encoded and (ii) processing the first signal 801 and sending the processed first signal (e.g., downsampled signal 807) to the base encoder 804 to be encoded. Such processing may include downsampling the first signal 801.

[0139] A second signal 808 is acquired. In this example, the second signal 808 includes a series of component signals 809. The second signal 808 can be acquired in a variety of different ways. For example, the second signal 808 can be received, retrieved from a memory, generated, or can be acquired in another way.

[0140] The second signal 808 includes an overlay region 810. The second signal 808 may include multiple overlay regions 810. The overlay regions 810 of the second signal 808 include overlay content. The overlay content may take a variety of different forms. In this example, the first signal 801 does not include the overlay content.

[0141] Overlay regions such as overlay region 810 of second signal 808 may be considered regions of interest (RoIs). For example, the RoI may include content that is of particular interest to an end consumer (e.g., a viewer) of second signal 808. For example, the size of the RoI may be at most half the size of second signal 808.

[0142] In some examples, the second signal 808 is generated by applying the superposition signal to the input signal. The superposition signal includes a superposition region 810. The applying may include replacing values ​​of signal elements of signal elements of the input signal in the superposition region 810 with values ​​of signal elements of the signal elements of the superposition signal.

[0143] In some examples, the generation of the second signal 808 includes identifying a ROI in the input signal. When the input signal is SDR, the identified ROI may be encoded using HDR while the rest is maintained in SDR.

[0144] The second signal 808 includes a non-overlapping region 811. The second signal 808 may include a plurality of non-overlapping regions 811.

[0145] In this example, the output of the base encoder 804 (i.e., the encoded base bitstream 805) is processed. In this example, such processing of the encoded base bitstream 805 includes decoding the encoded base bitstream 805 to generate a decoded version 812 of the encoded base bitstream 805. In this example, the base encoder 804 outputs both the encoded base bitstream 805 and the decoded version 812 of the encoded base bitstream 805. For example, the base encoder 804 may include a decoding function to decode the encoded base bitstream 805. However, in other examples, a separate decoder (not shown) is used to decode the encoded base bitstream 805. The decoded version 812 of the encoded base bitstream 805 is generally referred to as the decoded base bitstream 812.

[0146] In this example, the decoded base bitstream 812 is processed. In this example, such processing of the decoded base bitstream 812 includes upsampling the decoded base bitstream 812 by an upsampler 813. The output of the upsampler 813 is an upsampled signal 814. However, in other examples, such as if the first signal 801 has not been downsampled by the downsampler 806, the decoded base bitstream 812 is not upsampled in this manner.

[0147] At least the second signal 808 is processed to generate residual data 815. In this example, the second signal 808 and the third signal 816 are processed by a residual generator 817 to generate residual data 815. The third signal 816 may be a reproduction of the first signal 801, i.e., the first signal 801 or a signal derived based on the first signal 801. In this example, the third signal 816 is based on the decoded base bitstream 812. The third signal 816 may include the decoded base bitstream 812 or may include the processed decoded base bitstream 812, i.e., the upsampled signal 814. In this example, the third signal 814 includes the processed decoded base bitstream 812, i.e., the upsampled signal 814, which is an upsampled version of the decoded base bitstream 812. In this example, the processing of the second signal 808 and the third signal 816 (i.e., the upsampled signal 814) includes calculating the difference between the second signal 808 and the third signal 816 (i.e., the upsampled signal 814) by the residual generator 817.

[0148] In this example, the residual generator 817 calculates the difference between the values ​​of signal elements in at least the superposition region 810 of the second signal 808 and the values ​​of corresponding signal elements in the third signal 816 .

[0149] The corresponding signal element in the third signal 816 may or may not be located in the superposition region of the third signal 816. In particular, in this example, the third signal 816 (and the first signal 801 based on which the third signal 816 is derived) does not include the superposition region. Thus, the corresponding signal element in the third signal 816 is not in the superposition region of the third signal 816.

[0150] The residual generator 817 may or may not calculate the difference between the values ​​of signal elements outside the overlap region 810 of the second signal 808 and the values ​​of corresponding signal elements in the third signal 816 .

[0151] In this example, the residual generator 817 calculates a difference by subtracting the value of a signal element in the third signal 816 (ie, the upsampled signal 814 ) from the value of a corresponding signal element in the second signal 808 .

[0152] The residual data 815 may indicate the superimposed content of the superimposed region 810 of the second signal 808. For example, when the residual generator 817 calculates the difference between the value of the signal element in the superimposed region 810 and the value of the corresponding signal element in the third signal 816, the residual data 815 is affected by the superimposed content of the superimposed region 810 of the second signal 808.

[0153] In some cases, the residual value derived based on the value of the signal element in the superposition region 810 of the second signal 808 is significantly larger than the residual value derived based on the value of the signal element in the non-superposition region 811 of the second signal 808. The (larger) value of the residual derived based on the value of the signal element in the superposition region 810 of the second signal 808 may indicate the superposition content of the superposition region 810 of the second signal 808.

[0154] In this example, the residual data 815 generated by the residual generator 817 is sent to the encoder 818 to be encoded. In this example, the encoder 818 is different from the base encoder 804. In this example, the encoder 818 includes an LCEVC residual encoding module. In this example, the encoder 818 is at the enhancement level and is therefore generally referred to as an "enhanced encoder". The enhanced encoder 818 encodes the residual data 815 output by the residual generator 817 to generate encoded residual data. Therefore, in this example, the first codec is used to generate the encoded base bitstream 805, and the different second codec is used to generate the encoded residual data. In this example, the encoded residual data is included in the encoded enhanced bitstream 819.

[0155] In this example, the example encoding process 800 outputs at least encoded residual data. For example, the example encoding process 800 may also output an encoded base bitstream 805.

[0156] The example encoding process 800 may include Figure 8 For example, the residual data 815 output by the residual generator 817 can undergo a transform and / or quantization. In some examples, the encoding process 800 can correct for any encoder-decoder errors introduced by the base encoder 804 when creating the encoded base bitstream and / or introduced by a decoder (whether or not part of the base encoder 804) when decoding the encoded base bitstream 805.

[0157] Although in this example, the example encoding process 800 includes the base encoder 804 encoding the first signal 801 to generate the encoded signal 805 , in other examples, the example encoding process 800 sends the first signal 801 to the base encoder to be encoded, but does not perform encoding of the first signal 801 .

[0158] Additionally, while in this example the example encoding process 800 includes decoding an encoded base bitstream 805 output by the base encoder 804, in other examples the example encoding process 800 receives a decoded base bitstream 812 from a decoder (whether or not part of the base encoder 804) but does not perform decoding of the encoded base bitstream 805.

[0159] Furthermore, as described above, although in this example, the example encoding process 800 includes downsampling and upsampling, in other examples, the example encoding process 800 does not perform such downsampling and / or upsampling.

[0160] In this example, the first signal 801 includes content having a first dynamic range. In this example, the second signal 808 includes content having a second dynamic range. The content having the second dynamic range may be included in an overlapped region 810 of the second signal 808 and / or in a non-overlapped region 811 of the second signal 808. In this example, the second dynamic range is higher than the first dynamic range.

[0161] In this example, the overlay content (included in the overlay region 810 of the second signal 808) includes or consists of HDR content. In this example, the first signal 801 includes or consists of SDR content.

[0162] In some examples, the first codec (associated with the base encoder 804) will not be able to encode the overlay region 810 of the second signal 808. For example, the first codec (associated with the base encoder 804) may not be able to encode HDR content. However, the second codec (associated with the enhanced encoder 818) may be able to encode HDR content. The first codec (associated with the base encoder 804) is able to encode SDR content.

[0163] As a specific example, the ad server can encode ads using LCEVC in HDR.The overlay content may include HDR ads.

[0164] The base encoder 804 may use a base codec with SDR capabilities (e.g., 8-bit) that may be widely supported. The enhanced encoder 812 may use a base codec with HDR capabilities (e.g., 10-bit) that may not be widely supported. In practice, HDR content may be added via LCEVC.

[0165] In some examples, the overlay content included in the overlay region 810 of the second signal 808 is used to provide content to one group of one or more consumers (e.g., viewers) but not to another group of one or more consumers (e.g., viewers). Thus, differences in content type may be relevant to different target consumers (whether or not SDR and HDR differences also apply).

[0166] In some specific examples, the overlay content can actually be used to replace at least some instances of one or more colors in the first signal 801 with one or more other colors. For example, the first signal 801 may include one or more colors that some consumers (e.g., viewers) find difficult to visually distinguish from one or more other colors. In fact, the overlay content can replace such colors with other colors that such consumers (e.g., viewers) can more easily distinguish. This example can also involve detection in order to detect which parts of the signal are to be changed or superimposed. In contrast, the other examples described above (e.g., involving location-specific overlay content, local time, etc.) may be less complex because the overlay can be in a static position in the signal and may not involve any analysis of the signal before the overlay is selected.

[0167] In this example, the second signal 810 has a given quality level. In this particular example, the given quality level is a higher (or "enhanced") quality level. In this example, the given quality level of the second signal 810 is maintained during the processing of the second signal 810. Such processing can be performed by the residual generator 817 or in other ways. In this example, the given quality level of the second signal 810 is maintained at least when the second signal 810 is not downsampled to a lower quality level.

[0168] In some examples, another signal is acquired. The other signal includes another superposition area, and the other superposition area includes another superposition content. The other superposition content is different from the superposition content included in the superposition area 810 of the second signal 808. At least another signal is processed to generate another residual data. The other residual data is encoded to generate another encoded residual data. The other encoded residual data is output in association with the encoded basic bit stream 805. In such an example, different superposition contents can be provided in association with the same encoded basic bit stream 805. This can enable personalized superposition content to be provided.

[0169] In this way, the overlay content and / or another overlay content may be selected based on one or more overlay content selection criteria. Examples of such overlay content selection criteria include, but are not limited to, location (e.g., of a consumer), time zone (e.g., of a consumer), subscription level (e.g., of a consumer). It is emphasized again that while the example encoding process 800 is described for an image including HDR content (e.g., an image) being overlaid onto an SDR image, this is merely one example of overlay content. In other examples, the overlay content may include non-HDR content.

[0170] In some examples, the further encoded residual data is output together with the encoded residual data.As such, a decoder receiving the encoded residual data and the further encoded residual data may be able to select between the encoded residual data and the further encoded residual data, for example based on one or more selection criteria.

[0171] For example, multiple overlays may be embedded or otherwise provided. The decoder may use the user data information to select data to use based on region, user, etc. This may enable targeted content and / or regional content.

[0172] In terms of decoder-side processing, a decoded signal (e.g., a decoded version of the encoded base bitstream 805) is processed (e.g., upsampled) to generate a signal (e.g., corresponding to a third signal 816). The encoded residual data (e.g., included in the encoded enhancement bitstream 819) is processed by at least decoding the encoded residual data to generate decoded residual data (e.g., corresponding to residual data 815). At least the generated signal and the decoded residual data are processed to generate another signal. Such processing may include combining the generated signal and the decoded residual data. Another generated signal may correspond to the above-mentioned second signal 808. In particular, another generated signal includes an overlay region (corresponding to the above-mentioned overlay region 810), which includes overlay content. The generated signal does not include overlay content. The decoded residual data may indicate the overlay content of the overlay region of another generated signal.

[0173] In principle, the overlay content can be output as a separate overlay video stream (i.e., separate from the base video bitstream 805), where the separate overlay video stream includes transparency outside the overlay area 810. However, this may involve two complete decodes; one for the base video bitstream 805 and another for the overlay video stream. In addition, the two decodes increase the decoding process power requirements. In addition, most conventional codecs do not support transparency.

[0174] Overlay graphics can be sent as graphics drawn by a graphics engine. However, this again uses more processing power. Furthermore, in general, real-time graphics do not have the same visual quality as graphics generated by a (post) production system.

[0175] The examples described herein using LCEVC can achieve high quality, energy efficient, and legacy codec compatible overlays. The examples described herein can also use VC-6, which can also provide such effects if used with transparency.

[0176] According to the examples described herein using LCEVC, two input sources are used instead of one. See, for example Figure 6 , which uses one input video 600 instead of two. According to an example, one input source is the source video and the other input source is the overlay video. Using LCEVC, the residual can be calculated based on the combination of the source video and the overlay video, rather than calculating the residual based on the source video. As described herein, the residue can be additive or alternative. One or more signaling bits can be used to signal whether a given residual is additive or alternative. The embedded information can be used to signal that the intention of the enhancement (using the residual) is alternative rather than additive. Only a decoder that understands the embedded information can fully decode the video. Others are still able to decode some videos. In some cases, the alternative overlay residual can be sent as embedded signaling, while the additive residual is signaled in a non-embedded manner.

[0177] A specific example of how the example encoding process 800 may be used in relation to SDR and HDR content will now be provided. The example encoding process 800 uses an LCEVC residual encoding module 818 and a base codec encoding module 804. The example encoding process 800 receives an SDR video signal 801. The SDR video signal 801 includes a plurality of SDR images. The plurality of SDR images include a first SDR image 802. The SDR image 802 may be referred to as an image 802 of the SDR video signal 801. The example encoding process 800 also receives an HDR video signal 808. The HDR video signal 808 includes a plurality of HDR images. One of the plurality of HDR images 808 is an HDR image 809. The HDR image 809 may be referred to as an image 809 of the HDR video signal 808. The HDR image 809 may be referred to as 'HDR content'. The HDR image 809 may be received alone or as a combined image, wherein the combined image includes an image of the HDR video signal 808 superimposed on a version of the image of the SDR video signal 801. The version of the image of SDR video signal 801 may be a higher resolution version of the image of SDR video signal 801 .

[0178] The base codec encoding module 804 receives the SDR image 802. The base codec encoding module 804 encodes the SDR image 802 using the base codec to generate an encoded version 805 of the SDR image 802. Examples of the base codec encoding module 804 include modules configured to encode the SDR image 802 according to known codecs including VVC, EVC, AVC, HEVC. The base codec encoding module 804 is further configured to output the encoded SDR image 805. The encoded version of the output SDR image 805 may be referred to as a base bitstream 805. The base bitstream 805 may be decoded by a decoder to obtain a decoded version of the SDR image 802.

[0179] The base codec encoding module 804 is further configured to generate and output a decoded version 812 of the encoded version 805 of the image 802 of the SDR video signal 801. The decoded version 812 is output to a residual calculation module 817. The residual calculation module 817 receives the decoded version 812 (or a reproduction derived therefrom). The residual calculation module 817 also receives the HDR image 809. In particular, the residual calculation module 817 receives the HDR image 809 in the form of a reproduction of the image 809 of the HDR video signal 808, which is superimposed on a portion of the image 802 of the SDR image 801. In this example, the residual calculation module 817 is configured to receive the HDR image 809 in the form of a reproduction of the combined image. In an example, the reproduction of an image may refer to a quantized version of an image. Other types of image reproduction may be used.

[0180] The residual calculation module 817 generates a residual 815 based on the version of the combined image and the decoded reproduction 814. In this example, the residual calculation module 817 is configured to generate the residual 815 by calculating the difference between the decoded reproduction 814 and the reproduction of the combined image, wherein the combined image includes an image 809 of the HDR video signal 808 superimposed on a portion of the image of the version of the SDR image. In this way, the residual calculation module 817 generates a residual 815 that takes into account artifacts related to the base codec and further takes into account the HDR image 809. Therefore, by generating the residual 815 in this way, two advantages can be obtained through a single method.

[0181] The generated residual 815 is then processed by the LCEVC residual coding module 818. Details on how the LCEVC residual code in module 818 processes the residual can be found in the 'LCEVC standard', ISO / IEC 23094-2 (draft first published in January 2020), the contents of which are incorporated herein by reference. In general, such methods of processing residuals advantageously process the residuals in an efficient manner. Therefore, the residual 815 of the present disclosure can be processed using known methods of processing residuals as described in the LCEVC standard to obtain similar positive results.

[0182] The result of the LCEVC residual encoding module 818 is output as an enhancement bitstream 819. The enhancement bitstream 819 is considered an enhancement layer that can be used by a decoder in combination with the base bitstream 805 to generate a decoded SDR image with the (n overlay) portion being HDR.

[0183] Fig. 9 Another example signal processing method is presented in the form of an example encoding process 900. The example encoding process 900 may encode an image including HDR content (eg, an image) superimposed on an SDR image.

[0184] Fig. 9 The example encoding process 900 shown is similar to Figure 8 An example encoding process 800 is shown. Similar elements are shown using the same reference numerals, but in increments of '100'.

[0185] In this example, the first signal 901 includes an overlapped region 920. In contrast, Figure 8 The depicted first signal 801 does not include an overlay region. In this example, an overlay region 920 of the first signal 901 includes overlay content. In this example, in addition to including the overlay region 920, the first signal 901 also includes a non-overlay region 903.

[0186] In this example, the superposition region 920 of the first signal 901 may include SDR superposition content and the superposition region 910 of the second signal 908 may include HDR superposition content. Upsampling 913 may involve tone mapping, where the tone mapping maps an SDR-based color set (e.g., 8 bits) to an HDR-based color set (e.g., 10 bits). This enables the residual generator 917 to calculate the difference between the values ​​of the signal elements in the superposition regions 920, 910 of the first and second signals 901, 908, respectively.

[0187] A specific example of how the example encoding process 900 may be used in connection with SDR and HDR content will now be provided.

[0188] Example encoding process 900 receives SDR video signal 901. SDR video signal 901 includes multiple SDR images. Multiple SDR images include combined SDR image 902. SDR image 902 can be referred to as image 902 of SDR video signal 901. Combined SDR image 902 includes an SDR image superimposed with SDR content 920. SDR content 920 corresponds to HDR content 910, which will be described in further detail below. SDR content 920 is an SDR version of HDR content 910. HDR content 910 and SDR content 920 can be considered as superimposed.

[0189] The example encoding process 900 receives an HDR video signal 908. The HDR video signal 908 includes a plurality of images having at least a portion that is HDR. One of the plurality of HDR images is an HDR image 909. The HDR image 909 may be referred to as an image 909 of the HDR video signal. The HDR image 909 may be referred to as 'HDR content'. The HDR image 909 may be received by the example encoding process 900 alone or as a combined image, wherein the combined image includes the image 909 of the HDR video signal superimposed on the image 902 version of the SDR video signal 901. The image 902 version of the SDR video signal may be a higher resolution version of the image 902 of the SDR video signal 901. Therefore, the combined image corresponds to the SDR image 902. However, the combined image includes a portion that is HDR (i.e., 'HDR content' 910), while the SDR image 902 includes the SDR content 920. In addition, the combined image may be at a higher resolution than the SDR image 902.

[0190] The base codec encoding module 904 receives the SDR image 902. The base codec encoding module 904 encodes the SDR image 902 using a base codec to generate an encoded version 905 of the SDR image 902. Examples of the base codec encoding module 904 include modules configured to encode the SDR image 902 according to known codecs including VVC, EVC, AVC, HEVC. The base codec encoding module 904 is further configured to output the encoded SDR image 905. The output encoded version of the SDR image 902 may be referred to as a base bitstream 905. The base bitstream may be decoded by a decoder to obtain a decoded reproduction of the SDR image 902.

[0191] The base codec encoding module 904 generates and outputs a decoded rendition 912 of the encoded version 905 of the SDR image 902 .

[0192] The decoded version 912 is output to a residual calculation module 917. The residual calculation module 917 receives the decoded rendering 912 (or, in this example, an upsampled version of the decoded rendering 912). The residual calculation module 917 further receives the HDR image 909. In this example, the residual calculation module 917 receives the HDR image 909 in a rendered form of the combined image.

[0193] The residual calculation module 917 generates a residual 915 based on the version of the combined image and the decoded reproduction 912. In this example, the residual calculation module 917 generates the residual 915 by calculating the difference between the decoded reproduction 912 (or more specifically, the upsampled version 914 thereof) and the version of the combined image. In this way, the residual calculation module 917 generates a residual 915 that takes into account artifacts related to the base codec and also takes into account the HDR content 910.

[0194] Fig.10 Another example signal processing method is presented in the form of an example encoding process 1000. The example encoding process 1000 may encode an image including HDR content (eg, an image) superimposed on an SDR image.

[0195] Fig.10 The example encoding process 1000 shown is similar to Figure 8 An example encoding process 800 is shown. Similar elements are shown using the same reference numerals, but in increments of '200'.

[0196] In this example, as in the above example, the second signal 1008 includes a non-overlapping region 1011. However, in this example, the non-overlapping region 1011 of the second signal 1008 is a blank (non-overlapping) region. In this example, the blank region includes signal elements whose values ​​are all zero.

[0197] In this example, as in the above example, the second signal 1008 and the third signal 1016 are processed to generate residual data. However, in this example, the combiner 1021 combines the values ​​of the signal elements in at least the overlap region 1010 of the second signal 1008 with the values ​​of the corresponding signal elements in the third signal 1016 to generate a combined signal 1021. In this particular example, the combination includes adding the values ​​together.

[0198] Additionally, in this example, a difference is calculated between the values ​​of signal elements in at least the superposition region of the combined signal 1021 and the values ​​of corresponding signal elements in the third signal 1016. The superposition region of the combined signal 1021 corresponds to the superposition region 1010 of the second signal 1008. The superposition regions may correspond to each other because they are at the same position in their respective signals.

[0199] One particular feature of the example encoding process 1000 will now be described in more detail.

[0200] The combiner 1021 receives the second signal 1008 and the third signal 1016 which may be represented as 'A' and 'B' respectively, and outputs a combined signal 1022 which may be represented as 'C'. Therefore, the combination by the combiner 1021 may be mathematically represented as: A+B=C.

[0201] The residual generator 1017 receives the combined signal 1022 and the third signal 1016, which may be denoted as 'C' and 'B' respectively, and outputs a residual, which is the difference between the combined signal 1022 and the third signal 1016 and may be denoted as 'R'. The subtraction performed by the residual generator 1017 may be mathematically represented as: C - B = R.

[0202] Substituting A+B=C into C–B=R we obtain (A+B)–B=R, or A=R.

[0203] Thus, the difference calculations combined by combiner 1021 and performed by residual generator 1017 may be viewed as redundant, or even inefficient, since they actually cancel the third signal 1016 (denoted 'B' above). In particular, in principle, enhancement encoder 1018 may simply receive second signal 1008 directly.

[0204] However, the enhancement encoder 1018 (and more generally the system in which it is located) may be designed and optimized to receive a signal calculated as a difference between two other signals. The example encoding process 1000 may therefore maintain compatibility with such an enhancement encoder 1018.

[0205] Fig.11Another example signal processing method is presented in the form of an example encoding process 1100. The example encoding process 1100 may encode an image including HDR content (eg, an image) superimposed on an SDR image.

[0206] Fig.11 The example encoding process 1100 shown is similar to Figure 8 An example encoding process 800 is shown. Similar elements are shown using the same reference numerals, but in increments of '300'.

[0207] In this example, the third signal 1116 includes a reference signal 1123. In this example, the reference signal 1123 is independent of the first signal 1101. In this example, the reference signal 1123 includes a non-overlapping region 1124. In this particular example, the reference signal 1123 is a blank signal.

[0208] Again, the difference calculation performed by the residual generator 1117 may be viewed as redundant, or even inefficient, since it does not actually change the second signal 1108 .

[0209] However, the enhancement encoder 1118 (and more generally the system in which it is located) may be designed and optimized to receive a signal calculated as a difference between two other signals. The example encoding process 1100 may therefore maintain compatibility with such an enhancement encoder 1118.

[0210] In this example, the residual data 1115 output by the residual generator 1117 includes alternative residual data. The alternative residual data includes signal element values ​​to be substituted to replace corresponding signal element values ​​in another signal. The alternative residual data is different from additive residual data, which includes signal element values ​​to be added to corresponding signal element values ​​in another signal.

[0211] In this example, the decoder may decode (and possibly upsample) the base bitstream 1105. The decoder may also decode the enhancement bitstream 1119. The decoder may then replace corresponding signal element values ​​of the decoded (and possibly upsampled) base bitstream 1105 with signal element values ​​of the alternative residual data.

[0212] Fig.12 Another example signal processing method is presented in the form of an example encoding process 1200 .

[0213] Fig.12 The example encoding process 1200 shown is similar to Figure 8 An example encoding process 800 is shown. Similar elements are shown using the same reference numerals, but in increments of '400'.

[0214] However, the example encoding process 800 does not include upsampling or downsampling. The first signal 1210 is processed by sending at least the first signal 1210 to the base encoder 1204. In this example, the first signal 1210 is not downsampled before being encoded.

[0215] A second signal 1208 is acquired. The second signal 1208 includes an overlay region 1210 that includes overlay content. At least the second signal 1208 is processed to generate residual data 1215. In this example, a third signal 1216 is processed together with the second signal 1208 to generate residual data 1215. In this example, the third signal 1216 includes a decoded base bitstream 1212 instead of an upsampled version of the decoded base bitstream 1212. The residual data 1215 is encoded to generate encoded residual data. At least the encoded residual data is output.

[0216] In this example, the first signal 1201 has a given quality level. The given quality level of the first signal 1201 is maintained during processing of the first signal 1201. For example, in this example, the first signal 1201 is neither downsampled nor upsampled.

[0217] Therefore, LCEVC can be used for overlay even in the absence of scaling between the base layer and the enhancement layer.

[0218] Fig.13 Another example signal processing method is presented in the form of an example encoding process 1300 .

[0219] Fig.13 The example encoding process 1300 shown is similar to Figure 8 An example encoding process 800 is shown. Similar elements are shown using the same reference numerals, but in increments of '500'.

[0220] In this example, the residual generator 1317 uses mask data 1325 when generating the residual data 1315. In this example, the mask data 1325 identifies the superposition region 1310 of the second signal 1308. In this example, the residual generator 1317 uses the mask data 1325 so that (i) the difference between the value of the signal element in the non-superposition region 1311 of the second signal 1308 and the value of the corresponding signal element of the third signal 1316 is calculated, and (ii) the value of the signal element in the superposition region 1310 of the second signal 1308 replaces (or "replaces") the value of the corresponding signal element of the third signal 1316. Therefore, the second signal 1308 can include a portion intended to be used as a substitute (the superposition region 1310) and a portion intended to be used as an additive (the non-superposition region 1311).

[0221] Fig.14Another example signal processing method is presented in the form of an example encoding process 1400 .

[0222] Fig.14 The example encoding process 1400 shown is similar to Figure 8 An example encoding process 800 is shown. Similar elements are shown using the same reference numerals, but in increments of '600'.

[0223] In this example, the first signal 1401 includes a blank area 1426. The blank area 1426 can be considered as an overlap area. In this example, the blank area 1426 of the first signal 1401 corresponds to the overlap area 1410 of the second signal 1408, at least because they are at the same position in the first signal 1401 and the second signal 1408, respectively.

[0224] In this example, the residual generator 1417 calculates (i) the difference between the value of the signal element in the non-overlapping region 1411 of the second signal 1408 and the value of the corresponding signal element of the third signal 1416 and (ii) the difference between the value of the signal element in the overlapping region 1410 of the second signal 1408 and the value of the corresponding signal element in the blank region 1426 of the first signal 1401. Since the values ​​of the signal elements in the blank region 1426 of the first signal 1401 are all zero, the residual generator 1417 actually replaces the (zero) values ​​of the signal elements in the blank region 1426 of the first signal 1401 with the values ​​of the corresponding signal elements in the overlapping region 1410 of the second signal 1408.

[0225] However, in this example, the first signal 1401 includes a blank area 1426. This may negatively impact the viewing experience of viewers who do not have the enhanced bitstream 1419. This may or may not be desirable.

[0226] The above examples provide particularly effective applications of LCEVC. They can enable overlaying of stock, advertising or other content onto other video content. The overlay content can be encoded differently than the main video. The residual portion of LCEVC can be used to encode content in HDR. The main portion of the video may be in SDR, leaving only the overlay content in HDR.

[0227] Other changes

[0228]

[0066] Certain other variations of the examples described herein will now be described.

[0229] In some specific examples above, the overlay region of the second signal includes HDR overlay content, and the first signal includes SDR content. In such examples, the residual data enables the HDR content to be transmitted and used. Alternatively, the encoder-decoder signaling can signal that a given SDR content (e.g., overlay content) will use HDR.

[0230] In some specific examples above, the decoder is capable of producing an output image with both SDR and HDR content. In such examples, the image can be displayed by a display device that can display both SDR and HDR content together. In other examples, the output image may include only SDR or only HDR content. This may be effective in the case where the display device cannot display both SDR and HDR content at the same time, but can display one or both of SDR and HDR content separately.

[0231] Example devices for implementing a decoder or encoder

[0232] refer to Fig.15 , a schematic block diagram of an example of device 1500 is shown.

[0233] Examples of apparatus 1500 include, but are not limited to, a mobile computer, a personal computer system, a wireless device, a base station, a telephone device, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, an application server, a storage device, a consumer electronic device such as a camera, a camcorder, a mobile device, a video game console, a handheld video game device, a peripheral device such as a switch, a modem, a router, a vehicle, etc., or generally any type of computing or electronic device.

[0234] In this example, device 1500 includes one or more processors 1501 configured to process information and / or instructions. One or more processors 1501 may include a central processing unit (CPU). One or more processors 1501 are coupled to bus 1511. Operations performed by one or more processors 1501 may be implemented by hardware and / or software. One or more processors 1501 may include multiple co-located processors or multiple fully remotely located processors.

[0235] In this example, the device 1501 includes a computer usable memory 1512 configured to store information and / or instructions for one or more processors 1501. The computer usable memory 1512 is coupled to the bus 1511. The computer usable memory may include one or more of a volatile memory and a non-volatile memory. The volatile memory may include a random access memory (RAM). The non-volatile memory may include a read-only memory (ROM).

[0236] In this example, the device 1500 includes one or more external data storage units 1580 configured to store information and / or instructions. The one or more data external storage units 1580 are coupled to the device 1500 via the I / O interface 1514. The one or more external data storage units 1580 may include, for example, magnetic or optical disks and magnetic disk drives or solid state drives (SSDs).

[0237] In this example, the device 1500 further includes one or more input / output (I / O) devices 1516 coupled via an I / O interface 1514. The device 1500 also includes at least one network interface 1590. Both the I / O interface 1514 and the network interface 1517 are coupled to the system bus 1511. The at least one network interface 1517 can enable the device 1500 to communicate via one or more data communication networks 1590. Examples of data communication networks include, but are not limited to, the Internet and a local area network (LAN). The one or more I / O devices 1516 can enable a user to provide input to the device 1500 via one or more input devices (not shown). The one or more I / O devices 1516 can enable information to be provided to the user via one or more output devices (not shown).

[0238] exist Fig.15 , a (signal) processor application 1540-1 is shown as loaded into memory 1512. This may be executed as a (signal) processor process 1540-2 to implement the methods described herein (e.g., to implement a suitable encoder or decoder). The device 1500 may also include additional features not shown for clarity, including an operating system and additional data processing modules. The (signal) processor process 1540-2 may be implemented by computer program code stored in a memory location within a computer-usable non-volatile memory, a computer-readable storage medium within one or more data storage units, and / or other tangible computer-readable storage media. Examples of tangible computer-readable storage media include, but are not limited to: optical media (e.g., CD-ROM, DVD-ROM, or Blu-ray), flash memory cards, floppy disks, or hard disks, or any other medium capable of storing computer-readable instructions such as firmware or microcode in at least one ROM or RAM or programmable ROM (PROM) chip or as an application specific integrated circuit (ASIC).

[0239] Thus, the device 1500 may include a data processing module that can be executed by one or more processors 1501. The data processing module can be configured to include instructions that implement at least some of the operations described herein. During operation, the one or more processors 1501 initiate, run, execute, interpret or otherwise perform instructions.

[0240] Although at least some aspects of the examples described herein with reference to the accompanying drawings include computer processes executed in a processing system or processor, the examples described herein also extend to computer programs, such as computer programs on or in a carrier, which are suitable for putting the examples into practice. The carrier can be any entity or device capable of carrying the program. It should be understood that the device 1500 may include more, less and / or more Fig.15 The components shown are different components. Device 1500 can be located at a single location or can be distributed across multiple locations. Such locations can be local or remote.

[0241] The techniques described herein may be implemented in software or hardware, or may be implemented using a combination of software and hardware. They may include configuring a device to implement and / or support any or all of the techniques described herein.

[0242] Various measures for encoding a superimposed video signal are described herein. The superimposed video signal includes an HDR video signal superimposed on a portion (e.g., less than 50%) of the SDR video signal. Various actions are performed for each image of the SDR video signal (and in an example, for the HDR video signal). A decoded reproduction of an encoded version of the image of the SDR video signal is received. An image of the HDR video signal is received. The image of the HDR video signal corresponds in time to the decoded reproduction (of the encoded version of the image of the SDR video signal). The image of the HDR video signal will be superimposed on the image of the SDR video signal. The residual is determined by calculating the difference between the following reproductions: (i) the decoded reproduction (of the encoded version of the image of the SDR video signal) and (ii) the reproduction of the image of the HDR video signal superimposed on a portion of the image of the SDR image. The residual is encoded to generate an enhancement layer. The enhancement layer is output.

[0243] The enhancement layer is configured to be processed by a decoder combining an encoded version of an image of the SDR video signal such that the decoder generates a superimposed image. The superimposed image comprises an image of the HDR video signal superimposed onto a portion of the image of the SDR video signal.

[0244] The superimposed video signal comprises the superimposed image and a plurality of further superimposed images derived from the further image of the SDR video signal and the further image of the HDR video signal.

[0245] This advantageously provides efficient encoding (due to the way LCEVC efficiently handles residuals) that corrects for imperfections of the base layer encoder while also enabling SDR video with HDR overlays.

[0246] Known approaches fail to solve both problems with a single approach (or provide both advantages in a single solution).

[0247] The image of the SDR video signal may include an SDR image superimposed with an SDR reproduction of the image of the HDR video signal.

[0248] The image of the HDR video signal may be received as a combined image. The combined image may comprise the image of the HDR video signal superimposed on a version of the image of the SDR video signal.

[0249] The reproduction of the image of the HDR video signal superimposed on the portion of the image of the SDR image may be generated by superimposing the image of the HDR video signal on a decoded reproduction (of an encoded version of the image of the SDR video signal).

[0250] The above described embodiments are to be understood as illustrative examples. Further embodiments are envisaged.

[0251] It should be understood that any feature described with respect to any embodiment may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other embodiment, or in combination with any combination of any other embodiment. In addition, equivalents and modifications not described above may also be adopted without departing from the scope of the invention as defined by the appended claims.

Claims

1. A method for signal processing, the method comprising: processing the first signal by sending at least the first signal to an encoder to generate an encoded signal; Acquire a second signal, where the second signal includes an overlay area, and the overlay area of ​​the second signal includes overlay content; processing at least the second signal to generate residual data; encoding the residual data to generate encoded residual data; as well as At least the encoded residual data is output. 2 . The method of claim 1 , wherein the processing of at least the second signal to generate the residual data comprises processing the second signal and a third signal to generate the residual data. 3 . The method of claim 2 , comprising receiving a decoded version of the encoded signal, wherein the third signal is based on the decoded version of the encoded signal. 4 . The method of claim 3 , comprising upsampling the decoded version of the encoded signal to generate an upsampled signal, wherein the third signal comprises the upsampled signal.

5. The method according to claim 3 or 4, wherein the processing of the second signal and the third signal to generate the residual data comprises: A difference between a value of a signal element in at least the superposition region of the second signal and a value of a corresponding signal element in the third signal is calculated. The method according to claim 5 , wherein the corresponding signal element in the third signal is included in a superposition region of the third signal. The method according to claim 5 , wherein the corresponding signal elements in the third signal are included in a non-overlapping region of the third signal.

8. The method according to claim 4 or 5, wherein the processing of the second signal and the third signal to generate the residual data comprises: combining values ​​of signal elements of at least the superposition region of the second signal with values ​​of corresponding signal elements in the third signal to generate a combined signal; as well as calculating a difference between a value of a signal element in at least a superposition region of the combined signal and a value of a corresponding signal element in the third signal, Wherein the superposition region of the combined signal corresponds to the superposition region of the second signal. 9 . The method according to claim 8 , wherein the second signal includes a non-overlapping area, and wherein the non-overlapping area of ​​the second signal is a blank area.

10. The method of claim 2, wherein the third signal comprises a reference signal, and wherein the reference signal is independent of the first signal. The method according to claim 10 , wherein the reference signal is a blank signal.

12. A method according to claim 10 or 11, wherein the residual data comprises replacement residual data comprising signal element values ​​to be substituted in place of corresponding signal element values ​​in another signal.

13. The method according to any one of claims 1 to 12, wherein the first signal comprises content having a first dynamic range, wherein the second signal comprises content having a second dynamic range, and wherein the second dynamic range is higher than the first dynamic range.

14. The method according to any one of claims 1 to 13, wherein: The overlay content includes or consists of high dynamic range, HDR, content; and / or The first signal comprises or consists of standard dynamic range, SDR, content.

15. The method of any one of claims 1 to 14, wherein the first signal does not include the overlay content.

16. A method according to any one of claims 1 to 15, wherein a first codec is used to generate the encoded signal and a second, different codec is used to generate the encoded residual data.

17. The method of claim 16, wherein the first codec is incapable of encoding the superimposed region of the second signal.

18. The method according to any one of claims 1 to 17, wherein the second signal has a given quality level, and wherein the given quality level of the second signal is maintained during the processing of the second signal.

19. A method according to any one of claims 1 to 18, comprising generating the second signal by applying a superposition signal to an input signal, the superposition signal comprising the superposition region.

20. The method according to any one of claims 1 to 19, comprising: Acquire another signal, the another signal comprising another superimposed area, and the another superimposed area comprising another superimposed content, the another superimposed content being different from the superimposed content included in the superimposed area of ​​the second signal; processing at least the further signal to generate further residual data; encoding the further residual data to generate further encoded residual data; and outputting the further encoded residual data, wherein the further encoded residual data is output in association with the encoded signal.

21. The method of claim 20, wherein the further encoded residual data is output together with the encoded residual data.

22. A method according to any one of claims 1 to 21, wherein the residual data is indicative of the superimposed content of the superimposed region of the second signal.

23. A method for signal processing, the method comprising: processing the encoded signal to generate a first signal; processing the encoded residual data by at least decoding the encoded residual data to generate decoded residual data; as well as At least the first signal and the decoded residual data are processed to generate a second signal, wherein the second signal comprises an overlay region comprising overlay content.

24. An apparatus configured to perform the method according to any one of claims 1 to 23.

25. A computer program arranged, when executed, to perform a method according to any one of claims 1 to 23.

Images