Correcting signal impairments using embedded signaling

Abstract

Examples described herein relate to decoding and encoding signals. A method of performing a signal enhancement operation on one or more portions of a signal is described, wherein the performing is based at least in part on information embedded in one or more values, the one or more values being received in one or more encoded data layers transmitted within an encoded data stream, and wherein the values are associated with transform coefficients intended to be processed by a decoder to derive elements of the signal, wherein the information is indicative of impairments associated with the portions of the signal.

Description

TECHNICAL FIELD

[0001] The present invention relates to methods for processing signals, such as by non-limiting example video, image, hyperspectral image, audio, point cloud, 3DoF / 6DoF and volumetric signals. Processing data can include, but is not limited to, obtaining, deriving, encoding, outputting, receiving and reconstructing signals in the context of a hierarchical (layer-based) encoding format, where signals are decoded in layers at successively higher quality levels, with subsequent layers (“stages”) of reconstructed data being utilized and combined. Signals of different layers can be encoded with different encoding formats, such as by non-limiting example, traditional single-layer DCT-based codecs, ISO / IEC MPEG-5 Part 2 Low Complexity Enhancement Video Coding, SMPTE VC-6 2117, etc., with the aid of different elementary streams that can or can not be multiplexed in a single bitstream. BACKGROUND

[0002] In a layer-based encoding format, such as ISO / IEC MPEG-5 Part 2 LCEVC (hereinafter “LCEVC”) or SMPTE VC-6 2117 (hereinafter “VC-6”), a signal is decomposed into multiple data “stages” (also referred to as “strata”), each corresponding to a “quality level” (also referred to herein as “LoQ”) of the signal, from a highest stage of a sampling rate of the original signal to a lowest stage that typically has a lower sampling rate than the original signal. In a non-limiting example, when the signal is a picture in a video stream, the lowest stage can be a thumbnail of the original picture, e.g., a low-resolution frame in the video stream, or even a single image element. Other stages contain information about corrections to apply to a reconstructed rendition in order to produce a final output. Stages can be based on residual information, e.g., a difference between a version of the original signal at a particular quality level and a reconstructed version of the signal at the same quality level. The lowest stage can not contain residual information, but can contain a lowest sampling of the original signal. A decoded signal at a given quality level is reconstructed by first decoding the lowest stage (thereby reconstructing the signal at a first lowest quality level), then predicting a signal rendition at a second, higher quality level, then decoding reconstructed data of the corresponding second stage (also referred to as “residual data” of the second quality level), then combining the prediction with the reconstructed data in order to reconstruct the signal rendition at the second, higher quality level, and so on until the given quality level is reconstructed.

[0003] The reconstructed signal can include decoded residual data and use this data to correct a version of a particular quality level derived from a version of the signal at a lower quality level. Different levels of data can be encoded using different encoding formats, and different quality levels can have different sampling rates (e.g., resolutions, in the case of image or video signals). Subsequent levels can refer to the same signal resolution (i.e., sampling rate) of the signal, or to progressively higher signal resolutions. Examples of these approaches are described in more detail in the available specifications for LCEVC and VC-6.

[0004] When using a layer-based encoding format, residual data for one or more quality layers can be used to correct for impairments introduced by the lower encoding method. For example, in LCEVC, image artifacts introduced by the processing of the base encoding and decoding (e.g., using a known codec such as H.264 or H.265) can be corrected by an enhancement data stream that includes residual data for one or more sub-layers. However, the more encoding artifacts that are introduced, the higher the target bit rate required to carry the residual data. This can be problematic if the network bandwidth is limited and the available bit rate is low. In this case, there is a trade-off between the bit rate of the encoded enhancement or higher level residual data stream and the quality of the reconstructed signal at the decoder. Therefore, it is desirable to improve encoding schemes such as LCEVC and VC-6 in cases where the target bit rate is low and / or a low level encoding method is used that uses a low quality output. SUMMARY

[0005] The non-limiting embodiments shown herein refer to a signal as a sequence of samples (i.e., a two-dimensional image, a video frame, a video field, a sound frame, etc.). In the description, the terms “image”, “picture” or “plane” (meaning the broadest meaning of “hyperplane”, i.e., an array of elements having arbitrary dimensionality and a given sampling grid) will often be used to identify a digital representation of a signal sample along a sequence of samples, where each plane has a given resolution for each of its dimensions (e.g., X and Y) and contains a set of plane elements (or “elements”, or “pels”, or display elements of a two-dimensional image, often referred to as “pixels”, or display elements of a volumetric image, often referred to as “voxels”, etc.) characterized by one or more “values” or “settings” (e.g., color settings in a suitable color space, settings indicating a level of density, settings indicating a level of temperature, settings indicating a pitch of audio, settings indicating an amplitude, settings indicating a depth, settings indicating a level of alpha channel transparency, etc., as non-limiting examples). Each plane element is identified by a suitable set of coordinates indicating the integer position of the element in the sampling grid of the image. The signal dimensionality can include only spatial dimensions (e.g., in the case of an image) or temporal dimensions (e.g., in the case of a signal evolving over time, such as a video signal).

[0006] As non-limiting examples, the signal can be an image, an audio signal, a multi-channel audio signal, a telemetry signal, a video signal, a 3DoF / 6DoF video signal, a volumetric signal (e.g., medical imaging, scientific imaging, holographic imaging, etc.), a volumetric video signal, or even a signal with more than four dimensions.

[0007] For simplicity, the non-limiting embodiments shown herein generally refer to signals displayed as 2D setup planes (e.g., 2D images in a suitable color space), such as for example video signals. The terms “picture”, “frame” or “field” will be used interchangeably with the term “image” in order to indicate a temporal sample of a video signal: any concept and method shown for video signals composed of frames (progressive video signals) can be easily applied to video signals composed of fields (interlaced video signals) and vice versa. Although the embodiments shown herein focus on image and video signals, the skilled person can easily understand that the same concepts and methods apply to any other type of multi-dimensional signal (e.g., audio signals, volumetric signals, stereoscopic video signals, 3DoF / 6DoF video signals, plenoptic signals, point clouds, etc.).

[0008] Examples described herein relate to signaling one or more impairments associated with a portion of a signal to be reconstructed using embedded information in one or more values received in one or more encoded data layers transmitted within an encoded data stream. In particular, examples use information embedded in transform coefficient values to signal one or more impairments such that these impairments can be at least partially corrected by signal enhancement operations applied at the time of reconstructing the signal (e.g., at a decoder). For example, both LCEVC and VC-6 encode residual data by using transform and quantization encoding blocks applied sequentially. In the present examples, the quantized transform residual values (referred to herein as transform coefficients) are used to carry signaling information related to signal impairments, in addition to the residual data itself. Thus, the transform coefficients are used to provide a form of embedded signaling in the encoded data stream. The signal enhancement operations can be enhancement operations not defined in one or more LCEVC or VC-6 video encoding standards, for example, containing so-called non-standard compliant operations, but which are still compatible with the standards. Thus, the present examples can be implemented as optional extensions to the above-mentioned standards, or other layer-based hierarchical encoding schemes.

[0009] According to certain described embodiments, a signal processor (e.g., computer processor hardware) is configured to receive data and decode it (“decoder”). The decoder implements a method of decoding data. The decoder can implement a layer-based decoding method, such as LCEVC or VC-6. In these cases, the decoder obtains a signal rendition at a first (e.g., lower) level of quality, and produces a predicted rendition of the signal at a second (e.g., higher) level of quality. For example, the second level of quality can have a higher resolution (i.e., signal sampling rate) than the first level of quality. In these cases, as described in layer-based decoding methods, the decoder receives and decodes a level of residual data to apply to the predicted rendition of the signal, producing a corrected rendition of the signal at the second level of quality. In certain examples described herein, when decoding a particular group of residual data coefficients and finding a particular group of quantized symbols, the decoder does not interpret the symbols as residual data, but rather performs a signal enhancement operation according to the received symbols. These signal enhancement operations can then help to remove or reduce impairments. In cases where bit rate is constrained, rather than attempting to correct all impairments using residual data, a better quality picture can be generated at the decoder by employing known classes of signal enhancement operations. For example, a signal to emit one of four signal enhancement operations can only require 2 bits, and a signal to emit one of sixty-four signal enhancement operations can only require 6 bits. Since the signaling of impairments is carried within the encoded data, the signaling can be localized to specific portions of the signal, e.g., a signal to go a slice can be emitted within only some, but not all, coding units of a full picture. Thus, the present approach provides a way to apply local corrections of impairments using local signal enhancement operations. The present approach can be used to apply signal enhancement operations at one or more of the first and second levels of quality, based on embedded signaling data carried in residual data at one or more of these levels.

[0010] In some non-limiting embodiments, the bits in the decoded byte stream signal to the decoder that additional information can have been embedded in some of the residual data coefficients, and thus that a particular group of symbols in a particular group of residual data should not be interpreted as actual residual data, but rather as contextual information informing a signal enhancement operation. In non-limiting embodiments, some reserved symbols correspond to a particular type of impairment, signaling to the decoder a post-processing operation that can be applied to a corresponding region of the signal in order to improve the quality of the final signal reconstruction.

[0011] In some non-limiting embodiments, the decoder implements signal enhancement operations in different ways (including sometimes not implementing them at all) based on the processing power available to the decoder device at any given time. For example, even if a signal is signaled to emit a particular signal enhancement operation and / or impairment, the one or more signal enhancement operations are only performed if a resource metric at the decoder indicates that available resources are above a pre-defined threshold.

[0012] In some non-limiting embodiments, the decoder applies the signal enhancement operation “in-loop”. This means that the signal enhancement operation is performed as part of the decoding loop operating on the partial signal, rather than after the full reconstructed frame is output. For example, the signal enhancement operation that corrects the impairment can be signaled and applied before applying the residual data decoded from the data level containing the embedded information about the impairment of the tile. In other words, the enhancement data containing the residual can also carry signaling to indicate the impairment correction before applying the residual. In other non-limiting embodiments, the decoder applies the signal enhancement operation after combining the preliminary rendition of the signal at the second quality level with the decoded residual data (e.g., this can be an “in-loop” adjustment to the decoding loop related to the second quality level), such as the sub-level 2 decoding operation in LCEVC. In other non-limiting embodiments, the decoder applies the signal enhancement method at the end of the decoding process after the rendition of the signal at the final (highest) quality level is produced (i.e., “out-of-loop” after the video frame is output (e.g., for rendition)).

[0013] In a certain non-limiting embodiment, the format in which the at least partial signal and the embedded context information are encoded is MPEG-5 Part 2 LCEVC (“Low Complexity Enhancement Video Coding”). For example, the signal can be encoded using a base encoder to create a base stream, and an LCEVC encoder to create an enhancement stream (which can have two sub-layers). In this case, it is the decoding of the enhancement stream, where the signal enhancement operation is applied based on the signaling of the impairment embedded in the enhancement stream (e.g., in the signaling in one or more sub-layers embedded in the transform coefficients before entropy encoding). In other non-limiting embodiments, the format in which the at least partial signal and the embedded context information are encoded is SMPTE VC-6 ST-2117. For example, VC-6 can be implemented as a full codec, where the lowest level data is encoded as the lowest resolution image (e.g., in a video stream of images). In this case, the residual data for the levels above the lowest level can have the signaling for the enhancement at a given quality level embedded in the level at that quality level or a level below it.

[0014] According to certain other non-limiting embodiments, a signal processor (e.g., computer processor hardware) is configured to receive data and encode it (“encoder”). The encoder implements an encoding method, e.g., one that corresponds to the decoding method described herein. In encoding, transform coefficients used for signal reconstruction are reserved for signal enhancement operations that will be performed on one or more portions of the signal to correct one or more impairments detected in the signal. For example, one or more quantization signs of a given set of coefficients (such as one of the A, H, V, D coefficients of the 2x2 Hadamard transform described in LCEVC and VC-6, or coefficients resulting from a larger 4x4 transform (e.g., HH coefficients)) are used to provide embedded signaling to identify detected impairments and indicate one or more signal enhancement operations to correct them. This is in addition to the regular use of transform coefficients (e.g., to provide encoded residual data to be added to a preliminary reconstruction generated from a lower quality level.

[0015] In one set of examples, an encoder produces a first (e.g., lower) quality level of signal reproduction and encodes it with a first encoding method. This can be the base encoding in LCEVC or the lowest level encoding in VC-6. The encoder then produces a predicted reproduction of a second (e.g., higher) quality level of signal and correspondingly produces a second quality level of residual data level and encodes it for application to the predicted reproduction of the second quality level of signal to produce a corrected reproduction of the second quality level of signal. For example, the predicted reproduction of the second quality level of signal can be produced from the first quality level of signal reproduction (e.g., can include up-sampled output of sub-layer 2 in LCEVC). When the process of encoding the first quality level of signal produces one or more impairments that cannot be properly corrected with the residual data at the target bitrate (e.g., the bitrate of a particular file format that can be used for transmission to a decoder or for storage), the encoder utilizes a set of reserved signs in a set of residual data of the second quality level of residual data level to signal to the decoder the type and / or location of impairments it should expect. The decoder is then able to selectively apply signal enhancement operations based on the type and / or location of impairments. In this case, the location of the impairments can be determined by applying embedded signaling within the data of a particular encoding unit (e.g., a grid of residual values such as a 2x2 or 4x4 encoding unit in LCEVC) where different encoding units have different signal impairments (or no impaired signal, so the residual data is as per regular application of the decoding method).

[0016] In some non-limiting embodiments, bits in the encoded byte stream are toggled in order to signal to the decoder whether a given set of signs in a given set of residual data should be interpreted as actual residual data or additional contextual information to inform image enhancement operations.

[0017] In some non-limiting embodiments, the format used to encode at least part of the signal (e.g., the residual data of the signal and the embedded impairment information) is MPEG-5 Part 2 LCEVC (“Low Complexity Enhancement Video Coding”). In other non-limiting embodiments, the format used to encode at least part of the signal and the embedded impairment information is SMPTE VC-6 ST-2117. For example, the signaling indicative of one or more signal enhancement operations can be embedded in transform coefficients received and decoded according to these standards.

[0018] According to other non-limiting embodiments, the impairment information embedded in the residual data comprises a sign corresponding to a blocking impairment. In non-limiting embodiments, the decoder implements a deblocking post-processing operation in the signal area corresponding to residual coefficients containing the preserved sign.

[0019] In non-limiting embodiments, the decoder applies different degrees of deblocking strength based on the received sign.

[0020] In non-limiting embodiments, the decoder deblocks the signal by means of a deblocking method such as described in patent US9445131B1 “De-blocking and de-banding filter with adjustable filter strength for video and image processing” (this patent is incorporated by reference), in which the QP information of a given neighboring area is embedded in the sign.

[0021] In some non-limiting embodiments, the decoder applies a deblocking method in loop before applying the residual data decoded from the data level containing embedded information about the blocking impairment. In other non-limiting embodiments, the decoder applies the deblocking method after combining the preliminary rendition of the signal at the second level of quality with the decoded residual data. In other non-limiting embodiments, the decoder applies said post-processing method at the end of the decoding process, after the rendition of the signal at the final (e.g., highest) level of quality has been produced.

[0022] According to other non-limiting embodiments, the signal information embedded in the residual data comprises a symbol corresponding to a banding, ringing and softening impairment. In non-limiting embodiments, the decoder implements a signal enhancement operation in signal areas corresponding to residual coefficients containing the preserved symbol, said signal enhancement operation comprising debanding, de- ranging, edge enhancement, range equalization and sharpening post-processing operations. For example, the video frame can be divided into several 2x2 or 4x4 coding unit areas, and the impairment information can be specific to the coding unit. In other cases, areas of the video frame can be defined, and within each area, the embedded signaling is provided in one of the set of transform coefficients of the area (e.g. the transform coefficients of the coding units of the corners or center of the area). It should be noted that the reference to a “frame” of video data also includes a reference to one or more planes of color data (e.g. luma and chroma planes), as known from layer-based encoding methods such as LCEVC and VC-6.

[0023] In some non-limiting embodiments, the decoder applies the post-processing method in the loop before applying the residual data decoded from the data level containing the embedded information about the blocking impairment. In other non-limiting embodiments, the decoder applies the post-processing method after combining the preliminary rendition of the signal at the second level of quality with the decoded residual data. In other non-limiting embodiments, the decoder applies the post-processing method at the end of the decoding process, after the rendition of the signal at the final (e.g. highest) level of quality has been produced.

[0024] According to certain non-limiting embodiments, the impairment information embedded in the residual data comprises a symbol corresponding to the risk of chroma flipping impairment in case of color conversion from wide color gamut to standard color gamut. Said impairment is due to the limitations of the conversion LUT (“Look-Up Table”). In non-limiting embodiments, the decoder clamps color values in signal areas corresponding to residual coefficients containing the preserved symbol, before applying the color conversion method.

[0025] According to additional non-limiting embodiments, the impairment information embedded in the residual data comprises a symbol corresponding to a quantization noise impairment. In some non-limiting embodiments, the decoder applies a denoising method in signal areas corresponding to residual coefficients containing the preserved symbol.

[0026] In some non-limiting embodiments, the decoder applies the denoiser in the loop before applying the residual data decoded from the data level containing the embedded information about the blocking impairment. In other non-limiting embodiments, the decoder applies the denoiser after combining the preliminary rendition of the signal at the second level of quality with the decoded residual data.

[0027] According to certain non-limiting embodiments, context signal information embedded in the residual data includes a sign corresponding to film grain and / or camera noise loss. In some non-limiting embodiments, the decoder applies a statistical dithering method in signal regions corresponding to residual coefficients containing the preserved sign.

[0028] In some non-limiting embodiments, statistical dithering is applied in loops of multiple levels in a layered system, e.g., at a given level of resolution and subsequent (e.g., higher) levels of resolution.

[0029] According to further non-limiting embodiments, impairment related context signal information is embedded in encoded data generated with a non-layered encoding format. In non-limiting embodiments, the sign is embedded at the macroblock level using a set of preserved signs in quantized coefficients.

[0030] Other features and advantages will be apparent from the following description, made with reference to the drawings, and only given as examples. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A block diagram showing an example of a decoding system according to an embodiment is shown;

[0032] Figure 2 A block diagram showing an example of a hierarchical encoding system according to an embodiment is shown;

[0033] Figure 3 A block diagram showing a hierarchical decoding system according to an embodiment is shown;

[0034] Figure 4 A block diagram showing another example of a hierarchical decoding system according to an embodiment is shown;

[0035] Figure 5 A block diagram showing another example of an encoding and decoding system according to an embodiment is shown; and

[0036] Figure 6 A block diagram showing an example of an apparatus according to an embodiment is shown. DETAILED DESCRIPTION

[0037] As additional background, based on the predicted fidelity of the reproduction at a given level of quality (LoQ), the signal can require varying amounts of correction, which can be provided by "residual data" or simply by "residue". This residual data (as taught in LCEVC and VC-6, for example) enables a decoder to generate a reconstruction of the signal at a given LoQ that most closely resembles (or even losslessly reconstructs) the original signal. Especially when reconstructing lower LoQs using relatively high compression ratios and / or less efficient encoding formats, certain impairments can arise at lower LoQs. The approach of layer-based encoding schemes such as LCEVC and VC-6 means that the residual data can operate in accordance with the scheme to correct for these impairments. However, when bitrates are constrained, it has been found that additional signal enhancement operations can be applied to enhance the picture quality within the final signal reconstruction at full resolution and full quality. In the examples described herein, the potential for impairments is detected at least at the encoder and a signal is signaled to the decoder using embedded signaling, so that the decoder can maintain a standardized decoding process but add additional signal enhancement operations that are selectively performed based on the content of the embedded signaling to reduce and / or correct impairments in the output reconstruction at the decoder. For the case of image and video signals, non-limiting examples of impairments that can be corrected include blocking, banding, softening, chroma impairments due to color space conversion, loss of low-contrast texture, quantization noise, loss of film grain noise, etc.

[0038] Certain methods described herein allow impairments to be addressed at the decoder in two ways: by directly correcting them (e.g., up to lossless correction and reconstruction) using suitable residual data, and by applying post-processing operations such as sharpening, de-noising, image enhancement, etc. Since impairments can be signaled in contextual data that can be carried within localized encoding units or blocks, localized signal enhancement operations can also be signaled, allowing for more efficient encoding, decoding, and correction than a comparative "blanket" approach that applies non-selective global post-processing. By comparison, with respect to correcting impairments via residual data only, while the approach can achieve lossless fidelity, it is typically very costly in terms of bitrate to correct artifacts such as blocking or banding properly, as it requires very small quantization steps. On the other hand, with respect to post-processing operations at the end of the decoding process, it is typically very costly to correctly identify to what extent they should be used in order to avoid corrupting signal portions that do not benefit from them, from the decoding process perspective. In the current case, the combination of some correction via residual data and signaled signal enhancement operations allows for more efficient residual encoding with larger quantization steps, and simple and efficient selective and possibly localized correction based on the signaling of impairments from the encoder within the residual data (e.g., in addition to and separate from the residual data itself).

[0039] The embodiments described herein allow for efficient generation, signaling and utilization of context information that a decoder can use along with residual data to properly correct signal reconstruction in order to improve the quality of the reconstructed signal. Such information is efficiently embedded in the coefficients of the residual data of one or more levels of the encoded signal, allowing for avoiding the need of additional signaling overhead (e.g. separate sideband signaling) and efficiently distinguishing situations that can benefit from a range of quality enhancement operations. Moreover, for some non-limiting embodiments described herein, in the context of a hierarchical encoding scheme, the application of certain non-essential signal enhancement operations at lower signal resolution also results in a substantial reduction of the required processing power at the decoder device.

[0040] Reference is made to Figure 1 An example of a method implemented in a decoding system is shown. A set of quantized symbols 100-1 to 100-N is received and processed. These quantized symbols contain quantized transform coefficients, where quantization can be optional and / or vary in extent based on the encoding configuration. The quantized symbols can contain symbols produced from one or more encoded streams described in the following figures. In the examples described herein, information is embedded in one or more values received in one or more encoded data layers, where the values are associated with transform coefficients of elements intended to be processed by the decoder to derive a signal. The reception of exemplary quantized and transform coefficients symbols is described in LCEVC and VC-6. Depending on whether symbol 100-1 is intended as a reserved symbol, the decoder follows two different approaches. In this context, the term “reserved symbol” can be considered to refer to a symbol that is reserved to carry context information such as impairment information including instructions about signal enhancement operations to be performed at the decoder to reduce and / or remove specific detected impairments.

[0041] If symbol 100-1 is not intended as a reserved symbol, e.g. is intended to carry residual data for reconstructing a signal, its decoding follows the normal processing implemented for the other symbols in the set: dequantization and inverse transform according to block 110, resulting in a set of decoded data 130. The decoded data is further processed by means of decoding operations 150 to produce a decoded signal 160. This normal processing can be, for example, the processing described in the decoding specification such as LCEVC or VC-6.

[0042] If the symbol 100-1 is intended to be used as a reservation symbol, its decoding follows a different process, as indicated by the comparison block 105. In block 120, the embedded information is decoded by the processing symbol 100-1 to produce information 140 about the enhancement operation to be performed. This information 140 is impairment information, as it signals information about the impairment detected at the decoder. The decoded signal 160 is processed with the information 140 about the enhancement operation to be performed by means of the enhancement operation 170, resulting in an enhanced reconstruction of the signal 180.

[0043] For example, the information 140 about the enhancement operation to be performed can be determined based on the embedded signaling information corresponding to one or more of the following impairment categories: banding, ringing, softening, color conversion flip, and quantization noise impairment. The enhancement operation 170 can correspondingly contain one or more of the following: deblocking, debanding, deringing, denoising, sharpening, dithering, color clipping, range equalization, and edge enhancement. The impairment can be detected by computing image quality metrics at the encoder, for example, using known metrics to detect the impairment.

[0044] In some non-limiting embodiments, a bit or a bit in the decoded bitstream (not shown in the figures) signals to the decoder that the symbol 100-1 is to be used as a reservation symbol. For example, this bit can be contained in a "user data" flag that is toggled "on" or "off" in the global configuration information, which will be described in more detail below.

[0045] In some non-limiting embodiments, the decoder implements the signal enhancement operations in different ways (including sometimes not implementing them at all) based on the processing power available to the decoder device at the time of processing. For example, even if the information 140 about the enhancement operation to be performed derived from the embedded signaling about the impairment indicates enhancement operations 170, these operations can not be performed if the resource metrics at the decoder are below a predefined threshold. For example, the enhancement operations 170 can only be performed if there is sufficient spare Central Processing Unit (CPU) or Graphics Processing Unit (GPU) capacity (e.g., measured in terms of one or more of utilization, throughput, available memory, etc.) and / or the remaining battery power for a mobile device is above a predefined threshold (e.g., 20%).

[0046] Reference Figure 2 is shown an example of a method implemented in an encoding system, in this case a layer-based hierarchical encoding method is implemented. Figure 2The blocks in FIG. 1 can be implemented by an exemplary encoder. A source signal 200 at quality level #2 (e.g., full resolution and quality) is received and processed by a down-sampler 210, resulting in a down-sampled signal 200-1. The down-sampled signal 200-1 is processed by an encoder 220, which applies a given encoding method (in some non-limiting embodiments, also a layer-based hierarchical encoding method, while in other non-limiting embodiments, a non-hierarchical encoding method), resulting in encoded data 225. The encoder 220 can be referred to as a "base" decoder.

[0047] The encoded data 225 and the down-sampled signal 200-1 are processed by a LOQ#1 residual data generator 230 to produce encoded data 235 and a signal reproduction at LOQ#1 237. The LOQ#1 residual data generator 230 can generate a residual signal by subtracting a reconstruction based on the encoded data 225 from the down-sampled signal 200-1. The LOQ#1 residual data generator 230 can also encode the residual signal by applying an encoding unit transform and quantizing the output of the transform. Another stage of entropy encoding can also be applied. The output of the transform and quantization can include (quantized) transform coefficients that are modified to include embedded signaling.

[0048] The signal reproduction at LOQ#1 237 is further processed by a LOQ#2 preliminary reproduction generator 240 to produce a preliminary reproduction of a signal at LOQ#2 245. This can include, for example, up-sampling the signal reproduction at LOQ#1 237 with optional modifications to generate a signal at the resolution and / or sampling rate of LOQ#2.

[0049] The preliminary reproduction of a signal at LOQ#2 245 is processed together with the source signal 200 by a LOQ#2 residual generator 260 to produce encoded data 265. The encoded data 265 can include a residual signal generated by the LOQ#2 residual generator 260 by subtracting the preliminary reproduction of a signal at LOQ#2 from the source signal 200. The LOQ#2 residual generator 260 can apply similar operations as the LOQ#1 generator 230, but based on a residual signal at the second quality level (e.g., at a higher resolution).

[0050] The multiple sets of encoded data 225, 235, and 265 are then processed by a multiplexer (Mux) 270 to produce encoded signal 280 in a level of encoded data. While all three sets of encoded data are processed in this example, in other non-limiting embodiments, only two sets of encoded data are processed, or only one set of encoded data is processed. Figure 2In some non-limiting embodiments, the LOQ#1 residual generator 230 triggers specific bits in the encoded bit or byte stream in order to signal to the decoder that a given set of symbols in the set of encoded data 235 should be interpreted as actual residual data or additional context information. In some non-limiting embodiments, the LOQ#1 residual generator 230 and the LOQ#2 residual generator 260 implement in-loop signal enhancement operations according to the information signaled with the reserved coefficients in order to simulate the reconstruction produced by the decoder and, if necessary, adapt the encoded data 265 appropriately. For example, the residual generated by the LOQ#2 residual generator 260 and encoded as encoded data 265 can correct the replacement of transform coefficient values with embedded impairment signaling.

[0051] In certain described embodiments, when the process of encoding the signal at the first quality level is detected to produce one or more impairments that cannot be properly corrected with residual data at the target bitrate, the LOQ#1 residual generator 230 produces encoded data 235 that signals to the decoder the type and / or location of impairments it should expect using a set of reserved symbols in the data. For example, the LOQ#1 residual generator 230 can process one or more reconstructions from the encoded data 225 and the residual signal to determine the one or more impairments that are present. This can include computing one or more image metrics on one or more frames of the signal. In one case, the LOQ#1 residual generator 230 replaces quantized transform coefficient values with embedded signaling data by multiplying specific elements in the generated vector of transform matrix. Only the value of one coefficient can be modified, other coefficients can remain unchanged, and encoded according to the comparative encoding of the standardized decoding process. The coefficient that minimizes the change in the reconstructed signal can be selected, such as the H or HH coefficients for 2x2 or 4x4 Hadamard transforms.

[0052] In some non-limiting embodiments, the LOQ#1 residual generator 230 triggers specific bits in the encoded bit or byte stream in order to signal to the decoder that a given set of symbols in the set of encoded data 235 should be interpreted as actual residual data or additional context information. In some non-limiting embodiments, the LOQ#1 residual generator 230 and the LOQ#2 residual generator 260 implement in-loop signal enhancement operations according to the information signaled with the reserved coefficients in order to simulate the reconstruction produced by the decoder and, if necessary, adapt the encoded data 265 appropriately. For example, the residual generated by the LOQ#2 residual generator 260 and encoded as encoded data 265 can correct the replacement of transform coefficient values with embedded impairment signaling.

[0053] In some non-limiting embodiments, the LOQ#1 residual generator 230 triggers specific bits in the encoded bit or byte stream in order to signal to the decoder that a given set of symbols in the set of encoded data 235 should be interpreted as actual residual data or additional context information. In some non-limiting embodiments, the LOQ#1 residual generator 230 and the LOQ#2 residual generator 260 implement in-loop signal enhancement operations according to the information signaled with the reserved coefficients in order to simulate the reconstruction produced by the decoder and, if necessary, adapt the encoded data 265 appropriately. For example, the residual generated by the LOQ#2 residual generator 260 and encoded as encoded data 265 can correct the replacement of transform coefficient values with embedded impairment signaling.

[0054] Reference Figure 3 , shows an example of a method implemented in a decoding system, also implementing a layer-based hierarchical encoding method. Figure 3The blocks in FIG. 1 can be implemented by an exemplary decoder. The encoded data 225 is received and processed by a lower LOQ decoder 300. The encoded data 225 can be obtained by demultiplexing the signals 280 encoded in the encoded data levels received from the encoder, such as shown in FIG. 2. Figure 2 The lower LOQ decoder 300 can be referred to as a base decoder. The lower LOQ decoder 300 outputs a preliminary rendition of the signal at LOQ#1 310.

[0055] The preliminary rendition of the signal at LOQ#1 310 is then processed by a LOQ#1 reconstructor 320 along with the encoded data 235 in order to produce a signal rendition at LOQ#1 337. The encoded data 235 can be obtained by demultiplexing the signals 280 encoded in the encoded data levels received from the encoder, such as shown in FIG. 2. Figure 2 In some non-limiting embodiments, the rendition 337 is the same as the corresponding rendition 237 of Figure 2 In other non-limiting embodiments, the decoder applies a different processing method (e.g., including a signal enhancement operation) to produce a rendition 337 that is different from the rendition 237. The encoded data 235 can be decoded by the LOQ#1 reconstructor 320 and then combined with the preliminary rendition of the signal at LOQ#1 310. In one case, the encoded data 235 can contain residual data as described in other examples herein. The decoding can include applying operations as shown in blocks 110 and 410 in FIGS. 1 and 4. Figure 1 and 4

[0056] The signal rendition at LOQ#1 337 is then processed by a preliminary rendition generator 340 at LOQ#2 (which can correspond to the preliminary rendition generator 240 at LOQ#2 in FIG. 2) to produce a preliminary rendition of the signal at LOQ#2 345. In some non-limiting embodiments, the rendition 345 is the same as the corresponding rendition 245 of Figure 2 In other non-limiting embodiments, the decoder produces a rendition 337 that is different from the rendition 237, and thus the rendition 345 is also different from the rendition 245. Figure 2

[0057] The preliminary rendition of the signal at LOQ#2 345 is processed by a LOQ#2 reconstructor 360 to produce a final rendition of the signal at LOQ#2 370.

[0058] In some non-limiting embodiments, when a particular set of data within the encoded data 235 is decoded and a particular set of quantization symbols is found, the decoder does not interpret the symbols as residual data, but rather performs a signal enhancement operation according to the received symbols.

[0059] In some non-limiting embodiments, the decoded byte stream (e.g., the decoded data 235) is a byte stream that is decoded from a byte stream that was encoded by the encoder. Figure 3 ​​The reconstructor 320 of LOQ#1 and / or the reconstructor 360 of LOQ#2 (not shown) in the decoder 300 emit a signal that additional information can have been embedded in some of the residual data coefficients, and more specifically that a certain set of quantization signs (or part of said signs) in a certain set of residual data should not be interpreted as actual residual data, but as contextual information informing the signal enhancement operation.

[0060] In non-limiting embodiments, some reserved signs correspond to a certain type of impairment, informing the decoder of a post-processing operation (either in the loop, i.e. within block 230, or at the end of the decoding process, i.e. within block 260), which can be applied to the corresponding region of the signal in order to improve the quality of the final rendition 370 of the signal at LOQ#2.

[0061] In some non-limiting embodiments, the decoder implements the signal enhancement operations in different ways (including sometimes not implementing them at all), based on the processing power available to the decoder device at the time of decoding.

[0062] In some non-limiting embodiments, the decoder applies the signal enhancement methods in the loop (i.e. within block 320) before applying the residual data decoded from the data level containing embedded information about the impairment of the block. In other non-limiting embodiments, the decoder applies the signal enhancement methods in the loop (i.e. still within block 320) after combining the preliminary rendition of the signal with the decoded residual data. In other non-limiting embodiments, the decoder applies the signal enhancement methods at the end of the decoding process (i.e. at or after the end of block 360) after having produced a rendition of the signal at the final (highest) level of quality. In yet other non-limiting embodiments, the decoder applies the signal enhancement methods in the loop (e.g. by means of a non-limiting example according to one of the above-mentioned alternatives) as well as at the end of the decoding process.

[0063] In preferred examples, the encoder or decoder is part of a layer-based scalable coding scheme or format. Examples of layer-based scalable coding schemes include LCEVC: MPEG-5 Part 2 LCEVC (“Low Complexity Enhancement Video Coding”) and VC-6: SMPTE VC-6 ST-2117, the former described in PCT / GB2020 / 050695 (and related standard documents), the latter described in PCT / GB2018 / 053552 (and related standard documents), all of which are incorporated by reference herein. However, the concepts shown herein need not be limited to these particular scalable coding schemes. The skilled person will know how the above-described encoder and decoder methods apply to base and enhancement layers in LCEVC (e.g. LOQ#1 corresponds to the base layer, LOQ#2 corresponds to the enhancement layer). As such, in some cases, the format in which the encoded data 235, the encoded data 265 and the corresponding embedded context information are encoded is MPEG-5 Part 2 LCEVC (“Low Complexity Enhancement Video Coding”). In this case, the encoded data 235 and the encoded data 265 can contain different enhancement sub-layers. In this case, the embedded context information can be referred to as “user data” in that it can contain information in addition to that required to reconstruct the signal according to the standard. In other cases, the format in which the encoded data 235, the encoded data 265 and the corresponding embedded context information are encoded is SMPTE VC-6 ST-2117. Similarly, the embedded context information can contain data that is not required to reconstruct the signal according to the standard’s definition.

[0064] Reference is made to Figure 4 , which shows an example of a method implemented in a decoding system, which implements a layer-based scalable coding method. The method is Figure 2 a variant of the method shown in Figure 3 In this variant, the impairment information can be used in processing one or more of the preliminary rendition of the signal at LOQ#1 (such as 310 in Figure 3 and the rendition of the signal at LOQ#2 (e.g. 370 in As such, the impairment information can be associated with one or more quality levels, including the base quality level.

[0065] As shown in Figure 2 In Figure 4 , the quantized symbol 400-1 is received and processed along with other quantized symbols 400-2...400-N. The quantized symbols can represent a residual data stream at the level of the encoded data 235. The decoder checks whether the symbol 400-1 should be treated as a reserved symbol. Depending on whether the symbol 400-1 is intended to be treated as a reserved symbol, the decoder follows two different approaches.

[0066] If symbol 400-1 is not intended as a reservation symbol, its decoding follows the normal processing implemented for the other symbols in the group: dequantization and inverse transform according to block 410, yielding a set of decoded residual data 420. Said residual data is further processed by means of a reconstructor 450 (e.g., together with other residual data of the rest of the signal samples, as a non-limiting example) to yield a signal rendition 460 at LOQ#1.

[0067] If symbol 400-1 is intended as a reservation symbol, its decoding follows a different processing. At block 430, a decode embedded information 430 operation is launched to process symbol 400-1. In this example, the signaling present in symbol 400-1 is decoded to yield information about the enhancement operation to be performed for one or more quality levels. A first set of information 432 about the enhancement operation to be performed is decoded and processed, by means of an enhancement operation 440, together with the preliminary rendition 310 of the signal at LOQ#1, yielding an enhanced preliminary rendition 445 of the signal at LOQ#1. In this case, the first set of information 432 about the enhancement operation to be performed indicates that a signal enhancement operation is to be performed on the signal decoded from the encoded data 225 in Figure 3 This can be the output of a basic decoder of an LCEVC implementation. Enhancement operation 440 takes place at the first quality level, before the addition of residual data. Then, enhanced preliminary rendition 445 is further processed by reconstructor 450 together with residual data 420, yielding an enhanced rendition 460 of the signal at LOQ#1. For example, reconstructor 450 can include processing similar to those performed by reconstructor 320 of the signal at LOQ#1 in Figure 3 Enhanced signal rendition 460 at LOQ#1 can correspond to signal rendition 337 at LOQ#1 in Figure 3 Enhanced signal rendition 460 at LOQ#1 can correspond to signal rendition 337 at LOQ#1 in Figure 4 As shown in, in some cases, reconstructor 450 can also use the first set of information 432 about the enhancement operation to be performed to provide a differential processing. For example, in some examples, enhancement operation 440 can be performed after the addition of residual data 420 based on the first set of information 432 about the enhancement operation to be performed. Different approaches can be applied, depending on the implementation.

[0068] In one case, decode embedded information block 430 can extract both impairment information and residual data from quantized symbol 400-1 (e.g., quantized symbol 400-1 can carry both impairment information and residual data). This can be achieved, for example, by splitting the bits of quantized symbol 400-1 and applying a higher level of quantization to the original symbol representing the transform coefficients of the residual data. This can be achieved by extracting the least significant bits of the symbol and then setting these bits to zero, which represents a rounding down operation with a level of rounding closest to 2 n(Where n is the number of bits of signaling information). For the bits representing transform coefficient data, the decoding embedded information frame 430 can perform operations similar to those of the dequantization and inverse transform frame 410, or alternatively, the non-signaling components of the quantization symbol 400-1 can be passed back to the quantization and inverse transform frame 410 to derive residual data. In either case, the reconstructor 450 can additionally receive the residual data of the quantization symbol 400-1.

[0069] Therefore, the reconstructor 450 generates an enhanced signal reproduction 460 at LOQ#1, which is further processed by the decoding operation 465 to produce a signal reproduction 470 at LOQ#2. The decoding operation 465 can be... Figure 2 The operation is associated with the generator 340 and the reconstructor 360 of LOQ#2 in the initial reproduction of LOQ#2. Therefore, the signal reproduction 470 in LOQ#2 can be regarded as corresponding to Figure 3 Reproduction of 370 in [the text]. Figure 4 In this context, the second set of information 434 regarding the enhancement operation to be performed is decoded by box 430 and used to indicate the enhancement operation 480 performed at the second quality level. Therefore, in Figure 4 In the process, the signal reproduction 470 in LOQ#2 is processed by the enhancement operation 480 together with information 434 about the enhancement operation to be performed, so as to produce the final signal reproduction 490 of the enhancement in LOQ#2.

[0070] exist Figure 4 In the example, there are two sets of information 432 and 434 regarding the enhancement operation to be performed, and three positions 440, 450, and 480 where the enhancement operation can be performed. Different implementations can use different combinations of the described signaling and enhancements. For example, in one case, only the first set of information 432 regarding the enhancement operation to be performed can be sent, and only enhancement operation 440 can be performed; in another case, only the second set of information 434 regarding the enhancement operation to be performed can be sent, and only enhancement operation 480 can be performed. In one case, embedded signaling can indicate whether one or more of enhancement operations 440 and 480 are activated and whether they will be performed. For example, this can be indicated in global signaling data. This global signaling data can be embedded within the initial coded block of the signal and / or within separate sideband signaling for a picture or picture group.

[0071] In some cases, if the quantized sign 400-1 still carries transform coefficient values (e.g., by splitting the bit capacity between impairment information and transform coefficient values) and / or transform coefficients are selected as coefficients from a larger (e.g., 4x4) transform that is found to have a reduced visual impact (e.g., HH coefficients in a 4x4 Hadamard transform), the visual impact of the resulting output signal (e.g., final rendition 490 of the signal at LOQ#2) is minimized. Additionally, if the coefficient values of the transform have reduced precision or are replaced at the LOQ#1 level, the generator 260 of the LOQ#2 residual is able to generate residual data that becomes the encoded data 265 that accounts for the difference between the preliminary rendition 245 of the signal at LOQ#2 and the source signal 200 at LOQ#2. Thus, this means that the visual effect of the embedded signaling is minimized while ensuring that the encoding process does not modify the embedded signaling (as the information is embedded after the transform and quantization but before the entropy encoding). Even if there is a minimal or small effect on the visual quality (e.g., if the embedded signaling replaces the values of the transform coefficients), this effect can be smaller than the perceived improvement in picture quality obtained with the signaling enhancement operation; thus, the overall picture quality can be improved at a limited bit rate (e.g., on a poor quality communication channel and / or when limited to use a low quality base encoding).

[0072] Reference is made to Figure 5 , which shows an example of a method implemented in an encoding and decoding system that utilizes the innovative methods described herein. An encoder 510 processes an original signal 500 to produce a data stream 520. The encoder 510 can encode the data stream 520 as explained with reference to Figure 2 , and can embed impairment information within one or more values received within one or more encoded data layers transmitted within the data stream 520, wherein the values are associated with transform coefficients intended to be processed by a decoder to derive elements of a signal, e.g., as described with reference to Figure 2 or Figure 4 .

[0073] The data stream 520 is processed by two decoders. The decoder 530-0 implements a signal enhancement method based on the information signaled by the encoder 510 within the reserved symbols to decode the reconstructed signal 540-0. The decoder 530-1 ignores the information signaled by the encoder 510 within the reserved symbols and reconstructs the reconstructed signal 540-1. For example, the decoder 530-1 can process the embedded signaling values as if they were values of transform coefficients, can ignore these values (e.g., set them to 0) and / or can perform in standard user data processing (e.g., for LCEVC) but subsequently ignore user data containing embedded signaling. In some non-limiting embodiments, the reconstructed signal 540-1 is the preferred feasible reconstruction of the signal for a given purpose, such that the enhancement operations performed by the decoder 530-0 are entirely optional. For example, the decoder 530-1 can be a decoder that applies the decoding process set forth in the LCEVC standard, while the decoder 530-0 can be a decoder that implements a non-standard decoding process (in addition to the decoding process set forth in the LCEVC standard in some cases). Thus, additional functionality can be provided based on impairment information while remaining compliant with the LCEVC standard.

[0074] In some non-limiting embodiments, the decoder 530-0 can sometimes decide to ignore part of the information signaled by the encoder 510 within the reserved symbols. In non-limiting embodiments, the decoder 530-0 defines whether to ignore part of the information signaled within the reserved symbols based on information including resolution and frame rate of the signal, processing power load at the time of decoding and battery power status.

[0075] In some cases, backward compatibility is achieved (e.g., as explained above) because the decoder 530-1 treats the reserved symbols as normal quantized transform coefficient values and decodes them appropriately. The correction applied within the layer-based hierarchical format means that any errors can be corrected. Alternatively, bits in the encoded data stream 520 are used to signal to the decoder 530 that one or more values should be interpreted as said information rather than actual quantized values of transform coefficients. In yet another case, the bit depth assigned to a particular transform coefficient value (e.g., depth D is 8 or 16 bits) can be shared between the reserved symbol and the (quantized) transform coefficient value. For example, n least significant bits of the transform coefficient value (where n is less than the bit depth, e.g., 2 or 6 bits) can be used to carry the reserved symbol (i.e., impairment information) which indicates a more aggressive quantization applied to the transform coefficient value carrying this symbol, but still enables transmission of a coarse level of information (D-n bits) and use in reconstructing the residual data. The visual impact can be further minimized by selecting transform coefficients in the reconstructed output (e.g., H or HH in 2x2 or 4x4 Hadamard transform) that are determined (e.g., by experiment) to be less perceptual.

[0076] ReferenceFigure 6 A schematic block diagram of an example of device 600 is shown.

[0077] Examples of device 600 include, but are not limited to, mobile computers, personal computer systems, wireless devices, base stations, telephone devices, desktop computers, laptop computers, notebook computers, netbook computers, mainframe computer systems, handheld computers, workstations, network computers, application servers, storage devices, consumer electronic devices such as cameras, portable video cameras, mobile devices, video game consoles, handheld video game devices, peripheral devices such as switches, modems, routers, vehicles, etc., or generally any type of computing or electronic device.

[0078] In this example, device 600 includes one or more processors 612 configured to process information and / or instructions. The one or more processors 612 may include a central processing unit (CPU). The one or more processors 612 are coupled to a bus 611. Operations performed by the one or more processors 612 may be performed by hardware and / or software. The one or more processors 612 may include multiple co-located processors or multiple dissimilar processors.

[0079] In this example, device 600 includes computer-usable memory 613 configured to store information and / or instructions from one or more processors 612. Computer-usable memory 613 is coupled to bus 611. Computer-usable memory 613 may include one or more volatile and non-volatile memories. Volatile memory may include random access memory (RAM). Non-volatile memory may include read-only memory (ROM).

[0080] In this example, device 600 includes one or more external data storage units 680 configured to store information and / or instructions. The one or more external data storage units 680 are coupled to device 600 via I / O interface 614. The one or more data storage units 680 may, for example, include a hard disk or optical disk and a hard disk drive or solid-state drive (SSD).

[0081] In this example, the device 600 further includes one or more input / output (I / O) devices 616 coupled via I / O interface 614, which is configured to communicate information between one or more processors 612 and / or one or more I / O devices 616. The device 600 also includes at least one network interface 617. Both I / O interface 614 and network interface 617 are coupled to system bus 611. The at least one network interface can enable the device 600 to communicate via one or more data communication networks 690. Examples of data communication networks include, without limitation, the Internet and local area networks (LANs). The one or more I / O devices 616 can enable a user to provide input to the device 800 via one or more input devices (not shown). The one or more I / O devices 806 can enable information to be provided to a user via one or more output devices (not shown).

[0082] In Figure 6 In this example, the device 600 further includes one or more input / output (I / O) devices 616 coupled via I / O interface 614, which is configured to communicate information between one or more processors 612 and / or one or more I / O devices 616. The device 600 also includes at least one network interface 617. Both I / O interface 614 and network interface 617 are coupled to system bus 611. The at least one network interface can enable the device 600 to communicate via one or more data communication networks 690. Examples of data communication networks include, without limitation, the Internet and local area networks (LANs). The one or more I / O devices 616 can enable a user to provide input to the device 800 via one or more input devices (not shown). The one or more I / O devices 806 can enable information to be provided to a user via one or more output devices (not shown).

[0083] The device 600 can thus include data processing modules that can be executed by one or more processors. The data processing modules can be configured to include instructions to implement at least some of the operations described herein. During operation, the one or more processors launch, run, execute, interpret or otherwise perform the instructions.

[0084] While at least some aspects of the examples described herein with reference to the drawings comprise computer processes performed in processing systems or processors, examples described herein also extend to computer programs, e.g., computer programs on or in computer media or carrier waves, adapted to put Figure 6More, fewer and / or different components than those depicted can be utilized. Device 600 can be located in a single location, or can be distributed across multiple locations. Such locations can be local or remote.

[0085] The techniques described herein can be implemented in software or hardware, or can be implemented using a combination of software and hardware. They can include configuring a device to perform and / or support any or all of the techniques described herein.

[0086] As described in examples herein, a signal processor (e.g., computer processor hardware) is configured to receive data and decode it (“decoder”). The decoder obtains a signal rendition at a first (lower) level of quality, and detects a reservation symbol specifying impairment information. The decoder reconstructs a signal rendition at a second (next higher) level of quality, and can use the impairment information to further process the rendition, such as through signal enhancement operations indicated by the impairment information.

[0087] In certain examples described herein, the reservation symbol can be carried as so-called user data of the encoded data stream. In these examples, the signal processing information is embedded in one or more values received in one or more encoded data layers transmitted within the encoded data stream. The values are associated with transform coefficients of an element of the signal that is processed to derive the signal during decoding, e.g., they can contain values of predefined transform coefficients within a set of different transform coefficients generated by an encoding transform.

[0088] A bit in the bitstream of the encoded data stream can be used to signal that user data is present. The bit can contain a user_data_enabled bit, which can be present in a global configuration header of the encoded data stream. In certain examples, encoding of user data in place of one of the coefficients can be configured as follows. If the bit is set to “0”, the decoder interprets the data as the relevant transform coefficient. If the bit is set to “1”, the data contained in the relevant coefficient is considered to be user data, and the decoder is configured to ignore the data, or the relevant coefficient is considered to carry user data, and a relevant process is performed to extract the data. For example, if the bit is set to “1”, this can indicate that impairment information is being transmitted.

[0089] User data transmitted in this way can be used to enable the decoder to obtain supplemental information, including for example various feature extraction and derivation. Although examples claimed herein relate to impairment information, user data can also be used to signal other optional parameters related to implementations outside of standardized implementations.

[0090] In one case, the user_data_enabled variable can be a k-bit variable. For example, user_data_enabled can contain a 2-bit variable with the following values:

[0091] user_data_enabled Type Value 0 Disabled 1 Enabled 2 bits 2 Enabled 6 bits 3 Reserved

[0092] In this case, the user data specifying the impairment information can be embedded in the last n (least significant) bits of one or more sets of decoded coefficient data (e.g., within the encoded residual coefficient data).

[0093] When user data is enabled (e.g., to transmit signal processing information as described in the examples herein), the "in-loop" processing of the transform coefficients can be modified. Two examples of this aspect are shown in Figure 2 and 4 In addition, the decoding of the transform coefficients can also be adjusted so that when user data is enabled, the value of a particular transform coefficient (e.g., H or HH) is set to 0 before the transform coefficient is subjected to the inverse transform. In the cases listed in the table above, the value of the transform coefficient used to carry the user data can be right shifted (e.g., shifted) by 2 bits (» 2) if n = 2 (e.g., user data enabled = 1) or right shifted (e.g., shifted) by 6 bits (» 6) if n = 6 (e.g., user data enabled = 1) after the least significant n bits are extracted as the reserved symbol. In one case, if the length of the value of the transform coefficient is D bits, where D > n, n is the length of the user data in bits (e.g., 2 or 6 in the table above), the remaining D - n bits of the transform coefficient can be used to carry the value of the transform coefficient (e.g., an integer value that is more heavily quantized compared to a full D-bit representation). In this case, the user data and the value of the transform coefficient can be divided across the D bits. In other, simpler cases, the user data can be extracted and the value of the transform coefficient can be set to 0 (i.e., so that the value of the transform coefficient has no effect on the output of the inverse transform).

[0094] In certain examples, the user data used to implement the reserved symbol can be formatted according to a defined syntax. This defined syntax can divide the user data into header data and payload data. In this case, the decoding of the user data can include parsing a first set of values received in one or more layers of encoded data to extract the header data, and parsing a second set of subsequent values received in one or more layers of encoded data to extract the payload data. The header data can be set to a defined number of bits of the first set. For example, in the user data defined as 2 or 6-bit value examples above, the first x values can include the header data. In one case, x can equal 1 so that the first value of the user data (e.g., the transform coefficient value of the first encoded unit or data block of a given frame or plane of video) defines the header data (e.g., 2 bits or 6 bits of the first value define the header data).

[0095] In certain instances, the header data can indicate at least whether a signal enhancement operation is enabled. Generally, the header data can indicate global parameters of the signal enhancement operation, and the payload data can indicate local parameters of the signal enhancement operation, i.e., whereby impairment information can be localized to one or more coding units containing an mxm block of residual data (e.g., a 2x2 or 4x4 block). Since the impairment information is encapsulated within reserved symbols of a particular coding unit, a tag and / or classification can be applied to a local region of the signal, for example. The tag can indicate a location where a local region of the video signal is to be modified and / or replaced, for example.

[0096] The above embodiments are to be understood as illustrative examples. Further embodiments are envisaged.

[0097] It should be understood that any of the features described in relation to any one embodiment can be used alone, or in combination with other features described, and that the means may, further, be combined with one or more features of any of the other embodiments, or any combination of any other embodiments. Furthermore, equivalents and modifications not described above can also be employed without departing from the scope of the application, which is defined in the claims appended hereto.

Claims

1. A method of performing a signal enhancement operation on one or more portions of a signal, wherein said performing is based at least in part on information embedded in one or more values received in one or more encoded data layers transmitted within an encoded data stream, and wherein said values are associated with transform coefficients intended to be processed by a decoder to derive elements of said signal, wherein said information indicates one or more impairments associated with a portion of said signal; wherein said encoded data stream comprising a rendition of said signal at a first level of quality encoded with a first encoding method; and an encoded level of residual data at a second level of quality, which, in combination with a predicted rendition of said signal at said second level of quality, yields a corrected rendition of said signal at said second level of quality; wherein said one or more values are marked in a set of reserved symbols in a set of residual data of said second level of residual data, and indicate a type and / or location of said one or more impairments.

2. The method of claim 1, wherein said one or more values are interpreted by said decoder to derive said information, rather than values of transform coefficients.

3. The method of claim 1 or 2, wherein said encoded data stream is contained in an encoded bitstream, and wherein bits in said encoded bitstream are used to signal to said decoder that one or more values should be interpreted as said information, rather than actual quantized values of transform coefficients.

4. The method of claim 1 or 2, wherein said information indicates a particular type of impairment expected in the decoded signal.

5. The method of claim 4, wherein said information is used to inform said decoder of an enhancement operation that can be applied to a corresponding region of said signal, in order to improve the quality of the final signal reconstruction.

6. The method of claim 1 or 2, wherein said enhancement operation comprises one or more of the following operations: deblocking, debanding, deringing, denoising, sharpening, dithering, color clipping, range equalization, edge enhancement.

7. The method of claim 6, wherein at least one of said signal enhancement processing operations performed based on embedded signaling is performed in-loop on an intermediate rendition of said signal at a resolution lower than full resolution.

8. The method of claim 1 or 2, wherein said decoder implements signal enhancement operations based on one or more of a target level of processing power to be used by the decoder device and battery power consumption.

9. A method of encoding a signal, comprising: encoding transform coefficients for signal reconstruction by reserving one or more quantized symbols of a given set of coefficients to provide embedded signaling information for a signal enhancement operation to be performed on one or more portions of said signal to correct for one or more impairments detected in said signal; said method further comprising: producing a rendition of said signal at a first level of quality; encoding said rendition of said signal with a first encoding method; producing a predicted rendition of said signal at a second level of quality; generating and encoding a residual data level of the second quality level to be combined with the predicted rendition of the signal at the second quality level to thereby generate a corrected rendition of the signal at the second quality level; and when a process of encoding the signal at the first quality level is detected to generate one or more impairments that cannot be properly corrected with the residual data level at the target bitrate, using a set of reserved symbols in a set of residual data of the residual data level at the second quality level to signal the type and / or location of the detected impairments to a decoder.

10. The encoding method of claim 9, comprising, prior to encoding the transform coefficients: obtaining the rendition of the signal at the first quality level; encoding the rendition of the signal with the first encoding method to generate a first encoded signal; and transforming residual data of one or more quality layers to generate the transform coefficients, the residual data comprising corrections to a version of the signal reconstructed using the first encoded signal; detecting one or more impairments that cannot be properly corrected with the residual data at a target bitrate of the encoded data stream; and in response to the detection of the one or more impairments, modifying one or more quantization signs of a given set of transform coefficients to provide the embedded signaling to signal the signal enhancement operation to be performed on one or more portions of the signal at the decoder.

11. The method of claim 9 or 10, wherein bits in the encoded byte stream are toggled to signal to the decoder that a given set of signs in a given set of residual data are to be interpreted as residual data or as additional embedded signaling information to inform a signal enhancement operation.

12. The method of claim 9 or 10, wherein the embedded signaling information corresponds to one or more of the following impairment categories: banding, ringing, softening, color conversion flip, and quantization noise impairment.

13. The method of claim 9 or 10, wherein the signal is encoded using a layer-based hierarchical encoding method.

14. The method of claim 13, wherein the embedded signaling information is included in the residual data at a resolution lower than the full resolution of the signal.

15. The method of claim 9 or 10, wherein the format in which at least part of the signal and the embedded signaling information are encoded is MPEG-5 Part 2 LCEVC.

16. The method of claim 9 or 10, wherein the format in which at least part of the signal and the embedded signaling information are encoded is SMPTE VC-6 ST-2117.

17. A decoder configured to perform the method of any one of claims 1 to 8.

18. An encoder configured to perform the method of any one of claims 9 to 16.

Images