Low complexity enhanced video coding with signal element modification

Abstract

The signal I1 to be modified is modified using the modification signal I2 to produce a modified signal IM. The modified signal IM has an element corresponding to an element of the signal I1 to be modified. The element of the modified signal IM has a modified value relative to a value of the corresponding element of the signal I1 to be modified. The modified signal IM or a down-sampled modified signal derived based on down-sampling the modified signal IM is transmitted to be encoded. A decoded modified signal BD is received. The decoded modified signal BD is the encoded modified signal IM or a decoded version of the encoded down-sampled modified signal. The decoded modified signal BD is used to generate a processed signal U. Residual data R is generated based on at least a value of an element of the target signal I1 and a value of a corresponding element of the processed signal U.

Description

Technical Field

[0001] The present disclosure relates generally to scalable video coding schemes, and more particularly to encoding and decoding schemes (eg, codecs) with signal element modification capabilities. Background Art

[0002] A scalable encoded video signal can be decoded (by a decoder) at different quality levels (e.g., at different spatial dimensions). This is advantageous because it means that a single encoded video signal can be sent to multiple types of decoding devices (each with different operating capabilities), and each device can decode the encoded video signal according to the operating capabilities of the decoder. For example, a first decoding device may only be able to decode and present the encoded video signal using a base codec, while a second decoding device may be able to decode and present the encoded video signal using a base (or "base layer") codec and an enhancement (or "enhancement layer") codec. If the encoded video signal is not scalable, then a first encoded video signal requiring only a base codec will need to be sent to the first decoding device and a second encoded video signal requiring both a base layer codec and an enhancement layer codec will be sent to the second decoding device. Therefore, it is desirable to have a (single) encoded video signal that can be decoded by a first decoding device using a base codec and by a second decoding device using a base layer codec and an enhancement layer codec.

[0003] However, the first decoding device may in some cases be able to reconstruct a visually acceptable representation of the original video signal using only the base codec, although it does not have the visual quality of a representation reconstructed using both the base layer codec and the enhancement layer codec.

[0004] This may be undesirable in terms of content protection. For example, it may be desirable that decoding using only the base codec does not produce a visually acceptable representation of the original video signal.

[0005] Additionally, as described above, it may be desirable for the second decoding device to receive the same encoded video signal as the first decoding device, but to be able to accurately reconstruct the original video signal using the same base codec and enhancement layer codec as the first decoding device.

[0006] There is therefore a need for scalable video coding schemes and systems and encoded video signals with improved content protection. Summary of the invention

[0007] Various aspects and variations of the invention are set forth in the appended claims. Certain non-claimed aspects are further set forth in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a block diagram illustrating an example signal encoding system.

[0009] Figure 2 is a block diagram illustrating an example signal decoding system.

[0010] Figure 3 It shows that it can be Figure 1 and Figure 2 Schematic diagram of example signals processed in the example system shown.

[0011] Figure 4 is a block diagram illustrating another example signal encoding system.

[0012] Figure 5 It shows that it can be Figure 4 Schematic diagram of example signals processed in an example system and in an example signal decoding system is shown.

[0013] Figure 6 is a block diagram illustrating another example signal encoding system.

[0014] Figure 7 It shows that it can be Figure 6 Schematic diagram of example signals processed in an example system and in an example signal decoding system is shown.

[0015] Figure 8 is a block diagram illustrating another example signal encoding system.

[0016] Fig. 9 is a block diagram illustrating another example signal encoding system.

[0017] Fig.10 It shows that it can be Fig. 9 Schematic diagram of example signals processed in an example system and in an example signal decoding system is shown.

[0018] Fig.11 is a block diagram illustrating another example signal encoding system.

[0019] Fig.12 It shows that it can be Fig.11 Schematic diagram of example signals processed in an example system and in an example signal decoding system is shown.

[0020] Fig.13 is a schematic diagram illustrating example signals that may be processed in an example signal encoding and decoding system.

[0021] Fig.14 is a schematic diagram illustrating the generation of an example video bitstream.

[0022] Fig.15is a schematic diagram illustrating different portions of data that may be generated as part of an example video encoding.

[0023] Fig.16 is a schematic diagram illustrating an encoding system according to an example of the present disclosure. DETAILED DESCRIPTION

[0024] In the detailed description below, measures that provide improved content protection are described. These measures make it difficult or impossible for some decoders to accurately reconstruct the original signal. Such decoders may be able to create one or more versions of the original signal, but these versions have undesirable (from the viewer's perspective) visual artifacts. Other decoders may be able to create one or more versions of the original signal without producing undesirable visual artifacts and can better reconstruct the original signal.

[0025] Without loss of generality, measures are described for modifying a signal by introducing a watermark into the signal. The watermark is present as a visual artifact in a base layer reproduction of the signal. However, the watermark is completely or substantially removed in one or more enhancement layer reproductions of the signal. Without the correct enhancement layer data to remove the watermark, the watermark will appear as an undesirable visual artifact in the output signal.

[0026] Examples are presented herein with reference to a signal being a sequence of samples (i.e., a two-dimensional image, a video frame, a video field, a sound frame, etc.). For simplicity, the non-limiting embodiments described herein generally refer to a signal displayed as a set of 2D planes (e.g., a 2D image in a suitable color space), such as, for example, a video signal. In a preferred case, the signal comprises a video signal. Reference Fig.15 An example video signal is described in more detail.

[0027] The terms "picture", "frame" or "field" will be used interchangeably with the term "image" to indicate a temporal sample of a video signal: any concepts and methods described for a video signal consisting of frames (progressive video signals) can also be easily applied to a video signal consisting of fields (interlaced video signals), and vice versa. Although the embodiments described herein focus on images and video signals, a person skilled in the art can easily understand that the same concepts and methods are also applicable to any other type of multi-dimensional signal (e.g., audio signals, volumetric signals, stereoscopic video signals, 3DoF / 6DoF video signals, plenoptic signals, point clouds, etc.). Although image or video encoding examples are provided, the same methods can be applied to signals with dimensions less than two (e.g., audio or sensor streams) or greater than two (e.g., volumetric signals).

[0028] In the specification, the terms "image", "picture" or "plane" (meaning the broadest sense of "hyperplane", i.e. an array of elements with arbitrary dimensions and a given sampling grid) will be generally used to identify a digital reproduction of signal samples along a sequence of samples, wherein each plane has a given resolution for each of its dimensions (e.g. X and Y) and comprises a set of plane elements (or "signal elements", "elements" or "pixels" or display elements commonly referred to as "pixels" for two-dimensional images, display elements commonly referred to as "voxels" for volumetric images, etc.) characterized by one or more "signal element values", "values" or "settings" (e.g., by way of non-limiting example, a color setting in a suitable color space, a setting indicating a density level, a setting indicating a temperature level, a setting indicating an audio pitch, a setting indicating an amplitude, a setting indicating a depth, a setting indicating an alpha channel transparency level, etc.). Each plane element is identified by a set of suitable coordinates indicating the integer position of the element in the sampling grid of the image. The signal dimensions may contain only spatial dimensions (e.g., in the case of images) or temporal dimensions (e.g., in the case of signals evolving over time, such as video signals). In one case, it can be seen that a frame of a video signal includes a two-dimensional array having three color component channels, or a three-dimensional array having two spatial dimensions (e.g., spatial dimensions indicating resolution - lengths equal to the respective width and height of the frame) and one color component size (e.g., length 3). In some cases, the processing described herein is performed separately for each plane of color component values ​​that make up the frame. For example, planes representing pixel values ​​for each of the Y, U, and V color components can be processed in parallel using the methods described herein.

[0029] Certain examples described herein use a scalability framework that uses base coding and enhancement coding. The video coding system described herein operates on a received decoding of a base coding (e.g., frame-by-frame or complete base coding) and adds one or more of spatial, temporal or other quality enhancements via enhancement layers. The base coding can be generated by a base layer, which can use a different coding scheme than the enhancement layer, and in some cases can include a legacy or comparative (e.g., older) coding standard.

[0030] Example Scalable Coding System

[0031] Figure 1 , Figure 4 , Figure 6 , Figure 8 , Fig. 9 and Fig.11An example system configuration is shown for implementing a spatially scalable coding scheme that uses a downsampled signal encoded using a base codec, adds a first correction or enhancement layer data to the decoded output of the base codec to generate a corrected picture, and then adds another correction or enhancement layer data to an upsampled version of the corrected picture. Thus, the spatially scalable coding scheme can generate an enhancement stream having two spatial resolutions (higher and lower) that can be combined with a base stream at a lower spatial resolution.

[0032] In a spatially scalable coding scheme, the method and apparatus may be based on an overall algorithm that is built upon existing encoding and / or decoding algorithms (e.g., MPEG standards such as AVC / H.264, HEVC / H.265, and non-standard algorithms such as VP9, ​​AV1, etc.) that are used as a baseline for enhancement layers. The enhancement layers operate according to different encoding and / or decoding algorithms. The idea behind the overall algorithm is to encode / decode video frames in a hierarchical manner, as opposed to using a block-based approach as done in the MPEG family of algorithms. Encoding frames in a hierarchical manner includes generating residuals for full frames, and then generating residuals for reduced or decimated frames, etc.

[0033] Various measures (e.g., methods, devices, systems, and computer programs) are described herein, wherein a signal to be modified is modified using a modified signal to produce a modified signal. The signal to be modified may have been modified before or may not have been modified before, but needs to be modified by using a modification signal. The modified signal has elements corresponding to elements of the signal to be modified. These elements may be corresponding because they have the same position (e.g., coordinates) in their respective signals. The elements of the modified signal have modified values ​​relative to the values ​​of the corresponding elements of the signal to be modified. Thus, as a result of the modification, the values ​​are different from each other. The modified signal or the downsampled modified signal derived based on the downsampled modified signal is sent to be encoded. This may involve sending the modified signal or the downsampled modified signal to an encoder. The decoded modified signal is received. The decoded modified signal may be received from a decoder that may be associated with the encoder. The decoded modified signal is a decoded version of the encoded modified signal (i.e., the encoder to which it is sent) or the encoded downsampled modified signal (i.e., the encoder to which it is sent). The decoded modified signal is used to generate the processed signal. The residual data is generated based at least on the values ​​of the elements of the target signal and the values ​​of the corresponding elements of the processed signal. The target signal and the processed signal can take a variety of different forms, as will become more apparent from the following.

[0034] In some examples, such as the following reference Figure 1 , Figure 6 , Figure 8 , Fig. 9 and Fig.11 As described above, the modified signal is downsampled to produce a downsampled modified signal, and the downsampled modified signal is sent to be encoded. In other words, in such examples, the signal to be modified is modified before downsampling. In other examples, such as the following reference Figure 4 As described, the signal to be modified is modified after downsampling.

[0035] In some examples, such as the following reference Figure 1 , Fig. 9 and Fig.11 As described, the target signal includes the signal to be modified. In other examples, such as the following reference Figure 6 , Figure 8 and Fig. 9 As described, the target signal includes the modified signal. In other examples, such as the following reference Figure 4 As described, the target signal includes a signal based on which the signal to be modified is derived.

[0036] In some examples, such as the following reference Figure 1 , Figure 4 , Figure 6 , Figure 8 , Fig. 9 and Fig.11 As described, the residual data, or data derived based on the residual data, is sent to be encoded. The residual data, or data derived based on the residual data, may be sent to an encoder different from the encoder to which the modified signal or the downsampled modified signal is sent.

[0037] In some examples, such as the following reference Fig.11 As described, the correction data is generated based on the following values: (i) the value of the element of the downsampled modified signal, (ii) the value of the corresponding element of the decoded modified signal, and (iii) the value of the corresponding element of the modified signal or the value of the corresponding element of the signal based on the modified signal. Items (i) to (iii) can be used to generate the correction data in one or more operations.

[0038] In some examples, such as the following reference Figure 6 , Fig. 9 and Fig.11 As described, the signal based on the modified signal is derived by processing the modified signal. Such processing can take a variety of different forms. Examples include, but are not limited to, downsampling and / or creating a negative (or "inverse") version.

[0039] In some examples, such as the following reference Figure 1 , Figure 4 , Figure 6 , Figure 8 , Fig. 9 and Fig.11 As described, the correction data, or data derived based on the correction data, is sent to be encoded. The correction data, or data derived based on the correction data, may be sent to an encoder different from the encoder to which the modified signal or the downsampled modified signal is sent.

[0040] As mentioned above, in some examples, such as the following reference Figure 6 , Figure 8 and Fig. 9 As described, the target signal includes the modified signal.

[0041] In some examples, such as the following reference Figure 6 , Figure 8 and Fig. 9 As described, further residual data is generated based on: (i) the value of an element of the residual data, and (ii) the value of a corresponding element of the modified signal or the value of a corresponding element of a signal based on the modified signal.

[0042] In some examples, such as the following reference Figure 6 , Figure 8 and Fig. 9 As described, the additional residual data, or data derived based on the additional residual data, is sent to be encoded. The additional residual data, or data derived based on the additional residual data, may be sent to an encoder different from the encoder to which the modified signal or the downsampled modified signal is sent.

[0043] As mentioned above, in some examples, such as the following reference Figure 4 As described above, the target signal includes the signal to be downsampled.

[0044] In some examples, such as the following reference Figure 4 As described, the signal to be downsampled is downsampled to derive the signal to be modified.

[0045] In some examples, the modifying signal includes an overlay to be applied to the signal to be modified. An overlay may be anything that is overlaid, positioned on, or otherwise applied to the signal to be modified.

[0046] In some examples, the modification signal includes a watermark. A watermark is a superposition. A watermark may include text (e.g., "watermark"), an image, a pattern, etc.

[0047] In some examples, the modified signal has another element corresponding to another element of the signal to be modified, and the modified signal has another element having the same value as the value of the corresponding another element of the signal to be modified. Such other elements can be considered as unmodified elements. Unmodified elements may have relatively small associated residuals, which may be more efficiently encoded than larger residuals.

[0048] In some examples, the modification of the signal to be modified includes combining the signal to be modified with the modification signal. This combination can take a variety of different forms. Examples include, but are not limited to, addition-based combination, subtraction-based combination, and replacement-based combination.

[0049] In some examples, the signal to be modified includes a luminance component and a chrominance component. The luminance component of the signal to be modified is modified based on the modification signal. The chrominance component of the signal to be modified is not modified based on the modified signal.

[0050] In some examples, generating a processed signal using the decoded modified signal includes performing an upsampling operation.

[0051] Various measures (e.g., methods, apparatus, systems, and computer programs) are described herein, wherein a decoded modified signal is processed to obtain a processed signal, and the processed signal and residual data are combined to generate a reconstructed signal. The decoded modified signal is a decoded version of the encoded modified signal or the encoded downsampled modified signal. The downsampled modified signal has been derived based on the modified signal having been downsampled. The modified signal has been generated based on the signal to be modified having been modified using the modified signal. The modified signal has elements corresponding to elements of the signal to be modified. Elements of the modified signal have modified values ​​relative to the values ​​of corresponding elements of the signal to be modified.

[0052] Figure 1 A system configuration for an example spatially scalable encoder 100 is shown. The encoding process is split into two halves as shown by the dotted lines. Below the dotted lines is the base level, and above the dotted lines is the enhancement level, which can be usefully implemented in software. The encoder 100 may include only the enhancement level process, or a combination of the base level process and the enhancement level process as desired. The encoder 100 topology at the general level is as follows.

[0053] The encoder 100 comprises a device for receiving a first input signal I 1 The first input is 10.

[0054] The encoder 100 comprises a second receiving input signal I 2 The second input is 20.

[0055] The encoder 100 includes a modifier 30. The modifier 30 modifies the signal to be modified to generate a modified signal. The signal to be modified may or may not have been modified in some way before being input to the modifier 30. In this example, the signal to be modified includes a first input signal I 1 .

[0056] In the example, the modifier 30 modifies the signal to be modified using (or "based on") another signal. The other signal may be referred to herein as a "modification signal", "modifying signal", etc. In this example, the modification signal includes the second input signal I 2 The modifier 30 outputs the modified signal I M .

[0057] The modifier may be a combined modifier that modifies the signal to be modified by combining the signal to be modified with the modifying signal.Such a combination may take a variety of different forms, examples of which are described below.

[0058] The modifier may be an addition modifier that modifies the signal to be modified by adding the modification signal to the signal to be modified. Such addition may involve the modifier adding the signal element value of the signal element of the modification signal to the signal element value of the corresponding signal element of the signal to be modified. The result of such addition may be the signal element value of the corresponding signal element of the modified signal.

[0059] The modifier may be a subtractive modifier that modifies the signal to be modified by subtracting the modifying signal from the signal to be modified (or vice versa). Such subtraction may involve the modifier subtracting the signal element value of the signal element of the modifying signal from the signal element value of the corresponding signal element of the signal to be modified. The result of such subtraction may be the signal element value of the corresponding signal element of the modified signal.

[0060] The modifier may be a substitution (or "replacement", "exchange", "substitution", "overwrite", etc.) modifier that modifies the signal to be modified by replacing one or more values ​​of the signal to be modified with one or more corresponding values ​​from the modifying signal. Such a replacement may involve the modifier replacing signal element values ​​of signal elements of the signal to be modified with signal element values ​​of corresponding signal elements of the modifying signal. The result of such a replacement may be the signal element value of the corresponding signal element of the modified signal.

[0061] The modifier may be a shift (or "shift") modifier that modifies the signal to be modified by shifting the position of one or more values ​​of the signal to be modified within the signal to be modified. For example, the modified signal may indicate that all values ​​of the signal to be modified are to be shifted right by one signal element. In such an example, the value "one" from the modified signal is not added to or subtracted from any signal element value of the signal to be modified, does not replace any signal element value of the signal to be modified, but does affect the manner in which the signal to be modified is modified.

[0062] Other types of modifiers may be used.

[0063] Thus, in an example, the modified signal has an element (e.g., a signal element) corresponding to an element (e.g., a signal element) of the signal to be modified. The element (e.g., a signal element) of the modified signal has a modified value relative to the value of the corresponding element (e.g., a signal element) of the signal to be modified. Thus, the modified value is different from the value of the corresponding element of the signal to be modified. The modified value may have been modified due to addition, subtraction, substitution, shifting, or any other modification operation.

[0064] The modified input signal I M is provided to the downsampler 105D. The downsampler 105D outputs the modified input signal I based on the downsampling at the base level. M The downsampled signal B is derived. The downsampled signal B can be considered as a modified downsampled signal because it is a modified input signal I M The downsampler 105D outputs (or "sends") a downsampled version of B to an encoder, which in this example is the base encoder 120E at the base level of the encoder 100. The downsampler 105D also outputs to the residual generator 110-S. The encoded base stream B E Created directly by the base encoder 120E and may be quantized and entropy encoded as needed according to the base encoding scheme. Encoded base stream B E It may be referred to as the base layer or base level. The encoded base stream B E can be considered as an encoded modified signal, since it is an encoded version of the downsampled signal B, which itself is a modified input signal I M A downsampled version of .

[0065] To generate the coded sub-layer 1 enhancement stream, the coded base stream B is received from the encoder. E , which in this example is the base encoder 120E. The encoded base stream B E Decoded via a decoding operation applied at the elementary decoder 120D. Decoding generates a decoded elementary stream B DIn a preferred example, the base decoder 120D may be a decoding component that complements an encoding component in the form of a base encoder 120E within a base codec. In other examples, the base decoding block 120D may instead be part of an enhancement layer.

[0066] Then, the decoded elementary stream B D is processed to generate a processed signal. In this example, the processed signal is an upsampled signal U, which will be described in more detail below.

[0067] Via the residual generator 110-S, a decoded elementary stream B output from the elementary decoder 120D is created D The difference between the downsampled input video B and the decoded base stream B (i.e., applying the subtraction operation 110-S to the frame of the downsampled input video B and the decoded base stream B D Thus, the residual generator 110-S uses and processes the decoded base stream B D and the downsampled input video B to generate a set of residuals. Here, the residual represents the error or difference between the reference signal or frame and the expected signal or frame. The residuals used in the first enhancement layer can be regarded as correction signals because they are able to 'correct' the frames of the (future) decoded base stream. In order to distinguish them from the residuals used in the second enhancement layer, the residuals used in the first enhancement layer will be referred to as "correction data" hereinafter. The correction data is useful because it can correct for weird modes or other characteristics of the base codec. These characteristics include, among others, the motion compensation algorithm applied by the base codec, the quantization and entropy coding applied by the base codec, and the block adjustment applied by the base codec.

[0068] exist Figure 1 In, a correction data set is used and processed by transformation, quantization and entropy coding to produce an encoded sub-layer 1 stream. Figure 1 , a transform operation 110-1 is applied to a set of correction data; a quantization operation 120-1 is applied to the transformed set of correction data to generate a set of quantized correction data; and an entropy coding operation 130-1 is applied to the quantized set of correction data to generate a coded sublayer 1 stream of a first enhancement layer level. However, it should be noted that in other examples, only the quantization step 120-1, or only the transform step 110-1 may be performed. Entropy coding may not be used, or may optionally be used in addition to one or both of the transform step 110-1 and the quantization step 120-1. The entropy coding operation may be any suitable type of entropy coding, such as a Huffmann coding operation or a run length encoding (RLE) operation, or a combination of both a Huffmann coding operation and an RLE operation (e.g., RLE followed by Huffmann or prefix coding).

[0069] To generate the coded sub-layer 2 stream, another enhancement information level is created by generating and encoding another set of residuals via the residual generator 100-S.

[0070] The upsampler 105U upsamples the decoded base stream B D to produce a decoded base stream B D The upsampled version U of the corrected version of φ(R) is obtained by upsampling the signal U (via upsampler 105U). The upsampled version U may be referred to as a reference signal, a reference frame, etc. In this example, the upsampled version U is the processed signal mentioned above.

[0071] To achieve this as will be done in a decoder (e.g. Figure 2 1i) reconstructs a corrected version of the decoded base stream generated at the decoder 120D, undoing at least some of the sub-layer 1 encoding operations to emulate the decoder's process, and accounting for at least some of the losses and quirks of the transform and quantization processes. To this end, the correction data set is processed by a decoding pipeline including an inverse quantization block 120-1i and an inverse transform block 110-1i. The quantized correction data set is inverse quantized at the inverse quantization block 120-1i in the encoder 100 and inverse transformed at the inverse transform block 110-1i to regenerate a decoder-side version of the correction data set. The decoded base stream B from the decoder 120D D It is then combined with the decoder-side version of the correction data set (ie, the decoded base stream B D The summation operation 110-C is performed on the decoder-side version of the input video I and the correction data set. The summation operation 110-C generates the modified input video I as it would likely be generated at the decoder. M The reconstructed base codec video is then upsampled by an upsampler 105U to generate an upsampled signal U. The processing in this example is typically performed on a frame-by-frame basis.

[0072] The residual generator 100-S generates another residual set R based on at least the value of the element of the target signal and the value of the corresponding element of the processed signal. The target signal may also be referred to as a target frame, an expected signal, an expected frame, etc. In this example, the target signal is the first input signal I 1 , which is the signal to be modified in this example. As described above, in this example, the processed signal includes the up-sampled signal U.

[0073] Thus, in this example, another residual set R is the decoded base stream B D The upsampled version U (via upsampler 105U) of the corrected version of the (processed signal) is compared with the first input signal I 1 (target signal).

[0074] In more detail, the upsampled signal U (ie, the reference signal or frame) is combined with the first input signal I 1 (i.e., the desired signal or frame) is compared to create another residual set R (i.e., the difference operation is applied by the residual generator 100-S to the upsampled recreated frame U to generate another residual set R). The other residual set R is then processed by an encoding pipeline that is a mirror image of the encoding pipeline for the correction data set to become an encoded sub-layer 2 stream (i.e., the encoding operation is then applied to the other residual set R to generate another encoded enhancement stream). In particular, the other residual set R is transformed (i.e., the transform operation 110-0 is performed on the other residual set R to generate another transformed residual set). The transformed residual is then quantized and entropy encoded in the manner described above in connection with the correction data set (i.e., the quantization operation 120-0 is applied to the transformed residual set to generate another quantized residual set; and the entropy encoding operation 120-0 is applied to the quantized other residual set to generate an encoded sub-layer 2 stream containing another enhancement information level). In some cases, the operation can be controlled, for example so that only the quantization step 120-1, or only the transform and quantization steps can be performed. Optionally, entropy coding may be used as a supplement. Preferably, the entropy coding operation may be a Huffman coding operation or a run length encoding (RLE) operation or both (e.g., RLE followed by Huffman coding). The transform applied to blocks 110-1 and 110-0 may be a Hadamard transform applied to a 2x2 or 4x4 residual block. In this example, the encoder used to encode the further residual set R (or data derived based on the further residual set R) is different from the base encoder 120E used to encode the base stream.

[0075] Figure 1 The encoding operation in does not result in dependencies between local blocks of the input signal (e.g., in contrast to many known encoding schemes that apply inter-frame or intra-frame prediction to macroblocks and thus introduce macroblock dependencies). Figure 1 The operations shown in can be performed in parallel on 4x4 or 2x2 blocks, which greatly improves the encoding efficiency on a multi-core central processing unit (CPU) or graphics processing unit (GPU).

[0076] like Figure 1As shown, the output of the spatial scalable coding process is one or more enhancement streams at an enhancement level, which preferably includes a first enhancement level and another enhancement level. This can then be combined with the base stream at the base level (e.g., via multiplexing or other means). The first enhancement level (sub-layer 1) can be viewed as implementing the corrected video at the base level, that is, for example, correcting the encoder quirks. The second enhancement level (sub-layer 2) can be viewed as another enhancement level that is used to convert the corrected video to the original input video or a close approximation thereof. For example, the second enhancement level can add fine details lost during downsampling and / or help correct errors introduced by one or more of the transform operation 110-1 and the quantization operation 120-1.

[0077] Figure 2 A corresponding example decoder 200 is shown for an example spatially scalable coding scheme. The encoded base stream is decoded at the base decoder 220 to generate a modified input signal I M It should be understood that without the second input signal I 2 In the case of the modifier 30, decoding the encoded elementary stream will produce the first input signal I 1 The basis reconstruction, rather than the modified input signal I M The modified input signal I M The basis reconstruction can be used in practice to provide a modified input signal I M Again, it should be understood that the lower quality level of the visual reproduction is a modified input signal I M (with visual artifacts) instead of the first input signal I 1 The basic reconstructed signal (i.e., the modified input signal I M The main purpose of the basic reconstruction) is to modify the input signal I M As will become more apparent below, the first input signal I 1 To this end, a decoded base stream is provided for sub-layer 1 processing (ie, sub-layer 1 decoding). Figure 2 The sub-layer 1 processing in the decoder 200 includes an entropy decoding process 230-1, an inverse quantization process 220-1, and an inverse transform process 210-1. Optionally, one or more of these steps may be performed only depending on the operations implemented at the corresponding block 100-1 at the encoder. By performing these corresponding steps, the encoded sub-layer 1 stream including the correction data set is made available at the decoder 200. The correction data set is combined with the decoded base stream from the base decoder 220 (i.e., in the decoded base stream B DA summation operation 210-C is performed on the frames of the modified input video B and the frames of the decoded correction data set to generate a reconstruction of a downsampled version of the modified input video B - that is, the reconstructed base codec video). The frames of the reconstructed base codec video are then upsampled by an upsampler 205U.

[0078] In addition, and optionally in parallel, the encoded sublayer 2 stream is processed to produce another set of decoded residuals R. Similar to the sublayer 1 processing, the sublayer 2 processing includes an entropy decoding process 230-0, an inverse quantization process 220-0, and an inverse transform process 210-0. Of course, these operations will correspond to the operations performed at block 100-0 in the encoder 100, and one or more of these steps may be omitted as desired. Block 200-0 produces an encoded sublayer 2 stream including another set of residuals R, and these are summed at operation 200-C with the output U from the upsampler 205U to create the input signal I 1 The sub-layer 2 reconstruction of , which can be provided as the output O of the decoder. Therefore, if Figure 1 and Figure 2 As shown, the output of the decoding process can include up to three outputs: a base reconstruction, a corrected lower resolution signal, and a higher resolution reconstruction O of the original signal.

[0079] Figure 3 It shows that it can be Figure 1 and Figure 2 Example signal element values ​​of an example signal processed in the example system shown.

[0080] The digital signal element values ​​used in this and other examples should be understood to be examples only.

[0081] As mentioned above, the first and second input signals I 1 ,I 2 In this example, the combiner 30 is an additive combiner which combines the first and second input signals I 1 ,I 2 Added together to generate the modified input signal I M In this example, such addition involves adding the first input signal I 1 The signal element values ​​of the signal elements in the second input signal I 2 The signal element values ​​of corresponding signal elements in are added to create a modified input signal I M In this example, the corresponding signal elements have the same positions as each other within their respective signals.

[0082] The downsampler 105D receives and downsamples the modified input signal I Mto generate a downsampled input signal B. In this example, such downsampling involves modifying the input signal I M The signal element values ​​of each signal element in the 2x2 signal element block in A are averaged to give the signal element value of the corresponding signal element in the downsampled input signal B. However, the downsampling may be performed in other ways. In this example, the signal element in the downsampled input signal B corresponds to the signal element in the modified input signal I. M A 2x2 block of signal elements in .

[0083] In this example and other examples, for convenience and simplicity, it is assumed that the decoded basic stream B D is the same as the downsampled input signal B. In other words, assume that the downsampled input signal B is encoded by the base encoder 102E to generate the encoded base stream B E , the encoded basic stream B E Decoded by the elementary decoder 120D to generate a decoded elementary stream B D , and the decoded basic stream B D The signal element values ​​of the signal elements of are the same as the signal element values ​​of the corresponding signal elements of the downsampled input signal B. However, even if the decoded base stream B D A higher resolution reconstruction O of the original signal can still be achieved in a similar manner by differing the signal element values ​​of the signal elements of the downsampled input signal B. In particular, the above-mentioned correction data set can eliminate encoded-decoded errors.

[0084] In this example and other examples, for convenience and simplicity, it is also assumed that no errors are introduced in the sub-layer 1 processing. In other words, it is assumed that the output of the residual generator 110-S is the same as the input from the inverse transform block 110-1i to the summation operation 110-C. Likewise, even if errors are introduced in the sub-layer 1 processing, a higher resolution reconstruction O of the original signal can still be achieved in a similar manner.

[0085] The upsampler 105U upsamples the decoded base stream B D (regardless of the decoded elementary stream B D has been corrected) to generate an upsampled signal U.

[0086] In this example, the signal element values ​​of the signal elements of the upsampled signal U are equal to the signal element values ​​of the first input signal I 1 The signal element values ​​of the corresponding signal elements of are not exactly the same. This may be caused by the asymmetry between downsampling and upsampling.

[0087] The residual generator 100-S calculates the target signal (in this example, the first input signal I 1) and the difference between the processed signal (in this example, the upsampled signal U). In this example, such calculation involves subtracting the first input signal I 1 The signal element values ​​of the signal elements in the upsampled signal U are calculated using the signal element values ​​of the corresponding signal elements in , so as to give the values ​​of the corresponding residual elements in another residual set R.

[0088] As described above, for decoder-side processing, the decoder can use the encoded elementary stream B E and a correction data set to generate an upsampled signal U. The upsampled signal U is combined with another residual set R to create a higher resolution original signal reconstruction O. In this example, such a combination involves adding signal element values ​​of signal elements in the upsampled signal U with signal element values ​​of corresponding residual elements in another residual set R to give signal element values ​​of corresponding residual elements in the original signal reconstruction O.

[0089] It should be understood that although the effect of the modification is not significant in the base video stream (i.e., in the decoded base stream B D and the upsampled signal U), but not in the original signal reconstruction O.

[0090] Thus, if a decoder receives the encoded elementary stream B E , it will be able to restore the first input signal I at the basic level 1 The decoder can also upsample the decoded base stream B D And restore the first input signal I with higher spatial resolution 1 However, without another set of residual data R, the decoder will not be able to fully reconstruct the first input signal I 1 .

[0091] The absolute value of the residual in the lower right 2x2 block in the first residual set R is usually larger than the absolute value of the residual in the other 2x2 blocks in the first residual set R. In this example, this is important because the modification modifies the first input signal I 1 The values ​​of the signal elements in the corresponding 2x2 blocks make the first input signal I 1 and the upsampled signal U in particular differ in this 2x2 block in the first residual set R.

[0092] Since the residuals in the lower right 2x2 block in the first residual set R may have a more significant impact on the visual quality of the original signal reconstruction O than the residuals in other 2x2 blocks in the first residual set R, they can be processed differently from other residuals in the first residual set R. For example, the residuals in the lower right 2x2 block (corresponding to the first input signal I 1The modified signal element ) may not be subjected to any quantization, or at least may be subjected to a lower level of quantization than the other residuals in the first residual set R.

[0093] It will also be appreciated that in many spatially scalable coding schemes and systems, it is desirable to minimize the values ​​of all residuals in the first residual set R. This may make encoding such residuals more efficient and may reduce the amount of residual data to be processed.

[0094] Instead, the modification described above violates such general expectations by increasing the magnitude of the values ​​of some of the residuals in the first set of residuals R compared to the magnitude such values ​​would have had without such modification. As described above, a benefit of such modification is improved content protection.

[0095] In this example, the first, second and modified input signals I 1 ,I 2 ,I M is shown as a 4x4 array. In reality, the signal may have larger dimensions.

[0096] Furthermore, in this example, the modification occurs in the 2x2 block at the bottom right corner of the array. In the case where the signal may have much larger dimensions, the modification may occur in a more central and visually salient part of the signal to maximize the (negative) impact of the modification on the viewer of the base layer stream without enhancement.

[0097] In this example, based on the first input signal I 1 The value of the corresponding element of the lower right 2x2 block selects the second input signal I 2 The values ​​of the elements in the lower right 2x2 block make the modified input signal I M The values ​​of the elements in the lower right 2x2 block of are all '10'. For this example only, assuming that '10' is the maximum signal element value, it can be independent of the first input signal I 1 The value of the corresponding element in the lower right 2x2 block is used to select the second input signal I 2 The values ​​of the elements in the lower right 2x2 block of ; for example, they can all be set to '10'. 1 ,I 2 The modified input signal I is then added and if the sum of the values ​​of the corresponding signal elements exceeds 10. M+ The value of the corresponding signal element may be set to a maximum value of '10'.

[0098] Figure 4 A system configuration of another example spatially scalable encoding system 100 is shown.

[0099] Figure 4 The example spatial scalable encoder 100 shown is similar to Figure 1 An example spatially scalable encoder 100 is shown.

[0100] However, in Figure 1 In the example spatial scalable encoder 100 shown, the modifier 30 is in the enhancement layer and the second input signal I 2 is an enhancement level signal. The modifier 30 uses the second input signal I in the enhancement level 2 To modify the first input signal I 1 To generate a modified input signal I in the enhancement layer M , and then downsample the modified input signal I M To generate a downsampled (modified) signal B.

[0101] On the contrary, Figure 4 In the example spatial scalable encoder 100 shown, the modifier 30 is in the base layer and the second input signal I 2 Is the base level signal. The first input signal I 1 is downsampled to generate a downsampled signal B, and the modifier 30 uses the second input signal I 2 Modify the downsampled signal B in the base layer to generate a modified downsampled signal B M .

[0102] In addition, in this example, the target signal based on which the residual generator 100-S generates the residual data set R includes a signal to be downsampled. In this example, the signal to be downsampled is the first input signal I 1 The signal to be downsampled, i.e. the first input signal I 1 , is downsampled to derive the signal to be modified, namely the downsampled signal B.

[0103] Figure 5 It shows that it can be Figure 4 Example signal element values ​​for an example signal processed in an example system and corresponding decoder are shown.

[0104] As mentioned above, the first input signal I 1 is downsampled to create a downsampled signal B, for example by averaging.

[0105] In this example, the combiner 30 is an additive combiner that combines the downsampled signal B and the second input signal I 2 Added to generate a modified downsampled signal B M The base encoder 120E encodes the modified downsampled signal B M To generate the encoded elementary stream B E And the elementary decoder 120D decodes the encoded elementary stream B E To generate the decoded elementary stream B DIn this specific example, for convenience and simplicity, it is again assumed that the decoded elementary stream B D and the modified downsampled signal B M The same can be expressed mathematically as: D =B M .

[0106] The output B of the residual generator 110-S D1 is the downsampled signal B and the decoded downsampled signal B D The difference between the two can be expressed mathematically as: D1 =B–B D =B–B M In this example, the modified downsampled signal B M is the downsampled signal B and the second input signal I 2 The combination of can be expressed mathematically as: M =B+I 2 Combining these two mathematical expressions, B D1 =B–B M =B–(B+I 2 )=-I 2 .

[0107] Output B of sum operation 110-C D2 is the decoded elementary stream B D (which is equal to the modified downsampled signal B M ) and the output B of the residual generator 110-S D1 The sum of can be expressed mathematically as: D2 =B M +B D1 However, from the above, B D1 =BB M Therefore, B D2 =B M +B D1 =B M +(BB M )=B. Therefore, the second input signal I 2 The resulting modification is actually already in the output B of the summation operation 110-C. D2 was removed.

[0108] Output B of sum operation 110-C D2 Then it is upsampled to generate an upsampled signal U, and based on the first input signal I 1 Another residual set R is calculated by taking the difference between the upsampled signal U and the upsampled signal U.

[0109] Again, in terms of decoder-side processing, the decoder can use the encoded elementary stream BE The upsampled signal U is generated by combining the upsampled signal U with another set of residuals R to create a higher resolution reconstruction O of the original signal. In addition, although the effect of the modification is not visible in the base video stream (i.e., in the decoded base stream B M The absolute value of the residual in the lower right corner is Figure 4 The 2x2 blocks in the first residual set R shown are smaller than Figure 1 The lower right 2x2 block in the first residual set R shown. This is because, in fact, after 1 Before the upsampled signal U is subtracted from the first input signal I, the modification has been undone. Therefore, the upsampled signal U may be more similar to the first input signal I 1 , and the absolute value of the residuals in the first residual set R may therefore be smaller. As described above, this can lead to more efficient residual coding.

[0110] Figure 6 A system configuration of another example spatially scalable encoder 100 is shown.

[0111] Figure 6 The example spatial scalable encoder 100 shown is similar to Figure 1 An example spatially scalable encoder 100 is shown.

[0112] However, in Figure 1 In the example spatial scalable encoder 100 shown, the residual generator 100-S calculates the first input signal I 1 and the upsampled signal U.

[0113] On the contrary, Figure 6 In the exemplary spatial scalable encoder 100 shown, the residual generator 100-S calculates the modified input signal I M The difference between (target signal) and the upsampled signal U. This creates a first further set of residuals R 1 .

[0114] in addition, Figure 6 The illustrated example spatial scalable encoder 100 comprises a further modifier 40 which uses a third input signal I 3 To modify the output of the residual generator 100-S. In this example, the third input signal I 3 is the second input signal I 2 The negative value (or "reciprocal") of , which can be written mathematically as: 3 =-I 2 This creates a second set of residuals R 2 .

[0115] Figure 7It shows that it can be Figure 6 Example signal element values ​​for an example signal processed in an example system and corresponding decoder are shown.

[0116] The first other residual set R 1 is calculated as the modified input signal I M The difference between the upsampled signal U and the upsampled signal U. Thus, in this example, a first further residual set R is generated based on it 1 The target signal includes the modified input signal I M .

[0117] The second residual set R 2 is calculated as the first another residual set R 1 With the third input signal I 3 Therefore, based on the first another residual set R 1 The value of the element and based on the modification signal I 2 The value of the corresponding element of the signal is used to generate another residual data (ie, the second another residual set R 2 ). In this example, the signal I is modified by processing 2 To obtain the modified signal I 2 In this example, such processing involves calculating the modified signal I 2 The negative value (or "inverse") of 3 .

[0118] It should be noted that Figure 7 The second residual set R shown in 2 Corresponds to Figure 3 The residual set R shown in Figure 1 and Figure 6 The configurations of the example spatial scalable encoders 100 shown in FIGS.

[0119] Figure 8 A system configuration of another example spatially scalable encoder 100 is shown.

[0120] Figure 8 The example spatial scalable encoder 100 shown is similar to Figure 6 An example spatially scalable encoder 100 is shown.

[0121] However, in Figure 6 In the example spatial scalable encoder 100 shown, another modifier 40 converts the third input signal I 3 (ie, the second input signal -I 2 negative value) and the first other residual set R 1 Add together to generate a second residual set R2. This can be expressed mathematically as: 2=I 3 +R 1 =(-I 2 )+R 1 =R 1 –I 2 .

[0122] On the contrary, Figure 8 In the example spatial scalable encoder 100 shown, the further modifier 40 subtracts the second input signal I from the first further residual set R1. 2 To generate the second residual set R 2 This can be expressed mathematically as: R 2 =R 1 –(I 2 )=R 1 –I 2 .

[0123] Thus, in this example, the second further residual set R 2 is calculated as the first another residual set R 1 and modify the signal I 2 Therefore, based on the first another residual set R 1 The value of the element and the modification signal I 2 The value of the corresponding element of generates another residual data.

[0124] Therefore, it can be seen that Figure 6 and Figure 8 The example spatial scalable encoder 100 shown produces encoded elementary streams B that are identical to each other. E , a correction data set and a first and a second further residual set R 1 , R 2 .

[0125] Fig. 9 A system configuration of another example spatially scalable encoder 100 is shown.

[0126] Fig. 9 The example spatial scalable encoder 100 shown is similar to Figure 6 An example spatially scalable encoder 100 is shown.

[0127] However, in Figure 6 In the example spatial scalable encoder 100 shown, the modifiers 30 and 40 are additive modifiers. In particular, the first and second input signals I 1 ,I 2 Add, and the first residual set R 1 With the third input signal I 3 The third input signal is the second input signal -I 2 A negative value.

[0128] On the contrary, Fig. 9 In the example spatially scalable encoder 100 shown, the modifiers 30, 40 are replacement modifiers.

[0129] In particular, the modifier 30 uses the second input signal I 2 The signal element value of the corresponding signal element in the first input signal I is replaced (also referred to as "replacing" or "overwriting") 1 The signal element values ​​of the signal elements in the modified input signal I M The signal element value of the corresponding signal element in .

[0130] In this example, the first input signal I 1 All signal element values ​​of the signal elements in are not overwritten. However, in this example, the overwritten first input signal I 1 The signal element values ​​of the signal elements in form the third input signal I 3 Thus, in this example, the first input signal I 1 is processed to derive a third input signal I 3 .

[0131] In addition, another modifier 40 uses the third input signal I 3 The signal element value of the corresponding signal element is overwritten with the modified input signal I M The first residual set R corresponding to the modified signal element 1 Any value for the residual element in .

[0132] Again, in this example, the first residual set R 1 All values ​​of all residual elements in are not covered by another modifier 40; only those that are consistent with the modified input signal I M The residual elements corresponding to the modified signal elements of .

[0133] Fig.10 It shows that it can be Fig. 9 Example signal element values ​​for various example signals processed in the example system shown.

[0134] In this example, the modifier 30 uses the second input signal I 2 The signal element values ​​"10", "10", "10", and "10" in the lower right 2x2 block of the first input signal I 1 The modifier 40 uses the third input signal I 3 The signal element values ​​"7", "8", "8" and "9" in the lower right 2x2 block of 1The residual values ​​in the lower right 2x2 block are “2”, “2”, “2”, and “1”.

[0135] Instead of the third input signal I 3 Store the first input signal I 1 The signal element values ​​are "7", "8", "8" and "9", and the third input signal I 3 The first residual element set R can be indicated 1 Which residuals in will be derived from the first input signal I 1 The corresponding signal element value is overwritten.

[0136] Fig.11 A system configuration of another example spatially scalable encoding system 100 is shown.

[0137] Fig.11 The example spatial scalable encoder 100 shown is similar to Figure 1 An example spatially scalable encoder 100 is shown.

[0138] However, in Figure 1 In the illustrated example spatial scalable encoder 100, the residual generator 110-S outputs to a transform operation 110-1.

[0139] On the contrary, Fig.11 In the example spatial scalable encoder 100 shown, the residual generator 110-S outputs to another modifier 40. The other modifier 40 receives the third input signal I 3 As input 50. Another modifier 40 uses the third input signal I 3 The output of the residual generator 110 -S is modified and the result is output to the transform operation 110 - 1 .

[0140] Thus, in this example, correction data is generated based on: (i) the values ​​of the elements of the downsampled modified signal B, (ii) the decoded modified signal B D The value of the corresponding element of, and (iii) modifying the signal I 2 The value of the corresponding element or the signal I based on the modification signal 3 In this particular example, a signal based on the modification signal, i.e., the third input signal I 3 , but in other examples, the modification signal I 2 .

[0141] In this example, the signal based on the modification signal, that is, the third input signal I 3 , is to modify the signal I by processing 2 In this particular example, such processing involves downsampling the modified signal I 2and creates negative values ​​of the downsampled modified signal, as will become more apparent below.

[0142] The correction data or data derived based on the correction data may be sent to be encoded.

[0143] Fig.12 It shows that it can be Fig.11 Example signal element values ​​for an example signal processed in an example system and corresponding decoder are shown.

[0144] In this example, the combiner 30 is an additive combiner which combines the first and second input signals I 1 ,I 2 Added together to generate the modified input signal I M , and the downsampler 105D downsamples the modified input signal I M To generate a downsampled signal B. The base encoder 120E encodes the downsampled signal B to generate an encoded base stream B E And the elementary decoder 120D decodes the encoded elementary stream B E To generate the decoded elementary stream B D In this specific example, for convenience and simplicity, it is again assumed that the decoded elementary stream B D Same as the downsampled signal B, which can be expressed mathematically as: D =B.

[0145] The output B of the residual generator 110-S D1 is the downsampled signal B and the decoded downsampled signal B D The difference between the two can be expressed mathematically as: D1 =B–B D Since in this example, B D =B, so B D1 =0.

[0146] In this example, the third input signal I 3 Is the second input signal -I 2D The third input signal I 3 By downsampling the second input signal I 2 And produce the negative value of the result to obtain.

[0147] In this example, the modifier 40 adds a third input signal I 3 To modify the output B of the residual generator 110-S D1 To give output B D2 , where: B D2 =B D1 +(-I 2D )=0+(-I 2D)=-I 2D .

[0148] Output B of sum operation 110-C D3 is the decoded base signal B D With the output B of modifier 40 D2 The sum of can be expressed mathematically as: D3 =B D +B D2 However, due to B D is through downsampling I M Generated, where I M =I 1 +I 2 , so the summation operation 110-C effectively cancels out the downsampled I 2 , leaving the output B of the summation operation 110-C D3 As the downsampled I 1 .

[0149] Output B of sum operation 110-C D3 , that is, the downsampled I 1 , and then upsampled to generate an upsampled signal U, and based on the first input signal I 1 Another residual set R is calculated by taking the difference between the upsampled signal U and the upsampled signal U.

[0150] Again, in terms of decoder-side processing, the decoder can use the encoded elementary stream B E The upsampled signal U is combined with another set of residuals R to create a higher resolution reconstruction O of the original signal. In addition, although the effect of the modification is not visible in the base video stream (i.e., in the decoded base stream B D ), but not in the original signal reconstruction O.

[0151] By comparing the 2x2 block in the lower right corner, we can see that Fig.12 The absolute value ratio of the residual set R in Figure 3 The absolute value of the residual set R in is smaller. Therefore, Fig.12 The coding efficiency of the residual set R in may be better than Figure 3 The residual set shown in is more efficient.

[0152] Fig.13 Example signal element values ​​of an example signal that may be processed in an example signal encoding and decoding system are shown.

[0153] Fig.13The top portion above the dotted line shows how an unmodified example input signal I can be downsampled to generate a downsampled signal B, the downsampled signal B can be upsampled to create an upsampled signal U, and an additional residual set R can be generated based on the difference between the input signal I and the upsampled signal U.

[0154] Fig.13 The bottom dashed line shows an example first input signal I 1 (Its Fig.13 How can the input signal I at the top be modified using the shift modifier to generate the modified input signal I M In this example, the shift modifier is implemented by shifting the first input signal I 1 Each value in is shifted right by one signal element to modify the first input signal I 1 In each row, the final value becomes the new first value. For example, in the first row, the first input signal I 1 The value of the first signal element of becomes the modified input signal I M The value of the second signal element of the first input signal I 1 The value of the fourth signal element of becomes the modified input signal I M The value of the first signal element of , and so on.

[0155] Even if the decoder receives Fig.13 The encoded version of the base video stream B in the bottom part and can be used Fig.13 The decoded version of the base video stream B in the bottom part is used to generate Fig.13 The upsampled signal U in the bottom part still needs Fig.13 The first residual set R in the bottom part is used to recover Fig.13 The first input signal I 1 .Will Fig.13 The first residual set R of the top part is applied to Fig.13 The upsampled signal U of the bottom part will not be restored Fig.13 The first input signal I 1 .in addition, Fig.13 The base video stream B and the upsampled signal U in the bottom part are not the first input signal I 1 A particularly accurate reproduction of and contains visual artifacts due to shift modifications.

[0156] Thus, the shift modifier can also provide content protection.

[0157] The shift modifier can apply shifts of different sizes and / or directions to different encodings. This can provide a further degree of content protection.

[0158] In more detail, the first input signal I 1 The first encoding may use a first shift (in magnitude and / or direction) and the first input signal I 1 The second encoding of I may use a second different shift (in terms of size and / or direction). The residual data R associated with the first encoding will not match (or "align") with the underlying video stream B of the second encoding. Similarly, the residual data R associated with the second encoding will not match the underlying video stream B of the first encoding. Thus, even if the decoder is able to extract the residual data from the first input signal I 1 If one of the first and second encodings of the base video stream B has access to the base video stream B, they will also need the corresponding residual data R from the same encoding to generate a matched (or "aligned") reconstructed first input signal I 1 .

[0159] In addition, the first and second encodings may use different codecs, for example to encode the residual data R. If the decoder uses an incorrect codec to decode the residual data R (for example, if the decoder uses a codec for different encodings), the first input signal I 1 Any reconstruction may also be related to the first input signal I 1 No match.

[0160] The shift amount can be a smaller number, such as three or less. The shift direction can be up, down, left, right, or other directions.

[0161] By using small shifts, the residual values ​​can be kept relatively small for efficient encoding while still achieving content protection.

[0162] In this example, the shift modifier shifts the first input signal I 1 To generate the modified input signal I M , and then downsample it. However, in other examples, and with the same Figure 4 The first input signal I is downsampled in a manner similar to that described above. 1 , and a shift modifier shifts the resulting downsampled signal (possibly multiple times for different encodings using different shifts).

[0163] Fig.14 An alternative representation of a scalable coding scheme is shown in the form of an example signal coding system 300. The signal coding system 300 is a multi-layer or layer-based coding system in that the signal is encoded via multiple bit streams, each bit stream representing a different encoding of the signal at a different quality level (e.g., a different spatial resolution). Fig.14In the example of , there is a base layer 301 and an enhancement layer 302. The enhancement layer 302 (as well as the above-mentioned enhancement layers) can implement an enhanced coding scheme such as LCEVC. LCEVC is described in PCT / GB2020 / 050695 and related standard specification documents (including ISO / IEC 23094-2Ed 1 Low Complexity Enhancement Video Coding Text published in November 2021). Both documents are incorporated herein by reference. According to Figure 1 , Figure 2 , Figure 4 , Figure 6 , Figure 8 , Fig. 9 and Fig.11 For example, Fig.14 In, the enhancement layer 301 includes two sublayers: a first sublayer 303 and a second sublayer 304. Each layer and sublayer may be associated with a specific quality level. As used herein, a quality level may refer to one or more of the following: sampling rate, spatial resolution, and bit depth, etc. In LCEVC, the base layer 301 is at a base quality level, the first sublayer 303 is at a first quality level, and the second sublayer 304 is at a second quality level. The base quality level and the first quality level may include a common (i.e., shared or identical) quality level or different quality levels. In the case where the quality levels correspond to different spatial resolutions, such as in LCEVC, the input of each quality level may be obtained by downsampling and / or upsampling from another quality level. For example, a first quality level may be at a first spatial resolution, and a second quality level may be at a second higher spatial resolution, wherein the signal may be converted between quality levels by downsampling from the second quality level to the first quality level and by upsampling from the first quality level to the second quality level.

[0164] exist Fig.14 , the corresponding encoder 305 and decoder 306 portions of the signal encoding system 300 are illustrated. It should be noted that the encoder 305 and decoder 306 can be implemented as separate products, and the encoder and decoder need not originate from the same manufacturer or be provided as a single combined unit. The encoder 305 and decoder 306 are typically implemented in different geographical locations, thereby generating an encoded data stream for transmitting an input signal between the two locations. Each of the encoder 305 and decoder 306 can be implemented as part of one or more codecs - hardware and / or software entities capable of encoding and decoding signals. Reference to signal transmission as described herein also covers the encoding and decoding of files, where the transmission can be performed in time on a common machine (e.g., by generating an encoded file and accessing it at a later point in time), or by physical transmission on a medium between two devices.

[0165] In certain preferred implementations, the components of the base layer 301 may be provided separately to the components of the enhancement layer 302; for example, the base layer 301 may be implemented by a hardware accelerated codec, while the enhancement layer 302 may include a software implemented enhancement codec. The base layer 301 includes a base encoder 310. The base encoder 310 receives a version of an input signal to be encoded 306 (e.g., a signal that has undergone one or two rounds of downsampling) and generates a base bitstream 312. The base bitstream 312 is transmitted between the encoder 305 and the decoder 306. At the decoder 306, the base decoder 314 decodes the base bitstream 312 to generate a reconstruction of the input signal at a base quality level 316.

[0166] Both enhancement sublayers 303 and 304 include a set of common encoding and decoding components. The first sublayer 303 includes a first sublayer transform and quantization component 320 that outputs a set of first sublayer transform coefficients 322. The first sublayer transform and quantization component 320 receives data 318 derived from an input signal of a first quality level and applies a transform operation. The data may include a first residual set as described above. The first sublayer transform and quantization component 320 may also apply variable levels of quantization to the output of the transform operation (including being configured not to apply quantization). Quality scalability may be applied by changing the quantization applied in one or more enhancement sublayers. The first sublayer transform coefficient group 322 is encoded by a first sublayer bitstream encoding component 324 to generate a first sublayer bitstream 326. The first sublayer bitstream 326 is transmitted from the encoder 305 to the decoder 306. At the decoder 306, the first sublayer bitstream 326 is received and decoded by a first sublayer bitstream decoder 328 to obtain a set of decoded first sublayer transform coefficients 330. The decoded first sublayer transform coefficient group 330 is passed to the first sublayer inverse transform and inverse quantization component 332. The first sublayer inverse transform and inverse quantization component 332 applies further decoding operations, which include applying an inverse transform operation to at least the decoded first sublayer transform coefficient group 330. If the encoder 305 has applied quantization, the first sublayer inverse transform and inverse quantization component 332 can apply an inverse quantization operation before inverse transforming. Further decoding is used to generate a reconstruction of the input signal. In one case, the output of the first sublayer inverse transform and inverse quantization component 332 is a reconstructed first residual set 334, which can be combined with the reconstructed base stream 316 as described above.

[0167] In a similar manner, the second sublayer 304 also includes a second sublayer transform and quantization component 340 that outputs a set of second sublayer transform coefficients 342. The second sublayer transform and quantization component 340 receives data derived from the input signal of the second quality level and applies a transform operation. In some embodiments, the data may also include residual data 338, although the residual data may be residual data different from the residual data received by the first sublayer 303, for example, it may include another residual set as described above. The transform operation may be the same transform operation applied at the first sublayer 303. The second sublayer transform and quantization component 340 may also apply a variable level of quantization before the transform operation (including being configured not to apply quantization). The second sublayer transform coefficient group 342 is encoded by a second sublayer bitstream encoding component 344 to generate a second sublayer bitstream 346. The second sublayer bitstream 346 is transmitted from the encoder 305 to the decoder 306. In one case, at least the first sublayer bitstream 326 and the second sublayer bitstream 346 may be multiplexed into a single encoded data stream. In one case, the three bitstreams 312, 326, and 346 can all be multiplexed into a single encoded data stream. The single encoded data stream can be received at the decoder 306 and demultiplexed to obtain each individual bitstream.

[0168] At the decoder 306, the second sublayer bitstream 346 is received and decoded by the second sublayer bitstream decoder 348 to obtain a set of decoded second sublayer transform coefficients 350. As described above, the decoding here involves bitstream decoding and can form part of a decoding pipeline (i.e., the decoded transform coefficient groups 330 and 350 can represent a set of partially decoded values ​​that are further decoded by further operations). The decoded second sublayer transform coefficient group 350 is passed to the second sublayer inverse transform and inverse quantization component 352. The second sublayer inverse transform and inverse quantization component 352 applies further decoding operations, which include applying at least an inverse transform operation to the decoded second sublayer transform coefficient group 350. If the encoder 305 has applied quantization at the second sublayer, the second sublayer inverse transform and inverse quantization component 352 can apply an inverse quantization operation before inverse transforming. Further decoding is used to generate a reconstruction of the input signal. This can include outputting a reconstruction of another residual set 354 for combination with the reconstruction of the first residual set 334 and the upsampled combination of the base stream 316 (e.g., as described above).

[0169] The bitstream encoding components 324 and 344 can implement a configurable combination of one or more of entropy encoding and run-length encoding. Likewise, the bitstream encoding components 328 and 348 can implement a configurable combination of one or more of entropy encoding and run-length decoding.

[0170] Further details and examples of the two sub-layer enhancement encoding and decoding systems can be obtained from the published LCEVC documents.

[0171] Typically, the examples described herein operate within a coding and decoding pipeline that includes at least a transform operation. The transform operation may include a DCT or a variant of the DCT, a fast Fourier transform (FFT), or a Hadamard transform such as implemented by LCEVC. The transform operation may be applied block by block. For example, the input signal may be divided into a plurality of different continuous signal portions or blocks, and the transform operation may include a matrix multiplication (i.e., a linear transform) applied to the data from each of these blocks (e.g., as shown in a 1-dimensional vector). In this specification and in the art, it can be said that the transform operation produces a set of values ​​of a predefined number of data elements, which, for example, represent positions in a transformed composite vector. These data elements are referred to as transform coefficients (or sometimes simply referred to as "coefficients").

[0172] As described herein, in the case where the signal data includes residual data, a set of reconstructed coefficient bits may include transformed residual data, and the decoding method may further include indicating a combination of the residual data obtained by further decoding of the reconstructed coefficient bit group with a reconstruction of the input signal generated by a representation of the input signal at a lower quality level to generate a reconstruction of the input signal at a first quality level. The representation of the input signal at a lower quality level may be a decoded base signal (e.g., from a base decoder 314), and the decoded base signal may be optionally upgraded before being combined with the residual data obtained by further decoding of the reconstructed coefficient bit group, the residual data being at a first quality level (e.g., a first resolution). Decoding may further include receiving and decoding residual data associated with the second sublayer 304, for example, obtaining the output of the inverse transform and inverse quantization component 352 and combining it with data derived from the reconstruction of the input signal at the first quality level described above. The data may include data derived from an upgraded version of the reconstruction of the input signal at the first quality level, i.e., upgraded to a second quality level.

[0173] Although the examples have been described with reference to a layer-based layered coding scheme in the form of LCEVC, the methods described herein may also be applied to other layer-based layered coding schemes, such as VC-6 as described in PCT / GB2018 / 053552:SMPTE VC-6ST-2117 and / or associated published standard documents, both of which are incorporated herein by reference.

[0174] Fig.15 shows how a video signal can be decomposed into different components and then encoded. Fig.15In the example of , a video signal 402 is encoded. The video signal 402 includes a plurality of frames or pictures 404, for example, where the plurality of frames represent an action over time. In this example, each frame 404 is composed of three color components. The color components can be in any known color space. Fig.15 In , the three color components are Y (luminance), U (first chrominance opponent color) and V (second chrominance opponent color). Each color component can be considered as a plane of values ​​408. The plane 408 can be decomposed into a set of n×n signal data blocks 410. For example, in LCEVC, n can be 2 or 4; in other video coding techniques n can be 8 to 32.

[0175] In some examples, the signal to be modified includes a luma component (e.g., a Y component) and chroma components (e.g., U and V components). In some such examples, the luma component of the signal to be modified is modified based on the modified signal. However, the chroma components of the signal to be modified are not modified based on the modified signal. Modifying only the luma component may still produce the visual artifacts described herein. However, modifying only the luma component may result in less processing than modifying all three components while still providing the desired content protection.

[0176] In LCEVC and certain other coding techniques, the video signal fed back to a base layer such as 301 is a degraded version of the input video signal 302. In this case, the signals fed back to the two sub-layers include a residual signal including residual data. The residual data plane may also be organized into n×n signal data block groups 410. The residual data may be generated by comparing data derived from the encoded input signal (e.g., video signal 402) and data derived from a reconstruction of the input signal, the reconstruction of the input signal being generated from a representation of the input signal at a lower quality level. Fig.14In the example of , the reconstruction of the input signal may include decoding of the coded base bitstream 312 available at the encoder 305. Such decoding of the coded base bitstream 312 may include a lower resolution video signal, which is then compared to a video signal downsampled from the input video signal 402. The comparison may include subtracting the reconstruction from the downsampled version. The comparison may be performed on a frame-by-frame (and / or block-by-block) basis. The comparison may be performed at a first quality level; if the base quality level is lower than the first quality level, the reconstruction from the base quality level may be upgraded before the comparison. In a similar manner, the input signal of the second sub-layer, such as the input of the second sub-layer transform and quantization component 340, may include residual data generated by comparing the input video signal 402 at a second quality level (which may include a full quality original version of the video signal) and the reconstruction of the video signal at the second quality level. As previously described, the comparison may be performed on a frame-by-frame (and / or block-by-block) basis and may include subtraction. The reconstruction of the video signal may include a reconstruction generated by decoding of the coded base bitstream 312 and a decoded version of the first sub-layer residual data stream. The reconstruction may be generated at a first quality level and may be upsampled to a second quality level.

[0177] Therefore, the data plane 408 of the first sublayer 303 may include residual data arranged in n×n signal blocks 410 . Fig.15 Such a 2×2 signal block is shown in more detail in FIG. 4 (for ease of explanation, n is chosen to be 2), where for the color plane, the block may have a value 412 of a set bit length (e.g., 8 or 16 bits). Each n×n signal block may be represented as a length of n 2 The flattened vector 414 represents the signal data block. To perform the transform operation, the flattened vector 414 can be multiplied by the transform matrix 416 (i.e., the dot product is taken). This then generates a length n 2 Another vector 418 representing different transform coefficients for a given signal block 410. Fig.15 An example similar to LCEVC is shown, where the transform matrix 416 is a Hadamard matrix of size 4x4, resulting in a transform coefficient vector 418 having four elements with respective values. These elements are sometimes referred to by the letters A, H, V, and D, because they can represent the average, horizontal difference, vertical difference, and diagonal difference. This transform operation may also be referred to as directional decomposition. When n=4, the transform operation may use a 16×16 matrix and is referred to as directional decomposition squared.

[0178] like Fig.15As shown, a set of values ​​for each data element in the entire group of signal blocks 410 of the plane 408 can itself be represented as a plane or surface 420 of coefficient values. For example, the values ​​of the "H" data elements of the signal block group can be combined into a single plane, where the original plane 408 is then represented as four separate coefficient planes 422. For example, the coefficient plane 422 shown contains all "H" values. These values ​​are stored in a predefined bit length (e.g., bit length B), which can be 8, 16, 32 or 64 depending on the bit depth. A 16-bit example is given below, but this is not limiting. Thus, the coefficient plane 422 can be represented as a sequence of 16-bit or 2-byte values ​​424 (e.g., in memory), which represents the value of one data element from the transformed coefficient. These values ​​can be called coefficient bits.

[0179] Fig.16 An example encoder is shown that may be configured to perform any of the methods described herein.

[0180] Fig.16 An example encoder 1200 is shown that includes a controller 1205 to receive (1203) at least one input signal 1201. The controller 1205 may receive one input signal 1201, two input signals 1201, three input signals 1201, or more than three input signals 1201. The controller 1205 is communicatively coupled to a base encoder 1213 and an enhancement encoder 1211. The controller 1205 is configured to make some or all of the at least one input signal 1201 (and / or one or more signals derived from the at least one input signal 1201) and a base encoding configuration 1207 available to the base encoder and to make some or all of the at least one input signal 1201 (and / or one or more signals derived from the at least one input signal 1201) and an enhancement encoding configuration 1209 available to the enhancement encoder 1211. The enhancement encoder 1211 receives a decode 1215 of the output of the base encoder after being applied to the input video to perform enhancement encoding of the input video. Fig.15 In , the base encoder 1213 forms part of a base codec that includes a base encoder and a base decoder components (although these may be provided separately). The base encoder 1213 generates an encoded base stream 1221 and the enhancement encoder generates an encoded enhancement stream 1217.

[0181] The techniques described herein may be implemented in software or hardware, or may be implemented using a combination of software and hardware. They may include configuring a device to implement and / or support any or all of the techniques described herein.

[0182] The above examples should be understood to be illustrative. Further examples are contemplated.

[0183] For example, although the examples described above are primarily related to image or video signals, the spatially scalable coding schemes and systems described herein can be applied to audio signals. In this case, the audio signal can be modified, for example, by adding noise. The audio signal can be restored at a base level, but with the added noise present. The audio signal can be restored at an enhanced level, but with the added noise removed.

[0184] It should be understood that any feature described with respect to any one example may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other example, or in combination with any combination of any other example. In addition, equivalents and modifications not described in the above text may also be adopted without departing from the scope of the invention as defined by the appended claims.

[0185] The following lists the particularly preferred examples of the present disclosure in the form of a group of numbered clauses. It should be understood that these are examples that are helpful for understanding the present invention.

[0186] Numbered clauses 1. A method comprising:

[0187] modifying a signal to be modified using a modification signal to produce a modified signal, wherein the modified signal comprises a watermark, wherein the modified signal has elements corresponding to elements of the signal to be modified, and wherein the elements of the modified signal have modified values ​​relative to values ​​of the corresponding elements of the signal to be modified;

[0188] sending the modified signal, or a downsampled modified signal derived based on downsampling the modified signal, to be encoded;

[0189] receiving a decoded modified signal, the decoded modified signal being a decoded version of the modified signal after encoding or the downsampled modified signal after encoding, wherein an effect of the modification is visible in the decoded modified signal;

[0190] using the decoded modified signal to generate a processed signal; and

[0191] Generate residual data based on at least the following values:

[0192] the value of the element of the target signal; and

[0193] the value of the corresponding element of the processed signal,

[0194] wherein the processed signal and the residual data are combineable to generate a reconstructed signal, and

[0195] wherein the effect of the modification is not visible in the reconstructed signal.

[0196] Numbered clause 2. The method according to numbered clause 1, comprising:

[0197] downsampling the modified signal to produce the downsampled modified signal; and

[0198] The downsampled modified signal is sent to be encoded.

[0199] Item 3. A method according to item 2, wherein the target signal includes the signal to be modified.

[0200] Clause 4. A method according to clause 3, comprising sending the residual data, or data derived based on the residual data, to be encoded.

[0201] Numbered clause 5. A method according to any one of numbered clauses 2 to 4, comprising generating correction data based on the following values:

[0202] values ​​of elements of the downsampled modified signal;

[0203] the value of the corresponding element of the decoded modified signal; and

[0204] The value of the corresponding element of the modification signal or the value of the corresponding element of a signal based on the modification signal.

[0205] Numbered clause 6. The method of numbered clause 5, comprising deriving the signal based on the modified signal by processing the modified signal.

[0206] Numbered clause 7. The method according to numbered clause 5 or 6, comprising:

[0207] The correction data, or data derived based on the correction data, is sent to be encoded.

[0208] Numbered clause 8. The method of numbered clause 2, wherein the target signal comprises the modified signal.

[0209] Numbered clause 9. The method according to numbered clause 8, comprising:

[0210] Generate additional residual data based on the following values:

[0211] The values ​​of the elements of the residual data; and

[0212] The value of the corresponding element of the modification signal or the value of the corresponding element of a signal based on the modification signal.

[0213] Numbered clause 10. The method of numbered clause 9, comprising deriving the signal based on the modified signal by processing the modified signal.

[0214] Numbered clause 11. A method according to numbered clause 9 or 10, comprising:

[0215] The further residual data, or data derived based on the further residual data, is sent to be encoded.

[0216] Item 12. The method of item 1, wherein the target signal comprises a signal to be downsampled.

[0217] Numbered clause 13. The method of numbered clause 12, comprising downsampling the signal to be downsampled to derive the signal to be modified.

[0218] Numbered clause 14. A method according to any one of numbered clauses 1 to 13, wherein the modification signal comprises a superposition to be applied to the signal to be modified.

[0219] Numbered clause 15. A method according to any one of numbered clauses 1 to 14, wherein the modified signal has another element corresponding to another element of the signal to be modified, and wherein the another element of the modified signal has the same value as the value of the corresponding another element of the signal to be modified.

[0220] Numbered clause 16. A method according to any one of numbered clauses 1 to 15, wherein the modification of the signal to be modified comprises combining the signal to be modified with the modification signal.

[0221] Clause 17. A method according to any one of clauses 1 to 16, wherein the signal to be modified includes a luminance component and a chrominance component, wherein the luminance component of the signal to be modified is modified based on the modification signal, and wherein the chrominance component of the signal to be modified is not modified based on the modification signal.

[0222] Numbered clause 18. A method according to any of numbered clauses 1 to 17, wherein the use of the decoded modified signal to generate the processed signal includes performing an upsampling operation.

[0223] Numbered clause 19. A method comprising:

[0224] processing the decoded modified signal to obtain a processed signal; and

[0225] combining the processed signal and the residual data to generate a reconstructed signal,

[0226] wherein the decoded modified signal is a decoded version of an encoded modified signal or an encoded downsampled modified signal, the downsampled modified signal having been derived on the basis that the modified signal has been downsampled,

[0227] wherein the modified signal has been generated based on that the signal to be modified has been modified using a modification signal, the modification signal comprising a watermark,

[0228] wherein the modified signal has elements corresponding to elements of the signal to be modified, and wherein the elements of the modified signal have modified values ​​relative to the values ​​of the corresponding elements of the signal to be modified,

[0229] wherein the effect of said modification is visible in said decoded modified signal, and

[0230] wherein the effect of the modification is not visible in the reconstructed signal.

[0231] Numbered clause 20. A method according to any one of numbered clauses 1 to 19, wherein the element of the modified signal and the corresponding element of the signal to be modified are pixels.

[0232] Numbered clause 21. An apparatus configured to perform the method according to any one of numbered clauses 1 to 20.

[0233] Numbered clause 22. A computer program arranged to, when executed, perform the method according to any one of numbered clauses 1 to 20.

[0234] Numbered clause 23. A data stream comprising:

[0235] an encoded version of the modified signal or an encoded version of a downsampled modified signal, the downsampled modified signal having been derived on the basis that the modified signal has been downsampled; and

[0236] encoded residual data, wherein the modified signal has been generated based on that the signal to be modified has been modified using a modification signal, the modification signal comprising a watermark,

[0237] wherein the modified signal has elements corresponding to elements of the signal to be modified,

[0238] wherein the element of the modified signal has a modified value relative to the value of the corresponding element of the signal to be modified,

[0239] wherein the decoded modified signal can be processed to obtain a processed signal, the decoded modified signal being a decoded version of the encoded modified signal or the encoded downsampled modified signal, wherein the processed signal and the residual data can be combined to generate a reconstructed signal,

[0240] wherein the effect of said modification is visible in said decoded modified signal, and

[0241] wherein the effect of the modification is not visible in the reconstructed signal.

Claims

1. A method comprising: modifying a signal to be modified using a modification signal to produce a modified signal, wherein the modified signal has elements corresponding to elements of the signal to be modified, and wherein the elements of the modified signal have modified values ​​relative to values ​​of the corresponding elements of the signal to be modified; sending the modified signal, or a downsampled modified signal derived based on downsampling the modified signal, to be encoded; receiving a decoded modified signal, the decoded modified signal being a decoded version of the encoded modified signal or the encoded downsampled modified signal; using the decoded modified signal to generate a processed signal; and Generate residual data based on at least the following values: the value of the element of the target signal; and The value of the corresponding element of the processed signal.

2. The method according to claim 1, comprising: downsampling the modified signal to produce the downsampled modified signal; and The downsampled modified signal is sent to be encoded. The method according to claim 2 , wherein the target signal comprises the signal to be modified.

4. A method according to claim 3, comprising sending the residual data, or data derived based on the residual data, to be encoded.

5. A method according to any one of claims 2 to 4, comprising generating correction data based on the following values: values ​​of elements of the downsampled modified signal; the value of the corresponding element of the decoded modified signal; and The value of the corresponding element of the modification signal or the value of the corresponding element of a signal based on the modification signal. The method according to claim 5 , comprising deriving the signal based on the modification signal by processing the modification signal.

7. The method according to claim 5 or 6, comprising: The correction data, or data derived based on the correction data, is sent to be encoded. The method of claim 2 , wherein the target signal comprises the modified signal.

9. The method according to claim 8, comprising: Generate additional residual data based on the following values: The value of the element of the residual data; as well as The value of the corresponding element of the modification signal or the value of the corresponding element of a signal based on the modification signal.

10. The method of claim 9, comprising deriving the signal based on the modification signal by processing the modification signal.

11. The method according to claim 9 or 10, comprising: The further residual data, or data derived based on the further residual data, is sent to be encoded. The method according to claim 1 , wherein the target signal comprises a signal to be downsampled.

13. The method according to claim 12, comprising downsampling the signal to be downsampled to derive the signal to be modified.

14. A method according to any one of claims 1 to 13, wherein the modification signal comprises a superposition to be applied to the signal to be modified.

15. A method according to any one of claims 1 to 14, wherein the modification signal comprises a watermark.

16. A method according to any one of claims 1 to 15, wherein the modified signal has another element corresponding to another element of the signal to be modified, and wherein the another element of the modified signal has the same value as the value of the corresponding another element of the signal to be modified.

17. The method according to any one of claims 1 to 16, wherein the modification of the signal to be modified comprises combining the signal to be modified with the modification signal.

18. The method of claim 1, wherein the modification of the signal to be modified comprises shifting a position within the signal to be modified of one or more values ​​of the signal to be modified based on the modification signal.

19. The method according to any one of claims 1 to 18, wherein the signal to be modified comprises a luminance component and a chrominance component, wherein the luminance component of the signal to be modified is modified based on the modification signal, and wherein the chrominance component of the signal to be modified is not modified based on the modification signal.

20. The method of any one of claims 1 to 19, wherein the use of the decoded modified signal to generate the processed signal comprises performing an upsampling operation.

21. A method comprising: processing the decoded modified signal to obtain a processed signal; and combining the processed signal and the residual data to generate a reconstructed signal, wherein the decoded modified signal is a decoded version of an encoded modified signal or an encoded downsampled modified signal, the downsampled modified signal having been derived on the basis that the modified signal has been downsampled, wherein the modified signal has been generated based on the signal to be modified having been modified using the modification signal, wherein the modified signal has elements corresponding to elements of the signal to be modified, and wherein the elements of the modified signal have modified values ​​relative to the values ​​of the corresponding elements of the signal to be modified.

22. An apparatus configured to perform the method according to any one of claims 1 to 21.

23. A computer program arranged, when executed, to perform a method according to any one of claims 1 to 21.

24. A data stream comprising: an encoded version of the modified signal or an encoded version of a downsampled modified signal, the downsampled modified signal having been derived on the basis that the modified signal has been downsampled; as well as The encoded residual data, wherein the modified signal has been generated based on that the signal to be modified has been modified using a modification signal, the modification signal comprising a watermark, wherein the modified signal has elements corresponding to elements of the signal to be modified, wherein the element of the modified signal has a modified value relative to the value of the corresponding element of the signal to be modified, The decoded modified signal may be processed to obtain a processed signal, the decoded modified signal being a decoded version of the encoded modified signal or the encoded down-sampled modified signal.

25. The data stream of claim 24, wherein the processed signal and the residual data are combineable to generate a reconstructed signal.

26. A data stream according to claim 24 or 25, wherein the effect of the modification is visible in the decoded modified signal.

27. A data stream according to any one of claims 24 to 26, wherein the effect of the modification is not visible in the reconstructed signal.

Images