Use of transform coefficients for providing embedded signaling for watermarks

Abstract

The examples described herein relate to decoding and encoding signals. The present invention proposes a method for performing a signal decoding operation on one or more parts of a signal. The performing is at least partially based on information embedded in one or more values, the one or more values being received in one or more encoded data layers transmitted in an encoded data stream, wherein the values are associated with transform coefficients of elements of the signal intended to be processed by a decoder to derive the signal, and wherein the information includes an indication of watermark information associated with the signal.

Description

Technical Field

[0001] The present invention relates to a method for processing signals, such as, by way of non-limiting example, video, images, hyperspectral images, audio, point clouds, 3DoF / 6DoF, and volumetric signals. Processing the data may include, but is not limited to, obtaining, deriving, encoding, outputting, receiving, and reconstructing the signal in the context of a hierarchical (layer-based) coding format, where the signal is decoded in subsequent higher quality levels in layers, and utilizing and combining subsequent layers ("rungs") of the reconstructed data. Signals of different layers may be encoded in different elementary streams in different coding formats (by way of non-limiting example, such as, for example, a traditional single-layer DCT-based codec, ISO / IEC MPEG-5 Part 2 Low Complexity Enhanced Video Coding SMPTE VC-62117, etc.), and the elementary streams may or may not be multiplexed in a single bitstream. Background Art

[0002] In a layer-based coding format, such as ISO / IEC MPEG-5 Part 2 LCEVC (hereinafter referred to as "LCEVC") or SMPTE VC-62117 (hereinafter referred to as "VC-6"), the signal is decomposed into a plurality of "rungs" (also referred to as "hierarchical layers") of data from the highest rung of the original signal sampling rate to the lowest rung, which typically has a lower sampling rate than the original signal, and each rung corresponds to a "level of quality" (also referred to herein as "LoQ") of the signal. In a non-limiting example, when the signal is a frame of a video stream, the lowest rung may be a thumbnail of the original frame, such as a low-resolution frame in the video stream, or even just a single image element. Other rungs contain information about the corrections applied to the reconstructed rendition to produce the final output. The rungs may be based on residual information, such as the difference between a version of the original signal at a particular level of quality and a reconstructed version of the signal at the same level of quality. The lowest rung may not contain residual information, but may contain the lowest sampling of the original signal. The decoded signal at a given level of quality is reconstructed by first decoding the lowest rung (and thus reconstructing the signal at the first lowest level of quality), then predicting the rendition of the next higher level of quality—the next higher level signal, then decoding the corresponding second rung reconstructed data (also referred to as second level of quality "residual data"), and then combining the predicted data with the reconstructed data in order to reconstruct the rendition of the second higher level of quality signal, and so on, until the given level of quality is reconstructed.

[0003] Reconstructing the signal can include decoding the residual data and using the residual data to correct a version of a particular quality level that is derived from a lower quality level version of the signal. Data for different tiers can be encoded using different coding formats, and the sampling rates (e.g., resolution in the case of an image or video signal) for different quality levels can be different. Subsequent tiers can refer to the same signal resolution (i.e., sampling rate) of the signal, or to increasingly higher signal resolutions. Examples of these methods are described in more detail in the available specifications of LCEVC and VC-6.

[0004] Digital watermarking techniques are known in the art. An example digital watermarking technique is the Advanced Television Systems Committee (ATSC) Video Watermarking Transmit Standard A / 335. The document defining this standard, for example, as issued on September 20, 2016, is incorporated herein by reference.

[0005] In A / 335, the top one or two lines of the video signal are used to embed watermark information. In A / 335, the luminance values of the pixels within these lines are modified to carry the watermark information. The receiver extracts the first one or two lines of each frame of the video signal and applies a set of threshold operations to recover the encoded data.

[0006] A problem with A / 335 and similar digital watermarking methods is their requirement that the embedded data undergo various compression and transcoding operations, as well as transmission over a conventional consumer high-definition multimedia interface (HDMI). To achieve robustness, more complex threshold operations are applied at the receiver. A / 335 is also not intended to be tamper-proof or non-erasable; it can be intentionally removed by an intermediary. Summary of the Invention

[0007] The non - limiting embodiments shown herein refer to a signal as a sample sequence (i.e., a two - dimensional image, video frame, video field, sound frame, etc.). In the description, the terms "image", "picture", or "plane" (intended to have the broadest meaning of "hyperplane", i.e., an array of elements with any number of dimensions and a given sampling grid) will often be used to identify the digital reproduction of signal samples along a sample sequence, where each plane has a given resolution for each of its dimensions (e.g., X and Y) and contains a set of plane elements (or "elements" or "pixels" or display elements of a two - dimensional image, commonly referred to as "pixels", display elements of a volume image are commonly referred to as "voxels", etc.), characterized by one or more "values" or "settings" (by way of non - limiting examples, such as color settings in a suitable color space, settings indicating density levels, settings indicating temperature levels, settings indicating audio tones, settings indicating amplitudes, settings indicating depths, settings indicating alpha - channel transparency levels, etc.). Each plane element is identified by a suitable set of coordinates indicating the integer position of the element in the sampling grid of the image. The signal dimensions can include only spatial dimensions (e.g., in the case of an image) or can also include a time dimension (e.g., in the case of a signal that evolves over time such as a video signal).

[0008] As a non - limiting example, the signal can be an image, an audio signal, a multi - channel audio signal, a telemetry signal, a video signal, a 3DoF / 6DoF video signal, a volume signal (e.g., medical imaging, scientific imaging, holographic imaging, etc.), a volume video signal, or even a signal with more than four dimensions.

[0009] For simplicity, the non - limiting embodiments shown herein generally refer to a signal displayed as a 2D plane of settings (e.g., a 2D image in a suitable color space), such as a video signal. The terms "picture", "frame", or "field" will be used interchangeably with the term "image" to indicate the samples of a video signal over time: any concept and method shown for a video signal composed of frames (progressive video signal) can also be easily applied to a video signal composed of fields (interlaced video signal), and vice versa. Although the embodiments shown here focus on images and video signals, those skilled in the art can easily understand that the same concepts and methods also apply to any other type of multi - dimensional signal (e.g., audio signals, volume signals, stereoscopic video signals, 3DoF / 6DoF video signals, light - field signals, point clouds, etc.).

[0010] The embodiments described herein allow for the efficient generation, signaling, and utilization of context information that can be used by a decoder. The context information can be used together with residual data to appropriately control signal reconstruction. Such information can be efficiently embedded in the coefficients of one or more tiers of residual data of an encoded signal, thereby obviating the need for additional signaling overhead. Additionally, for some non-limiting embodiments described herein, applying certain non-essential signal enhancement operations at a lower signal resolution in the context of a hierarchical coding scheme also results in a substantial reduction in the processing capabilities required by the decoder device.

[0011] According to a first non-limiting embodiment, a signal processor (e.g., computer processor hardware) is configured to receive data and decode it (“decoder”). The decoder obtains a reproduction of a first (lower) quality level signal and produces a predicted reproduction of a second (higher) quality level signal, where the resolution (i.e., signal sampling rate) of the second quality level is higher than the first quality level. The decoder then receives and decodes a tier of residual data applied to the predicted reproduction of the signal to produce a corrected reproduction of the second quality level signal. When decoding a particular set of residual data coefficients and finding a particular set of quantization symbols, the decoder does not interpret the symbols as residual data but instead performs a watermarking operation based on the received symbols.

[0012] Thus, according to a non-limiting aspect of the present invention, the embedded information can include an indication of a feature associated with the signal, where the feature is watermark information. For example, the feature can be used to identify and authenticate the encoder that generated the data stream and / or contain information related to the time and location of encoding.

[0013] In some non-limiting embodiments, the watermark information can be used, for example, to identify the nature of the signal. The watermark information can indicate that the decoder should initiate the application of a watermark to the decoded signal. In other non-limiting embodiments, the feature corresponds to a state associated with the signal. The state includes compliance information associated with the signal. By way of non-limiting example, the compliance information can include any of the following information: the manner in which the signal was generated, a particular encoder version used to generate the signal, license information associated with the signal, and / or the encoder version that generated the signal. The compliance information can be useful for the decoder to initiate compliance actions such as generating a valid license for the signal when it detects a mismatch between the compliance information and a record. In such a case, for example, the decoder can initiate a compliance process for the signal, such as interrupting the display or playback of the signal, sending a request to the source that transmitted the signal to obtain a valid license, etc.

[0014] In other non-limiting embodiments, the feature identifies an object in the signal. In a non-limiting embodiment, the feature allows for the identification of an object in a video corresponding to a unique identifier known to the decoder.

[0015] As a non - limiting example, the feature can include a marker associated with one or more elements of a signal. The marker can include an indication of whether an element of the signal can be selected by an end - user of the signal. In other non - limiting embodiments, the marker can include an indication of whether an element of the signal can be associated with an action to be taken by the end - user of the signal, such as clicking on the element and / or linking to a different signal / web page. In another non - limiting embodiment, the marker includes an identification of an element of the signal that belongs to a classification, such as a video classification or an object classification. By way of non - limiting example, the element can represent a person, and the marker will identify who that person is. Alternatively, the element can represent an object, and the marker can identify what object it is. Alternatively, the marker can identify to which class the object belongs. Generally, the classification can include an association of the element with a class of identifiers, such as the class to which the element belongs.

[0016] In other non - limiting embodiments, reserved symbols are used to embed different secondary signals that are part of an encoded stream, where the different secondary signals are encoded by a given public key and can only be decoded by a decoder that knows both the existence of the secondary signals and the private key corresponding to the public key used to encrypt the secondary signals.

[0017] In some non - limiting embodiments, bits in a decoded byte stream signal to a decoder that additional information may have been embedded in some residual data coefficients, such that a particular set of symbols in a particular residual data set should not be interpreted as actual residual data, but rather as context information, for example, informing a signal reconstruction operation. In non - limiting embodiments, some reserved symbols correspond to specific regions of the signal, i.e., whereby a watermark can be differentially applied to local blocks of the reconstructed signal.

[0018] In some non - limiting embodiments, the decoder implements signal reconstruction operations in different ways based on the processing capabilities available to the decoder device at any given time (including sometimes not implementing these signal reconstruction operations at all).

[0019] In some non - limiting embodiments, before applying residual data decoded from the tier data containing embedded information, the decoder applies signal processing operations based on the context data loop. In other non - limiting embodiments, the decoder applies signal processing operations after combining a preliminary reproduction of a second quality - level signal with the decoded residual data. In other non - limiting embodiments, after a reproduction of a final (highest) quality - level signal has been produced, the decoder applies the signal processing operations at the end of the decoding process.

[0020] In some non - limiting embodiments, the format for encoding the residual data and the embedded context information is MPEG - 5 Part 2 LCEVC (“Low Complexity Enhancement Video Coding”). In other non - limiting embodiments, the format for encoding the residual data and the embedded context information is SMPTE VC - 6 ST - 2117. The signal processing operations using the watermark information can be one or more operations not defined in either the LCEVC or VC - 6 video coding standards, such as so - called non - standard operations that are still compatible with the standards. This example can thus be implemented as an optional extension to the aforementioned standards, or other layer - based hierarchical coding schemes.

[0021] According to a second non - limiting embodiment, a signal processor (e.g., computer processor hardware) is configured to receive data and encode it (“encoder”). The encoder produces a reproduction of a first (lower) quality level signal and encodes it using a first encoding method. Then, the encoder produces a predicted reproduction of a second (higher) quality level signal and correspondingly generates and encodes a second quality level residual data for a tier applied to the predicted reproduction of the second quality level signal in order to produce a corrected reproduction of the second quality level signal. In these embodiments, the encoder signals the watermark information to the decoder using a set of reserved symbols in a set of residual data of the tiered residual data.

[0022] In some non - limiting embodiments, bits in the switched - coding byte stream are switched to signal to the decoder whether a given set of symbols in a given set of residual data should be interpreted as actual residual data or as additional context information informing signal reconstruction operations such as watermark processing to be applied by the decoder.

[0023] In some non - limiting embodiments, the context information is embedded in more than one tier of residual data.

[0024] In some non - limiting embodiments, the format for encoding the residual data and the embedded context information is MPEG - 5 Part 2 LCEVC (“Low Complexity Enhancement Video Coding”). In other non - limiting embodiments, the format for encoding the residual data and the embedded context information is SMPTE VC - 6 ST - 2117.

[0025] According to other non - limiting embodiments, the context signal information is embedded in the encoded data generated in a non - hierarchical coding format. In non - limiting embodiments, the symbols are embedded at the macro - block level using a set of reserved symbols in the quantization coefficients.

[0026] Other features and advantages will become apparent from the following description given by way of example only with reference to the accompanying drawings. Description of the Drawings

[0027] Figure 1 A block diagram showing an example of a decoding system according to an embodiment;

[0028] Figure 2 A block diagram showing an example of a hierarchical coding system according to an embodiment;

[0029] Figure 3 A block diagram showing a hierarchical decoding system according to an embodiment;

[0030] Figure 4 A block diagram showing another example of a hierarchical decoding system according to an embodiment;

[0031] Figure 5 A block diagram showing another example of an encoding and decoding system according to an embodiment; and

[0032] Figure 6 A block diagram showing an example of a device according to an embodiment. DETAILED DESCRIPTION

[0033] Reference Figure 1 , an example of a method implemented within a decoding system is shown. A set of quantized symbols 100 - 1 to 100 - N is received and processed. These quantized symbols contain quantized transform coefficients, where quantization can be optional and / or vary maximally based on an encoding configuration. The quantized symbols can contain symbols generated by one or more in an encoding stream described with respect to subsequent figures. In the examples described herein, information is embedded in one or more values received in one or more encoded data layers, where the values are associated with transform coefficients of elements intended to be processed by a decoder to derive a signal. The reception of example quantization and transform coefficient symbols is described in LCEVC and VC - 6. Depending on whether symbol 100 - 1 is a reserved symbol, the decoder follows two different methods. In this case, the term "reserved symbol" can be considered to refer to a symbol reserved to carry context information such as watermark information.

[0034] If symbol 100 - 1 is not a reserved symbol, for example, intended to carry residual data for reconstructing the signal, its decoding follows the normal process implemented for other symbols in the set: dequantization and inverse transformation are performed according to block 110, generating a set of decoded data 130. The decoded data is further processed by a decoding operation 150 to generate a decoded signal 160. For example, this normal process can be the process described in a decoding specification such as LCEVC or VC - 6.

[0035] If symbol 100-1 is to be a reserved symbol, its decoding follows a different process as shown in comparison box 105. At box 120, the embedded information is decoded by processing symbol 100-1 to produce watermark information 140. At box 170, the decoded signal 160 is processed together with the watermark information 140, where one or more additional operations 170 are performed. These operations can include enhancement operations such as determining whether to interrupt the display of the decoded signal based on the watermark information 140, and / or initiating compliance processes such as checking whether the encoder is properly licensed and / or has correctly generated symbols 100-1 to 100-N. The output of the additional operations 170 can include an enhanced reconstruction 180 of the signal. For example, markers associated with one or more elements of the signal can be processed to enable user actions, where the one or more elements are encoded within the watermark information. The elements here can refer to pixels or regions of a video frame. For example, a user viewing the reconstructed video can perform actions regarding the marked elements. In other cases, the enhanced reconstruction of the signal can include modifying the content based on the watermark information.

[0036] In some non-limiting embodiments, a bit in a decoded bit or byte stream (not shown) signals to the decoder that symbol 100-1 is to be interpreted as a reserved symbol. For example, as described in more detail below, the bit can be a "user data" flag that is "turned on" or "turned off" within global configuration information. In some non-limiting embodiments, the decoder implements signal processing operations (including sometimes not implementing signal processing operations at all) in different ways based on the processing capabilities available to the decoder device at the time of processing, such as the additional operations 170.

[0037] Reference Figure 2 , an example of a method implemented within an encoding system is shown, in this case a layer-based hierarchical encoding method is implemented. Figure 2 The boxes in can be implemented by an example encoder. A source signal 200 of quality level #2 (e.g., full resolution and quality) is received and processed by a downsampler 210 to produce a downsampled signal 200-1. The downsampled signal 200-1 is processed by an encoder 220 that applies a given encoding method (in some non-limiting embodiments also a layer-based hierarchical encoding method, and in other non-limiting embodiments a non-hierarchical encoding method) to produce encoded data 225. The encoder 220 can be referred to as a "base" decoder.

[0038] The encoded data 225 and the downsampled signal 200-1 are processed by the LOQ#1 residual data generator 230 to produce the encoded data 235 and a reproduction 237 of the LOQ#1 signal. The LOQ#1 residual data generator 230 can generate a residual signal by subtracting a reconstruction based on the encoded data 225 from the downsampled signal 200-1. The LOQ#1 residual data generator 230 can also encode the residual signal by applying an encoding unit transform and quantifying the output of the transform. A further entropy encoding stage can also be applied. The output of the transform and quantization can comprise (quantized) transform coefficients that are modified to include embedded signaling. The reproduction 237 of the LOQ#1 signal is further processed by the LOQ#2 preliminary reproduction generator 240 to produce a preliminary reproduction 245 of the LOQ#2 signal. For example, this can comprise upsampling the reproduction 237 of the LOQ#1 signal with optional modification to generate a signal at the LOQ#2 resolution and / or sampling rate.

[0039] The preliminary reproduction 245 of the LOQ#2 signal, together with the source signal 200, is processed by the LOQ#2 residual generator 260 to produce the encoded data 265. The encoded data 265 can comprise a residual signal generated by the LOQ#2 residual generator 260 by subtracting the preliminary reproduction of the LOQ#2 signal from the source signal 200. The LOQ#2 residual signal generator 260 can apply similar operations to the LOQ#1 generator 230, but for the residual signal at a second quality level (e.g., higher resolution).

[0040] Then, the sets of encoded data 225, 235, and 265 are processed by a multiplexer (Mux) 270 to produce a signal 280 encoded with the tiered encoded data. Although all three encoded data sets are multiplexed in Figure 2 , in other instances, subsets can be multiplexed, e.g., the encoded data 235 and 265 can be transmitted separately from the encoded data 225. In VC-6, the encoded data stream can comprise all sets of encoded data 225, 235, 265; in LCEVC, the encoded (enhanced) data stream can comprise the encoded data 235 and 265, and the encoded data 225 can form a separate base stream.

[0041] In certain described embodiments, when encoding the first quality level signal, the LOQ#1 Residual Generator 230 generates encoded data 235 that utilizes a set of reserved symbols in order to signal watermark information to the decoder. This can include watermark information converted from known watermark standards such as A / 335 discussed in the background art. As is known in the art and explained in the standard literature of A / 335, watermark information can include data embedded in the noise-tolerant signal for use by the decoder. The uses of watermark information are diverse. These uses include supporting programming elements associated with the signal, such as those required to support interactivity, dynamic content replacement or overlay, service usage monitoring, and content identification. A common use of watermark information is to identify the ownership of intellectual property associated with the signal and / or elements within the signal. It should be noted that references to elements herein include references to pixels or planar elements associated with objects distinguishable by a human observer within the signal. Watermark information, also known as digital watermarking, can also be used to verify the authenticity or integrity of the encoded signal. This watermark information can be added during the encoding process and extracted during the decoding process.

[0042] In one case, the LOQ#1 Residual Generator 230 replaces the quantized transform coefficient value of a particular transform coefficient (e.g., a particular element in a vector generated by multiplying a transform matrix) with embedded signaling data. Only the value of one coefficient can be modified, and the other coefficients can remain unmodified and be encoded according to the comparison encoding used for the standard decoding procedure. Coefficients that minimize the change in the reconstructed signal can be selected, such as the H or HH coefficients for a 2×2 or 4×4 Hadamard transform.

[0043] In some non-limiting embodiments, the LOQ#1 Residual Generator 230 toggles specific bits in the encoded bitstream or byte stream to signal to the decoder whether a given set of symbols in the encoded data set 235 is to be interpreted as actual residual data or as additional context information informing the signal decoding operation. In some non-limiting embodiments, methods 230 and 260 perform in-loop signal processing operations based on the information signaled with the reserved coefficients in order to reconstruct the signal from the encoded data 265.

[0044] Reference Figure 3 shows an example of a method implemented within a decoding system that also implements a layer-based hierarchical encoding method. Figure 3 The blocks in Figure 2 can be implemented by an example decoder. The encoded data 225 is received and processed by the lower LOQ decoder 300. As Figure 2 shown, the encoded data 225 can be obtained by demultiplexing the signal 280 encoded with tiered encoding data received from the encoder. The lower LOQ decoder 300 can be referred to as the base decoder. The lower LOQ decoder 300 outputs a preliminary reproduction 310 of the LOQ#1 signal.

[0045] Then, the initial reproduction 310 of the LOQ#1 signal, together with the encoded data 235, is processed by the reconstructor 320 of LOQ#1 to produce a reproduction 337 of the LOQ#1 signal. As Figure 2 shown, the encoded data 235 can be obtained by demultiplexing the signal 280 encoded with hierarchical encoded data received from the encoder. The reproduction 337 of the LOQ#1 signal corresponds to Figure 2 the reproduction 237 of the LOQ#1 signal in Figure 1 and 4 The encoded data 235 can be decoded by the LOQ#1 reconstructor 320 and then combined with the initial reproduction 310 of the LOQ#1 signal. In one case, the encoded data 235 can include residual data as described in other examples herein. Decoding can include applying operations shown in blocks 110 and 410 in

[0046] The reproduction 337 of the LOQ#1 signal is then processed by the LOQ#2 initial reproduction generator 340 (which can correspond to the LOQ#2 initial reproduction generator 240 in Figure 2 to produce an initial reproduction 345 of the LOQ#2 signal. The initial reproduction 345 of the LOQ#2 signal is processed by the LOQ#2 reconstructor 360 to produce a final reproduction 370 of the LOQ#2 signal.

[0047] In some non - limiting embodiments, when encoding a specific group of data within the encoded data 235 and finding a specific group of quantization symbols, the decoder does not interpret the symbols as residual data, but instead performs signal processing operations based on the received symbols.

[0048] In some non - limiting embodiments, bits in a decoded byte stream (not shown in Figure 3 signal the LOQ#1 reconstructor 320 and / or the LOQ#2 reconstructor 360 that additional information may have been embedded in some of the residual data coefficients, and more specifically, that a specific group of quantization symbols in a specific group of residual data (or a portion of the symbols) should not be interpreted as actual residual data, but rather as context information informing signal enhancement operations.

[0049] In some non - limiting embodiments, the decoder implements signal processing operations in different ways (including sometimes not implementing these signal processing operations at all) based on the processing capabilities available to the decoder device at the time of decoding. For example, markers and / or classifications can be extracted from reserved symbols only when processing capabilities are available.

[0050] In some non-limiting embodiments, a signal processing method based on the watermark information is applied within the decoder loop before applying the residual data decoded from the tiered data containing the embedded watermark information. In other non-limiting embodiments, a signal processing method based on the watermark information is applied within the decoder loop after combining the preliminary reproduction of the second quality level signal with the decoded residual data. In other non-limiting embodiments, a signal processing method based on the watermark information is applied by the decoder at the end of the decoding process after the reproduction of the final (highest) quality level signal has been generated. In yet another non-limiting embodiment, the decoder applies the signal processing method based on the watermark information both within the loop and at the end of the decoding process. Certain examples regarding Figure 4 are described later.

[0051] In a preferred example, the encoder or decoder is part of a layer-based scalable coding scheme or format. Examples of layer-based scalable coding schemes include LCEVC: MPEG-5 Part 2 LCEVC (“Low Complexity Enhanced Video Coding”) and VC-6: SMPTE VC-6 ST-2117, the former being described in PCT / GB2020 / 050695 (and associated standard documents), and the latter being described in PCT / GB2018 / 053552 (and associated standard documents), all of which are incorporated herein by reference. However, the concepts shown herein need not be limited to these particular scalable coding schemes. Those skilled in the art will appreciate how the above encoder and decoder methods apply to the base layer and enhancement layer in LCEVC (e.g., LOQ#1 corresponds to the base layer and LOQ#2 corresponds to the enhancement layer). Thus, in some cases, the coding format of the encoded data 235, the encoded data 265, and the corresponding embedded context information is MPEG-5 Part 2 LCEVC. In such cases, the encoded data 235 and the encoded data 265 may include different enhancement sub-layers. In such cases, the embedded context information may be referred to as “user data” as it may contain information other than that required to reconstruct the signal according to the standard. In other cases, the coding format of the encoded data 235, the encoded data 265, and the corresponding embedded context information is SMPTE VC-6 ST-2117. Again, the embedded context information may contain data that is not required to reconstruct the signal according to the standard's definition. It should be noted that, as is known from layer-based coding methods such as LCEVC and VC-6, a reference to a “frame” of video data also includes a reference to one or more planes of color data (e.g., the luminance and chrominance planes).

[0052] Referring to Figure 4 , an example of a method implemented within a decoding system that also implements a layer-based scalable coding method is shown. The method is Figure 2Variant of the method shown. In this variant, the watermark information can be used to process one or more of the initial reproduction of the LOQ#1 signal (such as 310 in Figure 3 ), and the reproduction of the LOQ#2 signal (such as 370 in Figure 3 ). In this way, the watermark information can be associated with one or more quality levels, including the basic quality level.

[0053] As Figure 2 shown, in Figure 4 , the quantization symbol 400-1 is received and processed together with other quantization symbols 400-2 to 400-N. The quantization symbols can represent the residual data stream of the encoded data 235 level. The decoder checks whether the symbol 400-1 should be a reserved symbol. Depending on whether the symbol 400-1 is a reserved symbol, the decoder follows two different methods.

[0054] If the symbol 400-1 is not a reserved symbol, its decoding follows the normal process applied to the other symbols in the group: dequantization and inverse transformation are performed according to block 410, generating a set of decoded residual data 420. The residual data (for example, in a non-limiting embodiment, together with other residual data of the residual part of the signal samples) is further processed by a reconstructor 450 to generate a reproduction 460 of the LOQ#1 signal.

[0055] If symbol 400-1 is to be a reserved symbol, its decoding follows a different process. At block 430, an operation to decode embedded information 430 is initiated to process symbol 400-1 to produce watermark information 435. In this instance, the watermark information can be used to control one or more of the following: additional operation 440, reconstructor 450, and additional operation 480. For example, the watermark information 435 can contain compliance information, and additional operations 440 and 480 can contain respective compliance processes performed on the initial reproduction 310 of the LOQ#1 signal and the reproduction 470 of the LOQ#2 signal. For example, the watermark information can indicate that the user is only permitted to access a lower resolution video signal; in such a case, the watermark information 435 can indicate that the initial reproduction 310 of the LOQ#1 signal can be processed to output a reproduction 460 of the LOQ#1 signal, but not output the reproduction 470 of the LOQ#2 signal (e.g., to limit the final reproduction 490 of the LOQ#2 signal). In such a case, if the user is not permitted any access to either video stream, then both the initial reproduction 310 of the LOQ#1 signal and the reproduction 470 of the LOQ#2 signal can be restricted or not output based on the watermark information 435. In other cases, the watermark information 435 can contain markers associated with elements of signals of different quality levels. Additional operations 440 and 480 can thus contain generating a metadata signal associated with the primary signal (e.g., a set of markers for pixels in a video signal), which allows the user to initiate actions regarding the elements, such as clicking on an element within the reproduction of one or more of the initial reproduction 310 of the LOQ#1 signal and the reproduction 470 of the LOQ#2 signal. For example, the watermark information 435 can contain information associated with an object shown within the signal, such as a particular model of an actor or product, allowing the end user to click on a pixel associated with the object and be provided with further information about the object.

[0056] More specifically, additional operation 440 processes the initial reproduction 310 of the LOQ#1 signal and the watermark information 435. This produces a processed initial reproduction 445 of the LOQ#1 signal. This processed initial reproduction 445, together with the residual data 420, is further processed by reconstructor 450 to produce an enhanced reproduction 460 of the LOQ#1 signal. For example, reconstructor 450 can contain processes similar to those performed by Figure 3 the reconstructor 320 of LOQ#1 in Figure 3Reproduction of the LOQ#1 signal 337. The enhanced reproduction 460 of the LOQ#1 signal can be enhanced by adding residual data 420 and by any processing of the preliminary reproduction 310 of the LOQ#1 signal performed by the additional operation 440. In one case, the decoded embedded information box 430 can extract the watermark data 435 and the residual data for the quantization symbol 400-1. For example, the quantization symbol 400-1 can carry the watermark data 435 and the residual data. This can be achieved, for example, by partitioning the bits of the quantization symbol 400-1 and applying a higher level of quantization to the original symbols representing the transform coefficients of the residual data. For example, the decoded embedded information box 430 can perform operations similar to the dequantization and inverse transform box 410, or can alternatively pass the zero watermark component of the quantization symbol 400-1 back to the dequantization and inverse transform box 410 to derive the residual data. In any case, the reconstructor 450 can additionally receive the residual data of the quantization symbol 400-1.

[0057] The reconstructor 450 thus generates an enhanced reproduction 460 of the LOQ#1 signal, which is further processed by the decoding operation 465 to produce a reproduction 470 of the LOQ#2 signal. The decoding operation 465 can be an operation associated with Figure 2 the LOQ#2 initial reproduction generator 340 and the LOQ#2 reconstructor 360 in. Thus, it can be seen that the reproduction 470 of the LOQ#2 signal corresponds to Figure 3 the reproduction 370 in. In Figure 4 in, the reproduction 470 of the LOQ#2 signal together with the watermark information 435 is processed by the additional operation 480 to produce a final reproduction 490 of the LOQ#2 signal. As described above, the additional operation 480 can include implementing a compliance process to limit the output of the final reproduction 490 of the LOQ#2 signal and / or generating additional metadata such as a tag for an element like the final reproduction 490 of the LOQ#2 signal. The additional operation 480 can include interrupting the display of the final reproduction 490 of the LOQ#2 signal based on the result of the compliance process.

[0058] In some cases, if the quantization symbol 400-1 still carries the transform coefficient value (e.g., by splitting the bit capacity between the watermark information and the transform coefficient value) and / or the transform coefficient is selected as a coefficient from a larger (e.g., 4×4) transform where the perceived visual impact is reduced (e.g., the HH coefficient in a 4×4 Hadamard transform), then the visual impact on the resulting output signal (e.g., the final reproduction 490 of the LOQ#2 signal) is minimized. Additionally, if the precision of the transform coefficient value is reduced or replaced at the LOQ#1 level, then the LOQ#2 residual generator 260 can generate residual data that becomes the encoded data 265, which corrects the difference between the preliminary reproduction 245 of the LOQ#2 signal and the LOQ#2 source signal 200. This is an improvement over watermarking standards such as A / 335, where there are distinct black or gray lines and pixel variations at the top of the image.

[0059] An additional benefit of the proposed watermarking method is that the watermark is performed as part of the encoding, so it cannot be destroyed or modified by subsequent encoding or compression. For example, comparative watermarking standards such as A / 335 apply the watermark information to the LOQ#2 source signal 200, where the watermarked signal is then encoded. This means that more complex threshold processing is required at the receiver because the original luminance values can be modified as part of the encoding and decoding process (e.g., if this is a lossy process). The method currently described is also more secure because it is more difficult to access and alter the picture information; the encoding and decoding operations can be protected and / or the watermark information itself can be encrypted (e.g., using public key cryptography).

[0060] Reference Figure 5 shows an example of a method implemented within an encoding and decoding system that utilizes the innovative methods described herein. Figure 5 shows how the proposed method of embedding watermark information can be used with a traditional decoder. The encoder 510 processes the original signal 500 to produce a data stream 520. The encoder 510 can encode the data stream 520 as explained with reference to Figure 2 and can embed the watermark information within one or more values received in one or more encoded data layers transmitted within the data stream 520, where the values are associated with transform coefficients of elements intended to be processed by the decoder to derive the signal, e.g., as described with respect to Figure 2 or Figure 4 described.

[0061] The data stream 520 is processed by two decoders. The decoder 530-0 implements a signal processing method based on the information signaled by the encoder 510 within the reserved symbols, thereby decoding the reconstructed signal 540-0. The reconstructed signal 540-0 can be enhanced with metadata, such as markers or classification information derived from the watermark information within the reserved symbols.

[0062] In Figure 5 , the decoder 530-1 also reconstructs the reconstructed signal 540-1, but ignores the information signaled by the encoder 510 within the reserved symbols. In some non-limiting embodiments, the reconstructed signal 540-1 is a fully viable reconstruction of the signal for a given purpose, such that the additional operations performed by the decoder 530-0 are completely optional. For example, the decoder 530-1 can be a decoder that applies the decoding process proposed in the LCEVC standard, while the decoder 530-0 can be a decoder that implements a non-standard decoding process (in some cases, in addition to the decoding process proposed in the LCEVC standard). Thus, additional functionality can be provided based on the watermark information while remaining compliant with the LCEVC standard.

[0063] In some non-limiting embodiments, the decoder 530-0 can sometimes decide to ignore some of the information signaled by the encoder 510 within the reserved symbols. For example, the decoder 530-0 can define whether to ignore some of the information signaled within the reserved symbols based on information including one or more of the resolution and frame rate of the signal, the processing power load at decoding, and the battery power status.

[0064] In some cases, backward compatibility is achieved, such as the backward compatibility described above, because the decoder 530-1 treats the reserved symbols as normal quantized transform coefficient values and decodes them appropriately. The correction applied in a layer-based hierarchical format means that any errors can be corrected. Alternatively, the bits in the encoded data stream 520 are used to signal to the decoder 530 that one or more values should be interpreted as the said information rather than the actual quantized value of the transform coefficient. In yet another case, the bit depth (e.g., depth D is 8 or 16 bits) assigned to a particular transform coefficient value can be shared between the reserved symbols and the (quantized) transform coefficient values. For example, the n least significant bits of the transform coefficient value (where n is less than the bit depth, e.g., 2 or 6 bits) can be used to carry the reserved symbols (i.e., watermark information), which represents a more aggressive quantization of the transform coefficient value to which the symbol is applied, but still enables the coarser level of information (D - n bits) to be transmitted and used for reconstructing the residual data. By selecting the transform coefficients (e.g., H or HH in a 2×2 or 4×4 Hadamard transform) that are determined (e.g., through experimentation) to have a lower perceptual impact in the reconstructed output, the visual impact can be further minimized.

[0065] Referring Figure 6 , a schematic block diagram of an example of the device 600 is shown.

[0066] Examples of device 600 include, but are not limited to, mobile computers, personal computer systems, wireless devices, base stations, telephone devices, desktop computers, laptop computers, notebook computers, netbook computers, mainframe computer systems, handheld computers, workstations, network computers, application servers, storage devices, consumer electronic devices such as cameras, video cameras, mobile devices, video game consoles, handheld video game devices, peripherals such as switches, modems, routers, vehicles, etc. or generally any type of computing or electronic device.

[0067] In this example, device 600 includes one or more processors 612 configured to process information and / or instructions. One or more processors 612 may include a central processing unit (CPU). One or more processors 612 are coupled to bus 611. Operations performed by one or more processors 612 may be implemented by hardware and / or software. One or more processors 612 may include multiple co-located processors or multiple fully remotely located processors.

[0068] In this example, device 600 includes computer-usable memory 613 configured to store information and / or instructions for one or more processors 612. Computer-usable memory 613 is coupled to bus 611. Computer-usable memory 613 may include one or more of volatile and non-volatile memory. Volatile memory may include random access memory (RAM). Non-volatile memory may include read-only memory (ROM).

[0069] In this example, device 600 includes one or more external data storage units 680 configured to store information and / or instructions. One or more external data storage units 680 are coupled to device 600 via I / O interface 614. One or more data storage units 680 may include, for example, magnetic or optical disks and disk drives or solid state drives (SSDs).

[0070] In this example, device 600 also includes one or more input / output (I / O) devices 616 coupled via I / O interface 614, which is configured to communicate information to and / or from one or more processors 612. Device 600 also includes at least one network interface 617. Both I / O interface 614 and network interface 617 are coupled to system bus 611. At least one network interface 617 may enable device 600 to communicate via one or more data communication networks 690. Examples of data communication networks include, but are not limited to, the Internet and local area networks (LANs). One or more I / O devices 616 may enable a user to provide input to device 600 via one or more input devices (not shown). One or more I / O devices 616 may enable information to be provided to a user via one or more output devices (not shown).

[0071] In Figure 6 it, the (signal) processor application 640-2 is shown as loaded into the memory 1513. This can be performed as the (signal) processor procedure 640-1 executed by the processor 612 to implement the methods described herein (e.g., implementing a suitable encoder or decoder). The device 600 may also include additional features not shown for clarity, including an operating system and additional data processing modules. The (signal) processor procedure 640-1 can be implemented by computer program code stored in a memory location within a computer-usable non-volatile memory, a computer-readable storage medium within one or more data storage units, and / or other tangible computer-readable storage media. Examples of tangible computer-readable storage media include, but are not limited to: optical media (e.g., CD-ROM, DVD-ROM, or Blu-ray), flash memory cards, floppy disks, or hard disks, or any other medium capable of storing computer-readable instructions such as firmware or microcode in at least one ROM or RAM or programmable ROM (PROM) chip, or as an application-specific integrated circuit (ASIC).

[0072] The device 600 can thus include data processing modules that can be executed by one or more processors. The data processing modules can be configured to include instructions for implementing at least some of the operations described herein. During operation, one or more processors initiate, run, execute, interpret, or otherwise execute the instructions.

[0073] Although at least some aspects of the examples described herein with reference to the accompanying drawings involve computer processes executed in a processing system or processor, the examples described herein also extend to computer programs, such as a computer program on or in a carrier, which is adapted to put the examples into practice. The carrier can be any entity or device capable of carrying the program. It should be understood that the device 600 may include more, fewer, and / or different components than Figure 6 those shown. The device 600 can be located in a single location or can be distributed over multiple locations. Such locations can be local or remote.

[0074] As described in the examples herein, a signal processor (e.g., computer processor hardware) is configured to receive data and decode it (“decoder”). The decoder obtains a reproduction of a first (lower) quality level signal and detects reserved symbols that specify watermark information. The decoder reconstructs a reproduction of a second (next higher) quality level signal and can use the watermark information to apply further processing to the reproduction.

[0075] In some examples described herein, the reserved symbols may be carried as so-called user data of the encoded data stream. In these examples, the signal processing information is embedded in one or more values received in one or more encoded data layers transmitted in the encoded data stream. The values are associated with transform coefficients that are processed to derive elements of the signal during decoding. For example, the values may include the values of predefined transform coefficients within a set of different transform coefficients generated by an encoding transform.

[0076] Bits in the bitstream of the encoded data stream can be used to signal the presence of user data. This bit may include a user_data_enable bit, which may be present in the global configuration header of the encoded data stream. In some examples, the encoding of user data in place of one of the coefficients may be configured as follows. If the bit is set to "0", then the decoder interprets the data as the associated transform coefficient. If the bit is set to "1", then the data contained in the associated coefficient is treated as user data, and the decoder is configured to ignore the data, or the associated coefficient is treated as carrying user data and the relevant process for extracting the data is performed. For example, if the bit is set to "1", this may indicate that watermark information is being transmitted.

[0077] The user data transmitted in this way can be used to enable the decoder to obtain supplementary information, including, for example, various feature extractions and derivations. Although the examples claimed herein relate to watermark information, user data can also be used to signal other optional parameters related to implementations outside of the standardization implementations.

[0078] In one case, the user_data_enable variable can be a k-bit variable. For example, user_data_enable can include a 2-bit variable with the following values:

[0079] User data enabled Type value 0 Disabled 1 Enabled 2 digits 2 Enabled 6 digits 3 Reserved

[0080] In this case, the user data specifying the watermark information can be embedded in the last n (least significant) bits of one or more decoded coefficient data sets (e.g., within the encoded residual coefficient data).

[0081] As described in the examples herein, when user data is enabled, for example, to transmit signal processing information, then the "in-loop" processing of the transform coefficients can be modified. These two examples are shown in Figure 2 and 4In addition, the decoding of the transform coefficients may be adjusted such that when user data is enabled, the value of a specific transform coefficient (e.g., H or HH) is set to 0 before the inverse transform of the transform coefficients. In the case stated in the above table, after the n least significant bits are extracted as the reserved symbol, if n = 2 (e.g., user_data_enable = 1), the value of the transform coefficient used to carry user data may be right-shifted (e.g., shifted) by 2 bits (>>2), or if n = 6 (e.g., user_data_enable = 1), right-shifted (e.g., shifted) by 6 bits (>>6). In one case, if the value of the transform coefficient is D bits in length, where D > n and n is the length of the user data in bits (e.g., 2 or 6 in the above table), then the remaining D - n bits of the transform coefficient can be used to carry the value of the transform coefficient (e.g., an integer value with a higher quantization level compared to the full D-bit representation). In this case, the user data and the value of the transform coefficient can be partitioned over D bits. In other simpler cases, the user data can be extracted and the value of the transform coefficient can be set to 0 (i.e., such that the value of the transform coefficient has no effect on the output of the inverse transform).

[0082] In some instances, the user data used to implement the reserved symbol may be formatted according to a defined syntax. The defined syntax may partition the user data into header data and payload data. In this case, the decoding of the user data may include parsing a first set of values received in one or more coded data layers to extract the header data, and parsing a second subsequent set of values received in one or more coded data layers to extract the payload data. The header data may be set to a first defined number of bits. For example, in the above instances where the user data is defined by a 2- or 6-bit value, the first x values may include the header data. In one case, x may be equal to 1 such that the first value of the user data (e.g., the transform coefficient value of the first coded unit or data block of a given video frame or plane) defines the header data (e.g., the 2 or 6 bits of the first value define the header data).

[0083] In some instances, the header data may at least indicate whether the watermarking operation is enabled. Generally, the header data may indicate the global parameters of the watermarking information, while the payload data may indicate the local parameters of the watermarking information, i.e., whereby the watermarking information can be localized to one or more coded units containing m×m residual data blocks (e.g., 2×2 or 4×4 blocks). Since the watermarking information is encapsulated within the reserved symbol of a specific coded unit, marking and / or classification (e.g.) can be applied to the localized region of the signal. For example, the marking may indicate the location where a local region of the video signal will be modified and / or replaced.

[0084] The techniques described herein may be implemented in software or hardware, or may be implemented using a combination of software and hardware. These techniques may include configuring a device to implement and / or support any or all of the techniques described herein.

[0085] The above embodiments should be understood as illustrative examples. Additional embodiments are contemplated.

[0086] It should be understood that any feature described with respect to any one embodiment may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other embodiment, or in combination with any combination of any other embodiment. Additionally, equivalents and modifications not described above may also be employed without departing from the scope of the invention as defined by the appended claims.

Claims

1. A method for performing a signal decoding operation on one or more parts of a signal (500), wherein the performing is at least partially based on information embedded in one or more values received in one or more coded data layers transmitted within a coded data stream (520), wherein the values are associated with transform coefficients intended to be processed by a decoder (530) to derive elements of the signal, wherein the information includes an indication of watermark information (140) associated with the signal (500), the coded data stream (520) is included in a coded bitstream, and bits in the coded bitstream are used to signal to the decoder (530) that one or more values should be interpreted as the information rather than the actual quantization values of the transform coefficients.

2. The method according to claim 1, wherein the one or more values are interpreted by the decoder (530) to derive the information rather than to derive values of the transform coefficients.

3. The method according to claim 1 or 2, wherein the signal (500) is encoded by means of a layer-based hierarchical format.

4. The method according to claim 3, wherein the embedded signaling is included in a residual layer having a resolution lower than the full resolution of the signal.

5. The method according to claim 4, wherein at least one of the signal decoding operations performed based on the embedded signaling is performed within an intermediate reproduction loop of the signal having a resolution lower than the full resolution.

6. The method according to claim 1 or 2, wherein the decoder (530) selectively implements the signal decoding operation based on a target level of processing capacity or battery power consumption to be used by the decoder (530).

7. The method according to claim 1 or 2, wherein the watermark information (140) indicates that the decoder should initiate the application of a watermark operation to the decoded signal.

8. The method according to claim 1 or 2, wherein the watermark information (140) includes compliance information associated with the signal.

9. The method according to claim 8, wherein the compliance information includes any one of the following information: the manner in which the signal is generated, a specific encoder version used to generate the signal, license information associated with the signal, and / or the encoder version that generated the signal.

10. The method according to claim 8, comprising: when decoding the watermark information, prompting the decoder to initiate a compliance process for the signal.

11. The method according to claim 10, wherein the compliance process includes initiating an interruption of the display of the signal.

12. The method according to claim 1 or 2, wherein the watermark information (140) includes a mark associated with one or more elements of the signal.

13. The method according to claim 12, wherein the mark includes an indication of whether an element of the signal can be selected by an end user of the signal.

14. The method according to claim 12, wherein the mark identifies whether an element of the signal is related to an action to be taken by an end user of the signal.

15. The method according to claim 14, wherein the action comprises clicking on the element.

16. The method according to claim 12, wherein the marker identifies the element of the signal belonging to a class of objects.

17. A method for encoding a signal, comprising: Encoding transform coefficients for signal reconstruction by retaining one or more quantization symbols (100 - N) of a given group of coefficients to provide embedded signaling information for a signal decoding operation to be performed on one or more parts of the signal, wherein the information comprises an indication of watermark information (140) associated with the signal. Switching at least one bit in the encoded bitstream of the signal to indicate to a decoder that a given group of symbols in a given group of residual data is to be interpreted as residual data or as additional embedded signaling information carrying watermark information (140).

18. The method according to claim 17, wherein the encoding uses a layer - based hierarchical encoding method.

19. The method according to claim 17 or 18, wherein the format for encoding at least a part of the signal and the watermark information is MPEG - 5 Part 2 LCEVC.

20. The method according to claim 17 or 18, wherein the format for encoding at least a part of the signal and the watermark information is SMPTE VC - 6 ST - 2117.

21. A decoder (530) configured to perform the method according to any one of claims 1 to 16.

22. An encoder (510) configured to perform the method according to any one of claims 17 to 20.

Images