Use of transform coefficients to provide embedded signaling for watermarks

Abstract

Examples described herein relate to decoding and encoding signals. A method of performing a signal decoding operation on one or more portions of a signal is presented. The performing is based at least in part on information embedded in one or more values received in one or more encoded data layers transmitted within an encoded data stream, where the values are associated with transform coefficients intended to be processed by a decoder to derive elements of the signal, and wherein the information comprises an indication of watermark information associated with the signal.

Description

[0001] This application is a divisional application of Chinese patent application CN 202080083191.6, with the invention name "Use of transform coefficients to provide embedded signaling for watermarks" and the application date of October 2, 2020. Technical Field

[0002] The present invention relates to methods for processing signals, such as, by way of non-limiting example, video, images, hyperspectral images, audio, point clouds, 3DoF / 6DoF, and volumetric signals. Processing data may include, but is not limited to, obtaining, deriving, encoding, outputting, receiving, and reconstructing signals in the context of a hierarchical (layer-based) coding format (wherein the signals are decoded in layers at subsequently higher quality levels), utilizing and combining subsequent layers ("steps") of reconstructed data. Signals at different layers may be encoded using different coding formats (e.g., by way of non-limiting example, conventional single-layer DCT-based codecs, ISO / IEC MPEG-5 Part 2 Low Complexity Enhancement Video Coding (SMPTE VC-6 2117), etc.), using different elementary streams that may or may not be multiplexed into a single bitstream. Background Art

[0003] In layer-based coding formats, such as ISO / IEC MPEG-5 Part 2 LCEVC (hereinafter referred to as "LCEVC") or SMPTE VC-6 2117 (hereinafter referred to as "VC-6"), a signal is decomposed into multiple "levels" (also called "hierarchical layers") of data ranging from the highest level at the sampling rate of the original signal to the lowest level at a sampling rate that is typically lower than the original signal, with each level corresponding to a "quality level" (also referred to herein as "LoQ") of the signal. In a non-limiting example, when the signal is a frame of a video stream, the lowest level may be a thumbnail of the original frame, such as a low-resolution frame in the video stream, or even just a single image element. The other levels contain information about the corrections applied to the reconstructed reproduction in order to produce the final output. The levels may be based on residual information, such as the difference between a version of the original signal at a particular quality level and a reconstructed version of the signal at the same quality level. The lowest level may not contain residual information, but may contain the lowest sample of the original signal. The decoded signal of a given quality level is reconstructed by first decoding the lowest rung (thereby reconstructing the signal at the first lowest quality level), then predicting a reproduction of the signal at the second quality level - the next higher quality level, then decoding the corresponding second rung reconstruction data (also called the second quality level "residual data"), then combining the predicted data with the reconstructed data in order to reconstruct a reproduction of the second higher quality level signal, and so on, until a given quality level is reconstructed.

[0004] Reconstructing the signal may include decoding residual data and using the residual data to correct a version of the signal at a particular quality level that is derived from a version of the signal at a lower quality level. Data at different levels may be encoded using different coding formats, and the sampling rates (e.g., resolution in the case of image or video signals) may be different for different quality levels. Subsequent levels may refer to the same signal resolution (i.e., sampling rate) of the signal, or to increasingly higher signal resolutions. Examples of these methods are described in more detail in the available specifications for LCEVC and VC-6.

[0005] Digital watermarking technology is known in the art. One example digital watermarking technology is the Advanced Television Systems Committee (ATSC) Video Watermark Emission Standard A / 335. The document defining this standard, for example, as published on September 20, 2016, is incorporated herein by reference.

[0006] In A / 335, the top one or two lines of the video signal are used to embed watermark information. In A / 335, the brightness values ​​of the pixels within these lines are modified to carry the watermark information. The receiver extracts the previous one or two lines of each frame of the video signal and applies a set of threshold operations to recover the encoded data.

[0007] One problem with A / 335 and similar digital watermarking methods is that they require the embedded data to undergo various compression and transcoding operations, as well as transmission over conventional consumer High-Definition Multimedia Interface (HDMI). To achieve robustness, more complex thresholding operations are applied at the receiver. A / 335 is also not intended to be tamper-proof or inerasable; it can be intentionally erased by an intermediary. Summary of the Invention

[0008] The non-limiting embodiments described herein refer to signals as sequences of samples (i.e., two-dimensional images, video frames, video fields, sound frames, etc.). In the description, the terms "image," "picture," or "plane" (intended to have the broadest meaning of "hyperplane," i.e., an array of elements having arbitrary dimensions and a given sampling grid) will often be used to identify a digital representation of the signal samples along the sequence of samples, where each plane has a given resolution for each of its dimensions (e.g., X and Y) and contains a set of planar elements (or "elements" or "pixels," or display elements for two-dimensional images, often referred to as "pixels," for volumetric images, often referred to as "voxels," etc.), characterized by one or more "values" or "settings" (e.g., by way of non-limiting example, a color setting in a suitable color space, a setting indicating density levels, a setting indicating temperature levels, a setting indicating audio pitch, a setting indicating amplitude, a setting indicating depth, a setting indicating alpha channel transparency levels, etc.). Each planar element is identified by a suitable set of coordinates indicating the integer position of the element in the sampling grid of the image. The signal dimensions may comprise only spatial dimensions (eg in the case of images) or may also comprise temporal dimensions (eg in the case of a signal evolving over time like a video signal).

[0009] As non-limiting examples, the signal can be an image, an audio signal, a multi-channel audio signal, a telemetry signal, a video signal, a 3DoF / 6DoF video signal, a volumetric signal (e.g., medical imaging, scientific imaging, holographic imaging, etc.), a volumetric video signal, or even a signal with more than four dimensions.

[0010] For simplicity, the non-limiting embodiments shown herein generally refer to signals, such as video signals, displayed as arranged 2D planes (e.g., 2D images in a suitable color space). The terms "picture," "frame," or "field" will be used interchangeably with the term "image" to indicate samples of a video signal in time: any concepts and methods shown for a video signal consisting of frames (progressive video signals) can also be easily applied to a video signal consisting of fields (interlaced video signals), and vice versa. Although the embodiments shown here focus on images and video signals, those skilled in the art can easily understand that the same concepts and methods are also applicable to any other type of multi-dimensional signal (e.g., audio signals, volumetric signals, stereoscopic video signals, 3DoF / 6DoF video signals, plenoptic signals, point clouds, etc.).

[0011] The embodiments described herein allow for the efficient generation, signaling, and utilization of context information that can be used by a decoder. This context information can be used in conjunction with residual data to appropriately control signal reconstruction. Such information can be effectively embedded within the coefficients of the residual data for one or more steps of the coded signal, thereby avoiding the need for additional signaling overhead. Furthermore, for some non-limiting embodiments described herein, applying certain non-essential signal enhancement operations at lower signal resolutions within the context of a hierarchical coding scheme also results in a substantial reduction in the processing power required by the decoder apparatus.

[0012] According to a first non-limiting embodiment, a signal processor (e.g., computer processor hardware) is configured to receive data and decode it ("decoder"). The decoder obtains a reconstruction of the signal at a first (lower) quality level and produces a predicted reconstruction of the signal at a second (higher) quality level, the second quality level having a higher resolution (i.e., signal sampling rate) than the first quality level. The decoder then receives and decodes a step of residual data applied to the predicted reconstruction of the signal to produce a corrected reconstruction of the signal at the second quality level. When decoding a particular set of residual data coefficients and finding a particular set of quantization symbols, the decoder does not interpret the symbols as residual data, but instead performs a watermarking operation based on the received symbols.

[0013] Thus, according to a non-limiting aspect of the present invention, the embedded information may include an indication of a characteristic associated with the signal, wherein the characteristic is watermark information. For example, the characteristic may be used to identify and confirm the encoder that generated the data stream and / or contain information about the time and location of the encoding.

[0014] In some non-limiting embodiments, watermark information can be used, for example, to identify a characteristic of a signal. The watermark information can indicate to a decoder that the watermark should be applied to the decoded signal. In other non-limiting embodiments, the characteristic corresponds to a state associated with the signal. The state includes compliance information associated with the signal. By way of non-limiting example, the compliance information can include any of the following: the manner in which the signal was generated, the specific encoder version used to generate the signal, license information associated with the signal, and / or the encoder version used to generate the signal. The compliance information can be useful for a decoder to initiate compliance actions, such as obtaining a valid license to generate the signal, upon detecting a mismatch between the compliance information and a recording. In this case, for example, the decoder can initiate compliance procedures on the signal, such as interrupting display or playback of the signal, sending a request to the source transmitting the signal to obtain a valid license, and the like.

[0015] In other non-limiting embodiments, the feature identifies an object in the signal. In a non-limiting embodiment, the feature allows identification of an object in the video corresponding to a unique identifier known to a decoder.

[0016] As non-limiting examples, the feature can include a tag associated with one or more elements of the signal. The tag can include an identification of whether the element of the signal can be selected by an end user of the signal. In other non-limiting embodiments, the tag can include an identification of whether the element of the signal can be related to an action to be taken by an end user of the signal, such as clicking on the element and / or linking to a different signal / webpage. In another non-limiting embodiment, the tag includes an identification of the element of the signal belonging to a category, such as a video category or an object category. As a non-limiting example, the element can represent a person, and the tag identifies who that person is. Alternatively, the element can represent an object, and the tag can identify what that object is. Alternatively, the tag can identify which category the object belongs to. In general, the category can include an association of the element with a class of identifiers, such as a category to which the element belongs.

[0017] In other non-limiting embodiments, the reserved symbols are used to embed a different secondary signal as part of the encoded stream, said different secondary signal being encoded by a given public key and only decodable by a decoder aware of both the existence of the secondary signal and of a private key corresponding to the public key used to encrypt the secondary signal.

[0018] In some non-limiting embodiments, the bits in the decoded byte stream signal to the decoder that additional information can have been embedded in some of the residual data coefficients, so that the particular sets of symbols in particular sets of residual data should not be interpreted as actual residual data, but rather as contextual information informing the signal reconstruction operations, for example. In non-limiting embodiments, some of the reserved symbols correspond to particular regions of the signal, i.e., whereby the watermark can be applied differentially to local blocks of the reconstructed signal.

[0019] In some non-limiting embodiments, the decoder implements the signal reconstruction operations differently based on the processing power available to the decoder device at any time (including sometimes not implementing these signal reconstruction operations at all).

[0020] In some non-limiting embodiments, the decoder applies a signal processing operation based on contextual data in the loop before applying the decoded residual data containing the embedded information. In other non-limiting embodiments, the decoder applies a signal processing operation after having combined the preliminary rendition of the second quality level signal with the decoded residual data. In other non-limiting embodiments, the decoder applies said signal processing operation at the end of the decoding process, after having produced a rendition of the final (highest) quality level signal.

[0021] In some non-limiting embodiments, the format for encoding the residual data and embedded context information is MPEG-5 Part 2 LCEVC ("Low Complexity Enhancement Video Coding"). In other non-limiting embodiments, the format for encoding the residual data and embedded context information is SMPTE VC-6 ST-2117. The signal processing operations using the watermark information may be operations that are not defined in one or more of the LCEVC or VC-6 video coding standards, for example, including so-called non-standard operations that are still compatible with the standards. This example can therefore be implemented as an optional extension to the aforementioned standards, or other layer-based hierarchical coding schemes.

[0022] According to a second non-limiting embodiment, a signal processor (e.g., computer processor hardware) is configured to receive data and encode it ("encoder"). The encoder generates a rendition of a first (lower) quality level signal and encodes it using a first encoding method. The encoder then generates a predicted rendition of a second (higher) quality level signal and, in turn, generates and encodes a step of second quality level residual data applied to the predicted rendition of the second quality level signal to produce a corrected rendition of the second quality level signal. In these embodiments, the encoder uses a set of reserved symbols in the set of residual data for the step of residual data to signal watermark information to a decoder.

[0023] In some non-limiting embodiments, bits in the encoded byte stream are toggled to signal to a decoder whether a given set of symbols in a given set of residual data should be interpreted as actual residual data or as additional context information that informs a signal reconstruction operation such as watermarking to be applied by the decoder.

[0024] In some non-limiting embodiments, contextual information is embedded in the residual data of more than one level.

[0025] In some non-limiting embodiments, the format used to encode the residual data and embedded context information is MPEG-5 Part 2 LCEVC ("Low Complexity Enhancement Video Coding"). In other non-limiting embodiments, the format used to encode the residual data and embedded context information is SMPTE VC-6 ST-2117.

[0026] According to other non-limiting embodiments, the context signal information is embedded in the coded data generated in a non-hierarchical coding format.In a non-limiting embodiment, the symbols are embedded at the macroblock level using a set of reserved symbols in the quantized coefficients.

[0027] Other features and advantages will become apparent from the following description, given by way of example only, with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A block diagram illustrating an example of a decoding system according to an embodiment;

[0029] Figure 2 A block diagram illustrating an example of a hierarchical encoding system according to an embodiment;

[0030] Figure 3 shows a block diagram of a hierarchical decoding system according to an embodiment;

[0031] Figure 4 A block diagram showing another example of a hierarchical decoding system according to an embodiment;

[0032] Figure 5 A block diagram showing another example of an encoding and decoding system according to an embodiment; and

[0033] Figure 6 A block diagram illustrating an example of an apparatus according to an embodiment is shown. DETAILED DESCRIPTION

[0034] refer to Figure 1 , shows an example of a method implemented within a decoding system. A set of quantized symbols 100-1 to 100-N are received and processed. These quantized symbols contain quantized transform coefficients, where the quantization may be optional and / or vary based on the coding configuration. The quantized symbols may include symbols generated by one or more of the coded streams described with respect to the following figures. In the examples described herein, information is embedded in one or more values ​​received in one or more coded data layers, where the values ​​are associated with transform coefficients of elements intended to be processed by the decoder to derive a signal. Example quantization and reception of transform coefficient symbols are described in LCEVC and VC-6. The decoder follows two different approaches depending on whether the symbol 100-1 is a reserved symbol or not. In this context, the term "reserved symbol" may be considered to refer to symbols that are reserved to carry contextual information such as watermark information.

[0035] If symbol 100-1 is not a reserved symbol, for example, intended to carry residual data for reconstructing the signal, its decoding follows the normal process implemented for other symbols in the group: dequantization and inverse transformation according to block 110 are performed, resulting in a set of decoded data 130. The decoded data is further processed by means of a decoding operation 150 to produce a decoded signal 160. This normal process may be, for example, the process described in a decoding specification such as LCEVC or VC-6.

[0036] If symbol 100-1 is to be a reserved symbol, its decoding follows a different process than that shown in comparison to block 105. At block 120, the embedded information is decoded by processing symbol 100-1 to produce watermark information 140. At block 170, decoded signal 160 is processed along with watermark information 140, where one or more additional operations 170 are performed. These operations may include enhancement operations, such as determining whether to interrupt display of the decoded signal based on watermark information 140, and / or initiating compliance procedures, such as checking whether the encoder is properly licensed and / or has correctly generated symbols 100-1 through 100-N. The output of additional operations 170 may include an enhanced reconstruction 180 of the signal. For example, tags associated with one or more elements of the signal encoded within the watermark information may be processed to enable user actions. These elements may refer to pixels or regions of a video frame. For example, a user viewing the reconstructed video may be able to perform actions associated with the tagged elements. In other cases, the enhanced reconstruction of the signal may include modified content based on the watermark information.

[0037] In some non-limiting embodiments, a bit in the decoded bit or byte stream (not shown) signals to the decoder that symbol 100-1 is to be interpreted as a reserved symbol. For example, as described in more detail below, this bit may comprise a "user data" flag that is turned "on" or "off" in global configuration information. In some non-limiting embodiments, the decoder implements signal processing operations, such as additional operation 170, in different ways (including sometimes not implementing a signal processing operation at all) based on the processing power available to the decoder device at the time of processing.

[0038] refer to Figure 2 , shows an example of a method implemented within a coding system, in this case a layer-based hierarchical coding method is implemented. Figure 2 The blocks in FIG. 2 may be implemented by an example encoder. A source signal 200 at quality level #2 (e.g., full resolution and quality) is received and processed by a downsampler 210, producing a downsampled signal 200-1. The downsampled signal 200-1 is processed by an encoder 220, which applies a given encoding method (also a layer-based hierarchical encoding method in some non-limiting embodiments, and a non-hierarchical encoding method in other non-limiting embodiments), producing encoded data 225. The encoder 220 may be referred to as a "basic" decoder.

[0039] The coded data 225 and the downsampled signal 200-1 are processed by an LOQ#1 residual data generator 230 to produce the coded data 235 and a reconstruction 237 of the LOQ#1 signal. The LOQ#1 residual data generator 230 may generate a residual signal by subtracting a reconstruction based on the coded data 225 from the downsampled signal 200-1. The LOQ#1 residual data generator 230 may also encode the residual signal by applying a coding unit transform and quantizing the output of the transform. A further entropy coding stage may also be applied. The output of the transform and quantization may include (quantized) transform coefficients modified to include embedded signaling. The reconstruction 237 of the LOQ#1 signal is further processed by an LOQ#2 preliminary reconstruction generator 240 to produce a preliminary reconstruction 245 of the LOQ#2 signal. For example, this may include upsampling 237 the reconstruction of the LOQ#1 signal with optional modifications to generate a signal at the LOQ#2 resolution and / or sampling rate.

[0040] The preliminary reproduction 245 of the LOQ#2 signal is processed along with the source signal 200 by the LOQ#2 residual generator 260 to produce encoded data 265. The encoded data 265 may include a residual signal generated by the LOQ#2 residual generator 260 by subtracting the preliminary reproduction of the LOQ#2 signal from the source signal 200. The LOQ#2 residual signal generator 260 may apply similar operations to the LOQ#1 generator 230, but based on a residual signal at a second quality level (e.g., higher resolution).

[0041] The coded data sets 225, 235 and 265 are then processed by a multiplexer (Mux) 270 to produce a signal 280 encoded with the ladder coded data. Although all three coded data sets are in Figure 2 In VC-6, the coded data stream may contain all sets of coded data 225, 235, 265; in LCEVC, the coded (enhancement) data stream may contain the coded data 235 and 265, and the coded data 225 may form a separate elementary stream.

[0042] In certain described embodiments, when encoding a first quality level signal, LOQ#1 residual generator 230 generates encoded data 235 using a set of reserved symbols to signal watermark information to a decoder. This may include watermark information converted from the well-known A / 335 watermark standard, as discussed in the background. As is known in the art and explained in the A / 335 standard documentation, watermark information may comprise data embedded in a noise-tolerant signal for use by a decoder. Watermark information can be used for a variety of purposes. These include supporting programming elements associated with the signal, such as those required to support interactivity, dynamic content replacement or overlay, service usage monitoring, and content identification. A common use of watermark information is to identify the ownership of intellectual property associated with a signal and / or elements within the signal. It should be noted that references to elements herein include references to pixels or planar elements associated with objects in the signal discernible by a human observer. Watermark information, also known as digital watermarks, can also be used to verify the authenticity or integrity of the encoded signal. This watermark information can be added during the encoding process and extracted during the decoding process.

[0043] In one embodiment, the LOQ#1 residual generator 230 replaces the quantized transform coefficient values ​​of specific transform coefficients (e.g., specific elements of a vector generated by multiplying the transform matrix) with embedded signaling data. Only one coefficient value may be modified; the other coefficients may remain unmodified and be encoded according to a comparative encoding used in standardized decoding procedures. Coefficients may be selected that minimize the variance of the reconstructed signal, such as the H or HH coefficients for a 2×2 or 4×4 Hadamard transform.

[0044] In some non-limiting embodiments, LOQ#1 residual generator 230 switches specific bits in the coded bitstream or bytestream to signal to the decoder whether a given set of symbols in the coded data set 235 is to be interpreted as actual residual data or as additional context information to inform the signal decoding operation. In some non-limiting embodiments, methods 230 and 260 perform in-loop signal processing operations based on the information signaled by the retained coefficients to reconstruct the signal from the coded data 265.

[0045] refer to Figure 3 , shows an example of the method implemented in a decoding system, and also implements a layer-based hierarchical encoding method. Figure 3 The blocks in FIG. 2 may be implemented by an example decoder. The encoded data 225 is received and processed by the lower LOQ decoder 300. Figure 2 As shown, the coded data 225 can be obtained by demultiplexing the signal 280 encoded with the ladder-coded data received from the encoder. The lower LOQ decoder 300 can be referred to as a basic decoder. The lower LOQ decoder 300 outputs a preliminary reconstruction 310 of the LOQ#1 signal.

[0046] The preliminary rendition 310 of the LOQ#1 signal is then processed by a LOQ#1 reconstructor 320 along with the encoding data 235 in order to produce a rendition 337 of the LOQ#1 signal. As shown, the encoding data 235 can be obtained from the signal 280 received by the encoder encoding the ladder encoding data by demultiplexing. The rendition 337 of the LOQ#1 signal corresponds to the rendition 237 of the LOQ#1 signal in Figure 2 LOQ#1 signal in Figure 2 The encoding data 235 can be decoded by the LOQ#1 reconstructor 320 and then combined with the preliminary rendition 310 of the LOQ#1 signal. In one case, the encoding data 235 can contain residual data as described in other examples herein. The decoding can include applying operations as shown in blocks 110 and 410 in Figure 1 and 4 LOQ#1 signal in

[0047] The rendition 337 of the LOQ#1 signal is then processed by a LOQ#2 preliminary rendition generator 340 (which can correspond to the LOQ#2 preliminary rendition generator 240 in Figure 2 LOQ#2 signal in

[0048] In some non-limiting embodiments, when a particular set of data within the encoding data 235 is encoded and a particular set of quantization symbols is found, the decoder does not interpret the symbols as residual data, but rather performs a signal processing operation according to the received symbols.

[0049] In some non-limiting embodiments, bits in a decoded byte stream (not shown in Figure 3 LOQ#1 reconstructor 320 and / or the LOQ#2 reconstructor 360 signal that additional information can have been embedded in some of the residual data coefficients, more specifically, that a particular set of quantization symbols in a particular set of residual data (or part of the symbols) should not be interpreted as actual residual data, but rather as contextual information informing a signal enhancement operation.

[0050] In some non-limiting embodiments, the decoder implements the signal processing operations differently based on processing power available to the decoder device at the time of decoding, including sometimes not implementing these signal processing operations at all. For example, the flag and / or classification can be extracted from the reserved symbols only when processing power is available.

[0051] In some non-limiting embodiments, the decoder applies the watermark information based signal processing method in-loop, before applying the decoding of the residual data containing the stepped data with embedded watermark information. In other non-limiting embodiments, the decoder applies the watermark information based signal processing method in-loop, after the preliminary rendition of the second quality level signal has been combined with the decoded residual data. In other non-limiting embodiments, the decoder applies the watermark information based signal processing method at the end of the decoding process, after the rendition of the final (highest) quality level signal has been generated. In yet another non-limiting embodiment, the decoder applies the watermark information based signal processing method both in-loop and at the end of the decoding process. Certain examples regarding Figure 4 are described later.

[0052] In preferred examples, the encoder or decoder is part of a layer-based hierarchical coding scheme or format. Examples of layer-based hierarchical coding schemes include LCEVC: MPEG-5 Part 2 LCEVC (“Low Complexity Enhancement Video Coding”) and VC-6: SMPTE VC-6 ST-2117, the former described in PCT / GB2020 / 050695 (and associated standard documents), the latter described in PCT / GB2018 / 053552 (and associated standard documents), all of which are incorporated by reference herein. However, the concepts shown herein are not necessarily limited to these particular hierarchical coding schemes. The skilled person will appreciate how the above-described encoder and decoder methods apply to base and enhancement layers in LCEVC (e.g. LOQ#1 corresponds to the base layer and LOQ#2 corresponds to the enhancement layer). As such, in some cases, the encoded data 235, the encoded data 265 and the corresponding encoded format with embedded context information is MPEG-5 Part 2 LCEVC. In this case, the encoded data 235 and the encoded data 265 can contain different enhancement sub-layers. In this case, the embedded context information can be referred to as “user data” in that it can contain information in addition to that required to reconstruct the signal according to the standard. In other cases, the encoded data 235, the encoded data 265 and the corresponding encoded format with embedded context information is SMPTE VC-6 ST-2117. Again, the embedded context information can contain data that is not required to reconstruct the signal according to the standard’s definition. It should be noted that references to “frames” of video data also include references to one or more planes of colour data (e.g. luma and chroma planes) as is known from layer-based coding methods such as LCEVC and VC-6.

[0053] With reference to Figure 4 , an example of a method implemented within a decoding system that also implements a layer-based hierarchical coding method is shown. The method is Figure 2In this variation, the watermark information can be used to process the initial reconstruction of the LOQ#1 signal (e.g. Figure 3 310 in ) and the reproduction of LOQ#2 signal (as Figure 3 In this way, the watermark information can be associated with one or more quality levels, including a basic quality level.

[0054] like Figure 2 As shown, in Figure 4 In the decoder, quantized symbol 400-1 is received and processed along with other quantized symbols 400-2 through 400-N. The quantized symbols may represent a residual data stream at the level of the encoded data 235. The decoder checks whether symbol 400-1 should be considered a reserved symbol. Depending on whether symbol 400-1 is a reserved symbol, the decoder follows two different approaches.

[0055] If symbol 400-1 is not a reserved symbol, its decoding follows the normal process implemented for other symbols in the group: dequantization and inverse transformation are performed according to block 410, resulting in a set of decoded residual data 420. This residual data (e.g., along with other residual data for the residual portion of the signal samples, for example, in a non-limiting embodiment) is further processed by a reconstructor 450 to produce a reproduction 460 of the LOQ#1 signal.

[0056] If symbol 400-1 is to be a reserved symbol, its decoding follows a different process. At block 430, an operation to decode embedded information 430 is initiated to process symbol 400-1 to generate watermark information 435. In this example, the watermark information may be used to control one or more of: additional operation 440, reconstructor 450, and additional operation 480. For example, watermark information 435 may include compliance information, and additional operations 440 and 480 may include respective compliance processes performed on preliminary reproduction 310 of LOQ#1 signal and reproduction 470 of LOQ#2 signal. For example, the watermark information may indicate that a user is only permitted to access a lower resolution video signal; in this case, watermark information 435 may indicate that preliminary reproduction 310 of LOQ#1 signal may be processed to output reproduction 460 of LOQ#1 signal, but reproduction 470 of LOQ#2 signal may not be output (e.g., to restrict final reproduction 490 of LOQ#2 signal). In this case, if the user is not allowed any access to either video stream, both the preliminary reproduction 310 of the LOQ#1 signal and the reproduction 470 of the LOQ#2 signal may be limited or not output based on the watermark information 435. In other cases, the watermark information 435 may include tags associated with elements of the signals at different quality levels. Additional operations 440 and 480 may therefore include generating a metadata signal associated with the main signal (e.g., a set of tags for pixels in the video signal) that allows a user to initiate an action with respect to the element, such as clicking on an element within one or more of the preliminary reproduction 310 of the LOQ#1 signal and the reproduction 470 of the LOQ#2 signal. For example, the watermark information 435 may include information associated with an object shown within the signal, such as an actor or a specific model of a product, allowing the end user to click on a pixel associated with the object and be provided with further information about the object.

[0057] In more detail, additional operation 440 processes the preliminary reproduction 310 of the LOQ#1 signal and the watermark information 435. This produces a processed preliminary reproduction 445 of the LOQ#1 signal. The processed preliminary reproduction 445, along with the residual data 420, is further processed by a reconstructor 450 to produce an enhanced reproduction 460 of the LOQ#1 signal. For example, the reconstructor 450 may include a Figure 3 1, and the enhanced reconstruction 460 of the LOQ#1 signal may correspond to Figure 34. The enhanced reproduction 460 of the LOQ#1 signal may be enhanced by adding residual data 420 and by any processing of the initial reproduction 310 of the LOQ#1 signal performed by additional operations 440. In one embodiment, the decode embedded information block 430 may extract watermark data 435 and residual data for the quantized symbol 400-1, e.g., the quantized symbol 400-1 may carry both watermark data 435 and residual data. This may be achieved, for example, by partitioning the bits of the quantized symbol 400-1 and applying a higher level of quantization to the original symbols representing the transform coefficients of the residual data. For example, the decode embedded information block 430 may perform operations similar to those of the dequantization and inverse transform block 410, or may alternatively pass the zero watermark component of the quantized symbol 400-1 back to the quantization and inverse transform block 410 to derive the residual data. In any case, the reconstructor 450 may additionally receive the residual data for the quantized symbol 400-1.

[0058] The reconstructor 450 thus generates an enhanced reproduction 460 of the LOQ#1 signal, which is further processed by a decoding operation 465 to produce a reproduction 470 of the LOQ#2 signal. The decoding operation 465 may be a Figure 2 The operations associated with the LOQ#2 initial reconstruction generator 340 and the LOQ#2 reconstructor 360 in FIG. 4 are shown in FIG. 4 . Thus, it can be seen that the reconstruction 470 of the LOQ#2 signal corresponds to Figure 3 Reproduction in 370. Figure 4 4, the rendering of the LOQ#2 signal 470 is processed along with the watermark information 435 by additional operations 480 to produce a final rendering of the LOQ#2 signal 490. As described above, the additional operations 480 may include implementing a compliance process to limit the output of the final rendering of the LOQ#2 signal 490 and / or generating additional metadata, such as tags of elements of the final rendering of the LOQ#2 signal 490. The additional operations 480 may also include interrupting the display of the final rendering of the LOQ#2 signal 490 based on the results of the compliance process.

[0059] In some cases, if the quantized sign 400-1 still carries transform coefficient values (e.g., by splitting the bit capacity between the watermark information and the transform coefficient values) and / or the transform coefficients are chosen to be from coefficients of a larger (e.g., 4x4) transform that is found to reduce visual impact (e.g., the HH coefficients in a 4x4 Hadamard transform), then the visual impact on the resulting output signal (e.g., the final rendition 490 of the LOQ#2 signal) is minimized. Additionally, if the precision of the transform coefficient values is reduced or replaced at the LOQ#1 level, then the LOQ#2 residual generator 260 is able to generate residual data that becomes the encoding data 265 that corrects for the difference between the preliminary rendition 245 of the LOQ#2 signal and the LOQ#2 source signal 200. This is an improvement over watermarking standards such as A / 335, where there are noticeable black or gray lines and pixel changes at the top of the image.

[0060] An additional benefit of the proposed watermarking method is that the watermark is performed as part of the encoding, so it is not destroyed or modified by subsequent encoding or compression. For example, comparative watermarking standards such as A / 335 apply the watermark information to the LOQ#2 source signal 200, where the watermarked signal is subsequently encoded. This means that more complex thresholding is required at the receiver, as the original luminance values can be modified as part of the encoding and decoding process (e.g., if this is a lossy process). The presently described method is also more secure, as it is more difficult to access and alter the picture information; the encoding and decoding operations can be protected and / or the watermark information itself can be encrypted (e.g., using public key cryptography).

[0061] Reference Figure 5 shows an example of a method implemented within an encoding and decoding system that utilizes the innovative methods described herein. Figure 5 shows how the proposed method of embedding watermark information can be used with a legacy decoder. An encoder 510 processes an original signal 500 to produce a data stream 520. The encoder 510 can encode the data stream 520 as explained with reference to Figure 2 , and can embed watermark information in one or more values received in one or more encoded data layers transmitted within the data stream 520, where the values are associated with transform coefficients of elements intended to be processed by a decoder to derive a signal, as described with reference to Figure 2 or Figure 4 .

[0062] The data stream 520 is processed by two decoders. A decoder 530-0 implements a signal processing method according to information signaled within the preservation sign by the encoder 510, thereby decoding a reconstructed signal 540-0. The reconstructed signal 540-0 can be augmented with metadata, such as labeling or classification information derived from the watermark information within the preservation sign.

[0063] In Figure 5 , the decoder 530-1 also reconstructs the reconstructed signal 540-1, but ignores the information signaled by the encoder 510 within the reserved symbols. In some non-limiting embodiments, the reconstructed signal 540-1 is a fully viable reconstruction of the signal for the given purpose, such that the additional operations performed by the decoder 530-0 are fully optional. For example, the decoder 530-1 can be a decoder that applies the decoding process set forth in the LCEVC standard, while the decoder 530-0 can be a decoder that implements a non-standard decoding process (in some cases, in addition to the decoding process set forth in the LCEVC standard). Thus, additional functionality can be provided based on the watermark information, while remaining compliant with the LCEVC standard.

[0064] In some non-limiting embodiments, the decoder 530-0 can sometimes decide to ignore part of the information signaled by the encoder 510 within the reserved symbols. For example, the decoder 530-0 can define whether to ignore part of the information signaled within the reserved symbols based on information including one or more of the resolution and frame rate of the signal, the processing power load at the time of decoding, and the battery power state.

[0065] In some cases, backward compatibility is achieved, e.g., as described above, because the decoder 530-1 treats the reserved symbols as normal quantized transform coefficient values and decodes them appropriately. The correction applied in a layer-based hierarchical format means that any errors can be corrected. Alternatively, bits in the encoded data stream 520 are used to signal to the decoder 530 that one or more values should be interpreted as said information rather than actual quantized values of transform coefficients. In yet another case, the bit depth assigned to a particular transform coefficient value (e.g., depth D of 8 or 16 bits) can be shared between the reserved symbol and the (quantized) transform coefficient value. For example, n least significant bits of the transform coefficient value (where n is less than the bit depth, e.g., 2 or 6 bits) can be used to carry the reserved symbol (i.e., the watermark information), which indicates a more aggressive quantization applied to the transform coefficient value carrying this symbol, but still enables the coarse level of information (D-n bits) to be transmitted and used for reconstructing the residual data. By selecting transform coefficients determined (e.g., by experiment) to be less perceptually significant in the reconstructed output (e.g., H or HH in 2x2 or 4x4 Hadamard transforms), the visual impact can be further minimized.

[0066] Referring to Figure 6 , a schematic block diagram of an example of a device 600 is shown.

[0067] Examples of apparatus 600 include, but are not limited to, a mobile computer, a personal computer system, a wireless device, a base station, a telephone device, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, an application server, a storage device, a consumer electronic device such as a camera, a camcorder, a mobile device, a video game console, a handheld video game device, a peripheral device such as a switch, a modem, a router, a vehicle, etc., or generally any type of computing or electronic device.

[0068] In this example, device 600 includes one or more processors 612 configured to process information and / or instructions. One or more processors 612 may include a central processing unit (CPU). One or more processors 612 are coupled to bus 611. Operations performed by one or more processors 612 may be implemented by hardware and / or software. One or more processors 612 may include multiple co-located processors or multiple remotely located processors.

[0069] In this example, device 600 includes computer-usable memory 613 configured to store information and / or instructions for one or more processors 612. Computer-usable memory 613 is coupled to bus 611. Computer-usable memory 613 may include one or more of volatile and non-volatile memory. Volatile memory may include random access memory (RAM). Non-volatile memory may include read-only memory (ROM).

[0070] In this example, device 600 includes one or more external data storage units 680 configured to store information and / or instructions. The one or more external data storage units 680 are coupled to device 600 via I / O interface 614. The one or more data storage units 680 may, for example, include magnetic or optical disks and disk drives or solid-state drives (SSDs).

[0071] In this example, the device 600 also includes one or more input / output (I / O) devices 616 coupled via an I / O interface 614, which is configured to communicate information to and / or from the one or more processors 612. The device 600 also includes at least one network interface 617. Both the I / O interface 614 and the network interface 617 are coupled to the system bus 611. The at least one network interface 617 can enable the device 600 to communicate via one or more data communication networks 690. Examples of data communication networks include, but are not limited to, the Internet and a local area network (LAN). The one or more I / O devices 616 can enable a user to provide input to the device 600 via one or more input devices (not shown). The one or more I / O devices 616 can enable information to be provided to the user via one or more output devices (not shown).

[0072] exist Figure 6 , a (signal) processor application 640-2 is shown as loaded into memory 1513. This can be executed by processor 612 as a (signal) processor process 640-1 to implement the methods described herein (e.g., implementing an appropriate encoder or decoder). Device 600 may also include additional features not shown for clarity, including an operating system and additional data processing modules. (Signal) processor process 640-1 may be implemented by computer program code stored in a memory location within a computer-usable non-volatile memory, a computer-readable storage medium within one or more data storage units, and / or other tangible computer-readable storage media. Examples of tangible computer-readable storage media include, but are not limited to, optical media (e.g., CD-ROM, DVD-ROM, or Blu-ray), flash memory cards, floppy disks, or hard disks, or any other medium capable of storing computer-readable instructions, such as firmware or microcode, in at least one ROM or RAM or programmable ROM (PROM) chip, or as an application-specific integrated circuit (ASIC).

[0073] Device 600 can thus include a data processing module that can be executed by one or more processors. The data processing module can be configured to include instructions for implementing at least some of the operations described herein. During operation, one or more processors start, run, execute, interpret or otherwise execute instructions.

[0074] Although at least some aspects of the examples described herein with reference to the accompanying drawings include computer processes executed in a processing system or processor, the examples described herein also extend to computer programs, such as computer programs on or in a carrier, which are suitable for putting the examples into practice. A carrier can be any entity or device capable of carrying a program. It should be understood that the apparatus 600 may include more, fewer, and / or more computer programs. Figure 6 The components shown are different components. Device 600 can be located at a single location or can be distributed across multiple locations. Such locations can be local or remote.

[0075] As described in the examples herein, a signal processor (e.g., computer processor hardware) is configured to receive data and decode it (a "decoder"). The decoder obtains a reproduction of the signal at a first (lower) quality level and detects the remaining symbols that specify watermark information. The decoder reconstructs a reproduction of the signal at a second (next higher) quality level and may apply further processing to the reproduction using the watermark information.

[0076] In certain examples described herein, the reserved symbols can be carried as so-called user data of the encoded data stream. In these examples, the signal processing information is embedded in one or more values received in one or more encoded data layers transmitted in the encoded data stream. The values are associated with transform coefficients that are processed to derive elements of the signal during decoding, for example, the values can contain the value of a predefined transform coefficient within a set of different transform coefficients generated by the encoding transform.

[0077] A bit in the bitstream of the encoded data stream can be used to signal the presence of user data. The bit can contain a user_data_enable bit, which can be present in a global configuration header of the encoded data stream. In certain examples, the encoding of user data in place of one of the coefficients can be configured as follows. If the bit is set to "0", then the decoder interprets the data as the relevant transform coefficient. If the bit is set to "1", then the data contained in the relevant coefficient is treated as user data, and the decoder is configured to ignore the data, or the relevant coefficient is treated as carrying user data and a relevant process is performed to extract the data. For example, if the bit is set to "1", this can indicate that watermark information is being transmitted.

[0078] User data transmitted in this way can be used to enable the decoder to obtain supplemental information, including, for example, various feature extraction and derivation. Although the examples claimed herein relate to watermark information, user data can also be used to signal other optional parameters related to implementations outside of the standardized implementation.

[0079] In one case, the user_data_enable variable can be a k-bit variable. For example, user_data_enable can contain a 2-bit variable with the following values:

[0080] user_data_enable Type value 0 Disable 1 Enable 2 bits 2 Enable 6 bits 3 reserve

[0081] In this case, the user data specifying the watermark information can be embedded in the last n (least significant) bits of one or more sets of decoded coefficient data (e.g., within the encoded residual coefficient data).

[0082] As described in the examples herein, when user data is enabled, for example, to transmit signal processing information, then the "in-loop" processing of the transform coefficients can be modified. Two examples are shown in Figure 2 and 4In addition, the decoding of transform coefficients can also be adjusted so that when user data is enabled, the value of a particular transform coefficient (e.g., H or HH) is set to 0 before the transform coefficient is inversely transformed. In the case described in the table above, after extracting the n least significant bits as reserved symbols, the value of the transform coefficient used to carry user data can be right-shifted (e.g., bit-shifted) by 2 bits (>>2) if n=2 (e.g., user_data_enable=1), or by 6 bits (>>6) if n=6 (e.g., user_data_enable=1). In one case, if the value of the transform coefficient is D bits in length, where D>n, and n is the length of the user data in bits (e.g., 2 or 6 in the table above), the remaining Dn bits of the transform coefficient can be used to carry the value of the transform coefficient (e.g., a more quantized integer value than the full D-bit representation). In this case, the user data and the transform coefficient value can be divided across D bits. In other simpler cases, the user data may be extracted and the values ​​of the transform coefficients may be set to zero (ie, so that the values ​​of the transform coefficients have no effect on the output of the inverse transform).

[0083] In some instances, user data used to implement reserved symbols can be formatted according to a defined syntax. The defined syntax can divide the user data into header data and payload data. In this case, decoding of the user data can include parsing a first set of values ​​received in one or more coded data layers to extract the header data, and parsing a second subsequent set of values ​​received in one or more coded data layers to extract the payload data. The header data can be set to a first set of defined bits. For example, in the above-mentioned examples of defining user data as 2 or 6 bit values, the first x values ​​can contain the header data. In one case, x can be equal to 1, so that the first value of the user data (e.g., the transform coefficient value of the first coding unit or data block of a given video frame or plane) defines the header data (e.g., 2 or 6 bits of the first value define the header data).

[0084] In some instances, the header data may indicate at least whether a watermarking operation is enabled. Typically, the header data may indicate global parameters of the watermark information, while the payload data may indicate local parameters of the watermark information, i.e., the watermark information may be localized to one or more coding units containing m×m blocks of residual data (e.g., 2×2 or 4×4 blocks). Because the watermark information is encapsulated within reserved symbols for a particular coding unit, marking and / or classification (for example) may be applied to localized regions of the signal. For example, marking may indicate where a local region of the video signal is to be modified and / or replaced.

[0085] The techniques described herein may be implemented in software or hardware, or may be implemented using a combination of software and hardware. These techniques may include configuring a device to perform and / or support any or all of the techniques described herein.

[0086] The above embodiments are to be understood as illustrative examples. Further embodiments are contemplated.

[0087] It should be understood that any feature described with respect to any embodiment may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other embodiment, or in any combination of any other embodiment. In addition, equivalents and modifications not described above may also be employed without departing from the scope of the invention as defined by the appended claims.

Claims

1. A method of performing a signal decoding operation on one or more parts of a signal, wherein said performing is based at least in part on information embedded in one or more values ​​received in one or more coded data layers transmitted within a coded data stream, wherein said values ​​are associated with transform coefficients of elements intended to be processed by a decoder to derive a signal, wherein the information comprises an indication of watermark information associated with the signal.

2. The method of claim 1, wherein the one or more values ​​are interpreted by the decoder to derive the information rather than deriving values ​​of transform coefficients.

3. Method according to claim 1 or 2, wherein the signal is encoded by means of a layer-based hierarchical format.

4. The method of claim 3, wherein the embedded signaling is included in a residual layer having a resolution lower than a full resolution of the signal.

5. The method of claim 4, wherein at least one of the signal decoding operations performed based on the embedded signaling is performed in-loop on an intermediate reproduction of the signal at a resolution lower than full resolution.

6. The method of claim 1 or 2, wherein the decoder selectively performs signal decoding operations based on a target level of processing power or battery power consumption to be used by the decoder device.

7. The method of claim 1 or 2, wherein the watermark information indicates that the decoder should initiate application of a watermark operation to the decoded signal.

8. The method of claim 1 or 2, wherein the watermark information comprises compliance information associated with the signal.

9. The method of claim 8, wherein the dependency information comprises any of the following information: the manner in which the signal was generated, a specific encoder version used to generate the signal, licensing information associated with the signal, and / or the encoder version that generated the signal.

10. The method of claim 8, comprising: prompting the decoder to initiate a compliance procedure for the signal when decoding the watermark information. The method of claim 10 , wherein the compliance process includes initiating an interruption of display of the signal.

12. The method of claim 1 or 2, wherein the watermark information comprises a marker associated with one or more elements of the signal.

13. The method of claim 12, wherein the tag comprises an identification of whether an element of the signal is selectable by an end user of the signal.

14. The method of claim 12, wherein the tag identifies whether an element of the signal is relevant to an action to be taken by the end user of the signal. The method of claim 14 , wherein the action comprises clicking the element. The method of claim 12 , wherein the tag identifies elements of the signal that belong to a class of objects.

17. A method for encoding a signal, comprising: encoding transform coefficients for signal reconstruction by retaining one or more quantization signs of a given coefficient group to provide embedded signaling information for a signal decoding operation to be performed on one or more portions of the signal, wherein The information includes an indication of watermark information associated with the signal.

18. The method of claim 17, wherein the encoding uses a layer-based hierarchical encoding method.

19. The method according to claim 17 or 18, wherein the format in which at least a part of the signal and the watermark information are encoded is MPEG-5 Part 2 LCEVC ("Low Complexity Enhancement Video Coding").

20. The method of claim 17 or 18, wherein the format in which at least a portion of the signal and the watermark information are encoded is SMPTE VC-6ST-2117.

21. A decoder configured to perform the method according to any one of claims 1 to 16.

22. An encoder configured to perform the method of any one of claims 17 to 20.

Images