Method, device, computer program and computer-readable medium for processing configuration data

Abstract

For each image in a set of images (310), residual data is obtained based on a first representation of the image at a first quality level and a second representation of the image. The second representation is based on a representation of the image at a second, lower quality level. The residual data is usable by a decoder to reconstruct a first representation for each image in the set (310) using the second representation. (i) One or more configuration parameters (320) of configuration data related to processing of the residual data are obtained, (ii) the one or more configuration parameters are usable by a decoder to reconstruct the first representation for each image in the set (310), and (iii) the one or more configuration parameters correspond to a plurality of images in the set (310). The one or more configuration parameters (320) are output for processing by the decoder. The one or more configuration parameters (320) are output for the set (310) a number of times less than the number of images in the set (310).

Description

Technical Field

[0001] The present invention relates to methods, apparatus, computer programs and computer readable media. In particular, but not exclusively, the present invention relates to methods, apparatus, computer programs and computer readable media for processing configuration information relating to residual data that can be used for image reconstruction. Background Art

[0002] Compression and decompression of signals is a consideration in many known systems. Many types of signals, such as video, audio or volumetric signals, can be compressed and encoded for transmission, for example, over a data communications network. When decoding such a signal, it may be desirable to improve the quality level of the signal and / or to recover as much information as possible contained in the original signal.

[0003] Some known systems utilize scalable coding techniques. Scalable coding involves encoding a signal and information to allow reconstruction of the signal at one or more different quality levels, depending on the capabilities of the decoder and the available bandwidth, for example.

[0004] There are some considerations for signal reconstruction in scalable coding systems. One such consideration is the amount of information stored, used and / or transmitted. The amount of information can vary, for example, according to the desired quality level of the reconstructed signal, the nature of the information used during reconstruction and / or the way to configure such information. Another consideration is the ability of the decoder to accurately and / or reliably reconstruct the signal. A factor of the accuracy and / or reliability with which the decoder can perform reconstruction involves providing the decoder with up-to-date information indicating how to reconstruct the signal. Summary of the invention

[0005] Various aspects of the invention are set out in the accompanying claims.

[0006] Further features and advantages will become apparent from the following description of preferred embodiments, given by way of example only, with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A schematic block diagram showing an example of a signal processing system according to an embodiment of the present invention;

[0008] Figure 2A and Figure 2B A schematic block diagram showing another example of a signal processing system according to an embodiment of the present invention;

[0009] Figure 3 A schematic diagram showing an example of a data processing technique according to an embodiment of the present invention;

[0010] Figure 4A schematic diagram showing another example of a data processing technique according to an embodiment of the present invention;

[0011] Figure 5 shows a flowchart depicting an example of a method according to an embodiment of the present invention; and

[0012] Figure 6 A schematic block diagram of an example of an apparatus according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0013] refer to Figure 1 , shows an example of a signal processing system 100. The signal processing system 100 is used to process signals. Examples of signal types include, but are not limited to, video signals, image signals, audio signals, volumetric signals (such as those used in medical, scientific, or holographic imaging), or other multi-dimensional signals.

[0014] The signal processing system 100 includes a first device 102 and a second device 104. The first device 102 and the second device 104 may have a client-server relationship, wherein the first device 102 performs the function of a server device, and the second device 104 performs the function of a client device. The signal processing system 100 may include at least one additional device (not shown). The first device 102 and / or the second device 104 may include one or more components. One or more components may be implemented in hardware and / or software. One or more components may be located in the same location in the signal processing system 100 or may be remote from each other. Examples of device types include, but are not limited to, computerized devices, handheld or laptop computers, tablet computers, mobile devices, game consoles, smart TVs, set-top boxes, augmented and / or virtual reality headsets, etc.

[0015] The first device 102 is communicatively coupled to the second device 104 via a data communication network 106. Examples of the data communication network 106 include, but are not limited to, the Internet, a local area network (LAN), and a wide area network (WAN). The first device 102 and / or the second device 104 may have a wired and / or wireless connection to the data communication network 106.

[0016] The first device 102 includes an encoder 108. The encoder 108 is configured to encode data included in the signal, which is referred to as "signal data" hereinafter. For example, in the case where the signal is a video signal, the encoder 108 is configured to encode video data. The video data includes a sequence of multiple images or frames. The encoder 108 can perform one or more other functions in addition to encoding the signal data. The encoder 108 can be embodied in a variety of different ways. For example, the encoder 108 can be embodied as hardware and / or software.

[0017] Although in this example, the first device 102 includes the encoder 108, in other examples, the first device 102 is separate from the encoder 108. In such examples, the first device 102 is communicatively coupled to the encoder 108. The first device 102 can be embodied as one or more software functions and / or hardware modules.

[0018] The second device 104 includes a decoder 110. The decoder 110 is configured to decode signal data. The decoder 110 can perform one or more other functions in addition to decoding signal data. The decoder 110 can be embodied in various different ways. For example, the decoder 110 can be embodied as hardware and / or software.

[0019] Although in this example, the second device 104 includes the decoder 110, in other examples, the second device 104 is separate from the decoder 110. In such examples, the second device 104 is communicatively coupled to the decoder 110. The second device 104 may be embodied as one or more software functions and / or hardware modules.

[0020] The encoder 108 encodes the signal data and transmits the encoded signal data to the decoder 110 via the data communication network 106. The decoder 110 decodes the received encoded signal data and generates decoded signal data. The decoder 110 can output the decoded signal data or data derived using the decoded signal data. For example, the decoder 110 can output such data to be displayed on one or more display devices associated with the second device 104.

[0021] In some examples described herein, encoder 108 transmits a representation and information of a signal at a given quality level to decoder 110, which can use the information to reconstruct a representation of the signal at one or more higher quality levels. Such information can be referred to as "reconstructed data". In some examples, the "reconstruction" of a representation involves obtaining a representation that is not an exact copy of the original representation. The extent to which the representation is the same as the original representation may depend on various factors, including but not limited to the quantization level. The representation of a signal at a given quality level can be considered to be a reproduction, version or depiction of the data included in the signal at a given quality level. In some examples, the reconstruction data is included in the signal data encoded by encoder 108 and transmitted to decoder 110. For example, the reconstruction data can be in the form of metadata. In some examples, the reconstruction data is encoded and transmitted separately from the signal data.

[0022] The information used by the decoder 110 to reconstruct the representation of the signal at one or more higher quality levels may include residual data, as described in more detail below. Residual data is an example of reconstruction data. The information used by the decoder 110 to reconstruct the representation of the signal at one or more higher quality levels may also include configuration data related to the processing of the residual data. The configuration data may indicate the manner in which the residual data has been processed by the encoder 108 and / or the manner in which the residual data is to be processed by the decoder 110. The configuration data may be signaled to the decoder 110, for example, in the form of metadata.

[0023] refer to Figure 2A and Figure 2B , schematically illustrates an example of a signal processing system 200. The signal processing system 200 includes a first device 202 having an encoder and a second device 204 having a decoder. In each of the first device 202 and the second device 204, items are shown at two logical levels. The two levels are separated by a dotted line. The items on the highest first level are associated with data at a relatively high quality level. The items on the lowest second level are associated with data at a relatively low quality level. The relatively high and relatively low quality levels are associated with a hierarchical structure having multiple quality levels. In some examples, the hierarchical structure includes more than two quality levels. In such an example, the first device 202 and the second device 204 may include more than two different levels. There may be Figure 2A and Figure 2B One or more other levels above and / or below the levels depicted.

[0024] First reference Figure 2A , the first device 202 obtains a first representation 206 of an image at a relatively high quality level. The representation of a given image is a representation of the data included in the image. The image may be a given frame of a video. The first representation 206 of the image at a relatively high quality level will be referred to as "input data" hereinafter because, in this example, it is the data provided as input to an encoder in the first device 202. The first device 202 may receive the input data 206. For example, the first device 202 may receive the input data 206 from at least one other device. The first device 202 may be configured to receive consecutive portions of the input data 206, such as consecutive frames of a video, and to perform the operations described herein on each consecutive frame. For example, a video may include frames F1, F2, . . . , F T , and the first device 202 can process each of these frames in sequence.

[0025] The first device 202 derives data 212 based on the input data 206. In this example, the data 212 based on the input data 206 is a representation 212 of an image at a relatively low quality level. In this example, the data 212 is derived by performing a downsampling operation on the input data 206, and thus will be referred to as "downsampled data" hereinafter. In other examples, the data 212 is derived by performing an operation other than a downsampling operation on the input data 206.

[0026] In this example, the downsampled data 212 is processed at a relatively low quality level to generate processed data 213. In other examples, the downsampled data 212 is not processed at a relatively low quality level. In this way, the first device 202 can generate data at a relatively low quality level, wherein the data at the relatively low quality level includes the downsampled data 212 or the processed data 213.

[0027] In some examples, generating the processed data 213 involves encoding the downsampled data 212. Encoding the downsampled data 212 may produce an encoded image at a relatively low quality level. The first device 202 may output the encoded image, for example for transmission to the second device 204. For example, a series of encoded images that form an encoded video, as an output for transmission to the second device 204, may be referred to as a "basic" stream. The encoded image is not generated in the first device 202, but may be generated by an encoding device separate from the first device 202. The encoded image may be part of an H.264 encoded video. Generating the processed data 213 may, for example, include generating continuous frames of a video as an output of a separate encoding device such as an H.264 video encoder. In contrast to any intermediate data generated by a separate encoding device, an intermediate data set used to generate the processed data 213 may include the output of such an encoder.

[0028] Generating processed data 213 at a relatively low quality level may further involve decoding the encoded image at a relatively low quality level. As will become apparent below, a decoding operation may be performed to mimic a decoding operation at the second device 204. Decoding the encoded image may produce a decoded image at a relatively low quality level. In some examples, the first device 202 decodes the encoded image at a relatively low quality level to produce a decoded image at a relatively low quality level. In other examples, the first device 202 receives a decoded image at a relatively low quality level, for example, from an encoding and / or decoding device separated from the first device 202. The encoded image may be decoded using an H.264 decoder. Decoding by a separate decoding device may include: inputting an encoded video, such as an encoded data stream configured to be transmitted to a remote decoding device, into a separate black box decoder implemented with the first device 202 to generate continuous decoded frames of video. Thus, the processed data 213 may include frames of video data generated by a complex non-linear encoding and decoding process, wherein the encoding and decoding process may involve modeling spatiotemporal correlations according to a particular coding standard such as H.264. However, because the output of any encoder is fed to a corresponding decoder, this complexity is effectively hidden from the first device 202.

[0029] In an example, generating the processed data 213 at a relatively low quality level also involves obtaining correction data based on a comparison between the downsampled data 212 and the decoded image obtained by the first device 202, for example, based on the difference between the downsampled data 212 and the decoded image. The correction data can be used to correct errors introduced when encoding and decoding the downsampled data 212. In some examples, the first device 202 outputs the correction data and the encoded signal, and the correction data is used, for example, for transmission to the second device 204. This allows the recipient to correct errors introduced when encoding and decoding the downsampled data 212. The correction data may also be referred to as a "first enhancement" stream. Since the correction data may be based on the difference between the downsampled data 212 and the decoded image, it may be considered to be in the form of residual data (e.g., different from another set of residual data described later below).

[0030] In some examples, generating processed data 213 at a relatively low quality level also involves correcting the decoded image using correction data. For example, the correction data as output for transmission can be placed in a form suitable for combination with the decoded image and then added to the decoded image. This can be performed on a frame-by-frame basis. In other examples, the first device 202 uses downsampled data 212 instead of correcting the decoded image using correction data. For example, in some cases, only encoded and then decoded data can be used, while in other cases, other processing can be used instead of encoding and decoding.

[0031] In some examples, generating processed data 213 involves performing one or more operations in addition to the encoding, decoding, obtaining, and correcting actions described above.

[0032] The first device 202 obtains data 214 based on the data at the relatively low quality level. As described above, the data at the relatively low quality level may include processed data 213 or downsampled data 212, where the downsampled data 212 is not processed at a lower level. As described above, in some cases, the processed data 213 may include a reconstructed video stream corrected using correction data (e.g., from an encoding-decoding operation). Figure 2A and Figure 2B In the example of , data 214 is a second representation of an image at a relatively high quality level, and the first representation of the image at the relatively high quality level is input data 206. The second representation at the relatively high quality level can be considered to be an initial or predicted representation of the image at the relatively high quality level. In this example, the first device 202 derives data 214 by performing an upsampling operation on data at a relatively low quality level. Data 214 will be referred to as "upsampled data" hereinafter. However, in other examples, for example, in the case where data 212 is not derived by downsampling the input data 206, one or more other operations may be used to derive data 214. It should be noted that references to relatively high and relatively low quality levels may correspond to references to first and second quality levels, wherein the second quality level is higher than the first quality level. As described herein, in some cases, quality levels may correspond to different spatial resolutions.

[0033] Residual data 216 is obtained using input data 206 and upsampled data 214. Residual data 216 is associated with an image. Residual data 216 may be in the form of a set of residual elements. Residual elements in a set of residual elements 216 may be associated with corresponding image elements in input data 206. An example of an image element is a pixel.

[0034] In this example, a given residual element is obtained by subtracting the value of an image element in upsampled data 214 from the value of a corresponding image element in input data 206. As such, residual data 216 may be used in conjunction with upsampled data 214 to reconstruct input data 206. Residual data 216 may also be referred to as "reconstructed data" or "enhancement data." In one case, residual data 216 may form part of a "second enhancement" stream.

[0035] The first device 202 obtains configuration data related to the processing of the residual data 216. The configuration data indicates the manner in which the residual data 216 has been processed and / or generated by the first device 202 and / or the manner in which the residual data 216 is to be processed by the second device 204. The configuration data may include a set of configuration parameters. The configuration data may be used to control the manner in which the second device 204 processes data and / or reconstructs the input data 206 using the residual data 216. The configuration data may be related to one or more characteristics of the residual data 216. The configuration data may be related to one or more characteristics of the input data 206. Different configuration data may result in different processing being performed on the residual data 216 and / or using the residual data 216. Therefore, the configuration data may be used to reconstruct the input data 206 using the residual data 216. As described below, in some cases, the configuration data may also be related to the correction data described herein.

[0036] In this example, the first device 202 transmits data based on the downsampled data 212 , data based on the residual data 216 , and configuration data to the second device 204 to enable the second device 204 to reconstruct the input data 206 .

[0037] Now go to Figure 2B , the second device 204 receives data 220 based on the downsampled data 212 (e.g., derived from the downsampled data 212). The second device 204 also receives data based on the residual data 216. For example, the second device 204 may receive a "base" stream (data 220), a "first enhancement stream" (any correction data), and a "second enhancement stream" (residual data 216). The second device 204 also receives configuration data related to the processing of the residual data 216. The data 220 based on the downsampled data 212 may be the downsampled data 212 itself, the processed data 213, or data derived from the downsampled data 212 or the processed data 213. The data based on the residual data 216 may be the residual data 216 itself or data derived from the residual data 216.

[0038] In some examples, the received data 220 includes processed data 213, which may include coded images and / or correction data at a relatively low quality level. In some examples, the second device 204 processes the received data 220 to generate processed data 222, for example, when the first device 202 has processed the downsampled data 212 to generate the processed data 213. Such processing by the second device 204 may include decoding the coded images (e.g., coded images that form part of a "basic" coded video stream) to produce decoded images at a relatively low quality level. In some examples, the processing by the second device 204 includes correcting the decoded images using the obtained correction data. Therefore, the processed data 222 may include a first or relatively low quality level of correction data frame. In some examples, the coded images at the relatively low quality level are decoded by a decoding device separate from the second device 204. The coded images at the relatively low quality level can be decoded using an H.264 decoder.

[0039] In other examples, the received data 220 includes the downsampled data 212 and does not include the processed data 213. In some such examples, the second device 204 does not process the received data 220 to generate the processed data 222.

[0040] The second device 204 uses the data at the relatively low quality level to derive the upsampled data 214. As described above, the data at the relatively low quality level may include processed data 222 or received data 220, wherein the second device 204 does not process the received data 220 at the relatively low quality level. The upsampled data 214 is a preliminary representation of the image at a relatively high quality level. The upsampled data 214 may be derived by performing an upsampling operation on the data at the relatively low quality level.

[0041] The second device 204 obtains residual data 216. The residual data 216 may be used together with the upsampled data 214 to reconstruct the input data 206. The residual data 216 indicates a comparison between the input data 206 and the upsampled data 214.

[0042] The second device 204 also obtains configuration data related to processing of the residual data 216. The configuration data may be used by the second device 204 to reconstruct the input data 206. For example, the configuration data may indicate characteristics or attributes related to the residual data 216 that affect the manner in which the residual data 216 is to be used and / or processed, or whether the residual data 216 is to be used at all. In some examples, the configuration data includes the residual data 216.

[0043] There are several considerations regarding the use of such configuration data. One such consideration is the amount of information generated, stored, transmitted and / or processed. The more information that is used, the greater the amount of resources that may be involved in processing such information. Examples of such resources include transmission resources, storage resources, and processing resources. Compared to some known techniques, the examples described herein allow for the use of relatively small amounts of information. This can reduce the amount of data transmitted via the data communications network 106. The savings may be particularly significant in situations where the data is related to high-quality video data, where the amount of information transmitted in known systems may be particularly high.

[0044] Other considerations include the ability of the decoder to accurately and reliably perform image reconstruction. Accurately and reliably performing image reconstruction can affect the final visual quality of the displayed image, and therefore can affect the viewer's understanding of the image and / or the video including the image. One factor that may affect the accuracy and / or reliability with which the decoder can perform reconstruction is to provide the decoder with up-to-date and / or accurate information about how to perform reconstruction. For example, configuration data may change over time, and / or change between different images (e.g., different video frames). If the decoder is not informed of such changes in a timely manner, the resulting visual quality of the reconstructed image may be affected. Compared to some known techniques, the examples described herein allow configuration information to be signaled to the decoder relatively frequently, so that the decoder can use relatively up-to-date configuration information in image reconstruction.

[0045] refer to Figure 3 , a data processing technique 300 is shown. The data processing technique 300 may be performed by a device including an encoder, such as the first device 102 described above.

[0046] Data processing technique 300 involves a set of images 310. Although Figure 3 The group of images 310 depicted in FIG. 1 includes 6 images, but in other examples, other numbers of images may be included in the group of images 310. In some examples, the group of images 310 includes 75 images. The group of images 310 may be part of a video signal. For example, the group of images 310 may include consecutive frames of a video. The video may be a streaming video. An example of a video is a broadcast video. For example, the video may be broadcast via the data communication network 106. In some examples, the group of images 310 is encoded as a group of pictures (GOP). A GOP is an encoding of a sequence of images, and each image in the sequence of images can be decoded using information contained entirely within the GOP. A video may include multiple such GOPs.

[0047] For each image in the set of images 310, residual data is obtained. The residual data is based on a first representation of the image at a relatively high quality level in a hierarchical structure having a plurality of quality levels and a second representation of the image at the relatively high quality level. Figure 2A and Figure 2B As explained, the residual data for each image may be independent of the residual data for other images in the set of images 310. The second representation is based on a representation of the image at a relatively low quality level in the hierarchical structure. The residual data may be used by a decoder to reconstruct the first representation using the second representation for each image in the set of images 310. For a given image in the set of images 310, the first representation may be the input data 206, the representation at the relatively low quality level may be the downsampled data 212, the second representation may be the upsampled data 214, and the residual data may be the residual data 216, as described above with reference to FIG. 2 .

[0048] The residual data may be output to a decoder, enabling the decoder to reconstruct the first representation using the second representation for each image in the set of images 310 .

[0049] One or more configuration parameters 320 of configuration data are obtained. The configuration data is related to the processing of residual data. The configuration data may indicate how the decoder is to process the residual data. The one or more configuration parameters 320 of the configuration data may be used by the decoder to reconstruct a first representation for each image in a set of images 310. The one or more configuration parameters 320 may be referred to herein as "image set configuration data".

[0050] In some examples, one or more configuration parameters 320 include configuration parameters related to a transform operation performed on the residual data. The transform operation may be a directional decomposition operation. An example of a directional decomposition operation is a Hadamard transform. The transform operation may utilize spatial and / or directional correlations between residual elements in the residual data. Performing a transform operation on the residual data produces transformed residual data. The transformed residual data may be smaller than the residual data. The transformed residual data may have a lower coding entropy than the residual data. In this way, the encoder may transmit the transformed residual data more efficiently than the residual data. For example, transforming the residual data may make the residual data more efficiently quantized and encoded. The decoder receiving the transformed residual data may then perform an inverse transform operation on the transformed residual data to obtain the residual data. Therefore, the encoder signals the decoder which transform operation has been applied to the residual data through the configuration parameters in the one or more configuration parameters 320. In some examples, the configuration parameters related to the transform operation specify the transform operation that can be applied to the residual data from a plurality of possible transform operations.

[0051] In some examples, one or more configuration parameters 320 include configuration parameters related to upsampling operations. For each image in a set of images 310, an upsampling operation is performed on a representation of the image at a relatively low quality level. An upsampling operation is performed on a representation at a relatively low quality level to derive a representation of the image at a relatively high quality level. The encoder performs this upsampling operation to derive residual data. The decoder may perform a corresponding upsampling operation to reconstruct the first representation, i.e., by combining the second representation with the residual data. Therefore, the encoder signals the decoder which upsampling operation will be used through a configuration parameter in one or more configuration parameters. In some examples, the configuration parameters related to the upsampling operation specify an interpolation operation from a plurality of possible interpolation operations. Examples of possible interpolation operations include nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, and Lanczos interpolation. In some examples, the configuration parameters related to the upsampling operation specify whether upsampling is performed in one dimension or two dimensions. In other words, the configuration parameters related to the upsampling operation may specify whether one or both of horizontal and vertical upsampling are performed on a representation at a relatively low quality level. In some examples, configuration parameters related to upsampling operations may specify whether to perform horizontal upsampling or both horizontal and vertical upsampling on representations at relatively low quality levels.

[0052] In some examples, one or more configuration parameters 320 include configuration parameters related to bit depth. Bit depth is associated with residual data for a given image in a set of images 310. Bit depth may also be referred to as color depth. In some examples, configuration parameters related to bit depth indicate the number of bits used to indicate the color of a given pixel in a given image. In this way, the bit depth may indicate how finely a color may be represented in a given image. In some examples, the bit depth indicates the number of bits for each color component of a given pixel in a given image. In some examples, a given residual element in the residual data for a given image includes multiple color components. The bit depth may indicate the number of bits for each color component of a given residual element, such as the number of bits for each YUV or RGB component. In some cases, color components may have different bit depths.

[0053] In some examples, one or more configuration parameters 320 include a configuration parameter that specifies a resolution of a first representation of a given image in set of images 310 .

[0054] One or more configuration parameters 320 correspond to a plurality of images in set of images 310. In this example, one or more configuration parameters 320 include image set configuration data corresponding to all images in set of images 310. Thus, a single set of image set configuration data may apply to each image in set of images 310. Image set configuration data may also be referred to as "global configuration data."

[0055] One or more configuration parameters 320 are output to a decoder, enabling the decoder to reconstruct the first representation using the second representation and the residual data for each image in the set of images 310 .

[0056] The number of times that one or more configuration parameters 320 are output for a group of images 310 is less than the number of images in the group of images 310. Outputting one or more configuration parameters 320 for a number of images less than the number of images in the group of images 310 reduces the amount of information transmitted in a given period of time. In this example, the number of times that one or more configuration parameters 320 are output for a group of images 310 is once. One or more configuration parameters 320 may be output only for a first image in a group of images 310. The first image in a group of images 310 may be the first image in the group of images 310 to be output, received, processed and / or displayed. Outputting one or more configuration parameters 320 for a first image in a group of images 310 may enable a recipient to receive the one or more configuration parameters 320 as early as possible, and / or may enable a recipient to apply the one or more configuration parameters 320 to as many images in the group of images 310 as possible. In some examples, the one or more configuration parameters 320 are output together or in conjunction with the first image in the group of images 310. The one or more configuration parameters 320 may be output in the form of metadata. In some examples, one or more configuration parameters 320 are output for images other than the first image in a group of images 310. In some examples, one or more configuration parameters 320 are output for a group of images 310 a number of times greater than one but less than the number of images in the group of images 310. In one example, one or more configuration parameters 320 are output once for a group of images including 75 images.

[0057] In some examples, the number of times one or more configuration parameters 320 are output for a set of images 310 is based on an optimization metric. An example of an optimization metric is a rate-distortion metric. Rate-distortion optimization optimizes the amount of data transmitted for the amount of loss in visual quality. The less frequently one or more configuration parameters 320 are output, the less data is transmitted. However, outputting one or more configuration parameters 320 too infrequently may result in an undesirable loss of quality due to not providing the decoder 110 with the latest configuration data. Therefore, the number of times one or more configuration parameters 320 are output is optimized so as to reduce the amount of data sent while reducing any loss in visual quality.

[0058] In some examples, a ratio of the number of times one or more configuration parameters 320 are output for a group of images 310 to the number of images in the group of images 310 is less than a predetermined threshold. The predetermined threshold may be based on an optimization metric. An example of an optimization metric is the rate-distortion metric described above. In an illustrative example, it may be determined that a ratio of the number of times one or more configuration parameters 320 are output for a group of images 310 to the number of images in the group of images 310 is less than 1:10. That is, it may be determined that one or more configuration parameters 320 are output once every ten or more images.

[0059] In some examples, one or more configuration parameters 320 are arranged into an integer number of bytes. In some examples, one or more configuration parameters 320 are three bytes long. In other examples, one or more configuration parameters 320 are seven bytes long.

[0060] In some examples, byte-by-byte processing is performed on one or more configuration parameters 320. In byte-by-byte processing, one or more configuration parameters 320 are processed on a byte-by-byte basis. Processing configuration data in a byte-by-byte manner may be more efficient than processing configuration data in a bit-by-bit manner. Bit-by-bit processing may involve tracking which bit in a given byte is the next bit to be read or written. When a group of bits representing a given parameter crosses a byte boundary, additional processing may be performed to read the group of bits and obtain the given parameter. In the case of performing byte-by-byte processing, such processing may be reduced. In some examples, when data is processed in a byte-by-byte manner, the number of execution cycles may be reduced compared to a bit-by-bit manner.

[0061] In some examples, configuration data is written to a memory. If the configuration data is processed in a byte-by-byte manner, less processing may be involved when storing the configuration data in the memory and / or retrieving the configuration data from the memory than when the configuration data is processed in a bit-by-bit manner. The smallest unit of addressable memory may contain a byte. In this way, information may be stored in the memory in a byte-by-byte manner. Therefore, if the configuration data is packed into a sequence of bytes, fewer steps may be involved when storing the configuration data in the memory and / or retrieving the configuration data from the memory. By providing the configuration data to the memory in a byte-by-byte form, the amount of padding used to pack the configuration data into the addressable memory unit may also be reduced.

[0062] In addition, if the configuration data is arranged into a sequence of bytes, the transmission of the configuration data via a network such as the Internet can be performed more efficiently. Data can be transmitted over the Internet via the Transmission Control Protocol (TCP). TCP operates in the transport layer of the protocol stack. TCP obtains data from the stream, packages the data into TCP segments, and transmits each TCP segment to a recipient via the Internet using an Internet module (such as the Internet Protocol (IP)). The data stream can be obtained from a memory. The TCP segment can include an integer number of bytes. In this way, if TCP obtains the data to be packaged into the TCP segment from a byte-by-byte stream rather than from a bit-by-bit stream, less processing may be involved. By providing the data to TCP in a byte-by-byte form, the amount of padding used to package the data into the TCP segment can also be reduced.

[0063] refer to Figure 4 , a data processing technique 400 is shown. The data processing technique 400 may be performed by a device including an encoder, such as the first device 102 described above. Figure 4 Some of the items shown are Figure 3 The items shown are similar. Therefore, similar items have been given corresponding reference numerals (increased by 100).

[0064] The data processing technique 400 involves a set of images 410. These may be frames of a video signal.

[0065] For each image in the set of images 410, quantization data 420 is obtained. The quantization data 420 indicates a quantization level of data that can be used by a decoder to reconstruct a first representation of the image at a relatively high quality level using a second representation of the image at a relatively high quality level. The second representation is based on a representation of the image at a relatively low quality level. For a given image in the set of images 410, the first representation may be the input data 206, the representation at the relatively low quality level may be the downsampled data 212, and the second representation may be the upsampled data 214, as described above.

[0066] Quantization involves approximating a relatively large set of values ​​by a relatively small set of values, such as by rounding, thresholding, or truncating the values ​​in a relatively large set. The purpose of the quantization operation is to reduce the amount of data in the quantized output data compared to the amount of data in the input data of the quantization operation. The quantization level of given data may indicate how strongly the given data is quantized. The quantization level may indicate the granularity of the quantized data obtained via the quantization process. In other words, the quantization level may indicate how much information is lost due to such quantization.

[0067] In some examples, quantization data 420 indicates a quantization level of the residual data. For a given image in a set of images 410, the residual data is based on a first representation of the image and a second representation of the image, as described above with reference to FIG. 2. Therefore, the data that can be used by the decoder to reconstruct the first representation can be data at a relatively high quality level. The quantization level can guide the decoder 110 on how to perform inverse quantization on the residual data 216 that has been quantized. For example, at the decoder 110, a received set of residual data 216 can have values ​​within a first quantization range, which are then expanded to a larger range of values ​​after inverse quantization. Quantization data 852 can be used to perform this operation.

[0068] In some examples, the quantization data 420 indicates a quantization level of the correction data. For a given image in the set of images 410, the correction data can be used to correct errors when encoding and / or decoding a representation at a relatively low quality level, as described above with reference to FIG. 2. Therefore, the data that can be used by the decoder to reconstruct the first representation can be data at a relatively low quality level. Inverse quantization can be performed on the correction data in a similar manner, but in some cases, the inverse quantization can be based on a different portion of the quantization data 420.

[0069] In some examples, the quantization data 420 indicates a first quantization level and a second quantization level. The first quantization level is the quantization level applied to the residual data. The second quantization level is the quantization level applied to the correction data. In this way, both the residual data and the correction data can be quantized using separately defined quantization levels indicated by the quantization data 420. The quantization data 420 may therefore include the level at which one or more of the correction data and the residual data are quantized at the encoder (e.g., after a transform as described herein), which is then sent to the decoder to allow the original values ​​associated with the correction data and the residual data to be inversely quantized and reconstructed (due to information loss during the quantization process, the reconstructed values ​​may not match the original values ​​before quantization at the encoder).

[0070] Quantized data 420 may include an integer number of bytes. In some examples, quantized data 420 includes two bytes. In some examples, quantized data 420 includes four bytes.

[0071] The quantized data 420 is output for processing by a decoder. The quantized data 420 enables the decoder to reconstruct, for each image in the set of images 410, the first representation of the image using the second representation of the image.

[0072] Quantization data 420 is output for a plurality of images in a set of images 410. In some examples, the number of images in the plurality of images is equal to the number of images in the set of images 410. In other words, quantization data 420 may be output for each image in the set of images 410. Quantization data 420 may be output together or in conjunction with each image in the set of images 410. In other examples, the number of images in the plurality of images is less than the number of images in the set of images 410. The number of images in the plurality of images may be based on an optimization metric. The optimization metric may be a rate-distortion metric.

[0073] In some examples, in addition to the quantized data 420, a reference is output for a set of images 410. Figure 3 The one or more groups of configuration parameters 320. The number of times the one or more configuration parameters 320 are output for a group of images 410 is less than the number of times the quantitative data 420 are output for a group of images 410. For example, the one or more configuration parameters 320 may be output only once for a group of images 410, and the quantitative data 420 may be output for each image in the group of images 410. In this way, the quantitative data 420 is output more frequently than the one or more configuration parameters 320.

[0074] The quantitative data 420 may be included in image configuration data. Image configuration data corresponds to a specific image in a set of images. This is in contrast to image set configuration data which corresponds to an entire set of images.

[0075] The image configuration data may include one or more configuration parameters in addition to the quantization data 420. The one or more configuration parameters in the image configuration data for a given image may include a configuration parameter specifying whether the given image is interlaced or progressive.

[0076] Outputting quantized data 420 relatively frequently uses more data than outputting quantized data 420 relatively infrequently. However, outputting quantized data 420 relatively frequently provides the decoder with up-to-date information, which enables the decoder to perform image reconstruction with high accuracy and / or reliability. For example, quantized data 420 may change between different images. Therefore, there is a tradeoff between the amount of data sent to the decoder and the accuracy and / or reliability with which the decoder can perform reconstruction using the received data. For one or more configuration parameters 320 and quantized data 420, the best outcome of this tradeoff may be that quantized data 420 is sent more frequently than one or more configuration parameters 320. Therefore, sending one or more configuration parameters 320 at a relatively low frequency and sending quantized data 420 at a relatively high frequency can reduce the amount of information sent in a given period of time while enabling the decoder to perform reconstruction at a predetermined level of accuracy and / or reliability.

[0077] refer to Figure 5, shows an example of a method 500 for processing a representation of an image. The method 500 may be performed by a device including a decoder (such as the second device 104 described above).

[0078] At item 510, data is received that can be used to derive a representation of an image at a relatively low quality level in a hierarchical structure including a plurality of quality levels. The received data may be data 220 as described above. The received data may be encoded data. Before being received by a decoder, a first representation of the image at a relatively high quality level in the hierarchical structure is used by an encoder (e.g., encoder 108) to derive a representation at a relatively low quality level. The first representation may be input data 206 as described above. In some examples, the image is one of a group of images. A group of images may be part of a video signal. In some examples, the received data is processed to obtain a representation at a relatively low quality level. For example, the received data may be decoded to obtain a representation at a relatively low quality level.

[0079] At item 520, a determination is made as to whether predetermined configuration data has been received. The predetermined configuration data corresponds to an image.

[0080] If it is determined at item 520 that predetermined configuration data corresponding to the image has been received, then at item 530, a first representation 206 of the image is reconstructed. The first representation is reconstructed using the predetermined configuration data. The first representation is reconstructed using a second representation of the image at a relatively high quality level. In some examples, the representation at a relatively low quality level is processed to derive the second representation. For example, the second representation may be derived by upsampling the representation at the relatively low quality level. The first representation may be reconstructed by combining the second representation with the residual data.

[0081] At item 535, the first representation is output. For example, the first representation can be output for display on a display device.

[0082] If it is determined at item 520 that the predetermined configuration data corresponding to the image has not been received, the first representation of the image is not reconstructed. Instead, the representation of the image at the relatively low quality level is processed to derive a second representation at item 540. The second representation may be derived by upsampling the representation at the relatively low quality level.

[0083] At item 545, the second representation is output. For example, the second representation can be output for display on a display device.

[0084] In some examples, if it is determined that the predetermined configuration data corresponding to the image has not been received, the second representation of the image is not exported. In some examples, a representation at a relatively low quality level is output instead of the second representation. In some examples, data based on the representation at the relatively low quality level is output. The data based on the representation at the relatively low quality level is different from the first representation and the second representation.

[0085] In the case where the image is one of a group of images, the predetermined configuration data may correspond to multiple images in the group of images. The predetermined configuration data may not be received with each image in the group of images. In one example, the predetermined configuration data is received only with the first image in the group of images. The predetermined configuration data may apply to each subsequent image in the group of images. Thus, if the first image in the group of images is not received by the decoder, the predetermined configuration data for the group of images may not have been received yet. The predetermined configuration data may be image set configuration data. In some examples, the predetermined configuration data includes one or more configuration parameters 320 described above. The predetermined configuration data may be configuration data applicable to each image in the group of images.

[0086] In the case where a group of images is part of a streaming video, the decoder may enter the video stream midway through a group of images. In this case, although the first representation is not reconstructed due to the failure to receive the predetermined configuration data, a representation at a relatively high quality level, i.e., a second representation, is still output. Outputting the second representation for display reduces the potential loss of visual quality due to the failure to receive the predetermined configuration data compared to the case where the representation at the relatively high quality level is not output. Outputting the second representation can be considered as part of a "minimum operating mode", in contrast to outputting the first representation, which can be considered as part of a "full operating mode". In the case where the decoder enters the stream midway through a group of images, the "minimum operating mode" can be used for the remainder of the group of images. Of course, when the first image in a group of subsequent images is subsequently received, the predetermined configuration data for the group of subsequent images is also received. In this way, a group of subsequent images can be processed according to the "full operating mode".

[0087] refer to Figure 6 , a schematic block diagram of an example of a device 600 is shown.

[0088] In an example, the device 600 includes an encoder. In another example, the device 600 includes a decoder.

[0089] Examples of apparatus 600 include, but are not limited to, a mobile computer, a personal computer system, a wireless device, a base station, a telephone device, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, an application server, a storage device, a consumer electronic device such as a camera, a camcorder, a mobile device, a video game console, a handheld video game device, or generally any type of computing or electronic device.

[0090] In this example, device 600 includes one or more processors 601 configured to process information and / or instructions. One or more processors 601 may include a central processing unit (CPU). One or more processors 601 are coupled to bus 602. The operations performed by one or more processors 601 may be performed by hardware and / or software. One or more processors 601 may include multiple processors located at the same location or multiple processors located at different locations.

[0091] In this example, device 600 includes computer usable volatile memory 603 configured to store information and / or instructions for one or more processors 601. Computer usable volatile memory 603 is coupled to bus 602. Computer usable volatile memory 603 may include random access memory (RAM).

[0092] In this example, device 600 includes computer usable non-volatile memory 604 configured to store information and / or instructions for one or more processors 601. Computer usable non-volatile memory 604 is coupled to bus 602. Computer usable non-volatile memory 604 may include read only memory (ROM).

[0093] In this example, device 600 includes one or more data storage units 605 configured to store information and / or instructions. One or more data storage units 605 are coupled to bus 602. One or more data storage units 605 may include, for example, magnetic or optical disks and disk drives or solid state drives (SSDs).

[0094] In this example, the device 600 includes one or more input / output (I / O) devices 606 configured to transmit information to and / or from one or more processors 601. The one or more I / O devices 606 are coupled to the bus 602. The one or more I / O devices 606 may include at least one network interface. The at least one network interface may enable the device 600 to communicate via one or more data communication networks. Examples of data communication networks include, but are not limited to, the Internet and a local area network (LAN). The one or more I / O devices 606 may enable a user to provide input to the device 600 via one or more input devices (not shown). The one or more input devices may include, for example, a remote control, one or more physical buttons, etc. The one or more I / O devices 606 may enable information to be provided to a user via one or more output devices (not shown). The one or more output devices may, for example, include a display screen.

[0095] Various other entities of the device 600 are depicted. For example, when present, an operating system 607, a data signal processing module 608, one or more other modules 609, and data 610 are shown as residing in one or a combination of computer usable volatile memory 603, computer usable non-volatile memory 604, and one or more data storage units 605. The data signal processing module 608 may be implemented by computer program code stored in a memory location within the computer usable non-volatile memory 604, a computer readable storage medium within the one or more data storage units 605, and / or other tangible computer readable storage media. Examples of tangible computer readable storage media include, but are not limited to, optical media (e.g., CD-ROM, DVD-ROM, or Blu-ray), flash memory cards, floppy disks, or hard disks, or any other medium capable of storing computer readable instructions, such as firmware or microcode in at least one ROM or RAM or programmable ROM (PROM) chip, or as an application specific integrated circuit (ASIC).

[0096] The device 600 may thus include a data signal processing module 608, which may be executed by one or more processors 601. The data signal processing module 608 may be configured to include instructions to implement at least some of the operations described herein. During operation, the one or more processors 601 initiate, run, execute, interpret or otherwise perform the instructions in the signal processing module 608.

[0097] Although at least some aspects of the examples described herein with reference to the drawings include computer processes executed in a processing system or processor, the examples described herein also extend to computer programs, such as computer programs on or in a carrier suitable for putting the examples into practice. The carrier may be any entity or device capable of carrying the program.

[0098] It should be understood that the device 600 may include Figure 6 More, fewer and / or different components than those depicted.

[0099] Device 600 may be located at a single location or may be distributed across multiple locations. Such locations may be local or remote.

[0100] The techniques described herein may be implemented in software or hardware, or may be implemented using a combination of software and hardware. The techniques may include configuring a device to perform and / or support any or all of the techniques described herein.

[0101] It should be understood that any feature described with respect to any embodiment can be used alone, or in combination with other features described, and can also be used in combination with one or more features of any other embodiment, or in combination with any combination of any other embodiment. In addition, equivalents and modifications not described above can also be adopted without departing from the scope of the invention as defined in the appended claims.

Claims

1. A method for processing configuration data, the method comprising: obtaining, for each image in a set of images, residual data based on a first representation of the image at a first quality level in a hierarchical structure having a plurality of quality levels and a second representation of the image at the first quality level, the second representation being a predicted representation of the image at the first quality level and obtained on the basis of a representation of the image at a second quality level in the hierarchical structure, the residual data being usable by a decoder to reconstruct, for each image in the set of images, the first representation using the second representation, the second quality level being lower than the first quality level; obtaining, for each image of the set of images, quantization data indicating a quantization level of data usable by the decoder to reconstruct the first representation of the image; obtaining one or more configuration parameters of configuration data relating to processing of the residual data, the one or more configuration parameters being usable by the decoder to reconstruct the first representation for each image in the set of images, the one or more configuration parameters being applicable to a plurality of images in the set of images: outputting the one or more configuration parameters for processing by the decoder to enable the decoder to reconstruct the first representation using the second representation and the residual data for each image in the set of images, wherein: the number of times the one or more configuration parameters are output for the set of images is less than the number of images in the set of images; and the number of times the one or more configuration parameters are output is based on a rate-distortion optimization metric that optimizes the amount of data sent for an amount of visual quality loss that would occur if the decoder was not provided with the most recent configuration data; and The quantized data is output for processing by the decoder, wherein the number of times the one or more configuration parameters are output for the set of images is less than the number of times the quantized data is output for the set of images.

2. The method of claim 1, wherein the number of times the one or more configuration parameters are output for the set of images is once.

3. A method according to claim 1 or claim 2, wherein the one or more configuration parameters are output only for a first image in the set of images.

4. A method according to claim 1 or claim 2, wherein the one or more configuration parameters include image set configuration data corresponding to all images in the set of images.

5. A method according to claim 1 or claim 2, the method comprising performing byte-by-byte processing of the one or more configuration parameters; and / or The one or more configuration parameters are arranged into an integer number of bytes.

6. The method of claim 1 or claim 2, wherein the one or more configuration parameters include configuration parameters related to a transform operation performed on the residual data. The method of claim 6 , wherein the transform operation comprises a directional decomposition operation performed on the residual data.

8. The method of claim 1 or claim 2, wherein the one or more configuration parameters include configuration parameters relating to an upsampling operation performed on the representation of the image at the second quality level for each image in the set of images; configuration parameters relating to a bit depth associated with said residual data; A configuration parameter specifies a resolution of the first representation of the image.

9. A method according to claim 1 or claim 2, wherein the quantified data is output for each image in the set of images.

10. The method of claim 1 or claim 2, wherein the quantization data indicates a quantization level of the residual data.

11. A method according to claim 1 or claim 2, comprising, for each image in the set of images, outputting the representation of the image at the second quality level for processing by the decoder.

12. A method for processing configuration data, the method comprising: receiving one or more configuration parameters of configuration data related to processing of residual data, the residual data being obtained, for each image of a set of images, based on a first representation of the image at a first quality level in a hierarchical structure having a plurality of quality levels and a second representation of the image at the first quality level, the second representation being a predicted representation of the image at the first quality level and being obtained on the basis of a representation of the image at the second quality level in the hierarchical structure, the residual data being usable for reconstructing, for each image of the set of images, the first representation using the second representation, wherein the one or more configuration parameters are applicable to a plurality of images of the set of images, the second quality level being lower than the first quality level; receiving quantization data indicating a quantization level of data that can be used to reconstruct the first representation for each image in the set of images; processing the one or more configuration parameters to reconstruct the first representation using the second representation and the residual data for each image in the set of images, wherein: the one or more configuration parameters are received a number of times for the set of images that is less than the number of images in the set of images; and the number of times the one or more configuration parameters are output is based on a rate-distortion optimization metric that optimizes an amount of data sent for an amount of visual quality loss that would occur if the decoder was not provided with the most recent configuration data; and The quantitative data is processed, wherein the number of times the one or more configuration parameters are received for the set of images is less than the number of times the quantitative data is received for the set of images.

13. A device for processing configuration data, configured to perform the method of any one of claims 1 to 12.

14. A computer program product comprising instructions which, when executed, cause a device to perform the method of any one of claims 1 to 12.

Images