Methods and apparatuses for data restructuring

Abstract

According to a first aspect, there is provided a computer-implemented method of data restructuring, the method comprising: receiving input data comprising input elements, each input element representing an input value; generating, for each input value, a corresponding clipped value; generating overflow data comprising, for each input value, a corresponding overflow element representing an overflow value; generating, for each input value, a corresponding modulus value; and generating restructured data comprising, for each input element, a corresponding restructured element comprising the corresponding clipped value and modulus value.

Description

[0001] METHODS AND APPARATUSES FOR DATA RESTRUCTURING

[0002] FIELD OF THE INVENTION

[0003] The present inventions relate generally to data restructuring, and more particularly to a method and apparatus for reversibly reducing the size of data.

[0004] BACKGROUND

[0005] When processing data, it may be advantageous to store received input data in a different format to that in which it is received, as restructured data. This may be because the format in which the input data is received requires a large amount of memory to store, or because further processing steps require the data to be stored in a specific format.

[0006] However, some methods of data processing may need access to the input data and the restructured data. Storing both the input data and the restructured data necessitates additional memory usage, decreasing the efficiency of the method.

[0007] Accordingly, it is desirable to implement a method of data restructuring, such that a representation of the input data may be recovered from the restructured data, without having to store the input data itself.

[0008] SUMMARY OF INVENTION

[0009] When restructuring data, the choice of format of the restructured data may impose limitations on the range of data, or the precision of the data, which can be stored. Usually, this can be mitigated by ensuring the input data can only comprise values in a specific, predetermined range, which the format of the restructured data is capable of representing without data loss. For instance, if the input data was 32- bit integers, but each individual integer was guaranteed to be between 0 and 65535 (the range which can be stored in a 16-bit integer), the restructured data could comprise 16-bit integers with no data loss or precision loss. However, it may be that the format of the restructured data cannot be freely chosen. For instance, certain software or hardware modules may require data to be stored in a particular format. In these cases, it may be important that the input data is maintained. This is because there may be some operations which require the full precision or original format of the input data.

[0010] A relevant scenario is that of an enhancement video codec, in which updates to enhancement data can be provided per frame relative to the previous frame. The base video data and enhancement data can be provided to the decoding device in a first format, suitable for transmission of the video data. However, due to the use of hardware modules to perform accelerated application of the enhancement data to the base video, it is possible that the enhancement data will need to be provided to the hardware module in a predetermined second format. This second format may not be the same as the first format.

[0011] In a comparative scheme addressing this issue, the video data received in the first format can be converted into the second format to provide to the hardware module. However, in this case, if the full range and precision of the input data were not maintained, errors could accumulate when updating the enhancement data. This is because when updates are provided as “relative” updates, i.e. updates which require knowledge of the current frame data to calculate the next frame data, minor errors in those updates can accumulate into larger errors. If these relative updates are applied to the converted data, the loss in precision caused by converting from the first format to the second format would then cause errors in the updated frame data.

[0012] In this comparative scheme, the data may be stored in the first format, preserving the input data, such that the updates can be applied accurately to the input data and then used to create new restructured data. However, this then requires the storage of the input data alongside the restructured data, which requires large amounts of memory, and further requires repeated conversions between the input and intermediate data. Contrastingly, according to a first aspect, there is provided a computer- implemented method of data restructuring, the method comprising: receiving input data comprising input elements, each input element representing an input value; generating, for each input value, a corresponding clipped value; generating overflow data comprising, for each input value, a corresponding overflow element representing an overflow value; generating, for each input value, a corresponding modulus value; and generating restructured data comprising, for each input element, a corresponding restructured element comprising the corresponding clipped value and modulus value.

[0013] This method of data restructuring allows for input data, of a certain size, to be stored as a smaller approximation of the data, whilst maintaining the capability, via the overflow data, to recreate data at the first size.

[0014] Optionally, instead of being generated for each (i.e. every) input value, the overflow data may (only) be generated for each input value outside of the clipping range. In those circumstances, a modulus value would only need to be generated for the respective or relevant input values, not for each input value. As an alternative, a default modulus value could be generated for each input value inside of the clipping range.

[0015] In some examples, the modulus value is the difference between the input value and the sum of the end value of the clipping range and the corresponding overflow value at the predetermined level of precision. In other words, the modulus has a correcting contribution, correcting for any inaccuracy caused by the overflow value being too low precision. This may occur because of storage constraints on the overflow value.

[0016] By the term “corresponding” above, it is intended to mean “respective”. For example, for the generating, for each input value, a corresponding modulus value, may mean generating, for each input value, a respective modulus value.

[0017] In some examples, the input data may be an integer or a series of integers, stored as a certain number of bytes each. In other examples, the data could be floating- point or fixed-point numbers, rationals, characters or character arrays, or any arbitrary data. The input data may correspond to signed or unsigned integers, as can the restructured data, output data, overflow data and modulus data.

[0018] Assuming the data corresponds to integers, an element of the clipped data is capable of representing a smaller range of numbers than the input data, as it is clipped, or clamped (i.e. such as being clipped by clamping), to the clipping range.

[0019] A value is inside the clipping range if it is greater than the lower end of the clipping range, and lesserthan the higherend of the clipping range. Avalue which is lesser than the lower end of the clipping range or greater than the higher end of the clipping range is outside the clipping range.

[0020] For a particular input value, if the input value is outside of the clipping range, the clipped value is whichever end value of the clipping range is closer to the input value. In other words, if the input value is less than a lower end of the clipping range, the clipped value is a lower end of the clipping range. If the input value is greater than an upper end of the clipping range, the clipped value is an upper end of the clipping range.

[0021] In some examples, this is desirable because such clipped data can be represented with a smaller number of bits than the input data. In other examples, it may be desirable due to existing hardware or software processes requiring numbers in a particular range or of a particular bit length, for example, particular network protocols. In further examples, the input data may lie mostly within the clipped range, and thus the clipping reduce the storage required at minimal cost. This may allow for faster processing of the clipped data.

[0022] However, in such cases, storing both the clipped data and the input data is inefficient, using large quantities of storage. Further, it may be desirable for algorithmic or performance reasons to be able to convert an element of the clipped data back to a representation of the input data, restoring the part of the range that was clipped away. This may allow, in some examples, the in-place updating of the clipped data with updates that are dependent on the corresponding input value.

[0023] The overflow data provides the ability to reconstruct a representation of the input data by storing a difference between an end value of the clipping range and the input value. The number of values which the overflow data represents is equal to the number of values which can be represented by an element of the input data minus the number of values inside the clipping range. However, in some examples, the overflow data is stored at a predetermined precision such that it cannot adequately represent the whole range of possible values for this difference at the same precision as the input data. This may be because the predetermined precision is too low. In some examples, the overflow data may only be able to represent every other possible value. In other examples, the predetermined precision may be such that the overflow data is only able to represent every fourth, eighth or sixteenth value. This may be due to the number of bits used to represent the overflow data.

[0024] Accordingly, the method also provides a modulus value corresponding to each input element. From the modulus value and the clipped value, the final restructured element is constructed. The modulus value may be used to encode the difference between end value of the clipping range plus the corresponding overflow value and the input value. In the examples above, where the overflow data can only represent every other possible value, the modulus value may represent an addition of 0 or 1 to the reconstructed data.

[0025] In this way the claimed method allows for fewer bytes to be used to represent each overflow element, as it can be less precise, due to the inclusion of the modulus value. This reduces the total amount of storage required.

[0026] A representation of the input data can be created based on the restructured data and the overflow data. In some examples, this may be the summation of the restructured data and the modulus data. In some examples, for some input value, if the input value is in the clipping range, the restructured value will equal the input value and the overflow value will be zero. However, if the input value is not in the clipping range, the input value may be calculated from the corresponding restructured value (comprising both the clipped value and the modulus value) and the corresponding overflow value. Accordingly, in some examples, the method may further comprise generating output data using the restructured data and the overflow data, the output data representing the input data.

[0027] In some examples, the output value may be a reconstruction of the input value.

[0028] In some examples, generating, for each input value, a corresponding clipped value comprises: if the input value is in a clipping range, setting the clipped value to be equal to the input value; or if the input value is outside of the clipping range, setting the clipped value to be equal to an end value of the clipped range.

[0029] In some examples, generating overflow data comprising, for each input value, a corresponding overflow element representing an overflow value comprises: if the input value is in a clipping range, setting the overflow value to be zero; or if the input value is outside of the clipping range, setting the overflow value, at a predetermined level of precision, to be the difference between an end value of the clipping range and the input value.

[0030] In some examples, if the input value is in the clipping range, the modulus value is zero; or if the input value is outside of the clipping range, the modulus value is the difference between the input value and the sum of the end value of the clipping range and the corresponding overflow value at the predetermined level of precision.

[0031] In the field of video encoding, data sets where the values lie frequently in a smaller range than the maximum range the data can encode are common. Accordingly, in some examples, the data may be video data.

[0032] Specifically, in enhancement codecs such as LCEVC, the enhancement data may lie frequently in a smaller range than the maximum range the data can encode. Accordingly, in some examples, the input data may be enhancement data, and the input elements may be residual elements. In this example, the restructured data, representing the input enhancement data, may be further used to enhance a base dataset. In some examples, this may be beneficial because the restructured data is represented with fewer bits, making operations on it more efficient. In other examples, it may be a requirement of a particular enhancement method. For example, some such methods use a blend unit to combine the base dataset and the enhancement data. In many such cases, hardware or software blend units require that the enhancement data has some prespecified format or a particular bit length. Accordingly, the restructured data may be clipped such that it conforms to the requirements of the blend unit.

[0033] In some examples, the modulus value of each restructured element could be stored in a predetermined number of bits of the restructured element. However, in preferred examples, the end value of the clipping range is less than a maximum restructured value, the maximum restructured value being the maximum value that can be represented by a restructured element. In this way, there is a portion of the range of values that can be represented by a restructured element that does not correspond to the clipped value. These remaining values can be used to store the modulus value, as the modulus value is only non-zero in the case where the clipped value is equal to an end value of the clipping range.

[0034] By way of example, and used by way of illustration only, an eight bit clipped element can represent 28= 256 (two raised to the power of eight) distinct numbers. If a corresponding example clipping range is between 1 and 254, the two values of 255 and 256 remain and may be used to encode a one bit modulus value.

[0035] In some examples, generating the restructured data may therefore comprise summing each clipped value and the corresponding modulus value.

[0036] In some examples such as those described above, it can be advantageous to reduce the size of the input elements to use less storage. Accordingly, in some examples, the input elements have a greater bit length than (e.g. may have a bit length greater than a bit length of) both of the restructured elements and the overflow elements. In other words, the restructured elements and the overflow elements may have a bit length that is less than a bit length of the input elements. In other examples, the overflow elements may have a greater bit length than the input elements or the restructured elements, when the utilisation of less storage space is not a desired outcome as compared to other concerns, e.g. bytealignment of the data. The use of the overflow elements may still provide significant advantages, by allowing the conversion of the restructured data to the input data directly, without having to store the input data elsewhere in memory. This may improve performance by allowing the restructured data to be modified in-place.

[0037] Optionally, the restructured data is stored in a restructured plane, and overflow data is stored in an overflow plane, where the restructured data and the overflow data are packed data. Packed data, in this example, is intended to refer to data which is stored without gaps or byte-alignment, for instance, two ten-bit numbers stored in twenty bits without alignment to byte boundaries. This can decrease the amount of storage required. Alternatively, the restructured data and overflow data may still be stored in restructured and overflow planes without being packed data, at the cost of potentially greater memory use.

[0038] Alternatively, in some examples, the restructured data and the overflow data are stored in a combined plane, where each restructured element and the corresponding overflow element are stored consecutively in the combined plane. In some examples, this may allow for more efficient use of storage. For example, if the restructured elements are ten-bit values stored as byte-aligned data, each restructured element is followed by six empty bits (so that the next restructured element can start at a byte boundary). These six bits could be used to store a six- bit overflow element, increasing efficiency.

[0039] Optionally, the restructured plane is stored in a predetermined format compatible with a hardware module, and the method further comprises copying the restructured plane to a predetermined location, such that it is suitable for processing by the hardware module. In some examples, copying the restructured plane to a predetermined location in memory allows the hardware module to process it. The memory may be the same memory in which the restructured plane is stored, or it may be a separate memory. This may improve the performance of the method by allowing the working data to be stored in the same format as required by the hardware module, instead of requiring conversions. In other examples, the restructured plane may be stored in a predetermined format compatible with a software module or software implementation.

[0040] In some examples, the input data itself may require some processing before it can be utilised in the method. In these cases, the step of receiving input data may further comprise receiving source data, (the source data typically including one or more source elements) wherein each source element of the source data has a greater bit length than the input elements; and for each source element of the source data, creating an input element of the input data by truncating the source element to the bit length of the input element.

[0041] Optionally, the source data may be the temporal layer of an LCEVC-compliant video signal. Alternatively, this processing step may not be required, and the input data may instead be the temporal layer of an LCEVC-compliant video signal.

[0042] In some examples, such as when considering an LCEVC-compliant video signal, it is desirable to clip some of the input data to create restructured data. However, in some cases, update data is provided, which is configured to allow the relative updating of the input data to generate new input data. As the update data is relative, it is necessary to know the input data at the original precision, rather than a clipped version of it. If a clipped version is used, errors may accumulate. An illustrative example of a relative update may be “add two to the input value”, which requires knowledge of the input value. An illustrative example of an absolute update may be “the input value is now two”, which does not require knowledge of the original input value. It may be desirable to not have to store the input data and the restructured data just to apply the updates.

[0043] Accordingly, the method may further comprise receiving update data comprising one or more update elements, the one or more update elements representing relative updates to a corresponding one or more of the input elements; and for each update element, updating the corresponding restructured element based on said update element and the corresponding output element. As the method allows for the restructured elements to be combined with the overflow elements to generate the output elements, which are a representation of the original input elements, the relative updates can be applied without accumulation of errors.

[0044] In other examples, some or all of the updates may not be relative updates, and may instead be absolute updates.

[0045] In some examples, the input data is in a first format, and the restructured data is in a second format. The input data may be received in the first format, and the restructured data generated in the second format. The first format and the second format may be predetermined. A format may refer to the structure of the data which gives the data its meaning, specifying how the bits of the data are used to encode information. It may specify the number of distinct pieces of information in the data, or the type of different elements of the data. For instance, an example format may specify that the each element of the data comprises a pair of 16-bit integers. A separate example format may specify that each element of the data comprises a single 32-bit floating-point value.

[0046] In some examples, the second format may be a format compatible with a hardware module, in particular wherein the second format is suitable for a display of an apparatus performing the method and the first format is a format of a bitstream received by the apparatus, said input data derived from the bitstream. This second format may be predetermined based on the hardware module which is to be used. The hardware module may be a display, screen, output device, chip, processor, co-processor, GPU, graphics card, or any other device capable of receiving a digital signal. Such a hardware module may be only capable of performing its function when provided with data in a particular format.

[0047] According to a second aspect, there is provided an apparatus configured to restructure data, the apparatus being configured to: receive input data comprising input elements, each input element representing an input value; generate, for each input value, a corresponding clipped value; generate overflow data comprising, for each input value, a corresponding overflow element representing an overflow value; generate, for each input value, a corresponding modulus value; and generate restructured data comprising, for each input element, a corresponding restructured element comprising the corresponding clipped value and modulus value.

[0048] In some examples, generating, for each input value, a corresponding clipped value comprises: if the input value is in a clipping range, setting the clipped value to be equal to the input value; or if the input value is outside of the clipping range, setting the clipped value to be equal to an end value of the clipped range.

[0049] In some examples, generating overflow data comprising, for each input value, a corresponding overflow element representing an overflow value comprises: if the input value is in a clipping range, setting the overflow value to be zero; or if the input value is outside of the clipping range, setting the overflow value, at a predetermined level of precision, to be the difference between an end value of the clipping range and the input value.

[0050] In some examples, the apparatus is configured to: if the input value is in the clipping range, set the modulus value to zero; or if the input value is outside of the clipping range, set the modulus value to the difference between the input value and the sum of the end value of the clipping range and the corresponding overflow value at the predetermined level of precision.

[0051] In some examples, the apparatus is further configured to generate output data using the restructured data and the overflow data, the output data representing the input data.

[0052] In some examples, the input data is enhancement data and the input elements are residual elements, wherein the apparatus is further configured to enhance a base dataset using the restructured data.

[0053] Optionally, the input data is the temporal layer of an LCEVC-compliant video signal. Optionally, the apparatus may further comprise a hardware module, wherein the restructured data is stored in a restructured plane in a predetermined format compatible with the hardware module, and the apparatus is further configured to copy the restructured plane to a predetermined location in memory, such that it can be acted on by the hardware module.

[0054] The apparatus may be further configured to receive update data, representing relative updates to one or more updated input elements, and further configured to, for each updated input element, update the corresponding restructured element based on the update data and the corresponding output element.

[0055] The apparatus may be a decoder.

[0056] According to a third aspect, there is provided a computer program comprising instructions, which, when the program is executed by a computer, cause the computer to carry out the methods of the first aspect.

[0057] According to a fourth aspect, there is provided a non-transitory computer-readable medium comprising said computer program.

[0058] It should be understood that any discussion of or benefits ascribed to a particular first aspect should be applied equally to any second aspect, where applicable.

[0059] BRIEF DESCRIPTION OF DRAWINGS

[0060] An example process and an example apparatus are described in detail herein with reference to the accompanying figures, in which:

[0061] Figure 1 shows a schematic diagram of decoding of the low complexity enhancement video codec (LCEVC);

[0062] Figure 2 shows a flow diagram of an example method;

[0063] Figure 3 shows a flow diagram of an example method;

[0064] Figure 4 shows a schematic of an apparatus configured to perform the methods described herein. DETAILED DESCRIPTION

[0065] This disclosure describes implementations of methods for restructuring data. In non-limiting examples, the disclosure relates to implementation and integration with a hybrid backward-compatible coding technology, previously proposed, for example in WO 2013 / 171173, WO 2014 / 170819, WO 2019 / 141987, and WO 2018 / 046940, the contents of which are incorporated herein by reference.

[0066] Further examples of tier-based coding formats include ISO / IEC MPEG-5 Part 2 LCEVC (hereafter ‘LCEVC’). LCEVC has been described in WO 2020 / 188273A1 , GB 2018723.3, WO 2020 / 188242, and the associated standard specification documents including the Draft Text of ISO / IEC DIS 23094-2 Low Complexity Enhancement Video Coding published at MPEG 129 meeting in Brussels, held Monday, 13 January 2020 to Friday, 17 January 2020, all of these documents being incorporated by reference herein in their entirety.

[0067] In these coding formats a signal is decomposed in multiple “echelons” (also known as “hierarchical tiers”) of data, each corresponding to a “Level of Quality”, from the highest echelon at the sampling rate of the original signal to a lowest echelon. The lowest echelon is typically a low quality rendition of the original signal and other echelons contain information on correction to apply to a reconstructed rendition in order to produce the final output.

[0068] LCEVC adopts this multi-layer approach where any base codec (for example Advanced Video Coding - AVC, also known as H.264, or High Efficiency Video Coding - HEVC, also known as H.265) can be enhanced via an additional low bitrate stream. LCEVC is defined by two component streams, a base stream typically decodable by a hardware decoder and an enhancement stream consisting of one or more enhancement layers suitable for software processing implementation with sustainable power consumption.

[0069] In the specific LCEVC example of these tiered formats, the process works by encoding a lower resolution version of a source image using any existing codec (the base codec) and the difference between the reconstructed lower resolution image and the source using a different compression method (the enhancement).

[0070] The remaining details that make up the difference with the source are efficiently and rapidly compressed with LCEVC, which uses specific tools designed to compress residual data. The LCEVC enhancement compresses residual information on at least two layers, one at the resolution of the base to correct artefacts caused by the base encoding process and one at the source resolution that adds details to reconstruct the output frames. Between the two reconstructions the picture is upscaled using either a normative up-sampler or a custom one specified by the encoder in the bitstream. In addition, LCEVC also performs some non-linear operations called residual prediction, which further improve the reconstruction process preceding residual addition, collectively producing a low-complexity smart content-adaptive (i.e., encoder driven) upscaling.

[0071] In this way, the video frame is encoded hierarchically as opposed to using blockbased approaches as done in the MPEG family of algorithms. Hierarchically encoding a frame includes generating residuals for the full frame, and then a reduced or decimated frame and so on. In the examples described herein, residuals may be considered to be errors or differences at a particular level of quality or resolution.

[0072] Since LCEVC and similar coding formats leverage existing decoders and are inherently backwards-compatible, there exists a need for efficient and effective integration with existing video coding implementations without complete redesign. Examples of known video coding implementations include the software tool FFmpeg, which is used by the simple media player FFplay.

[0073] Moreover, LCEVC is not limited to known codecs and is theoretically capable of leveraging yet-to-be-developed codecs. As such any LCEVC implementation should be capable of integration with any hitherto known or yet-to-be-developed codec, implemented in hardware or software, without introducing coding complexity. LCEVC is an enhancement codec, meaning that it does not just upsample well: it will also encode the residual information necessary for true fidelity to the source and compress it (transforming, quantizing and coding it). LCEVC can also produce mathematically lossless reconstructions, meaning all of the information can be encoded and transmitted and the image perfectly reconstructed. Creator’s intent, small text, logos, ads and unpredictable high-resolution details are preserved with LCEVC.

[0074] As an example:-

[0075] LCEVC can deliver 2160p 10-bit HDR video over an 8-bit AVC base encoder.

[0076] When using an HEVC base encoder for a 2160p stream, LCEVC can deliver the same quality at typically 33% less of the original bitrate i.e. , lower a typical bitrate of 20 Mbit / s (HEVC only) to 15 Mbit / s or lower (LCEVC on HEVC).

[0077] The many unique benefits of LCEVC can be summarised as follows. LCEVC ... rapidly enhances the quality and cost efficiency of all codec workflows, reduces processing power requirements for serving a given resolution, is deployable via software, resulting in much lower power consumption, simplifies the transition from older generation to newer generation codecs. improves engagement by increasing visual quality at a given bitrate, is retrofittable and backward compatible. is immediately deployable at scale via software update, has low battery consumption on user devices. reduces new codecs complexity and makes them readily deployable.

[0078] With a view to all of the above, LCEVC allows for some interesting and highly economic ways to utilise legacy devices / platforms for higher resolutions and frame rates without the need to swap the entire hardware, ignoring customers with legacy devices, or creating duplicate services for new devices. That way the introduction of higher quality video services on legacy platforms at the same time generates demand for devices with even better coding performance. In addition, LCEVC not only eliminates the need to upgrade the platform, but it also allows for delivery of higher resolution content over existing delivery networks that might have limited bandwidth capability.

[0079] The approach of LCEVC being a codec agnostic enhancer based on a software- driven implementation, which leverages available hardware acceleration, also shows in the wider variety of implementation options on the decoding side. While existing decoders are typically implemented in hardware at the bottom of the stack, LCEVC basically allows for implementation on a variety of levels i.e. , from Scripting and Application to the OS and Driver level and all the way to the SoC and ASIC. In other words, there is more than one solution to implement LCEVC on the decoder side. Generally speaking, the lower in the stack the implementation takes place, the more device specific the approach becomes. Except for an implementation on ASIC level, no new hardware is needed.

[0080] Challenges exist when attempting to integrate LCEVC decoding into video decoder pipeline chipsets without re-designing those chipsets. It is desirable, at least in the short term, to implement LCEVC in a simple manner using existing architectures and designs. There are particular implementation challenges in relation to secure decoding of protected (e.g. premium) content.

[0081] In general, one place to perform operations for the LCEVC reconstruction stage, i.e. the combination of the residuals of the decoded enhancement and the base decoded video, is in the video output path. This is because the video output path is the most secure but also because such use is memory efficient, involving direct operations being performed on secure memory.

[0082] However, such an implementation in the video output path involves dealing with inherent hardware limitations. These hardware limitations include for example low memory bandwidth and limitations on the type of operations that can be performed. Elements of the video output path such as the video shifter (alternatively referred to as the graphics feeder) are specifically designed for, and excel at, functions such as overlay and colour space conversion but are limited for wider use. Different blocks of the video output path have different limitations and trade-offs and different blocks from different manufacturers have different functionalities. For example, a hardware upscaler designed for that specific use might have different trade-offs to a video shifter. Identifying how to implement LCEVC reconstruction within the video output path involves compromises. These challenges are exacerbated when dealing with operations at UHD resolutions.

[0083] Alternative implementation of LCEVC reconstruction into the decoder CPU may be insecure as the CPU is not a protected pipeline, while implementations of LCEVC into the video output path are potentially limited by those inherent hardware limitations of the blocks of the path. Implementations thus have the potential to be inefficient.

[0084] Innovations are sought which address the limitations of video decoder chipsets and facilitate the introduction and implementation of enhancement decoders, such as LCEVC, into the wider video decoder ecosystems.

[0085] For context purposes only, as the detailed structure of LCEVC is known and set out in the approved draft standards specification, Figure 1 illustrates, in a logical flow, how LCEVC operates on the decoding side assuming H.264 as the base codec. Those skilled in the art will understand how the examples described herein are also applicable to other multi-layer coding schemes (e.g., those that use a base layer and an enhancement layer) based on the general description of LCEVC that is presented with reference to Figure 1. Turning to Figure 1 , the LCEVC decoder 10 works at individual video frame level. It takes as an input a decoded low-resolution picture from a base (H.264 or other) video decoder 11 and the LCEVC enhancement data to produce a decoded full-resolution picture ready for rendering on the display view. The LCEVC enhancement data is typically received either in Supplemental Enhancement Information (SEI) of the H.264 Network Abstraction Layer (NAL), or in an additional data Packet Identifier (PID) and is separated from the base encoded video by a demultiplexer 12. Hence, the base video decoder 11 receives a demultiplexed encoded base stream and the LCEVC decoder 10 receives a demultiplexed encoded enhancement stream, which is decoded by the LCEVC decoder 10 to generate a set of residuals for combination with the decoded low-resolution picture from the base video decoder 11.

[0086] An example method corresponding to an aspect is shown diagrammatically in Figure 2. Figure 2 shows an example method, with commentary below provided to give details to help the reader’s understanding of the invention.

[0087] In step S200, source data is received. The source data may be received over a network, or read from a disk or other memory. In some examples, the source data may be a stream or bitstream, or a part thereof.

[0088] In step S202, input data is created from the source data. In this example, the input data has a lower bit length than the source data. Here, it is assumed that the source data only contains values which can be represented by the input elements, i.e. are less than 2n, where n is the number of bits per input element, and the higher bits, missing from the input data, are simply truncated. In other examples, it may be necessary to perform a more complex conversion, by scaling the source data and / or checking for overflow, or other techniques.

[0089] In step S204, the clipped data is created. In some examples, the clipping range is a quarter of the range of values which can be represented by the input data. Accordingly, in this illustrative case, in order to be able to represent the whole range, the clipped elements can be two bits shorter than the input elements. This is because 2nis four times larger than 2(n-2), where n is the number of bits per input element, and n-2 is the number of bits per clipped element.

[0090] In step S206, each input value is checked against the clipping range. In some examples, as is shown here, overflow data and modulus data are only created if the input value is outside the clipping range. In other examples, this data may always be created, and further, may be set to zero or another default value.

[0091] If the input value is inside the clipping range, the left branch is taken and the restructured data is created (step S208). If the input value is outside the clipping range, overflow values and modulus values must first be determined, then the restructured data can be created. In step S208, if the input value is inside the clipping range, the restructured data is created. In this case, the restructured data is equal to the clipped data. There is no overflow value.

[0092] In step S210, if the input value is outside the clipping range, the overflow value is calculated. The overflow value is the difference between the input value and the corresponding clipped value, to some predetermined precision. In some examples, as is shown here, the predetermined precision is lower than is needed to precisely encode all possible overflow values. This means that the overflow value is rounded to some extent, i.e. lacks some precision. In this case the overflow value is always an even number, i.e. does not encode the least significant bit of the overflow. In other examples, the overflow may be missing the two least significant bits, three least significant bits, or another number of least significant bits. The overflow value is then stored in an array. In other examples, it should be understood that the overflow value could be stored in any kind of data structure, such as an array, plane, list, dictionary or table, which could be sparse or not sparse.

[0093] Then, in step S212, a modulus value is determined. The modulus value is determined by calculating the difference between the input value and the sum of the corresponding clipped and overflow values. In this way, the modulus value provides the additional precision required to recreate the input value. For example, the overflow value may be selected to be insufficient bit length to represent every possible input value, instead being able to represent only every other input value. This may provide advantages in requiring less memory, or by allowing the overflow elements to be byte-aligned in memory, improving performance. In this case, the modulus value may encode whether or not the input value is even or odd, i.e. being either 0 or +1 , a one-bit value.

[0094] In step S214, restructured data is created. In this example, the restructured data is created by summing the clipped value and the modulus value. In other examples, the modulus value and the restructured value may be stored in bitwise fashion, for instance, as a struct of two integers, or by storing the modulus value in the least significant bits of the clipped value. In step S216, output data is created. Each output value is created from the corresponding restructured value and overflow value. In this example, the output value is created by summing the restructured value and the overflow value. Further, in this example, the sum is performed such that the output data has the same bit length as the input data, such that the result is not truncated or overflowed.

[0095] An example of a specific implementation of an aspect is shown diagrammatically in Figure 3. In this example, the input data corresponds to enhancement data, which can be used to enhance a base dataset. In some examples, this is the temporal layer of an LCEVC-compliant video signal.

[0096] In this example, the input data comprises 11 -bit integers, and the restructured data comprises 10-bit integers.

[0097] Accordingly, assuming these integers are unsigned integers, the input data comprises integers between 0 and 211- 1 = 2047, and the restructured data comprises integers between 0 and 210- 1 = 1023.

[0098] Alternatively, considering these integers as signed integers, the input data could comprise integers between -1023 and +1023, and the restructured data could comprise integers between -511 and +511.

[0099] This distinction merely corresponds to a difference of interpretation and does not have a material impact on the functioning of the method. In this example, the input data is assumed to be signed data, in the range of -1023 to +1023 (signed 11 -bit data). The restructured data and output data, however, are assumed to be unsigned data in the range of 0 to 1023 (unsigned 10-bit data). This slightly complicates the method by requiring the incorporation of an offset step, described below.

[0100] While the relationship between the specific choices of bit lengths of input data and restructured data influences the maximum size of the modulus data, it should be understood that these bit lengths could be any bit length. For instance, the input data could equally well be 16-bit integers, 20-bit integers or 32-bit integers, or indeed not integers at all, but floating or fixed-point values, etc. The input data could also be any combination of signed and / or unsigned data.

[0101] Details of how the input data is received and processed previous to the application of this method are omitted for brevity, though, in one example, could comprise the truncation of 16-bit data from a bitstream to 11 -bit integers.

[0102] In step S300, the signed input data is shifted.

[0103] We consider here a single datum of the input data, i.e. a single 11 -bit signed integer, for brevity, which shall be denoted r. This integer is given the value r = 600.

[0104] In this example, as discussed above, the input data comprises signed 11 -bit integers and the restructured data is chosen to be 10-bit unsigned integers. As, in this example, most of the input data are small values, around zero, it is advantageous to shift the zero of the input data into the centre of the range of the 10-bit restructured data. This ensures that less of the data is clipped in the clipping stage, as more of the input data will lie naturally in the range of the restructured data.

[0105] This corresponds to a shift of the input data by +512. Accordingly, the shifted input datum, in this example, is r2 = r + 512 = 600 + 512 = 1112.

[0106] In step S302, the modulus value is determined. How the modulus value is determined, and the maximum size of the modulus value required, depend upon the choice of the bit length of the input data, the restructured data and the overflow data. This is because the modulus value is used to encode the “missing” precision that cannot fit in the overflow value.

[0107] As said above, in this example, the input data comprises 11 -bit integers. The restructured data is chosen to be 10-bit integers. From this, it may appear that only one additional bit is required as the overflow data, to encode the difference between the two, as 11 - 10 = 1. However, this is not the case, because the restructured data is not simply the input data with the least significant bit removed. The restructured data is the same precision as the input data but represents a smaller range. In other words, the elements of the restructured data are capable of representing data which falls within a clipping range with the same precision as the original input data, but, if it falls outside the clipping range, it cannot be represented and is clipped. This means there is not a one-to-one mapping between the bits which make up the 11 -bit input element and the corresponding 1O-bit restructured element, and it is accordingly not possible to simply add an additional bit of information to recover the missing data.

[0108] It may also appear that the overflow data should comprise 1 O-bit integers, as the sum of two 10-bit integers can equal any 11 -bit integer. However, this is also not the case. As the clipping range is in the middle of the range of the input data, there could be overflow above or below the clipping range. Based on where the clipping is happening, the sign of the overflow data can be determined. If the clipping is clipping at the bottom end of the clipping range, the overflow data must be negative, as the input value is below the bottom end of the clipping range. If the clipping is at the top of the clipping range, the overflow data must be positive, as the input value is above the top end of the clipping range.

[0109] This is discussed in more detail below, and halves the amount of data required to be stored (as the overflow data does not need a sign), reducing the number of bits required by 1. Accordingly, 9 bits of information are required to accurately store the overflow.

[0110] However, in this example, 8-bit integers are used to store the overflow information. This means there is 1 bit of “missing” precision in the overflow data, which is stored in the modulus data. This missing precision amounts to whether or not the input data was even or odd. Using a modulus value in this way means that 8-bit integers can be used for the overflow data (this may be advantageous due to the fact that there are 8 bits in a byte and therefore the data is inherently byte-aligned), while costing only a very small portion of the clipping range to store this additional data, as will be shortly seen. Turning back to the present example, the modulus value can be found as the least significant bit of r, which encodes whether r is even or odd. This is the precision lost by using an 8-bit overflow value - the 8-bit overflow value can be thought of as “counting in twos” out past the end of the clipping range, to cover double the distance, with the modulus adding in an “extra one” at the end if required. Therefore, the modulus value, which herein is denoted B, can be found, in the notation of the C programming language, by B = r2 & 0x1 = 0. This expression means to take only the least significant bit of r2. B in this case is zero, because r2 is even.

[0111] In step S304, the input data is clipped to produce clipped data. In some examples, the clipping range may just be between 0 and the maximum value the restructured elements can represent, in this case 1023. However, in this example, the modulus value is added to the clipped value to obtain the restructured value. Therefore, the clipping range must be reduced in order to account for the modulus value. As the maximum modulus value possible in this case is +1 , as discussed above, the clipping range must therefore be between 0 and 1023 - 1 = 1022.

[0112] In other examples, the modulus value may be represented by a bit or series of bits in the restructured element. However, the approach of the present example can be advantageous as it stores the modulus value as part of the restructured value, and accordingly does not require additional storage.

[0113] At this stage, as the least significant bit is already represented by the modulus value, the least significant bit of r2 can be discarded prior to clipping. Accordingly, first, a value A is calculated by taking all but the least significant bit of r2, where A = r2 & OxFFFE = 1112. This leaves r2 unchanged, because it is even. If r2 were odd, A = r2- 1.

[0114] Then, the clipped datum C can be created, where: if A < 0, C = 0; if A > 1022, C = 1022; otherwise, C = A.

[0115] In step S306, the values of C and B are combined to produce the final restructured data. The restructured data C2 = B + C = 1022. This corresponds to incorporating the modulus value B into the restructured data, i.e. adding back in the least significant bit.

[0116] In this example, the effect of the use of the modulus data is to preserve whether or not the input value is even or odd when it is used to create the restructured data. This allows the use of a modulus value which “counts in twos”, i.e. which is one bit smaller than would otherwise be required. In other examples, this method can be used to store more information in the modulus data. Instead of the single bit of information corresponding to even / odd input values, in some examples, two bits of information may be stored, corresponding to +0, +1 , +2 or +3 to the clipped data. In this way, overflow data which “counts in fours” could be used. In other examples, three bits of information may be stored in the modulus data, corresponding to +0, +1 , +2, +3, +4, +5, +6, or +7 to the clipped data. In this way, overflow data which “counts in eights” could be used.

[0117] In step S308, the overflow data is created. The overflow data is simply the difference between the restructured data C2 and the shifted input data r2, giving overflow data O = r2 - C2. In this case, O = 7772 - 7022 = 90.

[0118] In step S310, the overflow data is reduced in precision. The use of the modulus value in the previous steps means that fewer bits can be used to represent the overflow data. In this case, the modulus value incorporates one bit of information, allowing the overflow data to be halved. This corresponds to a bit shift one to the right. In other examples, depending on the number of bits of information stored in the modulus value, greater shifts and reductions in size of the overflow data may be used. Such a shifted overflow data may be written, in the notation of the C programming language, as 02 = O » 7 = 45.

[0119] If the overflow data is negative, in this example, the sign can be discarded and it can be made positive. This is because, when combining it with the restructured data in step S316, it can be determined if the overflow data should be negative. This is because, if the clipping is clipping at the bottom end of the clipping range, i.e. C = 0 or C = 1 , the overflow data must be negative. If the clipping is at the top of the clipping range, i.e. C = 1022 or C = 1023, the overflow data must be positive. In step S312, the restructured data C2 and the overflow data 02 are stored in a restructured plane and an overflow plane respectively. In this example, the restructured data and overflow data are stored as packed data, i.e. without bytealignment, as consecutive 10-bit and 8-bit values respectively. In other examples, they may be stored as byte-aligned data, or not stored in planes at all, but in other data structures, such as lists.

[0120] In step S314, relative update data is received for input data r. In this case, this is relative update data U = -200. In other words, new input data r’ = r + U = r- 200.

[0121] In order to calculate the corresponding change to the restructured data, without having to store or use the original input data, the output value is created. This uses the restructured data and the overflow data to reconstruct the input data, such that the relative update can be applied.

[0122] In step S316, the output data is created. First, the overflow data is read from the overflow plane, where 02 = 45. This is shifted left by 1 to undo the earlier right shift, providing, in the notation of the C programming language, O = 02 « 1 = 90. Then, the overflow data can be added to the restructured data C2, yielding output data s = C2 + O = 1112. In order to arrive back at the original input data, the +512 shift must be subtracted, yielding n = s - 512 = 600. In this way, the original input data of 600 is recovered.

[0123] In some examples, an apparatus is provided as generally illustrated at 1000 in Figure 4. This includes a non-transitory computer-readable medium 1010, such as a hard-drive, solid-state drive or some other form of storage medium, is provided. This is capable of holding or storing a computer program 1001. The computer program includes instructions, typically in the form of computer code, that, when implemented on a computing device, such as by executing the program, cause an apparatus to perform one or more of the methods and processes set out above or to provide one or more of the decoders set out above. The apparatus may be a computing device such as a chip, server or some other form of computer. The system also includes a multi-core CPU 1002 on which the decoder is able to be implemented in some examples. In other examples, the CPU may be a single-core CPU.

Claims

CLAIMS1. A computer-implemented method of data restructuring, the method comprising: receiving input data comprising input elements, each input element representing an input value; generating, for each input value, a corresponding clipped value; generating overflow data comprising, for each input value, a corresponding overflow element representing an overflow value; generating, for each input value, a corresponding modulus value; and generating restructured data comprising, for each input element, a corresponding restructured element comprising the corresponding clipped value and modulus value.

2. The method of claim 1 , wherein generating, for each input value, a corresponding clipped value comprises: if the input value is in a clipping range, setting the clipped value to be equal to the input value; or if the input value is outside of the clipping range, setting the clipped value to be equal to an end value of the clipped range.

3. The method of claim 1 or claim 2, wherein generating overflow data comprising, for each input value, a corresponding overflow element representing an overflow value comprises: if the input value is in a clipping range, setting the overflow value to be zero; or if the input value is outside of the clipping range, setting the overflow value, at a predetermined level of precision, to be the difference between an end value of the clipping range and the input value.

4. The method of any one of the preceding claims, wherein: if the input value is in the clipping range, the modulus value is zero; or if the input value is outside of the clipping range, the modulus value is the difference between the input value and the sum of the end value of the clippingrange and the corresponding overflow value at the predetermined level of precision.

5. The method of any one of the preceding claims, further comprising generating output data using the restructured data and the overflow data, the output data representing the input data.

6. The method of any one of the preceding claims, wherein the data is video data.

7. The method of any one of the preceding claims, wherein the input data is enhancement data, generated by an enhancement encoding process, and the input elements are residual elements of the enhancement data.

8. The method of claim 7, further comprising enhancing a base dataset using the restructured data.

9. The method of any one of the preceding claims, wherein the end value of the clipping range is less than a maximum restructured value, the maximum restructured value being the maximum value which can be represented by a restructured element.

10. The method of any one of the preceding claims, wherein generating restructured data comprises summing each clipped value and the corresponding modulus value.

11. The method of any one of the preceding claims, wherein the input elements have a greater bit length than both the restructured elements and the overflow elements.

12. The method of any one of the preceding claims, wherein the restructured data is stored in a restructured plane, and overflow data is stored in an overflow plane, where the restructured data and the overflow data are packed data.

13. The method of claim 12, where the restructured plane is stored in a predetermined format compatible with a hardware module, and the method furthercomprises copying the restructured plane to a predetermined location, such that it is suitable for processing by the hardware module.

14. The method of any one of claims 1 to 11 , wherein the restructured data and the overflow data are stored in a combined plane, where each restructured element and the corresponding overflow element are stored consecutively in the combined plane.

15. The method of any one of the preceding claims, wherein the step of receiving input data comprises: receiving source data, wherein each source element of the source data has a greater bit length than the input elements; and for each source element of the source data, creating an input element of the input data by truncating the source element to the bit length of the input element.

16. The method of claim 15, wherein the source data is the temporal layer of an LCEVC-compliant video signal.

17. The method of any one of claims 1 to 14, wherein the input data is the temporal layer of an LCEVC-compliant video signal.

18. The method of any one of the preceding claims, further comprising: receiving update data comprising one or more update elements, the one or more update elements representing relative updates to a corresponding one or more of the input elements; and for each update element, updating the corresponding restructured element based on said update element and the corresponding output element.

19. The method of any one of the preceding claims, wherein each output element represents an output value, the output value being the sum of the corresponding overflow value, clipped value and modulus value.

20. The method of any one of the preceding claims, wherein the input data is in a first format, and the restructured data is in a second format.21 . The method of claim 20, wherein the second format is a format compatible with a hardware module, in particular wherein the second format is suitable for a display of an apparatus performing the method and the first format is a format of a bitstream received by the apparatus, said input data derived from the bitstream.

22. An apparatus configured to restructure data, the apparatus being configured to: receive input data comprising input elements, each input element representing an input value; generate, for each input value, a corresponding clipped value; generate overflow data comprising, for each input value, a corresponding overflow element representing an overflow value; generate, for each input value, a corresponding modulus value; and generate restructured data comprising, for each input element, a corresponding restructured element comprising the corresponding clipped value and modulus value.

23. The apparatus of claim 22, wherein generating, for each input value, a corresponding clipped value comprises: if the input value is in a clipping range, setting the clipped value to be equal to the input value; or if the input value is outside of the clipping range, setting the clipped value to be equal to an end value of the clipped range.

24. The apparatus of claim 22 or claim 23, wherein generating overflow data comprising, for each input value, a corresponding overflow element representing an overflow value comprises: if the input value is in a clipping range, setting the overflow value to be zero; or if the input value is outside of the clipping range, setting the overflow value, at a predetermined level of precision, to be the difference between an end value of the clipping range and the input value.

25. The apparatus of any one of claims 22 to 24, wherein:if the input value is in the clipping range, the modulus value is zero; or if the input value is outside of the clipping range, the modulus value is the difference between the input value and the sum of the end value of the clipping range and the corresponding overflow value at the predetermined level of precision.

26. The apparatus of any one of claims 22 to 25, wherein the apparatus is further configured to generate output data using the restructured data and the overflow data, the output data representing the input data.

27. The apparatus of any one of claims 22 to 26, wherein the input data is enhancement data and the input elements are residual elements, wherein the apparatus is further configured to enhance a base dataset using the restructured data.

28. The apparatus of any one of claims 22 to 27, wherein the input data is the temporal layer of an LCEVC-compliant video signal.

29. The apparatus of any one of claims 22 to 28, further comprising a hardware module, wherein the restructured data is stored in a restructured plane in a predetermined format compatible with the hardware module, and the apparatus is further configured to copy the restructured plane to a predetermined location in memory, such that it can be acted on by the hardware module.

30. The apparatus of any one of claims 22 to 29, wherein the apparatus is further configured to receive update data, representing relative updates to one or more updated input elements, and further configured to, for each updated input element, update the corresponding restructured element based on the update data and the corresponding output element.31 . The apparatus of any one of claims 22 to 30, wherein the apparatus is a decoder.

32. A computer program comprising instructions, which, when the program is executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 21 .

33. A non-transitory computer-readable medium comprising the computer program according to claim 32.

Images