Parallel decoding

Abstract

There is provided a method of decoding an entropy encoded signal. The method comprises: separating at least a part of a received entropy encoded signal into a plurality of surfaces, each surface comprising entropy encoded data; generating an array of coefficients from each surface by entropy decoding, in parallel, each surface, wherein each coefficient in each array is an entropy decoded coefficient of the entropy encoded data of the respective surface; and processing the coefficients of each array into an output readable to create a decoded signal of the entropy encoded signal thereby replicating the signal encoded by entropy encoding. A corresponding apparatus is provided.

Description

[0001] PARALLEL DECODING

[0002] FIELD OF THE INVENTION

[0003] This disclosure relates to a method for decoding video data. In particular, but not exclusively, this disclosure relates to advantageous implementations of standardised hierarchical coding schemes.

[0004] BACKGROUND

[0005] The decoding technology for the Low Complexity Enhancement Video Codec / Coding (LCEVC) is set out in the Draft Text of ISO / IEC FDIS 23094-2 as published at Meeting 129 of MPEG in Brussels in January 2020, as well as the Final Approved Text and WO 2020 / 188273 A1 , each of which is incorporated by reference.

[0006] Enhancement codecs, such as LCEVC have been described in documents such as the LCEVC standard Draft Text of ISO / IEC FDIS 23094-2 as published at Meeting 129 of MPEG in Brussels in January 2020, as well as the Final Approved Text and WO 2020 / 188273 A1.

[0007] A working implementation of this standard is found in the LCEVC Test Model (‘LTM’), which decodes data in accordance with the decoding processes described in the LCEVC standard document.

[0008] The standard describes a decoding process comprising multiple operations such as entropy decoding, dequantization, and inverse-transformation. The LTM describes a process of taking the encoded data, processing the entirety of the data with a first set of operations, and then processing the output of that with a second set of operations, and so forth. In this way, an entire plane (or planes) of data is processed by a single operation before being fed into a further operation. The output of each operation may comprise one or more data structures representing a plane of video. These data structures may be stored in memory ready for access by a subsequent operation. For example, an entire plane(s) of quantised data is dequantized to create a plane of dequantized data. The plane(s) of dequantized data is then inverse transformed to generate a plane of residuals. The plane of residuals is then applied to an initial reconstructed signal. Acompliant decoder is created in this manner.

[0009] SUMMARY OF INVENTION

[0010] The Inventors have determined that a decoding process for a video signal comprises a number of different operations that need to be performed in sequence. For example, these operations may comprise entropy decoding (e.g., using run length and Huffman decoders), dequantization, and applying an inverse transformation to coding units. These operations are described in the LCEVC standard. In one case, planes of coefficients that result from an encoding transformation are encoded as separate “surfaces”. In an implementation, entropy decoding is performed for all the different surfaces (e.g., all the different directions resulting from the encoding transformation) in parallel. The result of the entropy decoding is stored as an intermediate signal or representation. The intermediate signal or representation comprises the entropy decoded coefficients for the set of surfaces. It is processed to produce residual values and their corresponding coordinate in an image to be reconstructed (e.g., a colour plane for a frame of video). The decoding process on the intermediate representation may comprise dequantization and an inverse transformation. The run-lengths from the run length encoding may be stored together with the decoded coefficients. This can aid skipping over portions of the signal where residual values are zero or null values. The dequantization and inverse transformation may be applied as a single thread. By storing residual values with a coordinate (e.g., so-called “geometry residuals”), the storage of a whole matrix of residuals to apply to a colour plane of a reconstructed signal may be avoided. For example, the residual data may be sparse (e.g., have many zero values) and zero values do not need to be applied to the reconstructed signal. As such, the geometry residual list may be much smaller and more efficient to apply than a complete matrix (e.g., each residual value may be applied individually based on the list and multiprocessing approaches may be applied to the list). Accordingly there is an aspect, which provides an apparatus configured to decode an entropy encoded signal, the apparatus being configured to: separate at least a part of a received entropy encoded signal into a plurality of surfaces, each surface comprising entropy encoded data; generate an array of coefficients from each surface by entropy decoding, in parallel, each surface, wherein each coefficient in each array is an entropy decoded coefficient of the entropy encoded data of the respective surface; and process the coefficients of each array into an output readable to create a decoded signal of the entropy encoded signal thereby replicating the signal encoded by entropy encoding.

[0011] There is thus described an improved apparatus that provides decoding. In particular, embodiments are described that have a quicker decode time than, for example, the LTM. various examples are described that can produce a compliant decoded bitstream whilst performing fewer calculations that, for example, the LTM. Described examples comprise features that synergistically produce both of these technical effects.

[0012] We describe a method of decoding. The method may comprise entropy decoding encoded data to obtain coefficients and run length indications. The method may comprise storing the obtained coefficients and run length indications as intermediate representations. Each coefficient and run length indication may correspond to a single layer of a plurality of layers. The method may comprise processing each layer separately to generate processed layers, wherein the processing may comprise performing at least one decoding operation. The method may comprise combining the processed coefficients to generate enhancement data.

[0013] Accordingly there is a further aspect that provides a method of decoding an entropy encoded signal, the method comprising: separating at least a part of a received entropy encoded signal into a plurality of surfaces, each surface comprising entropy encoded data; generating an array of coefficients from each surface by entropy decoding, in parallel, each surface, wherein each coefficient in each array is an entropy decoded coefficient of the entropy encoded data of the respective surface; and processing the coefficients of each array into an output readable to create a decoded signal of the entropy encoded signal thereby replicating the signal encoded by entropy encoding.

[0014] The apparatus of the above aspect may be configured to enact the steps of the method in use.

[0015] The method may further comprise receiving the entropy encoded signal. This may be instead of simply having the entropy encoded signal already available.

[0016] The entropy encoding may include run length encoding and the entropy decoding includes run length decoding. This allows the entropy decoding, and hence the array generation, to pass over portions of each surface that do not need to be processed, which reduces processing resource requirements and processing time.

[0017] The entropy encoded data of each surface may be represented by a plurality of elements, each element having a value, and each entropy decoding of each surface may include decoding a respective surface from the element with the lowest non-null value.

[0018] The entropy decoding of each surface may include reading the respective surface to identify the element with the lowest non-null value and leaving preceding elements encoded.

[0019] The entropy encoded data may include a plurality of run length indications. This simplifies identification of a location in a surface to which to skip during decoding.

[0020] Processing each layer (also referred to as a “surface”) separately may comprise processing each layer on (i.e. within) a separate thread. This implements parallel processing of each surface allowing efficient processing of all the surfaces without slowing overall processing.

[0021] The entropy encoding may include Huffman coding and the entropy decoding includes Huffman decoding. This allows the benefits of Huffman encoding and decoding to be implemented as part of the parallel processing of this aspect. The method may comprise performing the entropy decoding on a single thread. Typically, this may be implemented by entropy decoding of each surface being performed on a thread independent of concurrent entropy decoding each respective other surface. Further, on completion of entropy decoding a respective surface, each thread may be reinitialised with a further surface on which entropy decoding is performed. This allows the thread / reinitialised thread to be re-used when processing is completed, allowing the thread to be used multiple times and for processing speeds to be enhanced by making use of threads on which processing has been completed.

[0022] The method may comprise performing the combining on a single thread.

[0023] The decoding operation may comprise dequantizing. Typically, this may be implemented by the processing including dequantization of each array of coefficients.

[0024] The layers may correspond to outputs of a transform, for example (and as described in further detail below) there may be an ‘A’ layer, a ‘H’ layer, a V layer and a ‘D’ layer.

[0025] Enhancement data may comprise residuals. Typically, this may be implemented by the processing including writing the array of coefficients to a residual representation, the residual representation including residual values. Further, this may be achieved by the residual representation including a coordinate associated with each residual value.

[0026] Overall, the processing may include writing the array of coefficients to a buffer or operation list as alternatives to one or more residual representations.

[0027] The combining may comprise applying an inverse transform. Typically, this may be implemented by the processing including performing an inverse transform on each array of coefficients.

[0028] The output of the processing the coefficients of each array may comprise enhancement data. Enhancing a base image may be performed by using the enhancement data. Typically, this may be implemented by the method further comprising enhancing a base dataset using the enhancement data.

[0029] Each coefficient may be paired with a run length indication. In other words, the entropy data may include a plurality of encoded coefficients, each of the encoded coefficients being paired with a run length indication of the plurality of run length indications. This provides a link between respective coefficients and locations in a surface.

[0030] Combining the processed coefficients to generate enhancement data may comprise processing the processed coefficients to populate a command buffer with one or more instructions to modify a plane of data.

[0031] The plane of data may comprise a base image to be enhanced.

[0032] The plane of data may comprise a temporal buffer associated with the enhancement data.

[0033] Each instruction may comprise an indication of an area of the plane of data to enhance and an action. The indication of an area may comprise a coordinate. The action may indicate to: modify a base image; or modify a temporal buffer associated with the enhancement data. An action indicating to modify a temporal buffer associated with the enhancement data may comprise: an indication to increase or decrease a single value of the temporal buffer; or an indication to zero multiple values of the temporal buffer.

[0034] The multiple values may comprise all values of a coding unit. A coding unit may be a transform unit associated with the inverse transform.

[0035] The command buffer may comprise multiple separate buffers. The method may comprise, in response to determining that an instruction comprises an action to modify a base image, adding said instruction to a first buffer of the multiple separate buffers. The method may comprise, in response to determining that an instruction comprises an indication to increase or decrease a single value of the temporal buffer action to modify a base image, adding said instruction to a second buffer of the multiple separate buffers. The method may comprise, in response to determining that an instruction comprises an indication to zero multiple values of the temporal buffer, adding said instruction to a third buffer of the multiple separate buffers.

[0036] The method may comprise splitting the command buffer into a plurality of sublists. The method may comprise assigning a separate thread to each of the plurality of sublists.

[0037] Splitting the command buffer may comprise splitting the command buffer into approximately even sized sublists to reduce processing time.

[0038] Splitting the command buffer may comprise splitting the command buffer into approximately even sized sublists such that each of the separate threads will have approximately same amount of processing to perform.

[0039] While typically the entropy encoded signal is an entropy encoded video signal, the method can be applied to alternative forms of signal, such as still images, or photographic signals. When a video signal, the entropy encoded signal may be an LCEVC compliant signal.

[0040] Rather than following the general teachings of the standard by completing an operation of the decoding process on the entirety of (e.g. a frame of) data before commencing a next operation, aspects utilise parallel processing. In more detail, in the LTM, a relatively large amount of data (e.g. data relating to an entire plane) is transferred in and out of memory (e.g. a buffer) whilst the entire plane is being processed by an operation.

[0041] The entropy decoding may performed on a multi-core Central Processing Unit (CPU) In certain examples described herein, the operation discussed utilises multiple processing threads, commonly referred to as “threads”. Hence, parallel processing of multiple threads may provide increased speed and concurrency. For example, on a multi-core CPU, each core may process one thread for a given layer. In otherwords, typically, each core of the CPU may be configured to process one thread at a time. As set out above, typically, each surface may be performed on a thread independent of concurrent entropy decoding each respective other surface. Once a particular thread is finished with, it may either be terminated and respawned or reset (e.g., reinitialised with encoded data associated with a next coding unit). In other words, on completion of entropy decoding a respective surface, each thread may be reinitialised with a further surface on which entropy decoding is performed. This process may be particularly efficient if the residual data is sparse, as this will lead to sparse coefficients and many portions of encoded data that are effectively 0 and do not need to be applied.

[0042] As may be appreciated, references to frame herein may be taken as a reference to one of one or more colour planes making up a frame. In this description the term “thread” is used as per the art of computer science, e.g. to refer to a smallest sequence of programmed instructions that can be managed independently by a scheduler (e.g., a scheduler that forms part of an operating system). As each thread typically has an associated portion of random access memory assigned to it (e.g., based on the variables that are to be processed by the thread), loads to and from memory may be optimised using the present approach as throughout the decoding sequence data may be small enough to fit within processor cache thus reducing memory accesses as different versions of the data are generated during the decoding sequence.

[0043] This is a departure from other teachings relating to other coding standards. In many coding standards, there are coding dependencies between different units of encoded data (i.e. intra-coding) and thus do not allow for such parallel processing during parts of the decoding process. However, some coding standards such as LCEVC, units can be decoded independently of other units, in other words, there is no intra compression. This lack of dependency allows a plane of encoded data to be decoded in parallel to other planes.

[0044] One of the decoding operations may comprise applying an inverse transform to multiple planes of data (e.g. 4 or 16) to generate a single plane of residual data. Each plane may represent an element in a vector that is the result of a matrix transformation applied to a flattened area of pixels with associated residual values (e.g., a 2x2 area of residual values may be flattened into a 4x1 vector and then multiplied by a matrix such as a 4x4 Hadamard matrix). Each plane of the ‘encoded’ (in this example transformed data, also referred to as coefficients, i.e. prior to being inverse transformed) may be run length encoded. For example, in LCEVC a 4x4 or 16x16 Hadamard transform is used that produces 4 or 16 planes of coefficients. The coefficients for the 4x4 transform may be represented as Average, Horizontal, Vertical and Diagonal directional components (A, H, V, D). Each of these directional components may be encoded as a separate plane or “surface” and the distribution of transformed residual values may vary according to the directional component (e.g., an average or adjusted average component may have more runs of 0). This provides for parallel decoding.

[0045] An intermediate representation of the processed (e.g. decoded) encoded data can be seen as a ‘command buffer’, this is because the intermediate representation are representations of the enhancement data that are obtained (and stored - hence ‘buffer’) overtime as portions of encoded data are processed. In examples, an intermediate representation comprises a coordinate (i.e. a reference to a position in a frame) and a (residual) value to adjust a pixel of a base image (e.g. +4 or -10). This may be applied in parallel to all the colour planes of a frame. This may be advantageous because if an enhancement frame only enhances a small number of pixels (e.g. 2 pixels) then an intermediate representation only needs to store two ‘instructions’, this is in contrast to if the enhancement frame was stored as a plane which could involve storing thousands of values (e.g. if the plane is 1080p). The concept of intermediate representations works synergistically with embodiments, each (e.g. fully processed) ‘portion’ may correspond to a (e.g. single) intermediate representation.

[0046] Examples thus generally refer to frame by frame processing.

[0047] The encoding method may be such that the coding or transform units are decodable independently of other coding or transform units.

[0048] The encoding method may be such that the coding or transform units are decodable independently of a decoding of the base image. We describe a device (e.g. a decoder) adapted to perform the described method. An aspect thereby provides a device or decoder adapted accordingly.

[0049] We describe a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the described method. We describe a non-transitory computer-readable medium comprising the computer program. Aspects thereby provide a computer program comprising said instructions and / or said non-transitory-readable medium.

[0050] We describe a decoder configured to decode the encoded data and to combine an output of said decoding with a decoding of the base encoding to generate a reconstruction of the input video.

[0051] BRIEF DESCRIPTION OF DRAWINGS

[0052] One or more examples will now be described with reference to the accompanying drawings, in which:

[0053] Figures 1 and 2 are schematic diagrams respectively showing an example multilayer encoder and decoder configuration;

[0054] Figure 3 is a schematic diagram showing certain data processing operations performed by an example multi-layer encoder;

[0055] Figure 4 is a schematic diagram showing further certain data processing operations performed by an example multi-layer encoder;

[0056] Figure 5 is a schematic diagram showing certain data processing operations performed by an example multi-layer decoder;

[0057] Figure 6 is a schematic diagram showing further certain data processing operations performed by an example multi-layer decoder;

[0058] Figure 7 is a flow diagram showing an example method of decoding an entropy encoded signal; and Figure 8 is a schematic diagram of an example system configured to implement the example method.

[0059] DETAILED DESCRIPTION

[0060] Certain examples described herein act to encode or decode a signal into a set of one or more data streams, i.e. data that changes over time. Certain examples relate to an encoder or encoding process that generates a set of streams including at least an enhancement stream, where the enhancement stream provides enhancement to a base stream. The base stream may comprise an encoding with MPEG standards such as AVC / H.264, HEVC / H.265, etc. as well as algorithms such as VP9, AV1 , and others. The enhancement stream may comprise an LCEVC stream. It is worth noting that the base stream may be decodable by a hardware decoder while the enhancement stream may be suitable for a software processing implementation with suitable power consumption. Certain examples provide an encoding structure that creates a plurality of degrees of freedom that allow great flexibility and adaptability in many situations, thus making the coding format suitable for many use cases including over-the-top (OTT) transmission, live streaming, live Ultra High Definition (UHD) broadcast, and so on. It also provides for low complexity video coding.

[0061] In certain examples, a specific enhancement encoding scheme may be used. In one such case, the input video may be received at a first spatial resolution (e.g., High Definition, HD, or UHD) and the method may comprise a number of steps on a frame-by-frame basis. To start, a current frame of the input video may be downsampled to create a downsampled frame at a second spatial resolution that is lower than the first spatial resolution. For example, the second spatial resolution may be HD or Standard Definition (SD) content for respective original UHD or HD content.

[0062] As part of the method, the base encoding of the downsampled frame may be instructed using the base encoder to create a base encoded stream. For example, an enhancement encoder may call an operating system encoding method and / or execute a defined executable file representing a registered and selected base encoder. This instructing may comprise passing the base quality factor to the base encoder, e.g. as part of a command line parameter or base configuration file.

[0063] Following this, a reconstruction of the current frame at the second spatial resolution is reconstructed from a decoding of the base encoded stream. In other words, the reconstruction of the current frame at the second spatial resolution is reconstruction from a decoding of the current frame in the base encoded stream, which may depend on other encoded frames if motion compensation is applied.

[0064] Then an enhancement encoder computes residual data for a first sublayer of the plurality of sublayers as a difference between the reconstruction of the current frame at the second spatial resolution and the downsampled current frame. For example, this may comprise the process of constructing an LCEVC first sublayer. Then, the residual data for the first sublayer is encoded using a quantisation step width to generate encoded residual data for the first sublayer.

[0065] The quantisation step width (SW) may be computed using a function based on the form: SW = a + (1-a)*Q_factorA(-1 / 2), where a is determined empirically and Q_factor is a modulated quantisation (Q) factor. Different Q factors may be computed from the modulated Q factor for each sublayer. In another case, the step widths may be computed as a linear or power function with custom multipliers and constant factors. Caps and / or clamps may also be added to improve performance.

[0066] The modulated Q factor may be computed by a mapping function. In some examples, this may be computed as set out in PCT / GB2023 / 053320, which is incorporated by reference, making use of the disclosed mapping function, empirical lookup tables, base quality factor, initial encoding quality factor, base factor calculator and / or enhancement quality factor. The mapping function may comprise a non-linear function where the parameters from the empirical tables set a multiplication coefficient and an exponential for the function.

[0067] In various examples, the mapping function is configured to output the Q factor using a set of parameters and / or coefficients, such as a set of parameters and / or coefficients retrieves, for example using a base quality factor for a set of empirical lookup tables. In one case, the mapping function may first compute a modified base quality factor from a received base quality factor. This may comprise applying corrections or modulation for one or more of the following: resolution of the base and / or enhancement encoding, sharpness filtering parameters, and frame rate. The mapping function may then compute the Q factor based on the modified base quality factor. In one case, the mapping function may apply different computations for different ranges for an input encoding quality factor. For example, at or above a threshold computed based on the modified base quality factor, the Q factor may be set as a constant. Below said threshold, the Q factor may be computed as a non-linear function of the modified base quality factor and the encoding quality factor. In this case, coefficients including a multiplier and a power may be retrieved from a look-up table for a specified base encoder based on the modified base quality factor value. For example, below the threshold (if applied), the Q factor may be computed as a linear function of the modified base quality factor and the encoding quality factor as multiplied by the multiplier, with the result of the linear function being then raised to the power. The linear function may also comprise a framerate adjustment term. As noted above, in certain cases, constraints or caps on the modified base quality factor and / or the Q factor may be applied (e.g., applying minimum or maximum clamping).

[0068] Returning to the method, a decoding of the encoded residual data for the first sublayer is then combined with the reconstruction of the current frame to generate a corrected reconstruction of the current frame. The corrected reconstruction of the current frame is upsampled to the first spatial resolution and residual data for a second sublayer of the plurality of sublayers is computed as a difference between the upsampled corrected reconstruction of the current frame and the current frame.

[0069] Lastly, the residual data for the second sublayer is encoded using the quantisation step width for the second sublayer as described above to generate encoded residual data for the second sublayer. The enhancement encoder thus outputs an encoded enhancement stream. The encoded enhancement stream may be multiplexed with the encoded base stream that is output by the base encoder or they may be used as separate streams (e.g., stored and / or transmitted as appropriately configured bit streams). More detail of the process of generating this particular enhancement encoding is described with reference to Figures 1 to 3.

[0070] In certain variations, the method is applied on a frame-by-frame basis and the base encoding parameters comprise one or more of: a frame type; a frame size in bits; and a quantisation metric for the frame. The frame type may indicate one of: an Intra (I) frame, a Predicted (P) frame, and a Bidirectional (B) frame. The quantisation metric may comprise an average quantisation parameter (QP) for the frame. The quantisation metric may alternatively comprise QPs for different regions of interest.

[0071] In certain cases, mapping the encoding quality factor and / or mapping the base encoding parameters comprises using one or more look-up tables. For example, these look-up tables may store function parameters or mapped output values. Interpolation and rounding may be used where desired depending on implementation.

[0072] In one particular variation, mapping the encoding quality factor and / or mapping the base encoding parameters comprises using one or more trained neural network architectures. For example, training samples comprising tuples of base encoding parameters and an enhancement quantisation factor (e.g., a normalised scalar or integer output) may be generated and these may be used to train a feedforward neural network with at least one hidden layer using off-the-shelf backpropagation methods (e.g., as applied by the PyTorch or Tensorflow software libraries). The feed-forward neural network may then be used with the trained parameters in an inference mode to predict the enhancement quantisation factor from the input base encoding parameters. A similar method may be applied with training data in the form of encoding quality factors and base quality factors.

[0073] In one case, mapping the encoding quality factor to the base quality factor comprises: determining a base encoding type, the base encoding type being selected from a plurality of different available base encoding types based on the base encoder used for the base encoding; and configuring a mapping for the determined base encoding type.

[0074] The described methods may be used to encode or decode video data. For example, an encoder may be adapted to perform the methods as described herein. In a software implementation, a computer program may be provided comprising instructions which, when the program is executed by a computer, cause the computer to carry out the methods as described herein, i.e. when executed by one or more processors of a computing device. The computer program may be stored upon a non-transitory computer-readable medium. The described methods and / or apparatus may be used to generate an enhancement bit stream that is encoded using the enhancement encoding parameters as computed by the described methods and / or apparatus. A decoder may be provided that is configured to decode this enhancement bit stream and to combine an output of said decoding with a decoding of the base encoding to generate a reconstruction of the input video.

[0075] Certain general information relating to example enhancement coding schemeswill now be described. This information provides examples of specific multi-layer coding schemes.

[0076] It should be noted that examples are presented herein with reference to a signal as a sequence of samples (i.e., two-dimensional images, video frames, video fields, sound frames, etc.). For simplicity, non-limiting examples illustrated herein often refer to signals that are displayed as 2D planes of settings (e.g., 2D images in a suitable colour space), such as for instance a video signal. In an example case, the signal comprises a video signal. An example video signal is described in more detail with reference to Figure 3. Although examples are described specifically in relation to video, the approaches may also be used for other forms of media such as audio or subtitle data.

[0077] The terms “picture”, “frame” or “field” are used interchangeably with the term “image”, so as to indicate a sample in time of the video signal: any concepts and methods illustrated for video signals made of frames (progressive video signals) can be easily applicable also to video signals made of fields (interlaced video signals), and vice versa. Despite the focus of examples illustrated herein on image and video signals, people skilled in the art can easily understand that the same concepts and methods are also applicable to any other types of multidimensional signal (e.g., audio signals, volumetric signals, stereoscopic video signals, 3DoF / 6DoF video signals, plenoptic signals, point clouds, etc.). Although image or video coding examples are provided, the same approaches may be applied to signals with dimensions fewer than two (e.g., audio or sensor streams) or greater than two (e.g., volumetric signals).

[0078] In the description the terms “image”, “picture” or “plane” (intended with the broadest meaning of “hyperplane”, i.e., array of elements with any number of dimensions and a given sampling grid) will be often used to identify the digital rendition of a sample of the signal along the sequence of samples, wherein each plane has a given resolution for each of its dimensions (e.g., X and Y), and comprises a set of plane elements (or “element”, or “pel”, or display element for two-dimensional images often called “pixel”, for volumetric images often called “voxel”, etc.) characterized by one or more “values” or “settings” (e.g., by ways of non-limiting examples, colour settings in a suitable colour space, settings indicating density levels, settings indicating temperature levels, settings indicating audio pitch, settings indicating amplitude, settings indicating depth, settings indicating alpha channel transparency level, etc.). Each plane element is identified by a suitable set of coordinates, indicating the integer positions of said element in the sampling grid of the image. Signal dimensions can include only spatial dimensions (e.g., in the case of an image) or also a time dimension (e.g., in the case of a signal evolving over time, such as a video signal). In one case, a frame of a video signal may be seen to comprise a two-dimensional array with three colour component channels or a three-dimensional array with two spatial dimensions (e.g., of an indicated resolution - with lengths equal to the respective height and width of the frame) and one colour component dimension (e.g., having a length of 3). In certain cases, the processing described herein is performed individually to each plane of colour component values that make up the frame. For example, planes of pixel values representing each of Y, U, and V colour components may be processed in parallel using the methods described herein.

[0079] Certain examples described herein use a scalability framework that uses a base encoding and an enhancement encoding. The video coding systems described herein operate upon a received decoding of a base encoding (e.g., frame-by- frame or complete base encoding) and add one or more of spatial, temporal, or other quality enhancements via an enhancement layer. The base encoding may be generated by a base layer, which may use a coding scheme that differs from the enhancement layer, and in certain cases may comprise a legacy or comparative (e.g., older) coding standard.

[0080] Figure 1 to 3 show a spatially scalable coding scheme that uses a down-sampled source signal encoded with a base codec, adds a first level of correction or enhancement data to the decoded output of the base codec to generate a corrected picture, and then adds a further level of correction or enhancement data to an up-sampled version of the corrected picture. Thus, the spatially scalable coding scheme may generate an enhancement stream with two spatial resolutions (higher and lower), which may be combined with a base stream at the lower spatial resolution.

[0081] In the spatially scalable coding scheme, the methods and apparatuses may be based on an overall algorithm which is built over an existing encoding and / or decoding algorithm (e.g., MPEG standards such as AVC / H.264, HEVC / H.265, etc. as well as non-standard algorithms such as VP9, AV1 , and others, furthermore, a hierarchical coding scheme may serve as a ‘base’ to a further enhancement layer) which works as a baseline for an enhancement layer. The enhancement layer works accordingly to a different encoding and / or decoding algorithm. The idea behind the overall algorithm is to encode / decode hierarchically the video frame as opposed to using block-based approaches as done in the MPEG family of algorithms. Hierarchically encoding a frame includes generating residuals for the full frame, and then a reduced or decimated frame and so on. Figure 1 shows a system configuration for an example spatially scalable encoding system 500. In this sense, figure 1 shows an encoding system that may generate the aforementioned encoded data. The encoding process is split into two halves as shown by the dashed line. Each half may be implemented separately. Below the dashed line is a base level and above the dashed line is the enhancement level, which may usefully be implemented in software. The encoding system 500 may comprise only the enhancement level processes, or a combination of the base level processes and enhancement level processes as needed. The encoding system 500 topology at a general level is as follows.

[0082] The encoding system 500 comprises an input “I” for receiving an input signal 501 . The input I is connected to a down-sampler 505D. The down-sampler 505D outputs to a base encoder 520E at the base level of the encoding system 500. The down-sampler 505D also outputs to a residual generator 510-S. An encoded base stream is created directly by the base encoder 520E, and may be quantised and entropy encoded as necessary according to the base encoding scheme. The encoded base stream may be the base layer as described above, e.g. a lowest layer in a multi-layer coding scheme.

[0083] Above the dashed line is a series of enhancement level processes to generate an enhancement layer of a multi-layer coding scheme. In the present example, the enhancement layer comprises two sub-layers. In other examples, one or more sub-layers may be provided.

[0084] In Figure 1 , to generate an encoded sub-layer 1 enhancement stream, the encoded base stream is decoded via a decoding operation that is applied at a base decoder 520D. In some examples, the base decoder 520D may be a decoding component that complements an encoding component in the form of the base encoder 520E within a base codec. In other examples, the base decoding block 520D may instead be part of the enhancement level.

[0085] Via the residual generator 510-S, a difference between the decoded base stream output from the base decoder 520D and the down-sampled input video is created (i.e., a subtraction operation 510-S is applied to a frame of the down-sampled input video and a frame of the decoded base stream to generate a first set of residuals). Here, residuals represent the error or differences between a reference signal or frame and a desired signal or frame. The residuals used in the first enhancement level can be considered as a correction signal as they are able to ‘correct’ a frame of a future decoded base stream. This is useful as this can correct for quirks or other peculiarities of the base codec. These include, amongst others, motion compensation algorithms applied by the base codec, quantisation and entropy encoding applied by the base codec, and block adjustments applied by the base codec.

[0086] In Figure 1 , the first set of residuals are transformed, quantised and entropy encoded to produce the encoded enhancement layer, sub-layer 1 stream. In Figure 1 , a transform operation 510-1 is applied to the first set of residuals; a quantisation operation 520-1 is applied to the transformed set of residuals to generate a set of quantised residuals; and, an entropy encoding operation 530-1 is applied to the quantised set of residuals to generate the encoded enhancement layer, sub-layer 1 stream (e.g., at a first level of enhancement).

[0087] The quantisation operation 520-1 may use the quantisation parameters generated by the methods and apparatus described above. However, it should be noted that in other examples only the quantisation step 520-1 may be performed. Entropy encoding may not be used, or may optionally be used in addition to one or both of the transform step 510-1 and quantisation step 520-1. The entropy encoding operation can be any suitable type of entropy encoding, such as a Huffmann encoding operation or a run-length encoding (RLE) operation, or a combination of both a Huffmann encoding operation and an RLE operation (e.g., RLE then Huffmann or prefix encoding).

[0088] To generate the encoded enhancement layer, sub-layer 2 stream, a further level of enhancement information is created by producing and encoding a further set of residuals via residual generator 500-S. The further set of residuals are the difference between an up-sampled version (via up-sampler 505U) of a corrected version of the decoded base stream (the reference signal or frame), and the input signal 501 (the desired signal or frame). To achieve a reconstruction of the corrected version of the decoded base stream as would be generated at a decoder (e.g., as shown in Figure 2), at least some of the sub-layer 1 encoding operations are reversed to mimic the processes of the decoder, and to account for at least some losses and quirks of the transform and quantisation processes. To this end, the first set of residuals are processed by a decoding pipeline comprising an inverse quantisation block 520-1 i and an inverse transform block 510-1 i.

[0089] The quantised first set of residuals are inversely quantised at inverse quantisation block 520-1 i and are inversely transformed at inverse transform block 510-1 i in the encoding system 500 to regenerate a decoder-side version of the first set of residuals. The decoded base stream from decoder 520D is then combined with the decoder-side version of the first set of residuals (i.e. , a summing operation 510-C is performed on the decoded base stream and the decoder-side version of the first set of residuals).

[0090] Summing operation 510-C generates a reconstruction of the down-sampled version of the input video as would be generated in all likelihood at the decoder - i.e. a reconstructed base codec video). The reconstructed base codec video is then up-sampled by up-sampler 505U.

[0091] Processing in this example is typically performed on a frame-by-frame basis. Each colour component of a frame may be processed as shown in parallel or in series.

[0092] The up-sampled signal (i.e., reference signal or frame) is then compared to the input signal 501 (i.e., desired signal or frame) to create the further set of residuals (i.e., a difference operation is applied by the residual generator 500-S to the up- sampled re-created frame to generate a further set of residuals). The further set of residuals are then processed via an encoding pipeline that mirrors that used for the first set of residuals to become an encoded enhancement layer, sub-layer 2 stream (i.e., an encoding operation is then applied to the further set of residuals to generate the encoded further enhancement stream). In various examples, the further set of residuals are transformed (i.e. , a transform operation 510-0 is performed on the further set of residuals to generate a further transformed set of residuals). The transformed residuals are then quantised, and entropy encoded in the manner described above in relation to the first set of residuals (i.e., a quantisation operation 520-0 is applied to the transformed set of residuals to generate a further set of quantised residuals; and, an entropy encoding operation 530-0 is applied to the quantised further set of residuals to generate the encoded enhancement layer, sub-layer 2 stream containing the further level of enhancement information).

[0093] The quantisation operation 520-0 may use the quantisation parameters generated by the methods and apparatus described above. In certain cases, the operations may be controlled, e.g. such that, only the quantisation step 520-0 may be performed.

[0094] Entropy encoding may optionally be used in addition. In some examples, the entropy encoding operation may be a Huffmann encoding operation or a runlength encoding (RLE) operation, or both (e.g., RLE then Huffmann encoding). The transformation applied at both blocks 510-1 and 510-0 may be a Hadamard transformation that is applied to 2x2 or 4x4 blocks of residuals.

[0095] The encoding operation in Figure 1 does not result in dependencies between local blocks of the input signal (e.g., in comparison with many known coding schemes that apply inter or intra prediction to macroblocks and thus introduce macroblock dependencies). As mentioned, examples of the described methods utilise this feature of the encoding scheme to generate a low latency implementation of a decoder to decode encoded data that has been encoded by an encoding scheme such as illustrated in Figure 1. Hence, the operations shown in Figure 1 may be performed in parallel on 4x4 or 2x2 blocks, which greatly increases encoding efficiency on multicore central processing units (CPUs) or graphical processing units (GPUs).

[0096] As illustrated in Figure 1 , the output of the spatially scalable encoding process is one or more enhancement streams for an enhancement layer which preferably comprises a first level of enhancement and a further level of enhancement. This is then combinable (e.g., via multiplexing or otherwise) with a base stream at a base level, e.g. into an MPEG2 transport stream or as multiple tracks within another digital container. The first level of enhancement (sub-layer 1) may be considered to enable a corrected video at a base level, that is, for example to correct for encoder quirks. The second level of enhancement (sub layer 2) may be considered to be a further level of enhancement that is usable to convert the corrected video to the original input video or a close approximation thereto. For example, the second level of enhancement may add fine detail that is lost during the downsampling and / or help correct from errors that are introduced by one or more of the transform operation 510-1 and the quantisation operation 520-1.

[0097] Figure 2 shows a corresponding example decoding system 600 for the example spatially scalable coding scheme. This thus illustrates an example coding scheme performed by the described decoding method.

[0098] In Figure 2, the encoded base stream is decoded at base decoder 620 in order to produce a base reconstruction of the input signal 501. This base reconstruction may be used in practice to provide a viewable rendition of the signal 501 at the lower quality level.

[0099] However, the primary purpose of this base reconstruction signal is to provide a base for a higher quality rendition of the input signal 501 . To this end, the decoded base stream is provided for enhancement layer, sub-layer 1 processing (i.e. , sublayer 1 decoding).

[0100] Sub-layer 1 processing in Figure 2 comprises an entropy decoding process 630- 1 , an inverse quantisation process 620-1 , and an inverse transform process 610- 1 . In various examples, one or more of the operations in the sub layer 1 processing are performed in parallel. Optionally, only one or more of these steps may be performed depending on the operations carried out at corresponding block 500-1 at the encoder. By performing these corresponding steps, a decoded enhancement layer, sublayer 1 stream, comprising the first set of residuals, is made available at the decoding system 600. The first set of residuals is combined with the decoded base stream from base decoder 620. This is achieved in some examples by a summing operation 610-C performed on a frame of the decoded base stream and a frame of the decoded first set of residuals to generate a reconstruction of the down- sampled version of the input video - i.e. the reconstructed base codec video. In various examples, the operations the summing operation 610-C may be performed in parallel.

[0101] A frame of the reconstructed base codec video is then up-sampled by up-sampler 605U. In some embodiments, for example ones involving computationally complex upsampling, the upsampling cannot be processed in parallel because the upsampling process may require a complete plane of data (i.e. a full sub layer 1 reconstruction). However, with simpler ‘localised’ upsampling processes, the upsampling process may be processed in parallel.

[0102] Additionally, and optionally in parallel, the encoded enhancement layer, sub-layer 2 stream is processed to produce a decoded further set of residuals. Similar to sub-layer 1 processing, enhancement layer, sub-layer 2 processing comprises an entropy decoding process 630-0, an inverse quantisation process 620-0 and an inverse transform process 610-0. Of course, these operations will correspond to those performed at block 500-0 in encoding system 500, and one or more of these steps may be omitted as necessary.

[0103] Block 600-0 produces a decoded enhancement layer, sub-layer 2 stream comprising the further set of residuals, and these are summed at operation 600- C with the output from the up-sampler 605U in order to create an enhancement layer, sub-layer 2 reconstruction of the input signal 501. This may be provided as the output of the decoding system 600.

[0104] In some examples, the operations of the sublayer 2 processing are performed in parallel. Thus, as illustrated in Figure 1 and 2, the output of the decoding process may comprise up to three outputs: a base reconstruction, a corrected lower resolution signal and an original signal reconstruction for the multi-layer coding scheme at a higher resolution.

[0105] In general, examples described herein operate within encoding and decoding pipelines that comprises at least a transform operation. The transform operation may comprise a Directional Decomposition (DD) transform or a Directional Decomposition Squared (DDS) transform. More details on these transforms are included in patent applications WO 2013 / 171173 A1 and WO 2018 / 046941 A1 , which are incorporated by reference. Further, the transform operation may comprise the Discrete Cosine T ransform (DOT) or a variation of the DOT, a Fast Fourier Transform (FFT), or, in various examples, a Hadamard transform as implemented by LCEVC.

[0106] The transform operation may be applied on a block-by-block basis. For example, an input signal may be segmented into a number of different consecutive signal portions or blocks and the transform operation may comprise a matrix multiplication (i.e., linear transformation) that is applied to data from each of these blocks (e.g., as represented by a 1 D vector).

[0107] In this description and in the art, a transform operation may be said to result in a set ofvalues for a predefined number of data elements, e.g. representing positions in a resultant vector following the transformation. These data elements are known as transformed coefficients (or sometimes simply “coefficients”).

[0108] As described herein, where the enhancement data comprises residual data, a reconstructed set of coefficient bits may comprise transformed residual data, and a decoding method may further comprise instructing a combination of residual data obtained from the further decoding of the reconstructed set of coefficient bits with a reconstruction of the input signal generated from a representation of the input signal at a lower level of quality to generate a reconstruction of the input signal at a first level of quality. The representation of the input signal at a lower level of quality may be a decoded base signal and the decoded base signal may be optionally upscaled before being combined with residual data obtained from the further decoding of the reconstructed set of coefficient bits, the residual data being at a first level of quality (e.g., a first resolution).

[0109] Decoding may further comprise receiving and decoding residual data associated with a second sub-layer, e.g. obtaining an output of the inverse transformation and inverse quantisation component, and combining it with data derived from the aforementioned reconstruction of the input signal at the first level of quality. This data may comprise data derived from an upscaled version of the reconstruction of the input signal at the first level of quality, i.e. an upscaling to the second level of quality.

[0110] Further details and examples of a two sub-layer enhancement encoding and decoding system may be obtained from published LCEVC documentation. Although examples have been described with reference to a tier-based hierarchical coding scheme in the form of LCEVC, the methods described herein may also be applied to other tier-based hierarchical coding scheme, such as VC- 6: SMPTE VC-6 ST-2117 as described in PCT / GB2018 / 053552 (published as WO 2019 / 111010 A1) and / or the associated published standard document, all of which are incorporated by reference.

[0111] Figure 3 shows an example 700 of how a video signal may be decomposed into different components and then encoded. In the example of Figure 3, a video signal 702 is encoded. The video signal (v) 702 comprises a plurality of frames (F) or pictures 704, e.g. where the plurality of frames represent action over time. In this example, each frame 704 is made up of three colour components. The colour components may be in any known colour space. In Figure 3, the three colour components 706 are Y (luma), U (a first chroma opponent colour) and V (a second chroma opponent colour). Each colour component may be considered a plane 708 of values. The plane 708 may be decomposed into a set of n by n blocks of signal data 710. For example, in LCEVC, n may be 2 or 4; in other video coding technologies n may be 8 to 32.

[0112] In LCEVC and certain other coding technologies, a video signal fed into a base layer is a downscaled version of the input video signal, e.g. 501. In this case, the signal that is fed into both sub-layers of the enhancement layer comprises a residual signal comprising residual data. A plane of residual data may also be organised in sets of n-by-n blocks of signal data 710.

[0113] The residual data may be generated by comparing data derived from the input signal being encoded, e.g. the video signal 501 , and data derived from a reconstruction of the input signal, the reconstruction of the input signal being generated from a representation of the input signal at a lower level of quality. The comparison may comprise subtracting the reconstruction from the downsampled version. The comparison may be performed on a frame-by-frame (and / or block- by-block) basis. The comparison may be performed at the first level of quality; if the base level of quality is below the first level of quality, a reconstruction from the base level of quality may be upscaled prior to the comparison.

[0114] In a similar manner, the input signal to the second sub-layer, e.g. the input for the second sub-layer transformation and quantisation component, may comprise residual data that results from a comparison of the input video signal 501 at the second level of quality (which may comprise a full-quality original version of the video signal) with a reconstruction of the video signal at the second level of quality. As before, the comparison may be performed on a frame-by-frame (and / or block- by-block) basis and may comprise subtraction. The reconstruction of the video signal may comprise a reconstruction generated from the decoded decoding of the encoded base bitstream and a decoded version of the first sub-layer residual data stream. The reconstruction may be generated at the first level of quality and may be upsampled to the second level of quality.

[0115] Hence, a plane of data 708 for the first sub-layer may comprise residual data that is arranged in n-by-n signal blocks 710. One such 2 by 2 signal block is shown in more detail in Figure 3 (n is selected as 2 for ease of explanation) where for a colour plane the block may have values 712 with a set bit length (e.g., 8 or 16-bit).

[0116] Each n-by-n signal block may be represented as a flattened vector 714 of length n2 representing the blocks of signal data. To perform the transform operation, the flattened vector 714 may be multiplied by a transform matrix 716 (i.e., the dot product taken). This then generates another vector 718 of length n2 representing different transformed coefficients for a given signal block 710.

[0117] Figure 3 shows an example similar to LCEVC where the transform matrix 716 is a Hadamard matrix of size 4 by 4, resulting in a transformed coefficient vector 718 having four elements with respective values. These elements are sometimes referred to by the letters A, H, V and D as they may represent an average (A), horizontal difference (H), vertical difference (V) and diagonal difference (D). Such a transform operation may also be referred to as a directional decomposition. When n = 4, the transform operation may use a 16 by 16 matrix and be referred to as a directional decomposition squared.

[0118] As shown in Figure 3, the set of values for each data element across the complete set of signal blocks 710 for the plane 708 may themselves be represented as a plane or surface of coefficient values 720. For example, values for the “H” data elements for the set of signal blocks may be combined into a single plane, where the original plane 708 is then represented as four separate coefficient planes 722. For example, the illustrated coefficient plane 722 contains all the “H” values.

[0119] These values are stored with a predefined bit length, e.g. a bit length B, which may be 8, 16, 32 or 64 depending on the bit depth. A 16-bit example is considered below but this is not limiting. As such, the coefficient plane 722 may be represented as a sequence (e.g., in memory) of 16-bit or 2-byte values 724 representing the values of one data element from the transformed coefficients. These may be referred to as coefficient bits. These coefficient bits may be quantised and then entropy encoded as discussed to then generate the encoded enhancement or second layer data as described above.

[0120] Figures 2 and 3 thus show decoding processes. There are many ways to implement such processes on a computer. Examples provide for an advantageous method of implementing a decoding process such as the decoding process illustrated in figures 2 and 3. Figures 2 and 3 are thus useful schematics for aiding understanding of described examples. Considering specifics of aspects disclosed herein, in general terms, when using the LTM, the processing involves feeding all of the encoded data for a whole frame into entropy decoding process 630-0 processing it and then feeding the entirety of the output into inverse quantisation process 620-0. This causes all the data to be unpacked into a large array.

[0121] In contrast, some examples utilise parallel processing and storing of intermediate representations to speed up the decoding of data. As described, this provides numerous unexpected advantages (decoding speed as well as decrease in overall processes / calculations / workload if the intermediate representations are configured correctly) and moreover it is different to how other decoders generally operate (due to the historical trend of relying on intra coding methods, which teach away from implementing a decoding method as described).

[0122] Consistent with Figure 3, Figures 4, 5 and 6 detail a process able to be applied to a signal, such as, in some examples, a video signal.

[0123] As set out above, Figure 3 shows plane 708 of values of one colour component. Such a plane 708 is also shown in Figure 4. By the processes set out above, in some examples including a transform, this is turned into a plane of coefficients, each representing A, H, V and D elements of the original block as a value.

[0124] In the example shown in Figure 4, the plane 708 is processed in blocks. In various examples, such as the example shown in Figure 4, the blocks 402 are a group of 2 by 2 pixels of the image plane. As set out above, the blocks may be a different size, such as a group of 4 by 4 pixels. Due to this, the processing to turn the plane into A, H, V and D elements results in the original elements of each block being replaced with transform blocks 404. Consistent with the blocks being 2 by 2 pixels, the transform blocks are a group of 2 by 2 elements.

[0125] The transform blocks 404 are then encoded as set out above. In various examples this includes Huffman encoding and / or run length encoding. This results in each of the coefficient values of element type being encoded into a bitstream 406. Consistent with the above, the coefficient values of each element type are represented as a sequence of 16-bit or 2-byte values 724.

[0126] In Figure 4, the bitstream 406 is shown as being made up of sequences 724, in which there are encoded values 408. Overall, depending on whether the sequences are generated from sub-layer 1 or sub-layer 2, there are 16 or 17 sequences respectively, with additional surface for sub-layer 2 being due to the inclusion of temporal data. Sequence 16 (corresponding to the 17th sequence due to sequence 0 being present) is therefore optional.

[0127] In some examples, each sequence 724 includes a pointer 410 or indicator identifying the location of the lowest value non-null value. In various examples, this identifies where processing of each sequence is able to start without being detrimental to the final output.

[0128] In a number of examples, each sequence 724 includes signal data and, optionally, one or more signal bits, a zero run length low value (i.e. the minimum zero run length in the respective sequence) a zero run length high value (i.e. the maximum zero run length in the respective sequence).

[0129] This is intended to correspond to a standard process to pass image data into a bitstream in an LCEVC compliant manner.

[0130] Turning to Figure 5, this shows the same bitstream 406 as in Figure 4. To decode the bitstream various steps are performed. Typically these are performed by a decoder 600 or an apparatus configured to decode an encoded signal.

[0131] In some examples, as part of the decoding, the sequences 724 are separated into individual surfaces 502. Typically, these surfaces have differing lengths based on the data content of each surface. The lengths of the surfaces are typically longer the earlier in the bitstream sequence 724 on which the surface is based is located. This is due to, in some examples, Huffman encoding having been applied to the sequence data and the results from Huffman encoding. In some examples, as with sequence 16, surface 16 is optionally included, typically being included when the surfaces related to the sub-layer 2 encoding output.

[0132] As with the sequences 724, each surface 502 has an indicator 504 identifying a non-null value (i.e. a value that is not a null value). In some examples, this indicator is provided at the point in the surface that has the lowest non-null value.

[0133] In various examples, the surfaces are each decoded using decoding mechanisms corresponding to the encoding mechanisms used to encode the data in each surface. In some examples, this included run length decoding and / or Huffman decoding.

[0134] As shown in Figure 6, in some examples, this decoding includes generating an array of coefficients 602 from, in certain examples, each surface 502, and in other examples an array of coefficients for all the surfaces, Figure 6 showing a portion of one array of coefficients. The coefficients are grouped into 4 by 4 sections 604 of coefficient values with the array forming a grid of these sections.

[0135] In an example according to an aspect, this decoding is implemented for each of the surfaces across multiple threads of a processor, with a one-to-one correspondence of surfaces and threads. This causes parallel decoding of the surfaces.

[0136] Typically, each thread is processed a core of a CPU, with a one-to-one correspondence between threads and cores. In other examples, a Field Programmable Gate Array (FPGA) or Graphical Processing Unit (GPU) is used for processing the threads.

[0137] The decoding to extract the surface data into coefficients is conducted in line with the known process for each surface. In some examples, this is then processed into an output readable to create a decoded signal of the entropy encoded signal. In various examples, this is in the form of a residual representation 606. In some examples, the residual representation 606 is arrived at by dequantization and / or an inverse transform being carried out on the array of coefficients 602. The residual values in the residual representation are each paired in a number of examples with a pixel coordinate to form a geometric residual.

[0138] As an alternative, the array of coefficients may be further processed to be output to a buffer, such as a command buffer or list of operations.

[0139] In various examples, the residual representations 606 is then able to be output as a compliant decoded bitstream and combined with a decoded base to recreate an original image either at a lower resolution or upscaled to the original resolution depending on the sub-layer data used in the combining. This is achieved by applying the processes set out above in relation to Figure 2.

[0140] Overall, this provides an example of the process shown in Figure 7. This includes the steps of separating (S101 ) at least a part of a received entropy encoded signal into a plurality of surfaces, each surface comprising entropy encoded data; generating (S102) an array of coefficients from each surface by entropy decoding, in parallel, each surface, wherein each coefficient in each array is an entropy decoded coefficient of the entropy encoded data of the respective surface; and processing (S103) the coefficients of each array into an output readable to create a decoded signal of the entropy encoded signal thereby replicating the signal encoded by entropy encoding. In various examples, this is based on having received (S100) the entropy encoded signal prior to the separation step. Broadly, this applies the LTM decoding process. However, rather than unpacking the bitstream (e.g. the entropy encoded signal) into a single large array, this unpacks the bitstream into separate surfaces that are processed, separately, in parallel.

[0141] Certain methods and encoder components as described herein may be performed by instructions that are stored upon a non-transitory computer readable medium. The non-transitory computer readable medium stores code comprising instructions that, if executed by one or more computers, would cause the computer to perform steps of methods or execute operations of encoder components as described herein. The non-transitory computer readable medium may comprise one or more of a rotating magnetic disk, a rotating optical disk, a flash random access memory (RAM) chip, and other mechanically moving or solid-state storage media. Some examples may be implemented as: physical devices such as semiconductor chips; hardware description language representations of the logical or functional behaviour of such devices; and one or more non-transitory computer readable media arranged to store such hardware description language representations. Descriptions herein reciting principles, aspects, and embodiments encompass both structural and functional equivalents thereof.

[0142] In some examples, a system is provided as generally illustrated at 1000 in Figure 8. This includes a non-transitory computer-readable medium 1010, such as a hard-drive, solid-state drive or some other form of storage medium, is provided. This is capable of holding or storing a computer program 1001. The computer program includes instructions, typically in the form of computer code, that, when implemented on a computing device, such as by executing the program, cause an apparatus to perform one or more of the methods and processes set out above or to provide one or more of the decoders set out above. The apparatus may be a computing device such as a chip, server or some other form of computer. The system also includes a multi-core CPU 1002 on which the decoder is able to be implemented in some examples.

[0143] The above examples are to be understood as illustrative examples. Further examples are envisaged. It is to be understood that any feature described in relation to any one example may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the examples, or any combination of any other of the examples. Furthermore, equivalents and modifications not described above may also be employed within the scope of the accompanying claims.

[0144] Patent and non-patent documents that are referenced herein are deemed to be incorporated by reference into the present document.

[0145] Certain examples have been described herein and it will be noted that different combinations of different components from different examples may be possible. Salient features are presented to better explain examples; however, it is clear that certain features may be added, modified and / or omitted without modifying the functional aspects of these examples as described. Elements described herein as “coupled” or “communicatively coupled” have an effectual relationship realizable by a direct connection or indirect connection, which uses one or more other intervening elements. Examples described herein as “communicating” or “in communication with” another device, module, or elements include any form of communication or link. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims.

Claims

CLAIMS1 . A method of decoding an entropy encoded signal, the method comprising: separating at least a part of a received entropy encoded signal into a plurality of surfaces, each surface comprising entropy encoded data; generating an array of coefficients from each surface by entropy decoding, in parallel, each surface, wherein each coefficient in each array is an entropy decoded coefficient of the entropy encoded data of the respective surface; and processing the coefficients of each array into an output readable to create a decoded signal of the entropy encoded signal thereby replicating the signal encoded by entropy encoding.

2. The method according to claim 1 , wherein the entropy encoding includes run length encoding and the entropy decoding includes run length decoding.

3. The method according to claim 2, wherein the entropy encoded data of each surface is represented by a plurality of elements, each element having a value, and each entropy decoding of each surface includes decoding a respective surface from the element with the lowest non-null value.

4. The method according to claim 3, wherein the entropy decoding of each surface includes reading the respective surface to identify the element with the lowest non-null value and leaving preceding elements encoded.

5. The method according to any one of claims 2 to 4, wherein the entropy encoded data includes a plurality of run length indications.

6. The method according to claim 5, wherein the entropy data includes a plurality of encoded coefficients, each of the encoded coefficients being paired with a run length indication of the plurality of run length indications.

7. The method according to any one of the preceding claims, wherein the entropy encoding includes Huffman coding and the entropy decoding includes Huffman decoding.

8. The method according to any one of the preceding claims, wherein entropy decoding of each surface is performed on a thread independent of concurrent entropy decoding each respective other surface.

9. The method according to claim 8, wherein, on completion of entropy decoding a respective surface, each thread is reinitialised with a further surface on which entropy decoding is performed.

10. The method according to any one of the preceding claims, wherein the processing includes dequantization of each array of coefficients.

11. The method according to any one of the preceding claims, wherein the processing includes performing an inverse transform on each array of coefficients.

12. The method according to any one of the preceding claims, wherein the output of the processing the coefficients of each array comprises enhancement data.

13. The method according to claim 12, further comprising enhancing a base dataset using the enhancement data.

14. The method according to any one of the preceding claims, wherein the processing includes writing the array of coefficients to a residual representation, the residual representation including residual values.

15. The method according to 14, wherein the residual representation includes a coordinate associated with each residual value.

16. The method according to any one of claims 1 to 12, wherein the processing includes writing the array of coefficients to a buffer or operation list, or residual representation.

17. The method according to any one of the preceding claims, wherein the entropy encoded signal is an entropy encoded video signal.

18. The method according to claim 17, wherein the entropy encoded signal is an LCEVC compliant signal.

19. The method according to claim 17 or claim 18, wherein the data of each surface corresponds to outputs of a transform.

20. The method according to any one of the preceding claims, wherein entropy decoding is performed on a multi-core Central Processing Unit (CPU).21 . The method according to claim 20 and one of claim 8 or claim 9, wherein each core of the CPU is configured to process one thread at a time.

22. The method according to any one of the preceding claims, further comprising receiving the entropy encoded signal.

23. An apparatus configured to decode an entropy encoded signal, the apparatus being configured to: separate at least a part of a received entropy encoded signal into a plurality of surfaces, each surface comprising entropy encoded data; generate an array of coefficients from each surface by entropy decoding, in parallel, each surface, wherein each coefficient in each array is an entropy decoded coefficient of the entropy encoded data of the respective surface; and process the coefficients of each array into an output readable to create a decoded signal of the entropy encoded signal thereby replicating the signal encoded by entropy encoding.

24. The apparatus according to claim 23, wherein the entropy encoding includes run length encoding and the entropy decoding includes run length decoding.

25. The apparatus according to claim 24, wherein the entropy encoded data of each surface is represented by a plurality of elements, each element having a value, and apparatus being arranged in use, as at least a part of the entropy decoding to include decoding a respective surface from the element with the lowers non-null value in the entropy decoding.

26. The apparatus according to claim 25, wherein the apparatus being arranged in use, as at least a part of the entropy decoding, to read the respectivesurface to identify the element with the lowest non-null value and leaving preceding elements encoded.

27. The apparatus according to any one of claims 24 to 26, wherein the entropy encoded data includes a plurality of run length indications.

28. The apparatus according to claim 27, wherein the entropy data includes a plurality of encoded coefficients, each of the encoded coefficients being paired with a run length indication of the plurality of run length indications.

29. The apparatus according to any one of claims 23 to 28, comprising a multi-core CPU arranged in use to entropy decode each surface, each surface being entropy decoded on a thread independent of concurrent entropy decoding each respective other surface, each core processing only a single thread at a time.

30. The apparatus according to any one of claims 23 to 29, wherein the apparatus is a decoder.31 . A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 22.

32. A non-transitory computer-readable medium comprising the computer program according to 31 .

Images