Up-sampling filter for applying predicted mean modification

Abstract

The present disclosure describes an up-sampling filter for video coding. The upsampling filter is configured to upsample an input video frame from a first resolution to a second resolution, the second resolution being higher than the first resolution. The upsampling filter has a set of filter coefficients configured to modify an output of the upsampling filter to apply a predicted average modifier. In an example, the predicted average modifier is defined as a difference between an average of a data block of pixels of the second resolution after application of the up-sampling filter and a corresponding pixel value of the first resolution before application of the up-sampling filter.

Description

Technical Field

[0001] The present invention relates to upsampling for video coding. In particular, examples relate to modifying video frames to apply a predicted mean modified upsampling filter that improves the efficiency of encoding of residual data derived from the video frames. Background Art

[0002] EP 2850829 B1 describes a method of transforming element information (such as residual data) to allow efficient video coding. In particular, EP 2850829 B1 describes how directional decomposition can be applied to small residual data blocks and how the mean value of a given data block can be adjusted using a so-called "predicted mean value" to derive a transformed residual data set for further entropy coding and transmission or storage as a coded bitstream. In the example described therein, the predicted mean value is determined as the difference between the pixel values ​​of a first lower resolution and the mean value of a second higher resolution upsampled pixel set, where a set of upsampled pixels corresponds to the upsampling of the pixel values. The use of the predicted mean value allows the energy of the mean value component within the transformed residual data to be reduced, resulting in a smaller bitstream and more efficient coding.

[0003] In the example of EP 2850829 B1, during the decoding of the video frame, the estimated value of the predicted average value is calculated using the signal information available to the decoder. Therefore, the predicted average value does not need to be explicitly transmitted in the bitstream. In particular, the pixel value of the first lower resolution can be derived from the first encoded bitstream (e.g., the first coding layer), and the average value of the upsampled pixels of the second higher resolution can be derived from the upsampling performed at the decoder. In the example of EP 2850829B1, the predicted average value is added to the received increment or adjusted average value component of the decoded data block of the transformed residual to restore the original average value for the data block. The data block can then be reconstructed (e.g., via the application of the inverse decomposition) to obtain the residual value for the data block. These residual values ​​can then be added to the upsampling of the decoding of the first encoded bitstream to output the decoding of the second higher resolution video signal.

[0004] WO2020 / 188242A1 describes a form of modified upsampling in which a predicted mean modifier can be calculated and added to the output of the upsampled signal during decoding of a video signal. In this case, instead of calculating the predicted mean as part of the decoding process for the encoded bitstream layer and applying the predicted mean to restore the mean component before the reverse decomposition, the predicted mean is calculated using the input and output of the upsampling stage. Due to the main linear sequence of decoding operations, the predicted mean can be applied after upsampling, rather than as part of the decoding process (e.g., thereby moving operations within the sequence of decoding operations without adverse effects). The method of WO2020 / 188242A1 allows more efficient decoding because it avoids the need to apply reverse decomposition to the predicted mean component, thereby saving computing resources and increasing the speed of decoding (e.g., by reducing the number of operations). For example, maintaining the transformed adjusted mean component on the data block to zero during reverse decomposition can reduce the number of bit operations (even if not all adjusted components are zero).

[0005] Although the application of the predicted mean modification of EP 2850829 B1 improves the coding efficiency for the resulting bitstream and the application of the modifications in WO2020 / 188242A1 improves the decoding efficiency, the calculation and application of the predicted mean can increase the complexity of the encoding and decoding process. This is especially a problem for older so-called "traditional" hardware devices such as set-top boxes or built-in decoders. For example, due to hardware constraints at one or more of the encoders and decoders, it may be difficult to support the use of predicted mean modification. In these cases, the video distributor may choose to turn off the predicted mean function and weigh a less compressed bitstream in exchange for a wider range of traditional device support.

[0006] In the field of video coding, more efficient video coding is always desired, such as video coding that reduces the bit rate of a bitstream for a given decoded video quality and / or reduces computation or power consumption. Video coding is typically a resource-intensive operation involving hardware accelerators for common video coding standards. This also brings up the problem of improving video coding efficiency while maintaining support for older hardware devices. Summary of the invention

[0007] Various aspects and variations of the invention are set out in the following claims.

[0008] Certain non-claimed aspects are further described in the detailed description which follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A high level schematic diagram of the hierarchical encoding and decoding process is shown.

[0010] Figure 2 A high-level schematic diagram of the hierarchical deconstruction process is shown.

[0011] Figure 3 An alternative high-level schematic diagram of the hierarchical deconstruction process is shown.

[0012] Figure 4 A high-level diagram of an encoding process suitable for encoding the residual of a hierarchical output is shown.

[0013] Figure 5 Shows the application of Figure 4 A high-level diagram of the hierarchical decoding process for decoding each output level of .

[0014] Figure 6 A high-level schematic diagram of the encoding process of the hierarchical encoding technique is shown.

[0015] Figure 7 Shows the application of Figure 6 A high-level diagram of the decoding process for the output of .

[0016] Figure 8 An example method of upsampling is shown.

[0017] 9A to 9C An example separable five-tap filter for upsampling and example constraints for designing a predicted mean-preserving upsampling filter are shown.

[0018] Fig.10 An example of a set of non-predicted mean preserving upsampling filters converted into predicted mean preserving upsampling filters according to the methods described herein is shown.

[0019] Fig.11 An example training configuration for training a predicted mean-preserving upsampling filter is shown. DETAILED DESCRIPTION

[0020] Certain examples described herein relate to adaptive upsampling operations that can be used, for example, in video coding. In particular, certain examples described herein apply predicted mean calculations, such as the predicted mean calculations described in EP 2850829 B1 or WO2020 / 188242A1, as part of the upsampling operation. This is achieved by configuring a set of upsampling coefficients for the upsampling filter. For example, a set of upsampling coefficients for the upsampling filter can be optimized so that the upsampling operation provides upsampling of pixel data, which reduces the bit rate of the residual data stream and is equivalent to the output of the modified upsampling output described in WO2020 / 188242A1. This can be achieved by configuring the upsampling coefficients so that the average value of the pixel values ​​of the upsampled data block (e.g., a 2x2 or 4x4 data block for a particular luminance or chrominance plane) is equal to or approximately equal to (e.g., within a quantization tolerance range) the pixel values ​​that are upsampled to generate the upsampled data block. Then, this effectively sets the predicted mean modifier to zero, so that within the transformed residual data block, the mean value component is the same as the adjusted mean value component.

[0021] The adaptive upsampling operation described in the examples herein can be used at one or more of an encoder and a decoder. In one case, the adaptive upsampler of the example can be applied to both the encoder and the decoder, for example, to generate new encoded bitstreams and decode these bitstreams, respectively.

[0022] In some specific examples, the adaptive upsampling operation is implemented as a separable filter with less than five coefficients for each of the two image dimensions, for example, a four-tap separable filter. In the example, the general form of the upsampling coefficients providing the adaptive upsampling operation is described. Therefore, different existing upsampling filters can be adapted to provide predicted mean calculations. This means that traditional hardware devices (such as set-top boxes) that are limited to hardware-implemented filters with four coefficients can be applied in a computationally efficient manner to calculate predicted mean values. In the case where more than four coefficients are available, a one-dimensional filter with five or more coefficients can be used. This may be preferred in the case of fewer hardware restrictions for a more expressive upsampling filter that further reduces the bit rate of residual data.

[0023] Certain examples described herein may be implemented as part of an MPEG 5 Part 2 Low Complexity Enhancement Video Coding (LCEVC) implementation and / or a SMPTE VC-6 2117 implementation.

[0024] In the following, some features of example signal encoders and decoders are first described. These example signal encoders and decoders may use the adaptive upsampling operation as described herein. After this general description, some specific aspects of the adaptive upsampling operation will be described in detail. The adaptive upsampling operation may be more easily understood by first understanding an example of a layered coding scheme or format based on layers using upsampling, although the methods described in the examples below are not necessarily limited to such schemes.

[0025] General signal processing terms and definitions

[0026] The examples described herein relate to signal processing. A signal may be viewed as a sequence of samples (i.e., a two-dimensional image, a video frame, a video field, a sound frame, etc.). In the specification, the term "image", "picture", or "plane" (meaning the broadest meaning of "hyperplane", i.e., an array of elements with arbitrary dimensions and a given sampling grid) will generally be used to identify a digital reproduction of a signal sample along a sequence of samples, wherein each plane has a given resolution for each of its dimensions (e.g., X and Y) and includes a set of plane elements (or "elements" or "pixels" or display elements for two-dimensional images, commonly referred to as "pixels", display elements for volumetric images, etc.), which are characterized by one or more "values" or "settings" (e.g., by way of non-limiting example, color settings in a suitable color space, settings indicating density levels, settings indicating temperature levels, settings indicating audio pitch, settings indicating amplitude, settings indicating depth, settings indicating alpha channel transparency levels, etc.). Each plane element is identified by a set of suitable coordinates, indicating the integer position of the element in the sampling grid of the image. The signal dimensions may contain only spatial dimensions (eg, in the case of images) or only temporal dimensions (eg, in the case of a signal evolving over time, such as a video signal).

[0027] For example, the signal may be an image, an audio signal, a multi-channel audio signal, a telemetry signal, a video signal, a 3DoF / 6DoF video signal, a volumetric signal (e.g., medical imaging, scientific imaging, holographic imaging, etc.), a volumetric video signal, or even a signal in more than four dimensions.

[0028] For simplicity, the examples described herein generally refer to signals displayed as a 2D set plane (e.g., a 2D image in a suitable color space), such as a video signal. The terms "frame" or "field" will be used interchangeably with the term "image" to indicate a temporal sample of a video signal: any concepts and methods described for a video signal consisting of frames (progressive video signals) can also be easily applied to a video signal consisting of fields (interlaced video signals), and vice versa. Although the embodiments described herein focus on images and video signals, those skilled in the art can easily understand that the same concepts and methods are also applicable to any other type of multi-dimensional signal (e.g., audio signals, volumetric signals, stereoscopic video signals, 3DoF / 6DoF video signals, plenoptic signals, point clouds, etc.).

[0029] Certain layer-based hierarchical formats described herein use varying amounts of correction (e.g., also in the form of "residual data", or simply "residual") to generate a signal reconstruction that is most similar to (or even lossless reconstruction of) the original signal at a given quality level. The amount of correction can be based on the fidelity of the predicted reproduction at a given quality level.

[0030] To achieve high fidelity reconstruction, the encoding method may upsample a lower resolution reconstruction of the signal to the next higher resolution reconstruction of the signal. In some cases, different signals may be best processed by different methods, ie, the same method may not be optimal for all signals.

[0031] Examples of layer-based hierarchical coding schemes or formats

[0032] In a preferred example, the encoder or decoder is part of a layer-based hierarchical coding scheme or format. Examples of layer-based hierarchical coding schemes include LCEVC: MPEG-5 Part 2 LCEVC and VC-6: SMPTE VC-6ST-2117, the former described in PCT / GB2020 / 050695 (and associated standard documents), and the latter described in PCT / GB2018 / 053552 (and associated standard documents), all of which are incorporated herein by reference. However, the concepts shown herein are not necessarily limited to these specific hierarchical coding schemes.

[0033] Figures 1 to 7 An overview of different exemplary layer-based hierarchical coding formats is provided. These are provided as a context for the adaptive upsampling operations that are performed in Figure 7 This is explained in the following figure. Figures 1 to 5 provides an example similar to the implementation of SMPTE VC-6ST-2117, while Figure 6 and Figure 7Examples similar to the implementation of MPEG-5 Part 2 LCEVC are provided. It can be seen that both sets of examples utilize common underlying operations (e.g., downsampling, upsampling, and residual generation) and can share modular implementation techniques.

[0034] General Hierarchical Coding Scheme

[0035] Figure 1 A hierarchical encoding scheme is shown very generally. Data to be encoded 101 is retrieved by a hierarchical encoder 102 which outputs encoded data 103. The encoded data 103 is then received by a hierarchical decoder 104 which decodes the data and outputs decoded data 105.

[0036] Typically, the hierarchical coding scheme used in the examples herein creates a base or core layer, which is a representation of the original data at a lower quality layer, and one or more residual layers, which can be used to reconstruct the original data at a higher quality layer using a decoded version of the base layer data. In general, the term "residual" as used herein refers to the difference between the values ​​of a reference array or reference frame and the actual array or frame of data. The array can be a one-dimensional or two-dimensional array representing a coding unit. For example, a coding unit can be a 2x2 or 4x4 group of residual values ​​corresponding to a similarly sized area of ​​an input video frame.

[0037] It should be noted that the generalized example is agnostic to the nature of the input signal. Reference to "residual data" as used herein refers to data derived from a set of residuals, such as a set of residuals themselves or the output of a set of data processing operations performed on a set of residuals. Throughout this specification, in general, a set of residuals comprises a plurality of residuals or residual elements, each of which corresponds to a signal element, i.e., an element of the signal or original data.

[0038] In specific examples, the data may be an image or a video. In these examples, the set of residuals corresponds to an image or a frame of a video, wherein each residual is associated with a pixel of a signal, the pixel being a signal element.

[0039] The methods described herein can be applied to so-called data planes reflecting different color components of a video signal. For example, these methods can be applied to different planes of YUV or RGB data reflecting different color channels. Different color channels can be processed in parallel. The components of each stream can be arranged in any logical order.

[0040] First Example Hierarchical Encoder and Components

[0041] A hierarchical coding scheme will now be described in which the concepts of the present invention can be deployed. Figures 2 to 5and generally corresponds to the above-mentioned VC-6. In this type of coding technique, residual data is used to gradually increase the quality level. In the proposed technique, the core layer represents the image at a first resolution, and the subsequent layers in the hierarchical structure are the residual data or adjustment layers required for the decoding side to reconstruct the image at a higher resolution. Each layer or level can be referred to as a step index, so that the residual data is the data required to correct the low-quality information present in the lower step indices. Each layer or step index in this hierarchical technique, especially each residual layer, is typically a relatively sparse data group with many zero-valued elements. When a step index is mentioned, it refers to all steps or component groups of that level, for example, all subgroups produced in the transformation step performed at that quality level.

[0042] In this particular hierarchical manner, the described data structure eliminates any requirement or dependency on previous or subsequent quality layers. The quality layers can be encoded and decoded individually without reference to any other layer. Thus, in contrast to many other known hierarchical coding schemes, where the lowest quality layer needs to be decoded in order to decode any higher quality layer, the described method does not require decoding of any other layer. However, the principles of exchanging information described below may also be applicable to other hierarchical coding schemes.

[0043] like Figure 2 As shown in , the encoded data represents a set of layers or levels, generally referred to herein as step indices. The base or core level represents the original data frame 210, albeit at the lowest quality level or resolution, and subsequent residual data levels can be combined with data at the core step indices to reconstruct the original image at progressively higher resolutions.

[0044] To create the core step index, the input data frame 210 may be downsampled using a number of downsampling operations 201 corresponding to the number of levels or step indices to be used in the hierarchical encoding operation. The required downsampling operations 201 are one less than the number of levels in the hierarchy. In all the examples described herein, there are 4 levels or step indices of output coded data and corresponding 3 downsampling operations, but it will of course be appreciated that these are for illustration only. Where n indicates the number of levels, the number of downsamplers is n-1. The core level R 1-n is the output of the third downsampling operation. As mentioned above, the core level R 1-n A representation of the input data frame corresponding to the lowest quality level.

[0045] To distinguish the downsampling operations 201, each operation will be referenced by the order in which the operation is performed on the input data 210 or by the data represented by its output. 1-n It is also called a core downsampler because its output generates a core step index or step 1-n, that is, the indices of all rungs at this level are 1-n. Therefore, in this example, the first downsampling operation 201 -1 Corresponding to R -1 Downsampler, second downsampling operation 201 -2 Corresponding to R -2 Downsampler, and the third downsampling operation 201 1-n Corresponds to the core or R -3 Downsampler.

[0046] like Figure 2 As shown in the figure, it indicates the core quality level R 1-n The data undergoes an upsampling operation 202 1-n , here referred to as the core upsampler. The second downsampling operation 201 -2 Output (R -2 The output of the downsampler, ie the input of the core downsampler) and the core upsampler 202 1-n The difference between the output of 203 -2 is output as the first residual data R -2 This first residual data R -2 Correspondingly, the core level R -3 and the error between the signal used to create the level. Since the signal itself has undergone two downsampling operations in this example, the first residual data R -2 is an adjustment layer that can be used to reconstruct the original signal at a higher quality level than the core quality level, but a lower quality level than the input data frame 210 .

[0047] Figure 2 and Figure 3 It is conceptually shown how to create variations of residual data representing higher quality levels.

[0048] exist Figure 2 In the second downsampling operation 201 -2 (or R -2 Downsampler, which is used to create the first residual data R -2 The output of the signal is upsampled 202 -2 , and the second downsampling operation 201 -2 (or R -2 Downsampler, i.e. R -1 The difference between the output of the downsampler) and the input 203 -1 Create the first residual data R with -2 The calculation is performed in almost the same way. This difference is correspondingly the second residual data R -1 and represents an adjustment layer that can be used to reconstruct the original signal at a higher quality level using data from lower layers.

[0049] However, in Figure 3 In a variant of -2 (or R -2 The output of the downsampler) is related to the first residual data R -2 Combine or add 304 -2 , to recreate the core upsampler 202 1-n In this variant, the upsampling 202 -2 It is this recreated data that is used instead of the downsampled data. Similarly, the downsampled data is compared with the input of the second downsampling operation (or R -2 Downsampler, i.e. R -1 The output of the downsampler) is compared 203 -1 , to create the second residual data R -1 .

[0050] Figure 2 and Figure 3 The change between the implementations of results in slight changes in the residual data between the two implementations. Figure 2 Benefit from greater parallelization potential.

[0051] This process or cycle is repeated to create the third residual R0. Figure 2 and Figure 3 In the example of , the output residual data R0 (ie, the third residual data) corresponds to the highest level and is used to reconstruct the input data frame at the decoder. At this level, the difference operation is based on the same input data frame as the input of the first downsampling operation.

[0052] Figure 4 An example encoding process 401 is shown for encoding each of the levels or step indices of data to produce a set of encoded data steps with step indices. This encoding process is merely an example of a suitable encoding process for encoding each of the levels, but it should be understood that any suitable encoding process may be used. The input to the process is from Figure 2 or Figure 3 The output is residual data of a corresponding level, and the output is a set of levels of encoded residual data, which together hierarchically represent the encoded data.

[0053] In a first step, a transformation 402 is performed. The transform may be a directional decomposition transform as described in WO2013 / 171173. If a directional decomposition transform is used, a set of four components (also referred to as transform coefficients) may be output. For example, a 2x2 data block may be transformed to generate four components: three directional components approximately related to the horizontal, vertical and diagonal directions within the data block and an average value component representing an aggregate calculation applied to the entire data block. When a step index is referred to, it refers collectively to all directions, e.g., 4 steps. For example, a 2x2 data block may be flattened into a set of 4x1 values ​​and then transformed using a 4x4 Hadamard transform. In some cases, the normalization factor (e.g., 1 / 4, because normalization is implicitly applied by other processes such as quantization or entropy coding). Therefore, the mean value component can be generated for the 2x2 data block by summing the individual residual values ​​in the data block (e.g., multiplying by the Hadamard row {1,1,1,1}). The mean value component can be further adjusted by subtracting a "predicted mean value". This is described in detail in EP2850829B1. In summary, the predicted mean value is the difference between the lower layer pixel value and the mean value of the corresponding upper layer upsampled value set (e.g., for each output 2x2 data block from the upsampler 202, the input pixel minus the mean value of the upsampled pixel). Since the predicted mean value can be recovered at the decoder using the received data, subtracting the predicted mean value at the encoder and re-adding the predicted mean value at the decoder reduces the size of the mean value component of the transformed data block. In the latter example, a special upsampling operation is applied that can be used by the upsampler 202 to reduce the size of the mean value component without explicitly applying a predicted mean value modification, such as where an adaptive upsampling operation applies a predicted mean value modification that avoids the need to subtract the predicted mean value at the encoder and re-add at the decoder. Although examples are described for 2x2 data blocks, similar methods can also be applied to larger data blocks (e.g., 4x4 and above), where these data blocks will also have a "mean value" or "mean of means" component.

[0054] Return to Figure 4 , and then quantize 403 the component groups output by the transform 402 before entropy encoding. In this example, the entropy encoding operation 404 is coupled to a sparsification step 405, which exploits the sparsity of the residual data to reduce the overall data size and involves mapping the data elements to an ordered quadtree. Such coupling of entropy encoding and sparsification is further described in WO2019 / 111004, but the precise details of such a process are not relevant to an understanding of the present invention. Each residual array can be considered as a rung.

[0055] The process set forth above corresponds to an encoding process suitable for encoding data for reconstruction according to SMPTE ST 2117, VC-6 multiplane picture format. VC-6 is a flexible, multi-resolution, intra-frame only bitstream format capable of compressing any ordered set of integer element grids, each of which has an independent size, but also designed for picture compression. It uses a data agnostic technique for compression and is capable of compressing low or high bit depth pictures. The header of the bitstream may contain various metadata about the picture.

[0056] As will be appreciated, each step or step index may be implemented using a separate encoder or encoding operation. Similarly, the encoding module may be divided into steps of downsampling and comparison to produce residual data, and then encoding the residual, or alternatively, each of the steps of the step may be implemented in a combined encoding module. Thus, the process may be implemented, for example, using 4 encoders, one encoder for each step index, 1 encoder and multiple encoding modules operating in parallel or serially, or one encoder repeating operations on different data groups.

[0057] First example hierarchical decoder and components

[0058] An example of reconstructing an original data frame that has been encoded using the exemplary process described above is given below. This reconstruction process may be referred to as pyramidal reconstruction. Advantageously, the method provides an efficient technique for reconstructing an image encoded in a received data set that may be received via a data stream, for example, by separately decoding different component groups corresponding to different image sizes or resolution levels, and combining image details from one decoded component group with scaled-up decoded image data from a lower resolution component group. Thus, by performing the process on two or more component groups, structures or details therein may be reconstructed to obtain progressively higher resolutions or a greater number of pixels without the need to receive all or complete image details of the highest resolution component group. Instead, the method facilitates the gradual addition of increasingly high resolution details while reconstructing images from lower resolution component groups in a staged manner.

[0059] Furthermore, individually decoding each component group facilitates parallel processing of the received component groups, thereby increasing reconstruction speed and efficiency in implementations where multiple processes are available.

[0060] Each resolution level corresponds to a quality level or step index. This is a collective term associated with a plane (in this example a grid representation of integer-valued elements) describing all new inputs or groups of components received and the output reconstructed images of the index m cycle. For example, the reconstructed image with step index zero is the output of the last cycle of pyramidal reconstruction.

[0061] The cone reconstruction may be a process of reconstructing an inverted cone starting from an initial step index and using a cycle of new residuals to derive higher step indices until a maximum quality at a step index of zero, the quality being zero. A cycle may be considered a step in such a cone reconstruction, the step being identified by an index m. The step typically comprises upsampling the output data from a possible previous step, e.g. amplifying the decoded first component group, and taking new residual data as further input in order to obtain output data to be upsampled in a possible subsequent step. In the case where only the first component group and the second component group are received, the number of step indices will be two, and there will be no possible subsequent steps. However, in examples where the number of component groups or step indices is three or more, the output data may be upsampled stepwise in the following steps.

[0062] The first component group generally corresponds to an initial step index, which may be represented by step index 1-N, where N is the number of step indices in the plane.

[0063] Typically, the upscaling of the decoded first component group includes applying an upsampler to the output of the decoding process of the initial step index. In an example, this involves making the resolution of the reconstructed picture output from the decoding of the initial step index component group consistent with the resolution of the second component group corresponding to 2-N. Typically, the output of the upscaling from the lower step index component group corresponds to a predicted image at a higher step index resolution. Due to the lower resolution initial step index image and the upsampling process, the predicted image typically corresponds to a smooth or blurred picture.

[0064] Higher resolution detail is added to the predicted picture from the step index above, which provides a combined reconstructed image set. Advantageously, where the received component groups in one or more of the higher step index component groups include residual image data or data indicative of pixel value differences between the scaled up predicted picture and the original, uncompressed or pre-encoded image, the amount of received data required to reconstruct an image or data set of a given resolution or quality may be significantly less than the amount of data or data rate required to receive an image of the same quality using other techniques. Thus, according to the method, data rate requirements are reduced by combining low detail image data received at a lower resolution with progressively more detailed image data received at increasingly higher resolutions.

[0065] Typically, the group of encoded data includes one or more additional component groups, wherein each of the one or more additional component groups corresponds to a higher image resolution than the second component group, and wherein each of the one or more additional component groups corresponds to a progressively higher image resolution, the method comprising: for each of the one or more additional component groups, decoding the component group to obtain a decoded group, the method further comprising: for each of the one or more additional component groups, in ascending order of corresponding image resolution: scaling up the reconstruction group with the highest corresponding image resolution so that the corresponding image resolution of the reconstruction group is increased to be equal to the corresponding image resolution of the additional component groups, and combining the reconstruction group with the additional component groups to produce an additional reconstruction group.

[0066] In this way, the method may involve obtaining a reconstructed picture output for a given component group level or step index, scaling the reconstructed group, and combining it with the decoded output of the above component group or step index to produce a new, higher resolution reconstructed picture. It should be understood that this can be repeated for progressively higher step indices based on the total number of component groups in the received group.

[0067] In a typical example, each of the component groups corresponds to a progressively higher image resolution, wherein each progressively higher image resolution corresponds to four times the number of pixels in the corresponding image. Thus, typically, the image size corresponding to a given component group is four times the size or number of pixels of the image corresponding to the following component group (i.e., the component group having a step index one less than the step index in question), or twice its height and twice its width. For example, a received set of component groups in which each corresponding image has a linear size twice the size of the following image can facilitate a simpler upscaling operation.

[0068] In the example shown, the number of other component groups is two. Therefore, the total number of component groups in the received group is four. This corresponds to an initial step index of step 3.

[0069] The first component group may correspond to image data and the second component group and any other component groups may correspond to residual image data. As described above, where the lowest step index (i.e., the first component group) contains a low resolution or downsampled version of the image being transmitted, the method provides a particularly advantageous reduction in data rate requirements for a given image size. In this way, in each reconstruction cycle, starting with a low resolution image, the image is scaled up to produce a high resolution yet smooth version, and then the image is improved by adding the difference between the scaled up predicted image and the actual image to be transmitted at that resolution, and this additional improvement may be repeated for each cycle. Therefore, each component that is set to a higher than initial step index need only contain residual data in order to reintroduce information that may have been lost when the original image was downsampled to the lowest step index.

[0070] The method provides a way to obtain image data when receiving a set of data that has been compressed, for example, by decomposition, quantization, entropy coding and sparseness, which may be residual data. The residual may be the difference between an element of a first image and an element of a second image, which are usually located at the same position. Such residual image data may generally have a high degree of sparsity. This can be considered to correspond to an image in which detail areas are sparsely distributed in areas with minimal, negligible or non-existent details. Such sparse data can be described as a data array in which the data is organized in at least a two-dimensional structure (e.g., a grid), and most of the data so organized is zero (logically or digitally) or is considered to be below a certain threshold. Residual data is just an example. In addition, metadata may be sparse and therefore greatly reduced in size by this process. Sending already sparse data allows a significant reduction in the required data rate to be achieved by omitting sending such sparse areas and reintroducing them at appropriate locations within the received byte group at the decoder.

[0071] Typically, the entropy decoding, dequantization and directional synthesis transformation steps are performed according to parameters defined by an encoder or node transmitting the received encoded data groups. For each step index or component group, these steps are used to decode the image data to obtain groups that can be combined with different step indices according to the techniques disclosed above, while allowing the transmission of each level group in a data efficient manner.

[0072] A method for reconstructing a coded data set according to the method disclosed above may also be provided, wherein decoding of each of the first component group and the second component group is performed according to the method disclosed above. Thus, the advantageous decoding method disclosed herein may be used for each component group or step index in a received image data set and reconstructed accordingly.

[0073] refer to Figure 5, now a decoding example is described. A set of encoded data 501 is received, wherein the set includes four step indices, each step index includes four steps: step 0 of the highest resolution or quality level to step 1 of the initial step -3 .step -3 The image data carried by the component group corresponds to the image data, and the other component groups contain the residual data of the transmitted image. Although each level can output data that can be regarded as a residual, the initial ladder level (i.e., ladder -3 ) effectively corresponds to the actual reconstructed image. At stage 503, each component in the component group is processed in parallel to decode the encoded group.

[0074] Refer to the initial step index or core step index, and step each component group -3 Go to rung 0 and perform the following decoding steps.

[0075] At step 507, the component groups are de-sparsed. De-sparsed may be an optional step that is not performed in other layer-based hierarchical formats. In this example, de-sparsed results in the reconstruction of a sparse two-dimensional array from the encoded byte groups received at each rung. The zero values ​​grouped at positions that were not received within the two-dimensional array (due to being omitted from the transmitted byte groups in order to reduce the amount of data transmitted) are refilled by this process. The non-zero values ​​in the array retain their correct values ​​and positions in the reconstructed two-dimensional array, and the de-sparsed step refills the transmitted zero values ​​at the appropriate positions or at the positions therebetween.

[0076] At step 509, a range decoder is applied to the de-sparsened group of each rung to replace the coded symbols within the array with pixel values, the configuration parameters of the range decoder corresponding to the parameters used to encode the transmission data before transmission. The coded symbols in the received group are replaced with pixel values ​​based on an approximation of the distribution of pixel values ​​of the image. Using an approximation of the distribution, i.e., the relative frequency of each value of all pixel values ​​in the image, rather than the true distribution, allows the amount of data required to decode the set to be reduced, since the range decoder requires the distribution information to perform this step. As described in the present disclosure, the steps of de-sparseness and range decoding are interdependent, rather than sequential. This is indicated by the loop formed by the arrows in the flow chart.

[0077] The array of values ​​is dequantized at step 511. The process is performed again according to the parameters used to quantize the decomposed image before transmission.

[0078] After dequantization, the group is transformed at step 513 by a synthetic transform which comprises applying an inverse directional decomposition operation to the dequantized array. This results in the directional filtering being inverted, according to a group of operators including average or adjusted average, horizontal, vertical and diagonal operators, so that the resulting array is stepped.-3 Image data and ladder -2 to the residual data for step 0. Since the Hadamard transform is its own inverse transform, a common transform matrix can be applied to the forward and inverse transforms (plus any additional normalization, which may however be performed implicitly by quantization). In a comparative example, the inverse direction decomposition may also include adding the predicted mean calculated by the decoder to the adjusted mean component prior to the inverse transform. The latter example described herein provides a way to skip the predicted mean adjustment and implicitly apply such an adjustment by the upsampling operation.

[0079] Stage 505 shows several loops involved in reconstructing using the output of the composite transform for each of the step component groups 501. Stage 515 indicates the reconstructed image data for the initial step output from decoder 503. In the example, the reconstructed picture 515 has a resolution of 64x64. At 516, the reconstructed picture is upsampled to increase its constituent pixel count by a factor of four, thereby producing a predicted picture 517 having a resolution of 128x128. At stage 520, the predicted picture 517 is added to the output from the step component groups 501. -2 The decoded residual 518 of the output of the decoder. Adding these two 128x128 size images produces a 128x128 size reconstructed image, which contains the reconstructed image from the ladder. -2 The higher resolution details of the residuals are enhanced from the smoothed image details of the initial steps. If the desired output resolution is corresponding to the steps -2 The resulting reconstructed image 519 can be output or displayed if the resolution is 256x256. In this example, the reconstructed image 519 is used for further looping. At step 512, the reconstructed image 519 is upsampled in the same manner as step 516 to produce a predicted image 524 of size 256x256. Then, at step 528, it is compared with the decoded ladder. -1 The outputs 526 are combined, resulting in a 256x256 reconstructed picture 527, which is an upscaled version of the prediction 519 enhanced with the higher resolution details of the residual 526. At 530, the process is repeated a final time, and the reconstructed picture 527 is upscaled to a 512x512 resolution for combination with the level 0 residual at stage 532. A 512x512 reconstructed picture 531 is thus obtained.

[0080] In a comparative embodiment such as EP 2850829 B1, a predicted average value may be calculated and added as part of step 513. In other comparative embodiments, such as WO2020 / 188242A1, a predicted average value may be added as a modifier after each upsampling step (e.g., one or more of 526, 522, and 530). The use of the predicted average value may be a configurable parameter so that it can be turned on and off and indicated via configuration data in the bitstream. In the preferred example described herein, the predicted average value calculation is implicitly applied by appropriately configuring the coefficients of the upsampling filter that performs upsampling at one or more of steps 526, 522, and 530. In this case, the average value component after the transformation can be calculated without explicitly applying the predicted average value modification, but the energy or bit content of the average value component is still reduced within the encoded bitstream.

[0081] Second Example Hierarchical Encoder and Components

[0082] Figure 6 and Figure 7 Other hierarchical coding techniques that can utilize the principles of the present invention are shown. This technique is a flexible, adaptable, efficient and computationally inexpensive coding format that combines different video coding formats, base codecs (e.g., AVC, HEVC or any other current or future codecs) with at least two enhancement levels of the coded data.

[0083] The general structure of the coding scheme uses a downsampled source signal encoded with a base codec, adds a first level of correction data to the decoded output of the base codec to produce a corrected picture, and then adds another level of enhancement data to the upsampled version of the corrected picture. Therefore, the stream is considered to be a base stream and an enhancement stream, which can be further multiplexed or otherwise combined to generate a coded data stream. In some cases, the base stream and the enhancement stream can be transmitted separately. As described herein, references to coded data can refer to an enhancement stream or a combination of a base stream and an enhancement stream. The base stream can be decoded by a hardware decoder, and the enhancement stream can be suitable for software processing implementation with appropriate power consumption. This general coding structure creates multiple degrees of freedom that allow great flexibility and adaptability to many situations, making the coding format suitable for many use cases, including OTT transmission, live streaming, live ultra-high-definition UHD broadcasting, and so on. Although the decoded output of the base codec is not intended for viewing, it is a fully decoded video at a lower resolution, making the output compatible with existing decoders and can also be used as a lower resolution output if deemed appropriate.

[0084] In some examples, each or both enhancement streams may be encapsulated into one or more enhancement bitstreams using a set of Network Abstraction Layer Units (NALUs). A NALU is intended to encapsulate an enhancement bitstream so that the enhancement is applied to the correct base reconstructed frame. A NALU may, for example, contain a reference index to a NALU containing the base decoder reconstructed frame bitstream to which the enhancement must be applied. In this way, the enhancement may be synchronized to the base stream and the frames of each bitstream combined to produce a decoded output video (i.e., the residual of each frame of the enhancement level is combined with the frames of the base decoded stream). A group of pictures may represent multiple NALUs.

[0085] Returning to the initial process described above, where a base stream is provided along with two levels (or sub-levels) of enhancements within an enhancement stream, an example of a generalized encoding process is depicted in Figure 6 , a block diagram of . An input video 600 at an initial resolution is processed to generate various coded streams 601, 602, 603. A first coded stream (coded base stream) is produced by feeding a downsampled version of the input video to a base codec (e.g., AVC, HEVC, or any other codec). The coded base stream may be referred to as a base layer or base level. A second coded stream (coded layer 1 stream) is produced by processing a residual obtained by taking the difference between a reconstructed base codec video and a downsampled version of the input video. A third coded stream (coded layer 2 stream) is produced by processing a residual obtained by taking the difference between an upsampled version of a corrected version of the reconstructed base coded video and the input video. In some cases, Figure 6 The components of may provide a general low-complexity encoder. In some cases, the enhancement stream may be generated by an encoding process that forms part of the low-complexity encoder, and the low-complexity encoder may be configured to control an independent base encoder and decoder (e.g., encapsulated in a base codec). In other cases, the base encoder and decoder may be provided as part of the low-complexity encoder. In one case, Figure 6 The low-complexity encoder may be viewed as a form of wrapper for the base codec, where the functionality of the base codec may be hidden from the entity implementing the low-complexity encoder.

[0086] The downsampling operation illustrated by the downsampling component 105 can be applied to an input video to produce a downsampled video to be encoded by a base encoder 613 of a base codec. Downsampling can be performed in both vertical and horizontal directions, or alternatively only in the horizontal direction. The base encoder 613 and the base decoder 614 can be implemented by a base codec (e.g., as different functions of a common codec). The base codec and / or one or more of the base encoder 613 and the base decoder 614 may include appropriately configured electronic circuitry (e.g., a hardware encoder / decoder) and / or computer program code executed by a processor.

[0087] Each enhancement stream encoding process may not necessarily include an upsampling step. Figure 6 In , the first enhancement stream is conceptually the correction stream, while the second enhancement stream is upsampled to provide the enhancement level.

[0088] Referring to the process of generating an enhancement stream in more detail, to generate an encoded layer 1 stream, the encoded base stream is decoded by the base decoder 614 (i.e., a decoding operation is applied to the encoded base stream to generate a decoded base stream). The decoding may be performed by a decoding function or mode of the base codec. A difference is then created between the decoded base stream and the downsampled input video at the layer 1 comparator 610 (i.e., a subtraction operation is applied to the downsampled input video and the decoded base stream to generate a first set of residuals). The output of the comparator 610 may be referred to as a first set of residuals, such as a surface or frame of residual data, where the residual values ​​are determined for each image element at the resolution of the output of the base encoder 613, the base decoder 614, and the downsampling block 605.

[0089] The difference is then encoded by the first encoder 615 (ie, the layer 1 encoder) to generate the encoded layer 1 stream 602 (ie, the encoding operation is applied to the first set of residuals to generate the first enhancement stream).

[0090] As described above, the enhancement stream may include a first enhancement level 602 and a second enhancement level 603. The first enhancement level 602 may be considered a corrected stream, e.g., a stream that provides a correction level to a base encoded / decoded video signal at a lower resolution than the input video 600. The second enhancement level 603 may be considered to convert the corrected stream into another enhancement level of the original input video 600, e.g., it applies the enhancement level or correction to a signal reconstructed from the corrected stream.

[0091] exist Figure 6In the example of , a second enhancement layer 603 is created by encoding another set of residuals. Another set of residuals is generated by a layer 2 comparator 619. The layer 2 comparator 619 determines the difference between the upsampled version of the decoded layer 1 stream (e.g., the output of the upsampling component 617) and the input video 600. The input of the upsampling component 617 is generated by applying the first decoder (i.e., the layer 1 decoder) to the output of the first encoder 615. This generates a set of decoded layer 1 residuals. These residuals are then combined with the output of the base decoder 614 at the summing component 620. This effectively applies the layer 1 residuals to the output of the base decoder 614. It allows the losses in the layer 1 encoding and decoding process to be corrected by the layer 2 residuals. The output of the summing component 620 can be regarded as an analog signal representing the output of applying the layer 1 processing to the encoded base stream 601 and the encoded layer 1 stream 602 at the decoder.

[0092] As previously described, the upsampled stream is compared to the input video, which produces another set of residuals (i.e., a difference operation is applied to the upsampled reconstructed stream to generate another set of residuals). The other set of residuals is then encoded by the second encoder 621 (i.e., the layer 2 encoder) into an encoded layer 2 enhancement stream (i.e., the encoding operation is then applied to the other set of residuals to generate another encoded enhancement stream).

[0093] Therefore, if Figure 6 As shown in and described above, the output of the encoding process is a base stream 601 and one or more enhancement streams 602, 603, which preferably include a first enhancement layer and another enhancement layer. The three streams 601, 602 and 603 can be combined with or without additional information such as control headers to generate a combined stream representing the video coding framework structure of the input video 600. It should be noted that Figure 6 The components shown in the figure can operate on blocks or coding units of data, which correspond to, for example, 2x2 or 4x4 portions of a frame at a particular resolution level. The components operate without any inter-block dependencies, so they can be applied to multiple blocks or coding units within a frame in parallel. This is different from comparative video coding schemes, where there are dependencies (e.g., spatial dependencies or temporal dependencies) between blocks. The dependencies of comparative video coding schemes limit the level of parallelism and require much higher complexity.

[0094] Second Example Hierarchical Decoder and Components

[0095] exist Figure 7 The corresponding generalized decoding process is depicted in the block diagram of . It is said that Figure 7 Can show corresponding to Figure 6The low complexity decoder receives the three streams 601, 602, 603 generated by the low complexity encoder together with a header 704 containing further decoding information. The encoded base stream 601 is decoded by a base decoder 710 corresponding to the base codec used in the low complexity encoder. The encoded layer 1 stream 602 is received by a first decoder 711 (i.e., a layer 1 decoder) which decodes the encoded layer 1 stream 602 as generated by the low complexity encoder. Figure 1 The first set of residuals encoded by the first encoder 615 are decoded. At the first summing component 712, the output of the base decoder 710 is combined with the decoded residuals obtained from the first decoder 711. The combined video, which may be referred to as the layer 1 reconstructed video signal, is upsampled by an upsampling component 713. The encoded layer 2 stream 103 is received by a second decoder 714 (i.e., a layer 2 decoder). The second decoder 714 decodes the encoded layer 2 stream 103 as received by Figure 1 The second group of residuals encoded by the second encoder 621 are decoded. Although the header 704 is Figure 7 714, but it may also be used by the first decoder 711 as well as the base decoder 710. The output of the second decoder 714 is a second set of decoded residuals. These may be at a higher resolution than the first set of residuals and the input to the upsampling component 713. At a second summing component 715, the second set of residuals from the second decoder 714 is combined with the output of the upsampling component 713 (i.e., the upsampled reconstructed level 1 signal) to reconstruct the decoded video 750.

[0096] According to the low complexity encoder, Figure 7 The low complexity decoder of the video signal can operate in parallel on different blocks or coding units of a given frame of the video signal. In addition, the decoding performed by two or more of the base decoder 710, the first decoder 711 and the second decoder 714 can be performed in parallel. This is possible because there is no inter-block dependency.

[0097] During the decoding process, the decoder may parse the header 704 (which may contain global configuration information, picture or frame configuration information, and data block configuration information) and configure the low-complexity decoder based on those headers. In order to recreate the input video, the low-complexity decoder may decode each of the base stream, the first enhancement stream, and the other or second enhancement stream. The frames of the streams may be synchronized and then combined to derive the decoded video 750. Depending on the configuration of the low-complexity encoder and decoder, the decoded video 750 may be a lossy or lossless reconstruction of the original input video 100. In many cases, the decoded video 750 may be a lossy reconstruction of the original input video 600, where the loss has a reduced or minimal impact on the perception of the decoded video 750.

[0098] exist Figure 6 and Figure 7 In each of the above, the level 2 and level 1 encoding operations may include the steps of transform, quantization, and entropy encoding (e.g., in that order). These steps may be performed in a manner similar to Figure 4 and Figure 5 The encoding operation may also include residual scalaring, weighting and filtering. Similarly, in the decoding stage, the residual may be passed through an entropy decoder, a dequantizer and an inverse transform module (e.g., in that order). Any suitable encoding and corresponding decoding operations may be used. However, preferably, the level 2 and level 1 encoding steps may be performed in software (e.g., as performed by one or more central or graphics processing units in the encoding device).

[0099] In the examples described herein, the transform is a directional decomposition transform, such as a Hadamard-based transform. This may involve applying a small kernel or matrix to a flattened coding unit of the residual (i.e., a 2x2 or 4x4 residual block). For more details on the transform, see, for example, patent applications WO2020188273 A1 or WO2018046941A1, which are incorporated herein by reference. The encoder may select between different transforms to be used, for example, between the sizes of the kernels to be applied.

[0100] The transform may transform the residual information to four surfaces. For example, the transform may produce the following components or transform coefficients: average, vertical, horizontal, and diagonal. A particular surface may include all values ​​of a particular component, for example, a first surface may include all average values, a second surface may include all vertical values, and so on. As mentioned earlier in this disclosure, these components output by the transform may be considered as data to be quantized. The transform may include a Hadamard transform as described above. In a comparative example, the average component may be adjusted using a predicted average; in this example, the predicted average adjustment is not explicitly applied, but the predicted average adjustment is implicitly applied using an adjusted upsampling filter. Therefore, in the example described later, the benefits of the predicted average are provided without the need for additional predicted average calculations (e.g., as if the predicted average mode is turned off). A quantization scheme may be used to create a residual signal as a quantity so that a particular variable may take only specific discrete values. In this example, entropy coding may include run-length encoding (RLE), and then the encoded output is processed using a Huffman encoder. In some cases, when entropy coding is required, only one of these schemes may be used.

[0101] In summary, the methods and apparatus herein are based on an overall approach that builds upon existing encoding and / or decoding algorithms (e.g., MPEG standards such as AVC / H.264, HEVC / H.265, etc.; and non-standard algorithms such as VP9, ​​AV1, etc.) that are used as a baseline for enhancement layers for different encoding and / or decoding methods, respectively. The idea behind the overall approach of the example is to encode / decode video frames in a hierarchical manner, in contrast to using a block-based approach used in the MPEG family of algorithms. Encoding frames in a hierarchical manner includes generating a residual for a full frame, and then generating a residual for a decimated frame, and so on.

[0102] As described above, these processes can be applied in parallel to the coding units or blocks of the color components of a frame because there are no inter-block dependencies. The encoding of each color component within a set of color components can also be performed in parallel (e.g., such that the operation is repeated according to (number of frames) * (number of color components) * (number of coding units per frame)). It should also be noted that different color components can have different numbers of coding units per frame, for example, the luma (e.g., Y) component can be processed at a higher resolution than a set of chroma (e.g., U or V) components when human vision can detect a greater change in illumination than a change in color.

[0103] Therefore, as shown and described above, the output of the decoding process is (optional) base reconstruction, and the original signal reconstruction at a higher level. This example is particularly suitable for creating encoded and decoded videos at different frame resolutions. For example, the input signal 30 can be an HD video signal including a frame at a 1920x 1080 resolution. In some cases, both base reconstruction and level 2 reconstruction can be used by a display device. For example, in the case of network services, level 2 streams may be more disturbed than level 1 streams and base streams (because it may include up to 4 times the amount of data, where downsampling divides the size by 2 in each direction). In this case, when business occurs, the display device can recover to display the base reconstruction, and the level 2 stream is interrupted (for example, when the level 2 reconstruction is unavailable), and then recover to display the level 2 reconstruction when network conditions improve. Similar methods can be applied when the decoding device is subject to resource constraints, such as the set-top box performing system updates may have an operating base decoder 220 to output the base reconstruction, but may not have the processing capacity to calculate the level 2 reconstruction.

[0104] The encoding arrangement also enables a video distributor to distribute video to a heterogeneous set of devices; those devices with only the base decoder 720 view the base reconstruction, while those devices with the enhancement layer can view the higher quality layer 2 reconstruction. In the comparative case, two complete video streams at separate resolutions are required to serve the two sets of devices. Since the layer 2 and layer 1 enhancement streams encode residual data, the layer 2 and layer 1 enhancement streams can be encoded more efficiently, e.g., the distribution of residual data is typically mostly around 0 quality (i.e., no difference exists) and typically takes a small range of values ​​around 0. This is especially true after quantization. In contrast, the complete video streams at different resolutions will have different distributions of non-zero means or medians, which require higher bit rates for transmission to the decoder.

[0105] In the examples described herein, the residual is encoded by an encoding pipeline. This may include transform, quantization, and entropy encoding operations. It may also include residual grading, weighting, and filtering. The residual is then transmitted to a decoder, for example as L-1 and L-2 enhancement streams, which may be combined with the base stream as a hybrid stream (or transmitted separately). In one case, a bit rate is set for a hybrid data stream comprising a base stream and two enhancement streams, and then different adaptive bit rates are applied to individual streams based on the data being processed to meet the set bit rate (e.g., high-quality video perceived with low artifact levels can be constructed by adaptively assigning bit rates to different individual streams (even at a frame-by-frame level) so that constrained data can be used by the most perceptually influential individual streams, which may change as the image data changes).

[0106] The group of residuals as described herein can be considered as sparse data, e.g., in many cases there is no difference for a given pixel or region, and the resulting residual value is zero. When looking at the distribution of the residuals, a lot of probability mass is assigned to small residual values ​​located close to zero, e.g., for certain video values ​​of -2, -1, 0, 1, 2, etc., which occur most frequently. In some cases, the distribution of residual values ​​is symmetric or approximately symmetric about 0. In some test video cases, it was found that the distribution of residual values ​​was shaped similar to a logarithmic or exponential distribution about 0 (e.g., symmetrically or approximately symmetrically). The exact distribution of residual values ​​may depend on the content of the input video stream.

[0107] The residual can itself be processed as a two-dimensional image, such as a delta image of the difference. In this way, it can be seen that the sparsity of the data is related to features such as "dots", small "lines", "edges", "corners", etc. that are visible in the residual image. It has been found that these features are usually not completely correlated (e.g., spatially and / or temporally). The feature has characteristics that are different from the characteristics of the image data from which it is derived (e.g., the pixel characteristics of the original video signal).

[0108] Because the characteristics of the residual are different from the characteristics of the image data from which the residual is derived, it is generally not possible to apply standard encoding methods, such as those found in the traditional Moving Picture Experts Group (MPEG) encoding and decoding standards. For example, many contrasting schemes use large transforms (such as transforms of large pixel areas in normal video frames). Due to the characteristics of the residual, such as described above, it would be very inefficient to use these contrasting large transforms on the residual image. For example, it would be very difficult to encode a small point in the residual image using a large block designed for an area of ​​the normal image.

[0109] Certain examples described herein address these issues by using smaller and simpler transform kernels instead (e.g., 2x2 or 4x4 kernels as presented herein—directional decomposition and directional decomposition squared). The transforms described herein can be applied using Hadamard matrices (e.g., 4x4 matrices for flattening 2x2 coding blocks, or 16x16 matrices for flattening 4x4 coding blocks). This moves in a different direction from the contrasting video coding methods. Applying these new methods to residual blocks generates compression efficiency. For example, certain transforms generate uncorrelated transform coefficients (e.g., in space) that can be effectively compressed. Although correlations between transform coefficients can be exploited, such as for lines in residual images, these can result in coding complexity that is difficult to implement on traditional and low-resource devices, and often generates other complex artifacts that need to be corrected. Preprocessing the residual by setting certain residual values ​​to 0 (i.e., not forwarding these residual values ​​for processing) can provide a controllable and flexible way to manage bit rate and stream bandwidth, as well as resource usage.

[0110] Examples related to downsampling enhancement

[0111] The present invention relates to the implementation of upsampling filters. For example, the method described herein can be used in Figures 1 to 7 In the implementation of one or more upsamplers among upsamplers 202, 522, 526, 530, 617 and 713.

[0112] Adaptive upsampling filter

[0113] Figure 8 An example method of applying an upsampling filter configured in accordance with the present invention is shown. At block 805, an input video frame of a first resolution is acquired. This may be a reconstructed frame at a lower level or layer in the hierarchical or layer-based configuration described above (e.g., Figure 5 Data in 515, 520 or 528, Figure 6 The output of summation 620, or Figure 7At block 810, an upsampling filter is applied to output an upsampled frame at a second resolution 815. This may include Figure 5 Data 517 or data 534, Figure 6 The residual calculation in 619 is input or Figure 7 The reconstruction summation 715 is input in.

[0114] In this example, the upsampling filter is configured to apply a predicted average modifier. The predicted average modifier may include a modifier described in WO2020 / 188242A1 or a "predicted average" value described in EP 2850829 B1. In a comparative example, the predicted average modifier is derived as the difference between the average value of a data block of pixels at the second resolution after application of the upsampling filter and the corresponding pixel value of the first resolution before application of the upsampling filter. For example, for 2x upsampling, each of a set of input pixels in the value plane can be upsampled to a corresponding 2x2 value block (in some cases, edge cases are handled differently, such as using padding or cropping). The predicted average modifier modifies (e.g., reduces) the average value of a data block of the residual derived as the difference between a video frame of the second resolution after upsampling and an original video frame of the second resolution used to derive the input video frame for upsampling. For example, the residual plane is Figure 6 The level 2 comparator 619 generates or serves as Figure 2 and Figure 3 In this residual plane, the pixel values ​​can be grouped into nxn blocks to form coding units, where n is typically 2 or 4. These nxn residual blocks will have corresponding nxn blocks in the upsampled frame, for example, the 64x64 residual plane output by the level 2 comparator 619 will have 32x32 2x2 residual blocks generated by the 32x32 corresponding 2x2 blocks in the input video 600 and the output of the upsampling 617. Figure 6 Level 2 code 621 or Figure 4 During the ladder coding 401 in FIG. 4 , each n×n data block of the residual is transformed to produce a set of n 2directional components, where the directional components in these directional components are the values ​​in the mean value component data block calculated by summing. In a comparative example, for example as described in EP 2850829B1, the predicted mean value modifier can be calculated as the difference between the mean value of the data block of pixels of the second resolution after the application of the upsampling filter and the corresponding pixel value of the first resolution before the application of the upsampling filter, and then the difference is subtracted from the mean value of the residual value in the data block. However, by optimizing the coefficients of the upsampling filter, the mean value of the data block of pixels of the second resolution after the application of the upsampling filter can be close to (e.g., within the quantization range) or equal to the corresponding pixel value of the first resolution so that the predicted mean value modifier is zero, which is equivalent to implicitly applying the predicted mean value.

[0115] In certain examples described herein, a separable upsampling filter is provided, wherein a set of separable filter coefficients is configured to minimize the difference between the mean value of a residual data block and the predicted mean value of the residual data block. Here, the residual data block is derived as the difference between a video frame of a second resolution after upsampling and an original video frame of a second resolution used to derive an input video frame for upsampling. Therefore, the filter implicitly applies a predicted mean value during the upsampling process by changing the upsampling filter coefficients (to produce a "predicted mean value preserving filter"), wherein the predicted mean value represents the difference between the mean value of a data block of pixels of the second resolution after upsampling and the corresponding pixel value of the first resolution before upsampling, and the predicted mean value in the comparison example is subtracted from each data block in a "predicted mean value mode" during encoding and added to each data block in a "predicted mean value mode" during decoding. Therefore, the adaptive upsampling filter in the example provides this "predicted mean value mode", but there is no explicit calculation other than the upsampling operation. Or in other words, the predicted mean value is configured to have a value of 0 through upsampling, thereby avoiding the need to explicitly subtract (at the encoder) or add (at the decoder) the predicted mean value from the data block.

[0116] Example Separable Filter

[0117] Fig.9A and Fig. 9B An example separable filter is shown that can be configured to apply the predicted average values ​​described herein. Fig.9A and Fig. 9BIn the example of , a five-coefficient separable filter is shown. The concept of separable filters is known in the art: a separable filter is a filter in which multiple dimensions of a signal (e.g., the two spatial dimensions of the plane of a video frame) are filtered in series by corresponding low-dimensional filters (e.g., two one-dimensional filters), where each filter is applied in a different direction. In this case, there are two one-dimensional filters, each with five coefficients (i.e., 5-tap filters).

[0118] Fig.9A shows a lower resolution pixel value to be upsampled – d 22 The receptive field 905 is a 5x5 region around the lower resolution pixel and can be represented as 5 rows of pixel values ​​910 or a 5x5 grid 915. The size of the receptive field is 5x5 because the two 1D five-coefficient filters effectively perform a 5x5 2D filtering. Fig.9A Also shown is the definition of an example separable filter 920, which includes a first filter 922 and a second filter 924, with filter coefficients shown as 926 and 928. In this example, upsampling by a factor of two is performed so that filter F 920 applied to receptive field D 905 results in four output pixels, as shown by V 930. Fig. 9B It is shown how each of the four output pixels is derived from a calculation involving the data row 910 and the first and second filters 922, 924. The 2x2 grid of four output pixels 930 can therefore be calculated as: V = F T D T F, where F = [F0, F1] and D is the receptive field. Similar calculations can be applied to separable filters of different lengths, e.g., for a three-coefficient filter, the receptive field is a 3x3 grid centered on the current pixel; or for a four-coefficient filter, the receptive field is a 4x4 grid with the upsampled pixel located at one of the four center pixels (e.g., as defined when designing the filter).

[0119] Using the above framework, the mean value component of an nxn data block can be calculated as:

[0120]

[0121] where I is the nxn data block of the input frame (e.g., 210 or 600), and v i is an element of V, ie, the residual mean can be calculated as the mean of the input frame data block minus the mean of the upsampled data block.

[0122] In the above example, the input signal is downsampled and then upsampled to generate V. For example, in Figure 2 and Figure 3 201 of them and Figure 6Downsampling is applied at the encoder at 605 in . Downsampling of the input signal generates data of the receptive field D, where the data may be a lower resolution signal reconstructed after base encoding and decoding, such as Figure 6 If we take d as the pixel value upsampled to generate V, we can define the difference between this pixel found in the downsampled input frame and the reconstructed pixel value d as

[0123]

[0124] where V(...) is a function that downsamples a block of data derived from an input frame I. Now, if the downsampling function is the mean downsampler And we can define A as:

[0125]

[0126] And since the decoder has access to d and We can define the adjusted mean component as:

[0127]

[0128] The predicted mean is defined as:

[0129]

[0130] In the comparative encoder, the predicted average PA is therefore calculated at the encoder and subtracted from the average A, so that for each data block, the smaller is encoded instead of A. However, if the upsampling filter is configured so that the predicted average PA is zero, i.e. This is similar to applying the predicted mean modification implicitly as Therefore, in this example, the upsampling filter is configured such that:

[0131]

[0132] For a 2x2 block of data, this means that the filter coefficients are configured as follows:

[0133] v0+v1+v2+v3=4d

[0134] In such Fig.9A and Fig. 9B In the five-coefficient filter shown, when all data values ​​d mn m,nε[0,1,2,3,4] cancel out and leave the center value d 22 When , the above equation is satisfied. For example, Fig. 9C An example 942 is shown where Fig. 9B The upsampled pixel value v in i The D0 terms in the expression of 940 are expanded. In this case, each group of terms (such as group 944) can be constrained to sum to zero. Accordingly, a set of equations 950 can be defined so that the terms cancel (i.e., the columns or groups 944 sum to zero). This can then be used as Fig. 9C The set of equations 960 shown is extended from the D0 case to the general D n case, where n≠2. A set of nontrivial solutions to Equation 960 is when n≠2 n0 +f n1 = 0. For the case of n = 2, all equations except the third central equation can be set as shown in 960; then the third central equation can be expressed as:

[0135] f 20 (f 20 +f 21 )+f 21 (f 20 +f 21 )=4

[0136] Solving for f 20 and f 21 , we get f 20 +f 21 = ±2. This shows that the general solution of the five-coefficient predicted mean preserving separable filter is:

[0137]

[0138] Therefore, a separable filter of the form described above will satisfy the constraint that the mean value of the upsampled data block is equal to the lower resolution value being upsampled.

[0139] The above solution allows an existing non-predicted mean preserving filter to be adapted to provide a predicted mean preserving filter. For example, a separable five-coefficient non-predicted mean preserving cubic upsampler can be defined as:

[0140]

[0141] To adjust this filter to implicitly provide a predicted mean calculation, the coefficients can be adjusted to confirm the general form shown above. Therefore, the coefficients shown above are modified as follows:

[0142]

[0143] Although the above solution provides a filter that retains the predicted mean value (i.e., implicitly applies a filter calculated by the predicted mean value), the above filter is a five-coefficient (five-tap) filter. In some video decoders, such as set-top boxes and traditional devices, there are limitations on the resources available for the up-sampling filter. In particular, the limitation can be related to the number of available filter coefficients. For example, some video processing devices are limited to four-coefficient filters, such as due to hardware limitations.

[0144] Some video coding standards (such as LCEVC) also specify that the filter bank of the upsampling filter contains coefficients that conform to a specific defined pattern, i.e., filter coefficients that are limited to certain forms. In LCEVC, a set of filter coefficients for a specific upsampling filter is specified as follows:

[0145] F=[-a,b,…,y,-z]

[0146] That is, the first and last coefficients must be negative. This form of "mirror kernel" can avoid having to send negative values ​​in the bitstream (for example, a and z are sent in the bitstream as positive coefficients and are then automatically set to their negative counterparts at the decoder, equivalent to a*-1 and z*-1), which can improve compression and reduce complexity while allowing adaptive filters with varying coefficients. There may also be a constraint that the coefficient values ​​other than the first and last coefficients are positive. In the general solution of the five-coefficient predicted mean preserving separable filter described above, an implementation that meets this specification would have f0=-f0=0 and f4=-f4=0. However, this essentially reduces the five-coefficient filter to a three-coefficient filter while retaining the implementation complexity of the five-coefficient filter.

[0147] Given these additional constraints applicable to certain implementations, it is also desirable to have an approximation to a predicted mean-preserving (separable) filter with four coefficients and with negative values ​​for the first and last filter coefficients in each filter bank.

[0148] Turning to the four-coefficient filter, this can be specified to have the following general form:

[0149]

[0150] This four-coefficient filter can also be specified as a five-coefficient filter, where one coefficient of each filter phase is set to 0, i.e.:

[0151]

[0152] Taking this into account and recalling one of the solutions for applying the filter over the predicted mean, the following limitations exist:

[0153] When n≠2, fn0 +f n1 =0

[0154] f 20 +f 21 = ±2

[0155] Returning to the four-coefficient filter specified as a five-coefficient filter, one solution to these constraints is for a = -c and b = ±1 and for d = 0. This suggests that the general form of the four-coefficient predicted mean application filter can be written as:

[0156]

[0157] Compare this form with the five coefficients F above PA norm, which suggests that an approximation to the original four-coefficient filter that retains or applies a predicted mean can be defined as:

[0158]

[0159] Therefore, the above formula can be used to convert a non-predicted average application filter F to a predicted average application filter F' or F PA .

[0160] Fig.10 An example 1000 of a four-coefficient non-predicted average application filter set F 1005 and its counterpart F'1010 is shown, which are configured to implicitly apply predicted average calculations according to the above specifications. According to the above derivation, the existing four-coefficient or five-coefficient upsampling filter can therefore be applied to generate a predicted average (sometimes referred to as a predicted residual) to which an upsampling filter is applied. The filter applying the predicted average approximates the filtering of the original filter and the application of the predicted average. In these cases, the predicted average application filter is if the predicted average filter is applied, for example, before the inverse transform described in EP 2850829 B1 or before the post-upsample modifier described in WO2020 / 188242A1, the predicted average (i.e., any modifier) ​​will be zero. Therefore, the upsampler acts as if the predicted average is applied without explicitly performing a predicted average correction. This situation is because the upsampler is configured to limit the average value of the upsampled pixel data block to the corresponding pixel value before upsampling. This method allows any custom filter kernel to be easily converted into a predicted average application filter, for example, by applying the conversion shown above. By using a custom predicted average application filter, predicted average calculation can be disabled in LCEVC or VC-6 configurations, but predicted average calculation will still be applied (even if predicted average calculation is enabled, it will have no impact because the predicted average modifier will be zero in every case).

[0161] Configuring the upsampler to implicitly perform the predicted mean calculations can provide benefits regardless of where the processing is performed. For example, it can speed up processing on graphics processing units (GPUs) and central processing units (CPUs) by reducing the number of operations that need to be performed (e.g., upsampling is performed in all encoding and decoding processes anyway, and the current approach does not increase the number of coefficients used or the number of upsampling calculations). By reducing the number of operations performed for each block of data, it can also save battery power consumed on mobile devices (e.g., a battery consumption saving of approximately 5% has been observed in testing).

[0162] Training Adaptive Upsampling Filters

[0163] In some cases, as described above, a set of upsampling coefficients (e.g., upsampling coefficients for a separable upsampling filter) can be selected based on multiple constraints to implicitly apply the predicted mean calculation. In other cases, the coefficients of the adaptive upsampling filter that applies the predicted mean can also be trained or optimized. Fig.11 An example configuration of how this can be implemented is shown. For example, the upsampler can be implemented as a neural network upsampler, such as a convolutional neural network.

[0164] Fig.11 A training setup 1100 is shown, in which a trainable predicted mean application upsampler 1105 can be trained to simulate the operation of an existing non-predicted mean application upsampler 1110. The trainable upsampler 1105 can be any linear or non-linear upsampling filter, including separable and non-separable filters. The trainable upsampler 1105 can be implemented as a convolutional neural network with one or more filters, each filter having a set of trainable filter parameters. The existing upsampler 1110 can be any known upsampling filter, also including linear and non-linear filters, and can be separable or non-separable. In one case, the existing upsampler 1110 can include a known n-coefficient separable filter in the form of the above-mentioned (non-predicted mean application) five-coefficient and four-coefficient filters. In other cases, the existing upsampler 1110 can include a trained neural network sampler. In this case, the existing upsampler 1110 may have been pre-trained to upsample the downsampled sequence so as to match the ground truth sequence of the original input as closely as possible at the upsampled resolution. When training, any trainable coefficients of the existing upsampler 1110 are fixed or frozen—these coefficients are not trained or updated. Instead, the purpose of training is to configure the trainable coefficients of the trainable upsampler 1105.

[0165] In the training setting 1100, a reference truth video sequence 1120 is obtained. The reference truth sequence 1120 can be a set of frames of a video sequence (e.g., one or more of the luminance and chrominance planes). The video sequence can be selected to have a wide range of video features in order to provide robust training (e.g., static and dynamic scenes with various textures). In the training setting 1100, a downsampler 1125 is used to downsample the reference truth sequence 1120 of video frames to obtain a downsampled video frame sequence 1130. This can be performed frame by frame or in batches. Then, the downsampled video frame sequence 1130 is passed to each of the trainable upsampler 1105 and the existing upsampler 1110 for upsampling. The existing upsampler 1110 upsamples the downsampled video frame sequence 1130 using an upsampling filter with a set of fixed coefficients to generate a first upsampled sequence. Thereafter, a predicted mean modifier 1135 is applied. For example, the predicted mean modifier 1135 can be calculated as the difference between the element values ​​of the upsampled data block and the input downsampled elements, for example, as described in WO2020 / 188242A1. After modification with the predicted mean at block 1135, a modified first upsampled sequence 1140 is obtained. Again, the sequence can be generated frame by frame or batch by batch.

[0166] In parallel with the upsampling and modification performed by the existing upsampler 1110 and the predicted mean modifier 1135, the trainable upsampler 1105 also upsamples the downsampled sequence of video frames 1130 in a forward pass or inference mode to generate a second upsampled sequence 1145. Then, as part of the loss calculation 1160, the modified first upsampled sequence of video frames 1140 and the second upsampled sequence of video frames 1145 are compared to determine an update for the trainable upsampler 1105. For example, the coefficient values ​​of the trainable upsampler 1105 may be updated using gradient descent (in a known form, such as using stochastic gradient descent) and backpropagation. As part of the training, the trainable coefficients of the trainable upsampler 1105 are optimized so as to minimize the difference (i.e., the loss) between the two sequences of video frames 1140, 1145. The trainable upsampler 1105 thus learns to mimic the actions of the existing upsampler 1110 and the predicted mean modifier 1135, i.e., is trained as an upsampler that applies a predicted mean. In Fig.11 In the illustrated training setup 1100 , the trainable upsampler 1105 is therefore optimized to reconstruct a lossy version of a ground truth sequence 1120 , which itself has been downsampled, upsampled, and had a predicted mean applied.

[0167] Adaptive upsampling filters as described herein can be implemented as hardware and / or software filters. For example, custom coefficients can be loaded into application-specific upsampler filter banks present in devices such as set-top boxes or embedded devices (e.g., by firmware updates, etc.). In devices such as personal computers and mobile devices, filtering can be performed using computer program code. In this case, a memory can store a set of filter coefficients as described herein, such as a set of separable filter coefficients, including a first set of filter coefficients for filtering in a first direction and a second set of filter coefficients for filtering in a second direction, and a processor (such as a CPU and / or a GPU) can apply the set of filter coefficients to upsample an input video frame from a first resolution to a second resolution, the second resolution being higher than the first resolution.

[0168] One benefit of using the predicted averages as described above to apply the filter is that, whether configured according to the above specifications or as Fig.11 The optimization shown has the advantage that the advantages of predicted average calculation can be obtained even when the hardware limitations of the decoding device prevent the predicted average calculation from being performed explicitly. For example, some traditional devices (such as set-top boxes) have a limited number of possible decoding configurations, and although these devices may be able to support custom upsampling filter coefficients and LCEVC decoding pipelines, they may not be able to support predicted average calculation (e.g., applied before inverse transform or as a modifier added after upsampling). Therefore, this example allows the predicted average to be applied in these cases.

[0169] The above examples are to be understood as illustrative. Further examples are contemplated. It should be understood that any feature described with respect to any one example may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other example, or in combination with any combination of any other example. In addition, equivalents and modifications not described above may also be employed without departing from the scope of the present invention as defined by the appended claims.

Claims

1. A method for decoding image data, the method comprising: receiving first image data at a first resolution, the first image data being associated with a video frame; upsampling the first image data to generate second image data for the video frame at a second resolution, the second resolution being higher than the first resolution; receiving encoded residual data at the second resolution, the encoded residual data comprising a plurality of encoded data blocks, the encoded data blocks comprising one or more directional components and at least one component derived from an average of the residual data within the encoded data blocks; applying inverse decomposition to the block of encoded residual data to derive decoded residual data; as well as combining the decoded residual data with the second image data to generate an output for the video frame at the second resolution, wherein the at least one component derived from the mean value of the residual data within the coded data block is calculated as a mean value component of a data block of uncoded residual data adjusted using a predicted mean value, the predicted mean value being derived from the mean values ​​of pixel values ​​in the first image data and a corresponding data block of pixels in the second image data, the corresponding data block being derived from upsampling of the pixel values ​​in the first image data, and Wherein the upsampling comprises applying a set of upsampling coefficients selected to constrain the mean value of the data block of pixels in the second image data to the corresponding pixel values ​​in the first image data so as to apply the predicted mean value during the upsampling. 2 . The method of claim 1 , wherein the upsampling comprises applying a separable filter having fewer than five coefficients for each of the two image dimensions.

3. The method of claim 2, wherein the set of upsampling coefficients has the following form:

4. The method according to claim 3, wherein for a separable upsampling filter having the following form: The set of upsampling coefficients is determined as: 5 . The method of claim 1 , wherein the upsampling comprises applying a separable filter having five coefficients for each of the two image dimensions.

6. The method of claim 5, wherein the set of upsampling coefficients has the following form:

7. The method of claim 6, wherein for a separable upsampling filter having the following form: The set of upsampling coefficients is determined as:

8. A method according to any one of the preceding claims, wherein the method of decoding image data applies one of Low Complexity Enhancement Video Coding (LCEVC) or SMPTE VC-6 2117.

9. A method according to any one of the preceding claims, wherein the set of coefficients is determined according to a training process with a set of applied constraints.

10. A method of encoding image data, the method comprising: receiving first image data at a first resolution, the first image data being associated with an original video frame; upsampling the first image data to generate second image data for the video frame at a second resolution, the second resolution being higher than the first resolution; generating residual data of the second resolution as a difference between the second image data and the original video frame, the residual data comprising a plurality of data blocks; as well as transforming the plurality of data blocks to generate a plurality of encoded data blocks, the encoded data blocks comprising one or more directional components and at least one component derived from an average of the residual data within the data blocks being encoded, wherein the upsampling comprises applying a set of upsampling coefficients selected to constrain a mean value of a block of pixels in the second image data to a corresponding pixel value in the first image data, wherein the upsampling modifies the second image data to adjust the mean value of the residual data within the encoded data block, the mean value being adjusted using a predicted mean value representing the difference between the mean value of the data block of pixels in the second image and the corresponding pixel values ​​in the first image data.

11. The method of claim 10, wherein the upsampling comprises applying a separable filter having fewer than five coefficients for each of the two image dimensions.

12. The method of claim 11, wherein the set of upsampling coefficients has the following form:

13. The method of claim 12, wherein for a separable upsampling filter having the following form: The set of upsampling coefficients is determined as:

14. The method of claim 10, wherein the upsampling comprises applying a separable filter having five coefficients for each of the two image dimensions.

15. The method of claim 14, wherein the set of upsampling coefficients has the following form:

16. The method of claim 15, wherein for a separable upsampling filter having the following form: The set of upsampling coefficients is determined as:

17. The method according to any one of claims 10 to 16, wherein the method of encoding image data applies one of Low Complexity Enhancement Video Coding (LCEVC) or SMPTE VC-6 2117.

18. A method according to any one of claims 10 to 17, wherein the set of coefficients is determined according to a training process with a set of applied constraints.

19. A method for training a set of coefficients for an upsampling filter, the method comprising: Obtaining a baseline truth sequence of video frames; downsampling the reference truth sequence of video frames to obtain a downsampled video frame sequence; upsampling the downsampled sequence of video frames using a first upsampling filter having a set of fixed coefficients to generate a first upsampled sequence; applying a predicted mean modifier to data derived from the first upsampled sequence to output a modified first upsampled sequence, the predicted mean modifier representing a difference between the mean of the data block of pixels in the first upsampled sequence and the corresponding pixel values ​​in the downsampled sequence of video frames; upsampling the downsampled sequence of video frames using a second upsampling filter having a set of trainable coefficients to generate a second upsampled sequence; calculating a loss between the second upsampled sequence and the modified first upsampled sequence; as well as optimizing the set of trainable coefficients to reduce the loss, The optimized set of trainable coefficients provides a predicted mean-preserving version of the first upsampling filter.

20. An upsampling filter for video coding, the upsampling filter comprising: a memory comprising a set of separable filter coefficients, the set of separable filter coefficients comprising a first set of filter coefficients for filtering in a first direction and a second set of filter coefficients for filtering in a second direction; as well as a processor for applying the set of separable filter coefficients to upsample an input video frame from a first resolution to a second resolution, the second resolution being higher than the first resolution, wherein the set of separable filter coefficients is configured to modify the output of the upsampling filter to apply a predicted mean modifier, the predicted mean modifier being derived as the difference between the mean value of a data block of pixels at the second resolution after application of the upsampling filter and the corresponding pixel values ​​at the first resolution before application of the upsampling filter, the predicted mean modifier modifying the mean value of a data block derived as a residual of the difference between the video frame at the second resolution after upsampling and the original video frame at the second resolution used to derive the upsampled input video frame.

21. A method of upsampling a video frame for video encoding, the method comprising: Obtain an input video frame of a first resolution; applying an upsampling filter defined by a set of separable filter coefficients including a first set of filter coefficients for filtering in a first direction and a second set of filter coefficients for filtering in a second direction to obtain an output video frame at a second resolution, the second resolution being higher than the first resolution, wherein the set of separable filter coefficients is configured to modify the output of the upsampling filter to apply a predicted mean modifier, the predicted mean modifier being derived as the difference between the mean value of a data block of pixels at the second resolution after application of the upsampling filter and the corresponding pixel values ​​at the first resolution before application of the upsampling filter, the predicted mean modifier modifying the mean value of a data block derived as a residual of the difference between the video frame at the second resolution after upsampling and the original video frame at the second resolution used to derive the upsampled input video frame.

Images