Signal enhancement filtering parameter coding for reference picture resampling

Abstract

A method of performing an entropy coding scheme for a signal enhancement filter is provided. The method includes optimizing parametric coding based on an estimated rate-distortion cost by estimating a rate-distortion enhancement of a current frame and one or more subsequent frames.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to the subject of video processing and video coding, and more particularly to a method, decoder, encoder and computer readable medium for signal enhancement filtering for reference picture resampling and / or image upscaling. BACKGROUND

[0002] Current video coding schemes such as H.265 / HEVC (High Efficiency Video Coding) and H.266 / VVC (Versatile Video Coding) support spatial scalability of coded video streams. The resolution of the coded video is changed adaptively during encoding, which is referred to as Reference Picture Resampling (RPR) or Adaptive Resolution Change (ARC) in VVC. Furthermore, multi-resolution coding and multi-layer coding enable a coded video to have scalable resolution. Thus, the spatial resolution of the coded video can be changed adaptively without the need to equalize the output or input resolution of the video. The advantage of this additional flexibility is that coding a video at a lower resolution requires a lower bit rate and can reduce the computational complexity, at the cost of losing high frequency information in the down-sampling step.

[0003] Coded video at a resolution lower than its original resolution requires performing down-sampling and up-sampling steps in the signal processing chain. In the down-sampling step, an anti-aliasing filter is applied to prevent artifacts caused by high frequency components in the image. The up-sampling process applies an interpolation filter to reconstruct the intensity values at fractional sample positions.

[0004] In RPR, the resolution of the coded video stream can be changed adaptively. Thus, the encoder can code parts of the video stream at a lower resolution. RPR is applied in inter prediction whenever a reference picture with a different resolution than the current picture is used in inter prediction. In this step, a resampling operation needs to be applied in order to map the reference picture block to the same spatial resolution as the current picture.

[0005] In multi-layer coding, the video is coded at different resolution layers. In a first step, the video is coded at the lowest resolution layer. To generate the video stream of the next layer, the video is up-sampled and, possibly, the residuals are coded and further processing steps are applied. This processing can be applied multiple times based on the number of layers.

[0006] Finding the best high-resolution representation from a low-resolution image is an important part of the coding scheme described above. One common approach is to apply a set of polyphase Finite Impulse Response (FIR) interpolation filters. While these filters do provide a good approximation of the high-resolution image content, they cannot recover the information lost in the down-sampling process and are limited by the linear filtering operation. As a result, the up-sampled image is often blurry. An image sharpening operation can improve the image quality. However, linear high-pass filters often result in artifacts such as overshoot and ringing. Furthermore, the distortion caused by down-sampling and up-sampling depends on the image content and the coding quality of the video (affected by the Quantization Parameter (QP) value).

[0007] Applying a locally weighted adaptive filter is one approach to handle these issues. Local weighting can be applied to smoothly increase or decrease the strength of the filter in a local region. One way of weighting comes to mind: increase the filter strength at edge regions but decrease the filter strength at regions where ringing usually occurs. With this setup, the optimized filter can amplify the high-frequency components without causing noticeable ringing. This is especially helpful for image up-sampling scenarios that require amplifying high-frequency components to sharpen the blurry edges. An adaptive filter is needed to handle the different characteristics of the coding artifacts and the video content.

[0008] The proposed method requires sending some side information. These side information are some flags, filter coefficients, region partition information, and mode parameters. For video coding applications, minimizing the required transmission rate is crucial. This means that the extra number of bits should be minimized. To reduce the number of bits, a coefficient coding scheme that exploits the redundancy in the information that needs to be transmitted is proposed. SUMMARY

[0009] Embodiments of the present application provide a method, decoder, encoder and computer readable medium for reference image resampling and / or signal enhancement filtering for image upscaling that overcomes problems associated with conventional arrangements.

[0010] According to a first aspect, there is provided a computer-implemented method of performing an entropy coding scheme for a signal enhancement filter, the method being performed by an encoder. The method comprises optimizing parameter coding based on estimated rate-distortion cost by estimating rate-distortion enhancement for a current frame and one or more subsequent frames.

[0011] In some embodiments, the signal enhancement filter is an edge-directed signal enhancement filter.

[0012] In some embodiments, estimating the rate-distortion enhancement comprises calculating the rate-distortion enhancement and summing the rate-distortion enhancement for one or more subsequent frames containing the current frame in their reference picture list.

[0013] In some embodiments, the method further comprises: performing parameter quantization; performing filter parameter encoding; and performing entropy encoding on the filter parameters.

[0014] In some embodiments, the parameter quantization comprises mapping consecutive values to a finite set of codewords.

[0015] In some embodiments, the filter parameter encoding is performed using a new filter coding mode, a new filter intra coding mode, a new filter inter coding mode, and a copy filter coding mode, wherein each of the above modes is used to predict filter coefficients and / or filter parameters.

[0016] In some embodiments, the new filter coding mode is selected if the filter cannot be effectively predicted by any other coding mode.

[0017] In some embodiments, the new filter intra coding mode is based on other filter coefficients of the same filter and utilizes dependencies between filter coefficients of the same filter to predict filter coefficients.

[0018] In some embodiments, the new filter inter coding mode is based on filter coefficients of a previous filter to make prediction of filter coefficients of a next filter.

[0019] In some embodiments, the copy filter coding mode copies all filter coefficients and is used to apply the same filter to another frame.

[0020] In some embodiments, the new filter inter coding mode and the copy filter coding mode use filter parameters from a previous filter to predict a current filter, wherein the filter from which the current filter is predicted is indicated by an index, and the filter in the reference list is a filter that has already been sent for some previously coded frame.

[0021] In some embodiments, the filter parameters are signaled in an adaptive parameter set, and a systematic code is used to perform entropy encoding of the filter parameters.

[0022] In some embodiments, a simple structured code is used to perform entropy encoding of the filter parameters.

[0023] In some embodiments, exponential Golomb-code, Golomb-Rice code, or k-th order exponential Golomb code is used to perform entropy coding on filter parameters.

[0024] In some embodiments, separate chroma filters and luminance filters are used.

[0025] In some embodiments, on / off flags are used for the luminance filter and the chrominance filter.

[0026] In some embodiments, the method is used for an adaptive reference image upsampling scheme.

[0027] According to a second aspect, a computer-readable medium is provided having computer-executable instructions stored thereon, which, when executed by a computing device, cause the computing device to perform any of the methods discussed in the first aspect.

[0028] According to a third aspect, an encoder is provided. The encoder includes one or more processors; and a computer-readable medium having computer-executable instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform any of the methods discussed in the first aspect.

[0029] These and other aspects of this application will become more apparent from the following description of the embodiments. Attached Figure Description

[0030] Embodiments will now be described by way of example only, with reference to the accompanying drawings.

[0031] Figure 1A A flowchart illustrating the operation of an entropy coding scheme for a signal enhancement filter, performed by an encoder according to an embodiment of the present invention, is shown.

[0032] Figure 1B A flowchart illustrating the operation of parameter encoding based on estimated rate distortion cost optimization according to an embodiment of the present invention is shown.

[0033] Figure 2 Four filter coding modes for predicting filter coefficients and / or filter parameters according to embodiments of the present invention are shown.

[0034] Figure 3A An embodiment of the invention is shown using a separate encoding mode.

[0035] Figure 3B An embodiment of the invention illustrates the use of the same encoding pattern for the entire filter.

[0036] Figure 4 An example of filter prediction for an intra-frame filter coding mode according to an embodiment of the present invention is shown.

[0037] Figure 5 A filter prediction for an inter-frame filter coding mode is illustrated according to an embodiment of the present invention.

[0038] Figure 6 A filter selection for replicating filter coding modes according to an embodiment of the present invention is shown.

[0039] Figure 7 A schematic diagram of a decoder according to various embodiments of the present invention is shown.

[0040] Figure 8 A schematic diagram of an encoder according to various embodiments of the present invention is shown. Detailed Implementation

[0041] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0042] These techniques can be applied to H.265 / HEVC or H.266 / VVC video coding systems, and are particularly suitable for scenarios involving RPR, ARC, multi-resolution coding, and multi-layer coding. However, it should be understood that these techniques can be applied to any other video coding system involving upsampling. Furthermore, while these principles are primarily illustrated with reference to video processing, they are also applicable to other data formats, including image processing and even audio processing.

[0043] In this embodiment, "video" refers to one or more images. In other words, a video may include one or more images. An image may also be referred to as a "picture".

[0044] An encoder is a device that encodes data into a bitstream, while a decoder is a device that decodes the bitstream to obtain the encoded data or an approximation of the encoded data. A bitstream consists of a sequence of bits.

[0045] Intra-frame prediction and inter-frame prediction are two prediction operations that can be used within the HEVC and VVC frameworks for the decoder to process the received bitstream in order to obtain the original signal. In this embodiment, "original signal" or "original video" refers to the data prior to encoding at encoder 20. The reference sample in this embodiment may refer to image data used to predict the image (or image region) in space and / or time intervals. Intra-frame and inter-frame prediction operations are also used at the encoder to make rate-distortion (RD) determinations.

[0046] More specifically, intra-frame prediction involves spatial data prediction within a single image, without referencing other images (at different time intervals). In other words, data from a first region of an image is used to predict data from another region of the same image, but does not depend on images at another time interval. In this context, data from the first region of the image is considered a "reference sample."

[0047] Inter-frame prediction involves predicting data between images over multiple time intervals. In other words, data from a first region of a first image is used to predict data from a second region of a second image. The first and second regions may or may not be spatially separated. In this context, the data from the first region of the first image is considered a "reference sample." It should also be noted that inter-frame prediction can sometimes use multiple reference regions from different images at once (i.e., for a single prediction operation).

[0048] In the embodiments, "residual" may refer to the value obtained based on the original value of the image region and the predicted value of the image region (e.g., the difference between the original value and the predicted value).

[0049] In this embodiment, "block" can refer to a portion of an image. For example, an image can be divided into two or more blocks. However, this is just an example. If the image is not divided, then "block" can refer to the entire image.

[0050] "Signal enhancement filter" can refer to a filter used to enhance a signal, particularly an upsampled signal. Typically, in the described embodiments, the signal enhancement filter is a filter configured to reduce edge blur (i.e., sharpen image patches). However, embodiments are not limited thereto, and the signal enhancement filter may alternatively be configured to provide alternative or additional signal enhancement in other embodiments, such as removal of block artifacts and / or ringing artifacts.

[0051] Figure 1A A flowchart illustrating the operation of an entropy coding scheme for a signal enhancement filter is shown.

[0052] Figure 1A The flowchart in step 101 illustrates optimizing parametric coding based on the estimated rate-distortion cost by estimating the rate-distortion enhancement of the current frame and one or more subsequent frames. Several aspects of the invention will be explained in more detail before proceeding with the description of the accompanying drawings.

[0053] The proposed invention provides a parametric coding method for adaptive reference image upsampling. Therefore, it considers the transmission requirements of the adaptive parameter set and the characteristics of auxiliary information to form an effective coding scheme.

[0054] In some embodiments, the coding scheme is rate-distortion optimized, meaning that the coding scheme affects the distortion of the downstream task (image upsampling). It should be noted that there are two types of distortion. The first type is the distortion in the downstream task, i.e., the distortion resulting from applying enhancement filters to some of the coded signal. The second type is the distortion of the coding parameters.

[0055] Embodiments of the present invention rely on two main components / steps, including: a) Filter parameter prediction schemes; and b) Encode the filter coefficients using low-complexity entropy coding.

[0056] In some embodiments, the prediction scheme utilizes the interdependencies between different parameters. Thus, parameters can be predicted based on previously decoded parameters from the same parameter set, thereby transmitting the difference between the predicted and true coefficients. Transmitting the difference between the predicted and true coefficients is meaningful if the entropy of the residual is less than the entropy of the parameter. Furthermore, parameters from previous frames can be reused without encoding the residual. Finally, it is also possible to modify the residual or the transmitted parameters to reduce the encoding cost that affects distortion in downstream tasks. Therefore, rate-distortion optimization is required for the modifications.

[0057] Upsampling is required in video coding applications such as RPR and multi-layer / multi-resolution coding. These methods typically apply interpolation filters to generate pixel values ​​for the upsampling filter. Exemplarily, the method according to embodiments of the present invention is applied directly after the interpolation filter and before any other processing steps. The purpose of this method is to reduce distortion caused by low-resolution video coding. These distortions include the loss of high-frequency information and distortions caused by video coding. In this case, for example, a default upsampling filter would be used to change the initial resolution.

[0058] The method according to embodiments of the invention is then applied independently without modifying the existing resampling process, but with the addition of potentially optional enhancement steps. Embodiments of the invention specifically illustrate possible implementations of the parameter encoding scheme for the aforementioned upsampling enhancement scheme.

[0059] Typically, efficient encoding of any additional data transmitted is crucial for improving the overall efficiency of the method. Gain can only be achieved by improving rate-distortion performance through this method. This means that a smaller bit rate is required to achieve a specific distortion level d when this method is applied, compared to not applying it.

[0060]

[0061] Therefore, the rate at which the bitstream is encoded using the method according to embodiments of the present invention is defined in the following equation.

[0062]

[0063] The following text will focus primarily on the parameters directly used by this method. and distortion All other parameters are not directly affected by the proposed encoding method. However, indirect effects are possible due to changes in content. Calculating these indirect effects is difficult and requires actual encoding. Since this is computationally expensive and computationally complex, it is recommended to model these effects. For this purpose, the direct and indirect effects of the method are distinguished. The direct effect is understood as reducing distortion in the current image by applying the method according to embodiments of the invention. The indirect effect is twofold. The first indirect effect is that, due to the increase in the number of bits in the current frame, rate control can decide to encode subsequent frames at a slightly lower rate. The second indirect effect only takes effect when the current image is referenced by some upsampled image. In this case, inter-frame prediction is performed on the enhanced image content. It is assumed that in most cases, if the source image is of better quality, the prediction will also be better, resulting in smaller residuals. Therefore, the improvement in quality may positively affect other images. Both effects need to be considered to make the optimal rate-distortion decision.

[0064] The direct impact can be estimated by measuring the distortion compared to the ground truth after applying the enhancement filter and estimating the additional rate required to send the adaptive parameter set with the signal. To reduce computational complexity, simplified measurements or approximations of the distortion are used in some embodiments.

[0065] Indirect effects are more difficult to estimate accurately. Therefore, it is recommended to use a simplified model.

[0066] In some embodiments, when the proposed method is applied to the current frame, the model calculates the RD gain for other frames. In some embodiments, the RD enhancement is calculated and summed for all frames in the reference picture list that contain the current frame. These frames can be easily inferred from the encoder configuration. The model considers the quality of the frame before and after the filter is applied, as well as the rate and encoding configuration (e.g., QP, Reference Picture List (RPL), QP of all images in the RPL, image size of the images in the RPL). Thus, the RD enhancement for that image is calculated. A simplified model for calculating the RD enhancement for a P-frame or B-frame n is shown below.

[0067]

[0068]

[0069] If i is not equal to the current image If i equals the current image, then i is zero. This is equivalent to RD enhancement. (Function) Returns a scalar weighting factor. The weighting factor depends on the encoding parameter p and the current index i. For example, for frames closer to frame n (i.e., if...), the weighting factor is... (Small values) will be given higher weight. This is because frames that are closer to the prediction frame in the time dimension are generally more similar in content. Furthermore, the horizontal and vertical rescaling factors of the frame, as well as the QP (Quality of Frame) are considered. For example, it is assumed that frames encoded at high resolution with a low QP are used more frequently than frames with a higher QP or lower resolution because their quality is generally lower. That is, if it is assumed that the current frame and the reference frame (i.e., the reference image) have similar content and that the error in the reference frame is uncorrelated with temporal variations between the two frames, then distortion in the reference frame will lead to a larger error in the prediction signal.

[0070] As a result of this formula, the RD offset is obtained for each frame referencing the current frame. By summing these RD offsets, the overall RD improvement relative to the reference image is obtained. It should be noted that this scheme can also consider that the reference image in the RPL can reference the current frame. To illustrate this, the previous formula can be applied recursively. However, for simplicity, this is omitted here. Therefore, if Defined as applying the filter to the RD difference of the current frame, the following equation is obtained to calculate the total RD gain.

[0071]

[0072] Using this formula, It can be optimized to a minimum. Several methods are proposed to reduce the signaling cost of the enhancement filter. This improves the overall RD performance and also makes this type of filtering suitable for a wider range of content.

[0073] There are two different types of data that need to be transmitted and optimized using signals. The first type of data consists of flags and mode decision parameters. These parameters are compressed in lossless mode. The reason for this is that such parameters significantly affect the performance of on / off decisions, such as those in filters. These parameters include, for example, luminance / chrominance on / off flags, filter shape parameters (separable / non-separable and the overall shape of the filter), quantization parameters (step size, clipping, etc.), weighted graph type, and encoding mode.

[0074] On the other hand, there are filter coefficients and (conceptually continuous) weighted graph function parameters that are quantized or modified before encoding and transmission to achieve a finite number of well-compressible codewords. These parameters include the filter coefficients of the luminance and chrominance filters, as well as the weighted graph parameters. The quantization and encoding schemes will be described below.

[0075] The following sections contain a detailed description of the parameters that can be used in such a setup. This includes all potentially optional parameters that can be used to adapt and signal the behavior of the proposed filters. First, there are luma and chroma flags. These flags indicate whether a filter is applied to the luma and / or chroma components. If one of these flags is false, no parameters are encoded for that channel. Each of the luma and chroma filters has a weighted graph function (which may itself have parameters) and a filter. The weighted graph is specified by a parametric function computed from the encoded video. The parameters of the weighted graph function depend on the type of weighted graph and are signaled according to the type of weighted graph function. The set of parametric weighted graph functions is predefined. The selected weighted graph function can be signaled or selected depending on the content. In addition to the weighted graph, the parameters of the filters are also transmitted. These parameters are the filter shape, possible quantization parameters, and filter coefficients. Furthermore, encoded information can also be signaled. The encoded information can be information about the behavior of quantization processing, parameter prediction, or entropy coding. This information can be sent individually for the luma and chroma filters, or for both filters simultaneously. Finally, information about the location where each filter is applied can be transmitted via signal. This means that a filter may be applied only to a portion of the image. Therefore, the portion to which each filter is applied can be transmitted via signal in the bitstream or inferred from the video content and / or encoding information. Note that multiple filters can be encoded with different parameters. Furthermore, two filters can be applied to the same spatial location within a frame.

[0076] Now return to Figure 1B It illustrates the operational flowchart for optimizing parameter encoding based on estimated rate-distortion cost. At 201, parameter quantization is performed. At 202, filter parameter encoding is performed using four different filter encoding modes, which will be determined by referencing... Figure 2 Let's explain. Finally, at position 203, entropy encoding of the filter parameters is performed.

[0077] A close examination of parameter quantization step 201 clearly reveals that, for parameter transmission and calculation in video coding, filter parameters need to be mapped to integer values. In some embodiments, this mapping is performed by limiting floating-point values ​​to a certain range. Then, through a certain scaling factor. These values ​​are scaled and rounded to the nearest integer. In some implementations, the entire process is described by the following equation.

[0078]

[0079] Assumption , and It is an integer value, and the number of quantization intervals is given as .

[0080] In some embodiments, the applicable bit depth for parameter quantization is 12 bits. Thus, the number of bits in the quantized number will be 12, having... There are several values. However, it was found that any number of binary numbers in the range of 10 to 14 worked in the test scenario. On the one hand, a higher scaling factor leads to lower quantization error, resulting in a more accurate representation of the number. On the other hand, more binary numbers exist, which generally leads to higher encoding costs. The choice of upper and lower thresholds should ensure that the clipping error does not significantly affect performance. However, the smaller the range set, the fewer binary numbers are required to achieve the same quantization error, which reduces the encoding cost. Finding good quantization parameters is important for the final performance. For example, premature clipping filter coefficients can significantly affect performance. Note that this is just an example of a quantization scheme that can be used here. Rate-distortion optimized quantization can also be used.

[0081] Furthermore, in some embodiments, different quantization step sizes are applied based on the filter coefficients. For example, it may be meaningful to use a smaller quantization for the filter's average value because the error caused by the offset in the filter's low-frequency coefficients is generally greater than the error caused by the high-frequency coefficients.

[0082] In some embodiments, the parameters of the quantization scheme are either signaled in a parameter set or fixed for all coding settings, depending on the characteristics of the data. Adaptive quantization schemes may show benefits if there is high variability in the content, particularly regarding the dynamics of signaling and residuals in the coding. However, if there is less variability, it may be meaningful to pre-estimate the optimal parameters to reduce signaling costs.

[0083] Figure 2 The filter parameter encoding is illustrated, and four filter encoding modes are disclosed: new filter encoding mode 401, new filter intra-frame encoding mode 402, new filter inter-frame encoding mode 403, and duplicate filter encoding mode 404. Each of these modes is used to predict filter coefficients and / or filter parameters.

[0084] In some embodiments, filter parameters include at least one of filter coefficients, coding parameters, region partitioning information (if multiple filters are applied), weighted graph parameters, or other auxiliary information. In some embodiments, filter coefficients result in the greatest coding cost. This is why most effort is spent on reducing the coding cost of filter coefficients. Each mode is signaled via a mode flag. All other parameters are derived based on the mode flag. In some embodiments, mode parameters are signaled for each filter or for each of the Y, U, V channels, or individually for the luma and chroma channels. This means that in some embodiments, there is a coding scheme that first signals the luma and chroma flags, and then signals the mode parameters individually for each channel, provided the filter is activated on that channel. In other embodiments, the coded modes are signaled first.

[0085] Figure 3A The diagram illustrates a coding scheme where luminance and chromaticity indicators are first transmitted via signaling, and then, with the filter activated for that channel, mode parameters are transmitted via signaling individually for each channel. Figure 3B The encoding scheme in which the signal transmission encoding mode is first shown is illustrated. Figure 3B The embodiment shown has the advantage of lower encoding cost because the encoding pattern only needs to be sent once, and the luminance / chrominance flag is also derived from the encoding pattern.

[0086] Reference Figure 4 Before discussing the new filter intra-frame coding mode 402, we will discuss the new filter coding mode 401.

[0087] In some embodiments, if the filter cannot be effectively predicted by any other coding scheme, a new filter coding scheme 401 should be selected. In this case, the filter coefficients are encoded with their quantized representation. This means that, except in some embodiments where a coefficient is replaced by the mean of the filter, no prediction is applied to the filter coefficients. It is recommended that this prediction be applied to the coefficient with the highest entropy, as this prediction works reliably due to the characteristics of the coding scheme. In mathematical terms: all predictions assume the existence of some statistical dependencies on the filter coefficients. If these statistical dependencies hold with a high probability, then in most cases, the predictions and the reality are similar. However, if these assumptions do not hold, it may be less costly to skip the predictions altogether.

[0088] refer to Figure 4Now we will discuss the new filter intra-frame coding mode. This new mode utilizes the dependencies between filter coefficients of the same filter. Suppose we need to encode a series of n filter coefficients [a1, ..., an]. Without restricting generality, the encoding / decoding order can be set to the numerical order of the indices. That is, first encode filter coefficient a1, then encode filter coefficient a... i Encoded in ascending order. Therefore, the filter coefficients a i Filter coefficients can be used Make predictions. Furthermore, there may be a difference in the current filter coefficients a. i There are some dependencies. For each coefficient, some possible learned context / prediction model can be used. Therefore, the model does not need to be equivalent for every coefficient.

[0089] For example, a very simple model can be used, where the filter coefficients a1 are encoded without modification, and all subsequent coefficients are encoded as the difference c between each coefficient and the previous coefficient. i =a i -a (i-1) The last filter coefficient is encoded as the average of all filter coefficients. It can be assumed that the residual / coding error is typically close to a zero-mean distribution. Therefore, in most cases, the sum of all filter coefficients will also be close to zero. Consequently, in most cases, the entropy of the probability distribution of this sum will be less than the entropy of the probability distribution of the nth coefficient, which is why this encoding might be meaningful. This exemplary implementation... Figure 4 As shown in the image.

[0090] Figure 5The new filter inter-frame coding pattern 403, which predicts the next filter coefficient based on the previous filter, is shown in more detail. Thus, the filter coefficient at position i is predicted by the filter coefficient at time step t+j, where j is some offset in the temporal domain. Furthermore, multiple filters are used for combined prediction. A simple example of implementing the new filter inter-frame pattern is constructing a reference filter set that is adaptive to the Picture Order Count (POC). For example, all filters corresponding to filters in a reference image set could be included in the reference filter set. Another option might be to always store the n filters that are closest in time. The current filter can be predicted by sending the index of the predicted filter in the reference image list as a signal. Then, for example, the current filter can be predicted directly using the selected reference filter. The residual between the current filter and the reference filter is then encoded. This makes sense if it can be assumed that the filter coefficients remain similar for subsequent frames. This is an assumption that relies on the temporal dependence of the encoded video. If the error signal of the video has similar statistics for frames that are close in time / coding order, the filters are also likely to be similar. Therefore, small residual signals exist. Small residual signals can be utilized through entropy coding of the residuals, which reduces the cost per symbol. Figure 5 This illustrates how this prediction scheme might work. Therefore, there exists a set of k filters available for prediction, each filter used for prediction of filter a. The residual between the predicted value and the residual is then calculated. The coding cost of each reference filter is then estimated, and the filter with the lowest coding cost is selected. Note that some transformation can be optionally applied to the residual. A notable transformation is replacing a filter coefficient with the sum of the filter coefficients. Furthermore, some form of intra-frame prediction can be applied after the residual calculation. Finally, a combined prediction using multiple filters can also be chosen.

[0091] Figure 6 A filter duplication mode is illustrated, which copies all filter coefficients and potential other filter parameters. This makes the mode less expensive but less flexible. In some embodiments, this mode is used to apply the same filter to another frame, which may be suitable if the coded video and residuals of two or more frames are very similar. In this case, filters optimized for some previous frames can still improve the current frame. In most cases, there may be a reduced quality enhancement. However, this may be worthwhile in terms of RD cost due to the reduced coding cost. Figure 6 A schematic implementation of an optimized filter replication mode is shown. Therefore, a set of filters previously transmitted with the signal can be included in the reference filter list. The reference filter list contains a subset of the previously decoded available filters bi.

[0092] The "New Filter Inter-Frame" and "Filter Copy" modes utilize filter parameters from previous filters to predict the current filter. The filter on which the current filter is based is indicated by an index. Filters in the reference list can be filters that have already been sent for some previously encoded frames. Other options are pre-computed filters and filters inferred from the video content. Typically, there may be many available filters, from which prediction can be made after some frames. Since the number of bits required to encode filter indices increases with the number of indices that need to be represented, we recommend selecting only a subset of filters from the reference list. The selected filters can depend on parameters previously sent with the signal and other factors, such as the selected weighting graph, luma and chroma filter shapes, etc.

[0093] Finally, the steps for entropy coding of the filter parameters will be analyzed in more detail. The parameters of the enhancement filter are sent as signals in an Adaptation Parameter Set (APS). This is a high-level parameter set, meaning that not all entropy coding tools are available. For example, arithmetic coding is typically not used at this level. Therefore, the following methods will also be limited to entropy methods, which are applied to or can be applied to such a parameter set.

[0094] The following sections outline an exemplary implementation of entropy coding. Note that this is only one example, and many potentially suitable modifications and adjustments will also be discussed in the following description.

[0095] The primary goal of entropy coding is to reduce the average coding cost used to encode symbols. This is typically achieved by assigning longer codewords to less frequent symbols and shorter codewords to more frequent ones. Binary entropy is a measure of the minimum average number of bits required to encode symbols with some given probability distribution. In a given application, the residuals of the prediction filter coefficients are encoded. In this case, a code capable of encoding any possible symbol / filter coefficients as efficiently as possible is needed. If we choose to encode symbols using a code other than an entropy-optimal code, the efficiency of the code can be measured by the Kullback-Leibler divergence, which gives the average bit increase per symbol. Note that we cannot use entropy-optimal codes due to complexity reasons. This means that entropy-optimal coding or arithmetic coding may be too complex to apply at a given level. However, entropy-optimal coding or arithmetic coding remains a theoretical option and may be used if the design paradigm allows. At the current point in time, simple structured codes are used at the APS level. More specifically, exponential Golomb codes are frequently used to encode symbols in the APS. This code can be used for both signed and unsigned integers and is capable of encoding every possible integer. It assumes that the probability of a sign decreases exponentially with its magnitude. This makes sense for encoding residual signals, given their small size, if the fundamental assumptions of the prediction pattern hold. Therefore, there should be many small values ​​and relatively sparse large values ​​in terms of amplitude. However, other options exist, such as Golomb-Rice codes and k-order exponential Golomb codes, which are also suitable for structured encoding of filter coefficients. We recommend using structured codes (such as those mentioned above) where low-complexity implementation is required. Furthermore, we suggest estimating the optimal hyperparameters from a test set or encoding them using signal transmission entropy. An example of encoding data after prediction processing is using exponential Golomb codes. For our application, this code assumes a slightly higher probability of low amplitude values. This makes it not RD-optimal. However, this is also highly dependent on the type of data and the encoding method, which is why it cannot be considered a general conclusion.

[0096] Figure 7 A schematic diagram of decoder 10 according to an embodiment is shown. Specifically, Figure 7 A schematic diagram of decoder 10 configured to perform any of the decoder methods discussed herein is shown. For the sake of brevity, these detailed descriptions are omitted here.

[0097] like Figure 7As shown, decoder 10 includes processor 11 and computer-readable medium 12. Processor 11 and computer-readable medium 12 may be connected via a bus system. The computer-readable medium is configured to store programs, instructions, or code. Processor 11 is configured to execute the programs, instructions, or code in computer-readable medium 12 to perform operations in the decoder method embodiments of this document.

[0098] Therefore, in this embodiment, the computer-readable medium 12 is configured to store a computer program that can run in the processor 11, and the processor 11 is configured to run the computer program to perform the steps in any decoder method discussed herein.

[0099] Figure 8 A schematic diagram of an encoder 20 according to an embodiment is shown. Specifically, Figure 8 A schematic diagram of an encoder 20 configured to perform any of the encoder methods discussed herein is shown. For the sake of brevity, these detailed descriptions are omitted here.

[0100] like Figure 8 As shown, encoder 20 includes processor 21 and computer-readable medium 22. Processor 21 and computer-readable medium 22 may be connected via a bus system. The computer-readable medium is configured to store programs, instructions, or code. Processor 21 is used to execute the programs, instructions, or code in computer-readable medium 22 to perform operations in the decoder method embodiments of this document.

[0101] Therefore, in this embodiment, the computer-readable medium 22 is configured to store a computer program that can run in the processor 21, and the processor 21 is configured to run the computer program to perform the steps in any decoder method discussed herein.

[0102] Embodiments of the present invention may also provide a computer-readable medium having computer-executable instructions to cause one or more processors of a computing device to perform the methods of the embodiments of the present invention.

[0103] Examples of computer-readable media include volatile and non-volatile media, removable and non-removable media, and include, but are not limited to: solid-state storage; removable disks; hard disk drives; magnetic media; and optical disks. Generally, computer-readable media includes any type of media suitable for storing, encoding, or carrying a series of instructions executable by one or more computers to perform any or more of the processes and features described herein.

[0104] It should be understood that the functions of the various components discussed above can be combined in various ways, and are not limited to those discussed in the foregoing description. For example, in some embodiments, the functions of multiple devices in discussion can be integrated into a single device. In other embodiments, the functions of at least one device in discussion can be split into multiple independent (or distributed) devices.

[0105] Conditional language such as “may” is often used to indicate the use of features / steps in a particular embodiment, but alternative embodiments may include alternative features or omit these features entirely.

[0106] Furthermore, the method steps are not limited to the specific order described, and it should be understood that these steps can be combined in any other suitable order. In some embodiments, this can result in the parallel execution of some method steps. Additionally, in some embodiments, specific method steps may be omitted entirely.

[0107] While certain embodiments have been discussed, it should be understood that these embodiments are intended to illustrate the general teachings of the invention, and various modifications may be made without departing from the scope of the invention. The scope of the invention should be interpreted in accordance with the appended claims and any equivalents thereof.

[0108] Many further variations and modifications will arise in those skilled in the art upon reference to the exemplary embodiments described above. These embodiments are provided by way of example only and are not intended to limit the scope of the invention as defined by the appended claims.

Claims

1. A method for implementing an entropy coding scheme for a signal enhancement filter, the method being performed by an encoder, comprising: Parametric coding is optimized based on the estimated rate-distortion cost by estimating the rate-distortion enhancement of the current frame and one or more subsequent frames.

2. The method according to claim 1, wherein, The estimated rate-distortion enhancement includes: calculating the rate-distortion enhancement and summing the rate-distortion enhancements of the one or more subsequent frames in which the current frame is included in its reference image list.

3. The method according to claim 1, further comprising: Perform filter parameter quantization; Perform filter parameter encoding; as well as Perform entropy encoding of the filter parameters.

4. The method according to claim 3, wherein, The parameter quantization includes mapping continuous values ​​to a finite set of codewords.

5. The method according to claim 1, wherein, Perform filter parameter encoding using one of the following: New filter coding mode; New filter intra-frame coding mode; New filter inter-frame coding mode; or Copy the filter encoding mode; Each of the above modes is used to predict filter coefficients and / or filter parameters.

6. The method according to any one of claims 3 to 5, wherein, The filter parameters include at least one of filter coefficients, coding parameters, region partitioning information, weighted graph parameters, or other auxiliary information.

7. The method according to claim 5, wherein, If the filter cannot be effectively predicted by any other coding mode, then the new filter coding mode is selected.

8. The method according to claim 5, wherein, The new filter intra-frame coding mode predicts filter coefficients based on the other filter coefficients of the same filter and the dependencies between filter coefficients of the same filter.

9. The method according to claim 5, wherein, The new inter-frame coding mode predicts the filter coefficients of the next filter based on the filter coefficients of the previous filter.

10. The method according to claim 5, wherein, The copy filter encoding mode copies all filter coefficients and is used to apply the same filter to another frame.

11. The method according to claim 5, wherein, The new filter inter-frame coding mode and the replicated filter coding mode use filter parameters from previous filters to predict the current filter, wherein the filter on which the prediction of the current filter is based is indicated by an index, and the filters in the reference list are filters that have been sent for some previously coded frames.

12. The method according to claim 1, wherein, The filter parameters are transmitted via signals in an adaptive parameter set, and a system code is used to perform entropy encoding of the filter parameters.

13. The method according to claim 12, wherein, Simple structured codes are used to perform entropy coding of the filter parameters.

14. The method according to claim 12, wherein, Exponential Golomb codes, Golomb-Rice codes, or k-order exponential Golomb codes are used to perform entropy coding on the filter parameters.

15. The method according to any one of claims 12 to 14, wherein, Use separate chroma and luminance filters.

16. The method according to claim 15, wherein, Use on / off flags for the luminance filter and the chrominance filter.

17. The method according to any one of the preceding claims, wherein, The method is used for an adaptive reference image upsampling scheme.

18. The method according to any one of the preceding claims, wherein, The method is used after the interpolation filter and before any other processing steps.

19. A computer-readable medium having stored thereon computer-executable instructions, which, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 18.

20. An encoder, comprising: One or more processors; as well as A computer-readable medium having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 18.

21. A method for performing an entropy decoding scheme for a signal enhancement filter, the method being performed by a decoder, comprising: Parse the bitstream and perform entropy decoding of the filter parameters; Perform filter parameter decoding; as well as Perform inverse quantization of the filter parameters.

22. The method according to claim 21, wherein, Perform filter parameter decoding using one of the following: New filter decoding mode; New filter intra-frame decoding mode; New filter inter-frame decoding mode; or Copy filter decoding mode; Each of the above modes is used to predict filter coefficients and / or filter parameters.

23. The method according to claim 21 or 22, wherein, The filter parameters include at least one of filter coefficients, decoding parameters, region division information, weighted graph parameters, or other auxiliary information.

24. The method according to claim 22, wherein, If the filter cannot be effectively predicted by any other decoding mode, then the new filter decoding mode is selected.

25. The method according to claim 22, wherein, The new filter intra-frame decoding mode is based on other filter coefficients of the same filter and utilizes the dependencies between filter coefficients of the same filter to predict filter coefficients.

26. The method according to claim 22, wherein, The new inter-frame decoding mode predicts the filter coefficients of the next filter based on the filter coefficients of the previous filter.

27. The method according to claim 22, wherein, The copy filter encoding mode copies all filter coefficients and is used to apply the same filter to another frame.

28. The method according to claim 22, wherein, The new filter inter-frame decoding mode and the copy filter decoding mode use filter parameters from previous filters to predict the current filter, wherein the filter on which the prediction of the current filter is based is indicated by an index, and the filters in the reference list are filters that have been sent for some previously encoded frames.

29. The method according to claim 21, wherein, The filter parameters are determined based on an adaptive parameter set, and the system code is used to perform entropy decoding of the filter parameters.

30. The method according to claim 29, wherein, Simple structured codes are used to perform entropy decoding of the filter parameters.

31. The method according to claim 29, wherein, Exponential Golomb codes, Golomb-Rice codes, or k-order exponential Golomb codes are used to perform entropy decoding on the filter parameters.

32. The method according to any one of claims 29 to 31, wherein, Use separate chroma and luminance filters.

33. The method according to claim 32, wherein, Use on / off flags for the luminance filter and the chrominance filter.

34. The method according to any one of the preceding claims, wherein, The method is used for an adaptive reference image upsampling scheme.

35. The method according to any one of the preceding claims, wherein, The method is used after the interpolation filter and before any other processing steps.

36. A computer-readable medium having stored thereon computer-executable instructions, which, when executed by a computing device, cause the computing device to perform the method of any one of claims 21 to 35.

37. A decoder, comprising: One or more processors; as well as A computer-readable medium having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 21 to 35.

Images