Encoding of parameters for signal enhancement filtering for reference image resampling.

Abstract

A method is provided for performing an entropy coding scheme for a signal enhancement filter. The method includes estimating rate distortion enhancement for the current frame and one or more subsequent frames, and optimizing the coding of parameters based on the estimated rate distortion costs.

Description

[Technical Field]

[0001] This application relates to the field of computer vision, more specifically to video processing and video coding, and more specifically to signal enhancement filtering methods, decoders, encoders, and computer-readable media for reference image resampling and / or image upscaling. [Background technology]

[0002] Current video coding schemes such as H.265 / HEVC (High Efficiency Video Coding) and H.266 / VVC (Versatile Video Coding) support spatial scalability of coded video streams. The resolution of the coded video is adaptively changed during coding, a process known in VVC as Reference Picture Resampling (RPR) or Adaptive Resolution Change (ARC). Furthermore, multi-resolution coding and multi-layer coding allow coded videos to have scalable resolutions. Therefore, the spatial resolution at which the video is coded can change adaptively and does not need to be the same as the video's output or input resolution. The advantage of this added flexibility is that coding lower-resolution video requires a lower bitrate and reduces computational complexity at the expense of losing high-frequency information in the downsampling step.

[0003] To encode video at a lower resolution than the original, a downsampling step and an upsampling step are required in the signal processing chain. In the downsampling step, an anti-aliasing filter is applied to prevent artifacts caused by high-frequency components in the image. In the upsampling process, an interpolation filter is applied to reconstruct luminance values ​​at fractional sample positions.

[0004] In RPR, the resolution of the encoded video stream can change adaptively. Therefore, the encoder can encode portions of the video stream at a lower resolution. RPR is applied to interpretation whenever the image uses a reference image with a different resolution than the current image in interpretation. This step requires applying a resampling operation to map the reference image block to the same spatial resolution as the current image.

[0005] Multilayer encoding encodes video at different resolution layers. In the first step, the video is encoded at the lowest resolution layer. To generate the video streams for the next layer, the video is upsampled, and if necessary, the residuals are encoded to apply further processing steps. Depending on the number of layers, this processing can be applied multiple times.

[0006] Finding the optimal high-resolution representation from a low-resolution image is a crucial part of the encoding scheme described above. One common method is to apply a series of polyphase finite impulse response (FIR) interpolation filters. While these filters certainly provide a good approximation to high-resolution image content, they cannot recover information lost during the downsampling process and are constrained by linear filtering operations. As a result, upsampled images are typically blurred. Image sharpening operations can improve image quality. However, linear high-pass filters frequently introduce artifacts such as overshoot and ringing. Furthermore, distortion from downsampling and upsampling depends on the encoding quality of the image content and video (influenced by the quantization parameter (QP) value).

[0007] Applying adaptive filters with local weighting is a way to address these problems. Local weighting can be applied to smoothly increase or decrease the intensity of the filter in local regions. One possible weighting method is to increase the filter intensity in edge regions and decrease it in regions where ringing frequently occurs. With such a setting, the optimized filter can amplify high-frequency components without causing significant ringing. This is particularly useful in upsampling scenarios of images where amplification of high-frequency components is required to sharpen blurred edges. Adaptive filters are necessary to address encoding artifacts and the different characteristics of video content.

[0008] The proposed method requires the transmission of several pieces of side information. This side information includes several flags, filter coefficients, region partitioning information, and mode parameters. In video coding applications, minimizing the required transmission rate is essential. This means minimizing the number of extra bits. To reduce the number of bits, we propose a coefficient coding scheme that utilizes redundancy in the information that needs to be transmitted. [Overview of the Initiative]

[0009] Embodiments of this application provide a method, decoder, encoder, and computer-readable medium for signal enhancement filtering for reference image resampling and / or image upscaling, overcoming the problems associated with conventional arrangements.

[0010] According to a first aspect, a computer implementation method is provided for performing an entropy coding scheme for a signal enhancement filter, which is performed by an encoder. The method includes optimizing the coding of parameters based on the estimated rate distortion cost by estimating rate distortion enhancement for the current frame and one or more subsequent frames.

[0011] In some embodiments, the signal enhancement filter is an edge guide signal enhancement filter.

[0012] In some embodiments, estimating rate distortion enhancement includes calculating rate distortion enhancement and summing the rate distortion enhancement of one or more subsequent frames, where the current frame is included in the reference picture list of the one or more subsequent frames.

[0013] In some embodiments, the method further includes performing quantization of parameters, performing encoding of filter parameters, and performing entropy encoding on the filter parameters.

[0014] In some embodiments, quantization of parameters includes mapping continuous values to a finite set of codewords.

[0015] In some embodiments, encoding of filter parameters is performed using a new filter encoding mode, a new intra-filter encoding mode, a new inter-filter encoding mode, and a replicated filter encoding mode, where each of these modes is used to predict filter coefficients and / or filter parameters.

[0016] In some embodiments, when the filter cannot be effectively predicted by any other encoding mode, the new filter encoding mode is selected.

[0017] In some embodiments, the new intra-filter encoding mode predicts filter coefficients by utilizing the dependency between filter coefficients of the same filter based on other filter coefficients of the same filter.

[0018] In some embodiments, the new inter-filter encoding mode predicts the filter coefficients of the next filter based on the filter coefficients of the immediately preceding filter.

[0019] In some embodiments, the copy filter coding mode is used to copy all filter coefficients and apply the same filter to another frame.

[0020] In some embodiments, the new filter inter-coding mode and the copy filter coding mode predict the current filter using filter parameters from the previous filter, where the filter on which the prediction of the current filter depends is indicated by an index, and the filters in the reference list are the filters transmitted for previously encoded frames.

[0021] In some embodiments, the filter parameters are signaled in an adaptive parameter set, and a systematic code is used to perform entropy coding of the filter parameters.

[0022] In some embodiments, a simple structured code is used to perform entropy coding of the filter parameters.

[0023] In some embodiments, an exponential Golomb code, a Golomb-Rice code, or a k-th order exponential Golomb code is used to perform entropy coding on the filter parameters.

[0024] In some embodiments, separate chrominance filters and luminance filters are used.

[0025] In some embodiments, on / off flags for the luminance filter and the chrominance filter are used.

[0026] In some embodiments, the method is used for an upsampling scheme of an adaptive reference image.

[0027] According to a second aspect, a computer-readable medium is provided which stores computer-executable instructions that, when executed by a computing device, cause the computing device to perform any of the methods described in relation to the first aspect.

[0028] According to a third aspect, an encoder is provided. The encoder comprises one or more processors and a computer-readable medium storing computer-executable instructions, the computer-executable instructions causing one or more processors to perform any of the methods described in relation to the first aspect when executed by the one or more processors.

[0029] These and other aspects of the present application will become more readily apparent from the following description of the embodiments. [Brief explanation of the drawing]

[0030] [Figure 1A] A flowchart illustrating the operation of a method for performing an entropy coding scheme for a signal enhancement filter executed by an encoder, according to an embodiment of the present invention, is shown. [Figure 1B] A flowchart illustrating an embodiment of the present invention shows the operation of optimizing parameter coding based on estimated rate distortion costs. [Figure 2] The following describes four filter coding modes for predicting filter coefficients and / or filter parameters according to embodiments of the present invention. [Figure 3A] This example demonstrates the use of individual encoding modes according to an embodiment of the present invention. [Figure 3B] This embodiment of the present invention demonstrates the use of the same encoding mode across the entire filter. [Figure 4] An example of filter prediction for an intra-filter coding mode according to an embodiment of the present invention is shown. [Figure 5]This invention illustrates a filter prediction for an interfilter coding mode according to an embodiment of the present invention. [Figure 6] This example of the present invention shows filter selection for a replication filter encoding mode. [Figure 7] A schematic diagram of a decoder according to various embodiments of the present invention is shown. [Figure 8] A schematic diagram of an encoder according to various embodiments of the present invention is shown. [Modes for carrying out the invention]

[0031] The following describes an example of an embodiment, with reference to the drawings.

[0032] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the drawings.

[0033] These technical solutions can be applied to H.265 / HEVC or H.266 / VVC video coding systems, particularly for the implementation of RPR, ARC, multi-resolution coding, and multi-layer coding. However, it should be understood that these technical solutions can be applied to any other video coding system involving upsampling. Furthermore, although these principles are primarily explained in reference to video processing, they are also applicable to other data formats, including image processing or even audio processing.

[0034] In these embodiments, "video" refers to one or more images. In other words, a video can include one or more images. Images may also be called "pictures."

[0035] An "encoder" is a device that can encode data into a bitstream, while a "decoder" is a device that can decode a bitstream to obtain encoded data or an approximate value of the encoded data. A "bitstream" contains a sequence of bits.

[0036] "Intra-prediction" and "inter-prediction" are two prediction operations within the HEVC and VVC frameworks for the decoder to process the received bitstream to obtain the original signal. In the embodiment, "original signal" or "original video" refers to the data before encoding at encoder 20. A reference sample in the embodiment may refer to spatially and / or temporally spaced image data used to predict an image (or region of an image). In the encoder, intra- and inter-prediction operations are used to make a rate-distortion (RD) determination.

[0037] More specifically, intra-prediction involves predicting data in a space within a single image and does not refer to other images (which are temporally separated). In other words, data from the first region of an image is used to predict data from another region of the same image, but does not depend on other images that are temporally separated. In this context, the data from the first region of an image is considered a "reference sample."

[0038] Interpretation involves predicting data between images that are spaced apart in time. In other words, data from the first region of the first image is used to predict data from the second region of the second image. The first and second regions may or may not be spatially separated from each other. In this context, the data from the first region of the first image is considered a "reference sample." It should be further noted that interpretation may use multiple reference regions from different images at once (i.e., for a single prediction operation).

[0039] In the examples, "residual" may refer to a value obtained based on the original value of the image region and the predicted value of the image region (for example, the difference between the original value and the predicted value).

[0040] In the examples, "block" may refer to a portion of an image. For example, an image may be divided into two or more blocks. However, this is merely an example. If the image is not divided, "block" may refer to the entire image.

[0041] A “signal enhancement filter” may refer to a filter used to enhance a signal, particularly an upsampled signal. Typically, in the embodiments described, a signal enhancement filter is configured to reduce edge blurring (i.e., to sharpen image blocks). However, embodiments are not limited to this, and in other embodiments, a signal enhancement filter may be configured to provide alternative or additional signal enhancement, such as the removal of blocking artifacts and / or ringing artifacts.

[0042] Figure 1A shows a flowchart illustrating the operation of implementing an entropy coding scheme for a signal enhancement filter.

[0043] The flowchart in Figure 1A illustrates that in step 101, the parameter coding is optimized based on the estimated rate distortion cost by estimating the rate distortion enhancement of the current frame and one or more subsequent frames. Before continuing with a further description of the drawings, some aspects of the present invention will be described in more detail.

[0044] The proposed invention provides a method for encoding parameters for adaptive reference image upsampling. This enables an efficient encoding scheme that takes into account transmission requirements and side information characteristics in the adaptive parameter set.

[0045] In some embodiments, the encoding scheme optimizes rate distortion, which means that the encoding scheme affects distortion in downstream tasks (image upsampling). Note that there are two types of distortion. The first type of distortion is distortion in downstream tasks, i.e., distortion after applying enhancement filters to some encoded signals. The second type of distortion is distortion of the encoding parameters.

[0046] Embodiments of the present invention primarily rely on two parts / steps: a prediction scheme for filter parameters and coding of filter coefficients by low-complexity entropy coding.

[0047] In some embodiments, the prediction scheme uses the interdependence between different parameters. This involves predicting the parameters based on previously decoded parameters of the same parameter set, thereby transmitting the difference between the predicted and actual coefficients. Transmitting the difference between the predicted and actual coefficients is meaningful when the entropy of the residuals is less than the entropy of the parameters. Furthermore, parameters from previous frames can be reused without encoding the residuals. Finally, it is possible to choose to modify the residuals or the transmitted parameters to reduce the encoding cost that affects distortion in downstream tasks. Therefore, rate distortion optimization should be performed for modifications.

[0048] Video coding applications (e.g., RPR and multi-layer / multi-resolution coding) require upsampling. These methods typically apply interpolation filters to generate sub-pixel values ​​for the upsampling filter. Exemplarily, the method according to an embodiment of the present invention is applied immediately after the interpolation filter and before any other processing step. The objective of the method is to reduce distortion caused by low-resolution video coding. This distortion includes loss of high-frequency information and distortion caused by video coding. In this scenario, for example, a default upsampling filter is used for the initial resolution change.

[0049] The methods according to embodiments of the present invention can be applied independently thereafter and add an optional enhancement step rather than modifying an existing resampling process. Embodiments of the present invention specifically describe possible implementations of the parameter coding scheme for the aforementioned upsampling enhancement scheme.

[0050] Typically, efficiently encoding additional data being transmitted is crucial for improving the overall efficiency of the method. Gains are only gained if the method improves rate distortion performance. This means that applying the method requires a lower bitrate to achieve a specific distortion level d compared to not applying it.

[0051] d 方法 (r) <d 基準 (r) Therefore, the rate for encoding a bitstream using the method according to the embodiment of the present invention is defined by the following equation:

[0052] r=r エンコーダ +r 方法 Below, we will mainly discuss the parameter r, which is directly used in this method. 方法 and distortion d 方法 We focus on this. All other parameters are not directly affected by the proposed encoding method. However, changes in content can have indirect effects. Calculating these indirect effects is difficult and requires performing the encoding itself. This is very demanding in terms of computation time and program complexity, so it is recommended to model these effects. For this purpose, a distinction is made between the direct and indirect effects of the method. The direct effect is understood as a reduction in distortion in the current image by applying the method according to the embodiments of the present invention. There are two indirect effects. The first indirect effect is that an increase in the number of bits in the current frame allows rate control to decide to encode subsequent frames at a relatively lower rate. The second indirect effect occurs only when the current image is referenced by some upsampled image. In this case, interpretation is performed on the enhanced image content. In most cases, it is assumed that better quality source images lead to better predictions and consequently smaller residuals. Therefore, quality improvements can have a positive impact on other images. Both of these effects must be considered to make an optimal rate distortion determination.

[0053] The direct impact can be estimated by measuring the strain after applying the reinforcement filter and comparing it to the ground truth, and by estimating the additional rate required to signal the adaptive parameter set. To reduce computational complexity, some embodiments use simplified measurements or approximations of the strain.

[0054] Accurately estimating indirect effects is more difficult. Therefore, it is proposed to use a simplified model.

[0055] In some embodiments, when the proposed method is applied to the current frame, the model calculates the RD gain of other frames. In some embodiments, the RD enhancement is calculated and summed for all frames in the reference picture list, including the current frame. These frames can be easily inferred from the encoder settings. The model considers the quality of the frame before filtering is applied, and the quality, rate, and encoding settings (e.g., QP, Reference Picture List (RPL), QP of all images in the RPL, image size of the image in the RPL) of the frame after filtering is applied. The RD enhancement of the image is then calculated. A simple model for calculating the RD enhancement for a P-frame or B-frame n is as follows:

[0056]

number

number

number

[0057] As a result of this formula, an RD offset is obtained for each frame that references the current frame. By adding these RD offsets, an overall RD improvement for the reference image is obtained. Note that this scheme may also take into account that the reference image in RPL may reference the current frame. To address this, the previous formula can be applied recursively. However, for the sake of brevity, this point is omitted. Thus, ΔRD B / P If (n) is defined as the RD difference applied to the current frame, the total RD gain is calculated using the following formula:

[0058]

number

[0059] Furthermore, filter coefficients and (conceptually continuous) weighted map function parameters exist, which are quantized or modified before encoding and transmission in order to have a limited number of good, compressible codewords. These parameters include the filter coefficients of the luminance filter and chromaticity filter, as well as the weighted map parameters. The quantization and encoding scheme is described below.

[0060] The following section contains a detailed description of the parameters that may be used in such a configuration. This includes optional parameters that can be selected and used to adapt to and signal the behavior of the proposed filter. First, there are luminance and chromaticity flags. These flags indicate whether the filter is applied to the luminance and / or chromaticity components. If one of these flags is false, no parameters are encoded for that channel. Each of the luminance and chromaticity filters has a weighting map function (which may itself have parameters) and a filter. The weighting map is specified by a parameterized function calculated from the encoded video. The parameters of the weighting map function depend on the type of weighting map and are signaled based on the type of weighting map function. A set of parameterized weighting map functions is predefined. The selected weighting map function may be signaled or selected on a content-dependent basis. In addition to the weighting map, the filter parameters are transmitted. These parameters are the filter shape, and possibly quantization parameters and filter coefficients. Furthermore, encoding information may also be signaled. The encoded information may be related to the behavior of quantization, parameter prediction, or entropy coding. This information may be transmitted individually to the luminance filter and the chromaticity filter, or to both filters simultaneously. Finally, information about the application location of each filter may be signaled. This means that the filter may be applied to only a portion of the image. Thus, the portion to which each filter is applied may be signaled in the bitstream or inferred from the video content and / or encoded information. Note that multiple filters may be encoded using different parameters. Furthermore, two filters may be applied to the same spatial position in the frame.

[0061] Returning to FIG. 1B, a flowchart of an operation for optimizing the encoding of parameters based on the estimated rate distortion cost is shown. In step 201, quantization of the parameters is performed. In step 202, encoding of the filter parameters is performed using four different filter encoding modes, which will be described with reference to FIG. 2. Finally, in step 203, entropy encoding of the filter parameters is performed.

[0062] Looking more closely at step 201 of parameter quantization, it becomes clear that the filter parameters need to be mapped to integer values for transmission and calculation of parameters in video encoding. In some embodiments, the mapping is performed by min ,t max clipping the floating point values to a range [t scale . Then, these values are scaled by a scaling factor s scale and rounded to the nearest integer. In some embodiments, the overall process is described by the following equation.

[0063] a quantized =round(min(max(a,t min ),t max )s scale ) Thereby, the number of quantization intervals is given by n bins =(t max -t min )s scale +1.

[0064] [[ID=**36**]]t[[ID=**37**]] min [[ID=**38**]]、t[[ID=**39**]] max [[ID=**40**]]、およびs [[ID=**41**]] scale [[ID=**42**]]が整数値であると仮定される。

[0065] In some embodiments, the bit depth applied to parameter quantization is 12 bits. Thereby, the number of bits of the quantized value is 12, and n bins =2 12It has a number of values. However, in test scenarios, it was found that it also works with any number of bins in the range of 10 to 14. On the one hand, a higher scaling factor results in lower quantization error, thereby resulting in a more accurate representation of the number. On the other hand, it requires a larger number of binary numbers, which usually results in a higher coding cost. The upper and lower thresholds should be chosen so that the clipping error does not significantly affect performance. However, the smaller the set range, the fewer binary numbers are needed to achieve the same quantization error, thereby reducing the coding cost. Finding good quantization parameters is important for the final performance. For example, clipping the filter coefficients too early can significantly affect performance. Note that this is just one example of a quantization scheme that may be used here. Rate-distortion optimized quantization may also be used.

[0066] Furthermore, in some embodiments, different quantization step sizes are applied depending on the filter coefficients. For example, it may be reasonable to use a smaller quantization relative to the filter's mean value because the error due to the offset at the filter's low-frequency coefficients is usually larger than the error due to the high-frequency coefficients.

[0067] In some embodiments, depending on the characteristics of the data, the parameters of the quantization scheme are signaled within a parameter set or fixed for all encoding settings. Adaptive quantization schemes may be advantageous when the content, particularly in terms of the dynamics of the encoded signal and residuals, is highly variable. However, when the variation is less, it may be reasonable to pre-estimate the optimal parameters to reduce signaling costs.

[0068] Figure 2 shows the encoding of filter parameters and discloses four filter encoding modes: a novel filter encoding mode 401, a novel filter intra encoding mode 402, a novel filter inter encoding mode 403, and a duplicate filter encoding mode 404. Each of these modes is used to predict filter coefficients and / or filter parameters.

[0069] In some embodiments, the filter parameters include at least one of the following: filter coefficients, coding parameters, region segmentation information (if multiple filters are applied), weighting map parameters, or other side information. In some embodiments, the filter coefficients cause the greatest coding cost; therefore, the greatest effort is spent reducing the coding cost of the filter coefficients. Each mode is signaled by a mode flag. All other parameters are derived based on the mode flag. In some embodiments, the mode parameters are signaled individually for each filter, or for each of the Y, U, and V channels, or for the luminance and chromaticity channels. This means that in some embodiments, there is a coding scheme that first signals the luminance and chromaticity flags, and then sends the mode parameters individually for each channel when a filter is enabled on that channel. In other embodiments, the coding mode is signaled first.

[0070] Figure 3A shows an encoding scheme in which the luminance and chromaticity flags are first signaled, and then the mode parameters are individually signaled for each channel when the filter is enabled for that channel. Figure 3B shows an encoding scheme in which the encoding mode is first signaled. The embodiment shown in Figure 3B has the advantage of lower encoding cost because the encoding mode only needs to be signaled once, and the luminance / chromaticity flags are also derived based on the encoding mode.

[0071] Before describing the new filter-intra coding mode 402 with reference to Figure 4, we will first describe the new filter-intra coding mode 401.

[0072] In some embodiments, if the filter cannot be effectively predicted in any of the other encoding modes, a novel filter encoding mode 401 is selected. In this case, the filter coefficients are encoded in their quantized representation. This means that no prediction is applied to the filter coefficients, except in some embodiments where one coefficient is replaced by the mean value of the filter. It is proposed to use this prediction on the coefficient with the highest entropy because it works more reliably due to the properties of the encoding scheme. Mathematically, all predictions assume that there is a certain statistical dependence with respect to the filter coefficients. If these statistical dependencies hold with high probability, the predictions and the true values ​​will be similar in most cases. However, if these assumptions do not hold, it may be possible to skip the predictions entirely, but at a lower cost.

[0073] Referring to Figure 4, the novel filter-intra coding mode is described here. The novel filter-intra coding mode utilizes the dependence between the filter coefficients of the same filter. A series of n filter coefficients [a1, ... a n Assume that we need to encode ]. Without loss of generality, the encoding / decoding order can be set in numerical order of the index. That is, first the filter coefficient a1 is encoded, and then the filter coefficient a i These are encoded in ascending order. This results in the filter coefficient a i The filter coefficients are a1, ..., a i-1 It can be predicted using the current filter coefficient a. i There may be some dependency on the coefficients. For each coefficient, a learned context / predictive model may be used, depending on the case. Therefore, this model does not need to be equivalent for all coefficients.

[0074] For example, we can employ a very simple model where the filter coefficient a1 is encoded unchanged, and all subsequent coefficients are the difference c from the previous coefficient. i =a i -a (i-1) It is encoded as follows. The last filter coefficient is the average value of all filter coefficients.

number

[0075] Figure 5 shows in more detail a novel filter intercoding mode 403 that performs prediction of the next filter coefficient based on the previous filter. This allows for the prediction of the filter coefficient a t,iThe current filter is predicted by the filter coefficients at time step t+j, where j is some offset in the time domain. Furthermore, multiple filters are used for combined prediction. A simple example is having a reference set of picture order count (POC) adaptations of filters to realize a novel filter intermode. For example, the reference filter set contains all filters corresponding to filters in the reference image set. Another option is to always remember the n filters that are closest in time. The current filter can be predicted by signaling the index of the predicted filter in the reference image list. Then, for example, the current filter can be predicted directly by the selected reference filter. Subsequently, the residual between the current filter and the reference filter is encoded. This is meaningful if we can assume that the filter coefficients remain similar in subsequent frames. This depends on the assumption of temporal correlation of the encoded video. If the error signals of the video have similar statistics for frames that are close in time / encoding order, then the filters are also likely to be similar. As a result, there is a small residual signal. This small residual signal can be utilized by entropy coding of the residuals, thereby reducing the cost of each symbol. Figure 5 illustrates how such a prediction scheme might work. Here, there is a set of k filters used for prediction, each filter potentially used to make a prediction for filter a. Next, the residuals between the predicted values ​​and the residuals are calculated. Then, the coding cost of each reference filter is estimated, and the filter with the lowest coding cost is selected. Note that some transformation may be applied to the residuals. A notable transformation is replacing one filter coefficient with the sum of the filter coefficients. Furthermore, some kind of intra-prediction may be applied after the residual calculation, or a combination prediction using multiple filters may be performed.

[0076] Figure 6 shows the filter duplication mode, which duplicates all filter coefficients and other potential filter parameters, thereby reducing the cost of this mode but decreasing its flexibility. In some embodiments, this mode can be used to apply the same filter to another frame, and this method may be preferable when the encoded video and residuals of two or more frames are very similar. In such cases, a filter optimized for some previous frames may also improve the current frame. In most cases, the quality improvement may be reduced. However, due to the reduction in encoding cost, this may be preferable from the standpoint of RD cost. Figure 6 shows an exemplary implementation of the filter duplication mode optimization. Thus, a previously signaled set of filters may exist in the reference filter list. The reference filter list contains a subset of filters bi previously available for decoding.

[0077] The new filter intermode and filter duplication mode predict the current filter using the filter parameters from the previous filter. The filters on which the current filter is predicted are indicated by their index. Filters in the reference list may be filters that were previously sent for some frame that was encoded earlier. Other options are pre-calculated filters and filters inferred from the video content. Typically, predictions are made from these filters after several frames have been processed, and there may be many filters available. Since the number of bits required to encode the filter index increases with the number of indices that need to be represented, we propose selecting only a subset of filters for the reference list. The selected filters may depend on previously signaled parameters and other factors such as selected weighting maps, luminance and chromaticity filter shapes, etc.

[0078] Finally, we will analyze in more detail the step of performing entropy coding of the filter parameters. The parameters of the reinforcement filter are signaled by an Adaptive Parameter Set (APS). This is a high-level parameter set, meaning that not all entropy coding tools are available. For example, arithmetic coding is not typically used at this level. Therefore, the methods described below are also limited to the entropy methods that can be applied or applied within such parameter sets.

[0079] The following outlines an example implementation of entropy coding. Note that this is merely one example, and many suitable modifications are possible; several possible adjustments will also be mentioned in the following explanation.

[0080] The primary goal of the entropy coding step is to reduce the average coding cost for coding symbols. This is typically achieved by assigning longer codewords to less frequent symbols and shorter codewords to more frequent codewords. Binary entropy is a measure of the minimum average number of bits required to code symbols for a given probability distribution. In a given application, we code the residuals of predicted filter coefficients. In this case, we need a code that can code all possible symbols / filter coefficients as efficiently as possible. If we use a code to code symbols instead of an entropy-optimal code, the efficiency of that code can be measured by the Kullback-Leibler divergence, which gives the average increase in bits per symbol. It should be noted that, for reasons of complexity, an entropy-optimal code cannot be used. This means that entropy-optimal coding or arithmetic coding may be too complex to apply at a given level. However, entropy-optimal coding or arithmetic coding are still theoretical options and may be used if the design paradigm allows it. Currently, a simple structured code is used at the APS level. More specifically, exponential Golomb codes are frequently used to encode symbols within APS. These codes can be used with both signed and unsigned integers and can encode virtually any possible integer. The codes assume that the probability of a symbol decreases exponentially with its magnitude. This is significant for encoding residual signals, given the small size of the residual signal when the basic assumptions of the prediction mode hold. Therefore, in terms of magnitude, there should be a great many very small values ​​and a few very large, infrequent values. However, other options exist, such as Golomb-Rice codes and k-th exponential Golomb codes, which are also suitable for structured coding of filter coefficients. We propose using structured codes (e.g., the codes described above) when low complexity is required.Furthermore, we propose either estimating the optimal hyperparameters from the test set or signaling the entropy-coded hyperparameters. A good example of encoding data after the prediction process is using exponential Golomb coding. In our application, we assume this coding has a slightly higher probability of low magnitude values. This makes it suboptimal for RD. However, this cannot be considered a general conclusion as it is highly dependent on the data type and coding scheme.

[0081] Figure 7 shows a schematic diagram of a decoder 10 according to an embodiment. Specifically, Figure 7 shows a schematic diagram of a decoder 10 configured to perform any decoder-side method described herein. For brevity, a detailed description of these is omitted here.

[0082] As shown in Figure 7, the decoder 10 comprises a processor 11 and a computer-readable medium 12. The processor 11 and the computer-readable medium 12 may be connected by a bus system. The computer-readable medium is configured to store programs, instructions, or code. The processor 11 is configured to perform the operations in the decoder-side method embodiment herein by executing the programs, instructions, or code in the computer-readable medium 12.

[0083] Accordingly, in the embodiment, the computer-readable medium 12 is configured to store a computer program executable by the processor 11, and the processor 11 is configured to perform the steps in any decoder-side method described herein by executing the computer program.

[0084] Figure 8 shows a schematic diagram of an encoder 20 according to an embodiment. Specifically, Figure 8 shows a schematic diagram of an encoder 20 configured to perform any encoder-side method described herein. For brevity, detailed descriptions of these are omitted here.

[0085] As shown in Figure 8, the encoder 20 comprises a processor 21 and a computer-readable medium 22. The processor 21 and the computer-readable medium 22 may be connected by a bus system. The computer-readable medium is configured to store programs, instructions, or code. The processor 21 is configured to perform the operations in the decoder-side method embodiment herein by executing the programs, instructions, or code in the computer-readable medium 22.

[0086] Accordingly, in the embodiment, the computer-readable medium 22 is configured to store a computer program executable by the processor 21, and the processor 21 is configured to perform the steps in any decoder-side method described herein by executing the computer program.

[0087] Embodiments of the present invention can further provide a computer-readable medium having computer-executable instructions for causing one or more processors of a computing device to perform the method of the embodiment of the present invention.

[0088] Examples of computer-readable media include, but are not limited to, volatile and non-volatile media, removable and non-removable media, and include solid-state memory, removable disks, hard disk drives, magnetic media, and optical disks. Generally, computer-readable media include any kind of media suitable for storing, encoding, or carrying a set of instructions, which can be executed by one or more computers to perform one or more of the processes and features described herein.

[0089] It is understandable that the functions of each component described above are not limited to the methods described above, but can be combined in many ways. For example, in some embodiments, multiple functions of the described equipment may be incorporated into a single device. In other embodiments, at least one function of the described equipment may be divided among multiple independent (or distributed) devices.

[0090] Conditional language such as "may" is typically used to indicate that a feature / step is used in a particular embodiment, but that alternative embodiments may include alternative features or omit them entirely.

[0091] Furthermore, it should be understood that the method steps are not limited to the specific order described, and these steps can be combined in any other suitable order. In some embodiments, this may result in some method steps being performed in parallel. Moreover, in some embodiments, certain method steps may be omitted entirely.

[0092] While several embodiments have been described, these embodiments are used to illustrate the overall teachings of the invention and should be understood to be capable of various modifications without departing from the scope of the invention. The scope of the invention should be construed in accordance with the appended claims and their equivalents.

[0093] Those skilled in the art will find many further variations and modifications suggested by referring to the exemplary embodiments described above. These embodiments are provided merely as examples and are not intended to limit the scope of the invention as defined by the appended claims.

Claims

1. A method for performing an entropy coding scheme for a signal enhancement filter, which is performed by an encoder, A method comprising optimizing the encoding of parameters based on the estimated rate distortion cost by estimating rate distortion enhancement for the current frame and one or more subsequent frames.

2. Estimating the rate distortion enhancement includes calculating the rate distortion enhancement and summing the rate distortion enhancements of one or more subsequent frames, wherein the current frame is included in the reference image list of one or more subsequent frames. The method according to claim 1.

3. The aforementioned method, Performing quantization of filter parameters, Performing the encoding of filter parameters, This further includes performing entropy coding on the filter parameters, The method according to claim 1.

4. The quantization of the aforementioned parameters includes mapping continuous values ​​to a finite set of codewords. The method according to claim 3.

5. The encoding of filter parameters is, New filter coding mode, New filter intra coding mode, New filter intercoding mode, or It is performed using one of the replication filter coding modes, Each of the above-mentioned new filter coding mode, new filter intra coding mode, new filter inter coding mode, or replica filter coding mode is used to predict filter coefficients and / or filter parameters. The method according to claim 1.

6. The filter parameters include at least one of the following: filter coefficients, coding parameters, region segmentation information, weighted map parameters, or other side information. The method according to any one of claims 3 to 5.

7. If the aforementioned filter cannot be effectively predicted by any other encoding mode, the new filter encoding mode is selected. The method according to claim 5.

8. The novel filter intra coding mode predicts filter coefficients by utilizing the dependencies between filter coefficients of the same filter, based on other filter coefficients of the same filter. The method according to claim 5.

9. The aforementioned new filter intercoding mode performs a prediction of the filter coefficients of the next filter based on the filter coefficients of the previous filter. The method according to claim 5.

10. The aforementioned duplicate filter encoding mode is used to duplicate all filter coefficients and apply the same filter to another frame. The method according to claim 5.

11. The aforementioned new filter intercoding mode and the aforementioned duplicate filter coding mode predict the current filter using the filter parameters from the previous filter, and the filter on which to predict the current filter is indicated by an index, and the filters in the reference list are the filters sent for previously coded frames. The method according to claim 5.

12. The filter parameters are signaled in an adaptive parameter set, and a systematic code is used to perform entropy coding of the filter parameters. The method according to claim 1.

13. A simple structured code is used to perform entropy coding of the filter parameters. The method according to claim 12.

14. An exponential Golomb code, a Golomb-Rice code, or a k-th exponential Golomb code is used to perform entropy coding on the filter parameters. The method according to claim 12.

15. Individual chromaticity and luminance filters are used. The method according to any one of claims 12 to 14.

16. On / off flags are used for the luminance filter and the chromaticity filter. The method according to claim 15.

17. The above method is used for an upsampling scheme of an adaptive reference image. The method according to any one of the preceding claims.

18. The above method is used after the interpolation filter and before any other processing step. The method according to any one of the preceding claims.

19. A computer-readable medium storing computer-executable instructions that, when executed by a computing device, cause the computing device to perform the method according to any one of claims 1 to 18.

20. It is an encoder, One or more processors, An encoder comprising: a computer-readable medium storing computer-executable instructions for causing the one or more processors to perform the method according to any one of claims 1 to 18 when executed by the one or more processors;

21. A method for performing an entropy decoding scheme for a signal enhancement filter, which is carried out by a decoder, This involves analyzing the bitstream and performing entropy decoding of the filter parameters, Performing the decoding of filter parameters, A method that includes performing inverse quantization of filter parameters.

22. Decoding of filter parameters is New filter decoding mode, New filter intra decoding mode, New filter inter-decoding mode, or It is performed using one of the replication filter decoding modes, Each of the modes among the novel filter decoding mode, the novel filter intra decoding mode, the novel filter inter decoding mode, or the replica filter decoding mode is used to predict filter coefficients and / or filter parameters. The method according to claim 21.

23. The filter parameters include at least one of the following: filter coefficients, decoding parameters, region segmentation information, weighted map parameters, or other side information. The method according to claim 21 or 22.

24. If the aforementioned filter cannot be effectively predicted by any other decoding mode, the new filter decoding mode is selected. The method according to claim 22.

25. The aforementioned novel filter intra-decoding mode predicts filter coefficients by utilizing the dependencies between filter coefficients of the same filter, based on other filter coefficients of the same filter. The method according to claim 22.

26. The aforementioned new filter inter-decoding mode performs a prediction of the filter coefficients of the next filter based on the filter coefficients of the previous filter. The method according to claim 22.

27. The aforementioned duplicate filter encoding mode is used to duplicate all filter coefficients and apply the same filter to another frame. The method according to claim 22.

28. The aforementioned new filter inter-decoding mode and the aforementioned duplicate filter decoding mode predict the current filter using the filter parameters from the previous filter, and the filter on which to predict the current filter is indicated by an index, and the filters in the reference list are filters that were sent for previously encoded frames. The method according to claim 22.

29. The filter parameters are determined based on an adaptive parameter set, and a systematic code is used to perform entropy decoding of the filter parameters. The method according to claim 21.

30. A simple structured code is used to perform entropy decoding of the filter parameters. The method according to claim 29.

31. An exponential Golomb code, a Golomb-Rice code, or a k-th exponential Golomb code is used to perform entropy decoding on the filter parameters. The method according to claim 29.

32. Individual chromaticity and luminance filters are used. The method according to any one of claims 29 to 31.

33. On / off flags are used for the luminance filter and the chromaticity filter. The method according to claim 32.

34. The above method is used for an upsampling scheme of an adaptive reference image. The method according to any one of claims 21 to 33.

35. The above method is used after the interpolation filter and before any other processing step. The method according to any one of claims 21 to 34.

36. A computer-readable medium having stored computer-executable instructions that cause a computing device to perform the method according to any one of claims 21 to 35 when executed by the computing device.

37. It is a decoder, One or more processors, A decoder comprising: a computer-readable medium storing computer-executable instructions for causing one or more processors to perform the method according to any one of claims 21 to 35 when executed by the aforementioned one or more processors;

Images