Intra-frame sharpening filter and / or de-ringing filter for video coding
By using sharpening filters and dering filters to process reference samples in video encoding, the natural edge distortion and ringing effect problems caused by intra prediction are solved, and image quality and encoding efficiency are improved.
Patent Information
- Application Number
- CN202411039337.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-06-13
- Filing Date
- 2019-06-13
- Publication Date
- 2024-11-08
AI Technical Summary
In the existing video encoding technology, natural edge distortion and ringing effects caused by intra prediction mechanisms are difficult to maintain image quality while suppressing ringing.
A decoder and encoder are designed to filter reference samples using sharpening filters and/or de-ringing filters, improve prediction accuracy and perceived image quality of prediction blocks, and pass signaling or images in the bitstream. Attributes activate or disable these filters to optimize encoding efficiency.
It effectively suppresses the ringing effect, improves the subjective and objective quality of the image, reduces blurring of natural edges, and improves coding efficiency.
Smart Images

Figure CN118921456A_ABST
Abstract
Description
This application is a divisional application. The application number of the original application is 2019800390982, and the original application date is June 13, 2019. The entire contents of the original application are incorporated into this application by reference. Technical Field
[0001] The present invention relates to a decoder and a method for decoding a block of a current frame of a video from a bitstream. The present invention also relates to an encoder and a method for encoding a block of the current frame in a bitstream. In addition, the present invention also relates to a computer-readable storage medium storing a program code, the program code comprising instructions for executing the above method. Background Art
[0002] In order to reduce the bit rate of video signals, the International Organization for Standardization (ISO) and the International Telecommunication Union (ITU) coding standards apply hybrid video coding using inter-frame prediction and intra-frame prediction combined with transform coding of prediction errors. The current H.264 / AVC and H.265 / HEVC standards use intra-block prediction based on reference samples, which are from coded neighboring blocks. The difference between the original block and the predicted block (called the residual) can then be transformed to the frequency domain, quantized, and encoded by entropy coding using, for example, a discrete cosine transform (DCT) or a discrete sine transform (DST).
[0003] The effectiveness of the prediction greatly affects the amount of residual that needs to be encoded and transmitted. Improving the prediction quality can reduce the amount of residual information and reduce the overall bit rate of the encoded video sequence.
[0004] On the decoding side, information about the prediction type and prediction parameters can be extracted from the bitstream. A prediction block can be generated based on the above information. After that, the residual coefficients are extracted from the bitstream, inverse quantized, transformed, and added to the prediction block to obtain the reconstructed block. Once the complete picture is reconstructed, it passes through a loop filter. The filtered picture obtained goes to the output and is stored in the reference picture buffer for future inter-frame prediction.
[0005] In video coding, conventional intra-frame prediction mechanisms use reference samples from an already coded region to generate a prediction signal for the block to be coded. A series of methods to improve intra-frame prediction have been proposed, some of which have been adopted in the H.265 / HEVC video coding standardization.
[0006] However, known solutions lead to distortion of natural edges in the reference samples and, accordingly, in the predicted blocks generated from these samples. The use of quantization matrices designed to preserve coefficients of low-frequency components can cause more errors in blocks containing natural edges and lead to edge blurring and ringing effects near the reconstructed edges. To reduce the ringing effects, a linear smoothing filter can be used. The problem with this approach is that, while suppressing the ringing, the natural edges may also become blurred. Summary of the invention
[0007] An object of the present invention is to provide a decoder, an encoder, and a method for decoding and encoding, wherein the decoder, the encoder, and the method overcome one or more of the above-mentioned problems in the prior art.
[0008] A first aspect of the present invention provides a decoder for decoding a block of a current frame of a video from a bit stream, the decoder comprising:
[0009] - a reference sample selection unit, for selecting reference samples of the reconstructed part of the current frame,
[0010] - a filter unit for filtering the reference samples, and
[0011] - a block generation unit for generating a prediction of a block based on the filtered reference samples,
[0012] The filter unit includes a sharpening filter and / or a de-ringing filter.
[0013] The decoder of the first aspect comprises a filter unit, which can filter the reference samples using a sharpening filter and / or a de-ringing filter. Experiments show that sharpening and / or de-ringing the reference samples can improve the prediction accuracy and / or the perceived image quality of the prediction block.
[0014] The decoder of the first aspect may further comprise a reconstruction unit for generating a reconstructed video block based on the prediction of the block and a residual video block from the received bitstream.
[0015] In a first implementation of the decoder according to the first aspect, the sharpening filter and / or the deringing filter are non-linear filters.
[0016] Experiments by the inventors have shown that non-linear filters can achieve particularly good results in terms of prediction accuracy and / or perceived image quality.
[0017] In a second embodiment of the decoder according to the first aspect or according to the first embodiment of the first aspect, the decoder is used to activate the sharpening filter and / or the deringing filter based on signaling of information related to the sharpening filter and / or the deringing filter in the bitstream and / or based on image properties of one or more reconstructed blocks of the current frame.
[0018] The decoder of the second embodiment has the following advantages: the sharpening filter and / or the de-ringing filter can be activated or deactivated based on whether it is desired to improve the image quality in a given scene. Explicit signaling in the bitstream allows for an optimal activation / deactivation setting (e.g., based on setting optimization during the encoding process). On the other hand, the activation of the sharpening filter and / or the de-ringing filter based on the image properties of one or more reconstructed blocks of the current frame without explicit signaling can improve the overall coding efficiency.
[0019] In order to avoid the overhead of explicit signaling, the sharpening filter and / or the de-ringing filter may be activated based on, for example, the edge strength of the surrounding blocks, the size of the coded block, etc. For example, the edge strength of the surrounding blocks and / or the size of the coded block may be compared with a threshold, and the sharpening filter and / or the de-ringing filter may be activated based on the threshold comparison result. For example, if the edge strength of one or more nearby blocks exceeds a certain threshold, the sharpening filter and / or the de-ringing filter may be activated. This may be accomplished by, for example, calculating a reference sample (e.g., Figure 7 As shown, for example, the absolute value of the derivative of reference samples arranged in a horizontal row close to the upper edge (upper line) of the current block or reference samples arranged in a vertical column close to the left edge (left line) of the current block estimates the edge strength, and the reference samples are used for intra-frame prediction of the current block.
[0020] As with state-of-the-art video compression techniques, the compression rate and the corresponding level of compression artifacts can be controlled by a quantization parameter (QP). The quantization parameter can be set at the level of each sequence, frame, slice, or block. Ringing is one of the strongest compression artifacts caused by the quantization process. The higher the compression level (quantization parameter value), the stronger the ringing effect. In some embodiments, a de-ringing filter that suppresses the ringing effect is only applied to higher compression levels and corresponding higher QPs (or larger quantization parameter values). Therefore, the de-ringing filter can be activated based on the QP value of the current block, for example, when a predefined quantization parameter value (or quantization parameter threshold) is exceeded.
[0021] For example, in an embodiment, a QP may be obtained, and a de-ringing filter may or may not be applied based on (or depending on) a quantization parameter. For example, a QP threshold may be used to determine whether to apply a de-ringing filter (e.g., if the QP is below a threshold, the de-ringing filter is not applied, if the QP is above or exceeds a first threshold, the de-ringing filter is applied; if the threshold is a valid QP, an embodiment may be configured in either manner, such as applying a de-ringing filter or not applying a de-ringing filter).
[0022] In another embodiment, different de-ringing filters, for example, de-ringing filters of different strengths, may be applied based on (or depending on) the quantization parameter. For example, a first QP threshold may be used to determine whether to apply a de-ringing filter (e.g., if the QP is lower than the first threshold, the de-ringing filter is not applied, if the QP is higher than or exceeds the first threshold, the first de-ringing filter is applied; if the first threshold is an effective QP, the embodiment may be configured in either manner, such as applying the first de-ringing filter or not applying the first de-ringing filter), and a second threshold may be used to determine whether to apply a stronger de-ringing filter (e.g., if the QP is lower than the second threshold, the first de-ringing filter is applied, if the QP is higher than or exceeds the second threshold, the second de-ringing filter is applied, and the second de-ringing filter is stronger than the first de-ringing filter in reducing the ringing effect; if the second threshold is an effective QP, the embodiment may be configured in either manner, such as applying the first de-ringing filter or the second de-ringing filter).
[0023] Another embodiment may include more than two de-ringing filters and correspondingly more than two thresholds to determine whether and / or which de-ringing filter to apply.
[0024] Another embodiment may use a list or table (e.g., a lookup table) rather than a threshold to determine whether to apply a deringing filter based on (or dependent on) the QP. For example, the list may include all QPs to which the deringing filter should be applied (e.g., larger QPs). For example, another embodiment using more than one deringing filter may use multiple lists (e.g., one list per deringing filter, each list including the QPs to which that particular deringing filter should be applied) or a list that maps QPs to deringing filters (e.g., by index).
[0025] Another embodiment may activate or deactivate the sharpening filter based on the same or similar criteria as described above for the sharpening filter (eg, a different threshold or list of quantization parameters).
[0026] In a preferred embodiment, in addition to the activation / deactivation signaling of the sharpening filter and / or the de-ringing filter, other parameters may be signaled in the bitstream and / or may be derived from the image properties.
[0027] Activation of filters may be performed only for certain portions of the video content, such as sequences, groups of pictures, single pictures, arbitrary regions, and / or regular coding blocks.
[0028] In a third implementation according to the first aspect or according to any one of the preceding implementations of the first aspect, the filter unit comprises:
[0029] - a first derivative unit for determining the first derivative of a reference sample,
[0030] - absolute value unit, used to determine the absolute value of the first-order derivative,
[0031] - a second derivative unit for determining a second derivative based on the absolute value of the first derivative, and
[0032] - A warping unit for warping the reference sample based on the second-order derivatives.
[0033] The decoder in the third embodiment comprises a non-linear filter, which experiments have shown to show particularly good results in suppressing ringing effects and increasing the subjective sharpness of natural edges.
[0034] In a fourth embodiment of the decoder according to the first aspect, or in a fourth embodiment of the decoder according to any of the preceding embodiments of the first aspect, the distortion unit is used to move each reference sample by a corresponding displacement vector, which is obtained by scaling the second-order derivative using a scaling factor, wherein the scaling factor is the sharpening strength.
[0035] In a fifth implementation of the decoder according to the first aspect, or a fifth implementation of the decoder according to any of the preceding implementations of the first aspect, the decoder is configured to adjust the scaling factor based on signaling in the bitstream and / or based on local image properties.
[0036] For example, the value of the scaling factor may be explicitly indicated in the bitstream. Alternatively, the overhead of explicit signaling may be avoided by adjusting the scaling factor based on local image properties (eg, based on measured strengths of one or more edges of a block that has been decoded from the bitstream).
[0037] In some embodiments, the scaling factor may be controlled by a quantization parameter of the block. The higher the QP (or QP value), the stronger the ringing effect caused by quantization.
[0038] Embodiments may be configured to obtain a QP (or a QP value or any other measure representing a degree of quantization) and apply a stronger de-ringing filter for a higher quantization parameter to more effectively suppress the ringing effect. In other words, embodiments may be configured to adjust the strength of the de-ringing filter based on the quantization parameter (e.g., by adjusting a scaling factor), and in particular, such that when the quantization parameter increases, the strength of the de-ringing filter increases (e.g., once or gradually or stepwise, such as increasing the scaling factor) (or vice versa), or such that for a small (smaller) quantization parameter, the strength of the de-ringing filter is low (lower), and for a large (larger) quantization parameter, the strength of the de-ringing filter is high (higher).
[0039] Embodiments may be configured to treat the size of a block as another indication of the quantization level. For example, due to the limitation of the amount or size of information sent by signal in the bitstream, embodiments may assume that at higher compression rates, the size of a block tends to be larger, and the larger the size of a block means the less information represented by the block splitting flag and the less prediction information for each block. Therefore, embodiments may be configured to obtain the size of the current block and control or adjust the scaling factor based on the size of the block. For example, embodiments may be configured to apply a stronger de-ringing filter (or sharpening filter) to blocks of larger size. For example, if the size of the block is obtained (e.g., estimated) to be the sum of the width and height of the block (e.g., width+height in samples), and the size of the block is equal to or greater than a block size threshold (or block size threshold value), such as 64, a first scaling factor (or first scaling factor value) k1 is used, and if the size of the block is less than the block size threshold, another scaling factor (or scaling factor value) k2 is used. Embodiments may also be configured to have a scaling factor k of zero when the size of the block is at a certain level. This actually means deactivating the de-ringing filter for such blocks.
[0040] Another embodiment may include more than one threshold, and correspondingly more than two scaling factors, to apply different de-ringing filters for different block sizes.
[0041] In a sixth implementation of the decoder according to the first aspect or a sixth implementation of the decoder according to any of the preceding implementations of the first aspect, the filter unit further comprises a clipping unit for limiting the absolute value to be above a threshold value.
[0042] The decoder of the sixth embodiment has the advantage that very high values of the absolute value of the first derivative are clipped, thereby avoiding extreme outliers that may lead to image degradation. Furthermore, limiting the absolute value (with a threshold) allows the position of edges to be kept constant, for example by using a warping procedure or algorithm. The filter processes only plane to edge transitions that may produce ringing.
[0043] The restrictive threshold value controls the neighborhood of the edge to be enhanced. On the other hand, as described above, the ringing is much higher due to the quantization parameter set for the current block. The higher the QP, the wider the corresponding ringing area. In some embodiments, the limiting value (threshold) can be controlled by the quantization parameter of the block, which will allow the filter to be adjusted to adapt to the artifact area. For example, an embodiment can be configured to apply a higher restrictive threshold value for higher QP values, which allows a wider area to be used for edge enhancement.
[0044] As mentioned above, another indication of the quantization level may be the size of the block. At higher compression rates, the size of the block tends to be larger. Therefore, embodiments may be configured to adjust or control the restrictive threshold value based on (or depending on) the size of the block, wherein a higher restrictive value is applied when the size of the block is larger.
[0045] In a seventh implementation of the decoder according to the first aspect, or in a seventh implementation of the decoder according to any one of the preceding implementations of the first aspect, a blur filter for smoothing absolute values is further included.
[0046] For example, the blur filter may be a Gaussian filter.
[0047] A second aspect of the invention relates to an encoder for encoding a block of a current frame in a bitstream, the encoder comprising:
[0048] - a reference sample selection unit, for selecting reference samples of the reconstructed part of the current frame,
[0049] - a filter unit for filtering the reference samples,
[0050] - a block generation unit for generating a prediction of the block based on the filtered reference samples, wherein the filter unit comprises a sharpening filter and / or a de-ringing filter, and
[0051] - A control unit for controlling whether a sharpening filter and / or a de-ringing filter is applied.
[0052] The encoder of the second aspect may be used to encode a block of a current frame in a bitstream such that it can be decoded by a decoder according to the first aspect of the invention.
[0053] The encoder of the second aspect may further include a reconstruction unit for generating a reconstructed video block based on the prediction of the block and the residual video block.
[0054] In a first embodiment of the encoder of the second aspect, the control unit is configured to select one of the following by performing rate distortion optimization, by minimizing a prediction error criterion, and / or based on one or more local image properties:
[0055] -Selective bypass, and
[0056] -Selective application of sharpening filters and / or de-ringing filters.
[0057] In this way, the encoder of the second aspect may determine the best filtering (or bypassing of filtering).The encoder may be configured to encode this selection into the bitstream, for example by explicit signaling of this selection.
[0058] In a second embodiment of the encoder of the second aspect, or in a second embodiment according to the first embodiment of the second aspect, the filter comprises one or more parameters, and the encoder further comprises a parameter selection unit for selecting the one or more parameters by performing rate-distortion optimization, by minimizing a prediction error criterion, and / or based on one or more local image properties.
[0059] In a third implementation of the encoder of the second aspect, or a third implementation of the first implementation of the second aspect, the encoder is used to encode a sharpening filter flag, a sharpening coefficient, and / or one or more parameters of a sharpening filter and / or a de-ringing filter into a bitstream.
[0060] Other embodiments of the encoder according to the second aspect correspond to other embodiments described according to the decoder of the first aspect.
[0061] A third aspect of the present invention relates to a method for decoding a block of a current frame of a video from a bitstream, the method comprising:
[0062] - select reference samples for the reconstructed part of the current frame,
[0063] - filtered reference samples, and
[0064] - generate a prediction for the block based on the filtered reference samples,
[0065] The filtering of the reference samples comprises the steps of sharpening and / or removing ringing of the reference samples.
[0066] The decoder according to the first aspect of the invention may perform the method according to the third aspect of the invention. Other features or embodiments of the method according to the third aspect of the invention may perform the functions of the decoder according to the first aspect of the invention and its different embodiments.
[0067] A fourth aspect of the present invention relates to a method for encoding a block of a current frame of a video in a bitstream, the method comprising:
[0068] - select reference samples for the reconstructed part of the current frame,
[0069] - filtered reference samples, and
[0070] - generate a prediction for the block based on the filtered reference samples,
[0071] Therein, the method further comprises the step of determining whether to filter the reference sample by sharpening and / or de-ringing.
[0072] The encoder according to the second aspect of the invention may perform the method according to the fourth aspect of the invention. Other features or embodiments of the method according to the fourth aspect of the invention may perform the functions of the encoder according to the second aspect of the invention and its different embodiments.
[0073] A fifth aspect of the present invention relates to a computer-readable storage medium storing program code, wherein the program code includes instructions for executing the method of the third aspect or the fourth aspect.
[0074] A sixth aspect of the present invention provides a decoder for decoding a block of a current frame of a video from a bit stream, the decoder comprising:
[0075] - a reference sample selection unit, for selecting reference samples of the reconstructed part of the current frame,
[0076] - a filter unit for filtering the reference samples, and
[0077] - a block generation unit for generating a prediction of a block based on filtered reference samples, wherein the filter unit comprises a sharpening filter and / or a de-ringing filter for application based on a flag bit in the bitstream.
[0078] The decoder of the sixth aspect has the following advantages: the sharpening filter and / or the de-ringing filter can improve the prediction efficiency, thereby improving the objective image quality and the perceived image quality. In particular, since the sharpening filter and / or the de-ringing filter can be applied based on the flag bit in the bitstream, these filters can be selectively applied when the image quality is improved.
[0079] The decoder of the sixth aspect may further comprise a reconstruction unit for generating a reconstructed video block based on the prediction of the block and a residual video block from the bitstream.
[0080] In a first implementation of the decoder according to the sixth aspect, the filter unit for filtering the prediction of the block and / or the block generation unit and / or the post-prediction filtering unit comprises a smoothing filter.
[0081] The decoder according to the first embodiment has the following advantages: the decoder includes both a sharpening filter / de-ringing filter and a smoothing filter. Therefore, the decoder can use sharpening or smoothing processing according to the situation, thereby achieving optimal image reproduction.
[0082] The decoder can be configured to apply one of the following based on a flag bit in the bitstream:
[0083] - a smoothing filter, or
[0084] - Sharpening filter and / or de-ringing filter.
[0085] For example, if a flag bit in the bitstream is true, the decoder may be configured to apply a smoothing filter, and if the flag bit is false, the decoder may be configured to apply a sharpening filter and / or a de-ringing filter.
[0086] This has the advantage of avoiding the opposing actions of applying smoothing and sharpening simultaneously.
[0087] If the sharpening flag in the bitstream is not true, the decoder may be configured to apply a smoothing filter based on a rule, the rule being based on one or more block properties.
[0088] In a second implementation of the decoder according to the sixth aspect, or in a second implementation of the decoder according to the first implementation of the sixth aspect, the decoder is configured to:
[0089] - Determine the smoothing filter conditions,
[0090] - parse the sharpness flags from the bitstream, and
[0091] - If the smoothing filter condition is true and the sharpening flag is false, apply the smoothing filter.
[0092] In other words, according to the second embodiment, the smoothing filter is applied only when the smoothing filter condition is satisfied and sharpening is deactivated. Thus, the opposite actions of performing smoothing and sharpening simultaneously are avoided.
[0093] In a third implementation of the decoder according to the sixth aspect, or a third implementation of the decoder according to any preceding implementation of the sixth aspect, the decoder is configured to:
[0094] - parse the sharpness flags from the bitstream, and
[0095] - if the sharp flag is false, parse the smooth flag from the bitstream,
[0096] and / or, wherein the decoder is configured to:
[0097] - parse the smoothing flag from the bitstream, and
[0098] - If the smooth flag is false, parse the sharp flag from the bitstream.
[0099] In other words, if the sharpening flag is true (which will cause the sharpening filter to be applied), then parsing the smoothing flag can be skipped. Similarly, if the smoothing flag is true (which will cause the smoothing filter to be applied), then parsing the sharpening flag can be skipped. The advantage of this is that less information needs to be parsed from the bitstream. Therefore, coding efficiency can be improved.
[0100] In a fourth implementation of the decoder according to the sixth aspect, or a fourth implementation of the decoder according to any preceding implementation of the sixth aspect, the block generation unit comprises a plurality of prediction modes including a DC mode, a planar mode, and / or one or more angular modes, and the decoder is further configured to:
[0101] - parse the prediction mode from the bitstream, and
[0102] - If the parsed prediction mode is neither DC mode nor planar mode, parse the sharpening flag from the bitstream.
[0103] In other words, if the parsed prediction mode is DC mode or planar mode, the decoder skips parsing the sharpening flag from the bitstream. The reason for this is that DC mode and planar mode can be regarded as smoothing filters. Therefore, the decoder can assume that if DC mode or planar mode is used for prediction, sharpening should be avoided. Therefore, if the prediction mode is DC mode or planar mode, there is no need to parse the sharpening flag. Therefore, the advantage of the fourth embodiment is that unnecessary flag parsing can be avoided and the coding efficiency can be improved.
[0104] In a fifth implementation of the decoder according to the sixth aspect, or in a fifth implementation of the decoder according to any preceding implementation of the sixth aspect, the block generation unit comprises a plurality of interpolation filters having different frequency response characteristics for obtaining sample values at fractional pixel positions, and the decoder is further configured to:
[0105] - parse the sharpness flags from the bitstream, and
[0106] - If the sharpening flag is true, then the interpolation filter among multiple interpolation filters that can pass the largest number of high frequencies is applied.
[0107] For example, if a cubic filter and a blur filter (eg, Gaussian or bilinear) are available, then if the sharpen flag is true, the cubic filter is applied because the cubic filter passes more high frequencies.
[0108] In a sixth implementation of the decoder according to the sixth aspect, or in a sixth implementation of the decoder according to any preceding implementation of the sixth aspect, the smoothing filter comprises:
[0109] - DC mode and / or planar mode of the block prediction method,
[0110] - boundary smoothing filters,
[0111] - Position dependent intra prediction combination, and / or
[0112] -Multi-parameter intra prediction.
[0113] Position-dependent intra prediction combination (PDPC) is a post-processing for block prediction that calls for a combination of HEVC block prediction with filtered and unfiltered boundary reference samples. The boundary samples are smoothed using some 7-tap low-pass filters. Thus, the pixels of the predicted block are smoothed according to their position in the block.
[0114] Multi-parameter intra prediction (MPI) is a post-processing tool for intra prediction that calls for additional smoothing of decoded boundaries. The purpose of MPI is to generate more natural patterns by applying different post-processing filters to the prediction results, while maintaining directional patterns even after smoothing.
[0115] In a seventh embodiment of the decoder according to the sixth aspect, or a seventh embodiment of the decoder according to any preceding embodiment of the sixth aspect, the smoothing filter is a Gaussian filter, and when a sharpening filter is used, the block generation unit is configured to use a cubic filter instead of a Gaussian filter.
[0116] A seventh aspect of the present invention relates to an encoder for encoding a block of a current frame of a video in a bitstream, the encoder comprising:
[0117] - a reference sample selection unit, for selecting reference samples of the reconstructed part of the current frame,
[0118] - a filter unit for filtering the reference samples, wherein the filter unit comprises a sharpening filter and / or a de-ringing filter,
[0119] - a block generation unit for generating a prediction of a block based on the filtered reference samples, and
[0120] - A control unit for controlling whether a sharpening filter and / or a de-ringing filter is applied.
[0121] The encoder may be configured to calculate a target criterion for each of a plurality of encoding modes, wherein each of the plurality of encoding modes applies a sharpening filter or a smoothing filter, and wherein the encoder is configured to select a preferred encoding mode based on the calculated target criterion.
[0122] The encoder may further comprise a writing unit for writing a flag bit in the bitstream, wherein the flag bit indicates whether a sharpening filter and / or a de-ringing filter should be applied during decoding.
[0123] In a first implementation of the encoder of the seventh aspect, the filter unit and / or the block generation unit and / or the post-filtering unit for filtering for prediction of the block comprises a smoothing filter.
[0124] In a second implementation of the encoder of the seventh aspect, or in a second implementation of the encoder according to the first implementation of the seventh aspect, the encoder is used to:
[0125] - write the sharpness flag into the bitstream, and
[0126] - When the sharp flag is false, write the smooth flag to the bitstream,
[0127] and / or, wherein the encoder is used to:
[0128] - write the smoothing flag into the bitstream, and
[0129] - When the smooth flag is false, write the sharp flag to the bitstream.
[0130] In a third implementation of the encoder of the seventh aspect, or in a third implementation of the encoder according to any of the preceding implementations of the seventh aspect, the block generation unit comprises a plurality of prediction modes including a DC mode, a planar mode, and / or one or more angular modes, and the decoder is further configured to:
[0131] - determine the forecast pattern, and
[0132] - If the determined prediction mode is neither DC mode nor planar mode, write the sharpening flag bit in the bitstream.
[0133] An eighth aspect of the present invention relates to a method for decoding a block of a current frame of a video from a bitstream, the method comprising:
[0134] - select reference samples for the reconstructed part of the current frame,
[0135] - filtered reference samples,
[0136] - generate a prediction for the block based on the filtered reference samples,
[0137] The method further comprises a step of determining whether to filter the reference sample based on a flag bit in the bit stream, including a step of applying a sharpening filter and / or a de-ringing filter.
[0138] The decoder according to the sixth aspect of the invention may perform the method according to the eighth aspect of the invention. Other features or embodiments of the method according to the eighth aspect of the invention may perform the functions of the decoder according to the sixth aspect of the invention and its different embodiments.
[0139] A ninth aspect of the present invention relates to a method for encoding a block of a current frame of a video in a bitstream, the method comprising:
[0140] - select reference samples for the reconstructed part of the current frame,
[0141] - filtered reference samples,
[0142] - generate a prediction for the block based on the filtered reference samples,
[0143] Therein, the method further comprises the step of determining whether to filter the reference sample, including the step of applying a sharpening filter and / or a de-ringing filter.
[0144] The method may further include: determining a target criterion for predicting the block; and selecting a preferred encoding mode based on the determined target criterion.
[0145] The encoder according to the seventh aspect of the invention may perform the method according to the ninth aspect of the invention. Other features or embodiments of the method according to the ninth aspect of the invention may perform the functions of the encoder according to the seventh aspect of the invention and its different embodiments.
[0146] A tenth aspect of the present invention relates to a computer-readable storage medium storing program code, the program code comprising instructions for executing the methods of the eighth and ninth aspects. BRIEF DESCRIPTION OF THE DRAWINGS
[0147] In order to more clearly illustrate the technical features of the embodiments of the present invention, the following is a brief introduction to the drawings used to describe the embodiments. The drawings described below are only some embodiments of the present invention, and these embodiments can be modified without departing from the scope of the present invention defined by the claims.
[0148] Figure 1 is a block diagram showing a decoder for decoding blocks of a current frame of a video from a bitstream;
[0149] Figure 2 is a block diagram illustrating an encoder for encoding a block of a current frame in a bitstream;
[0150] Figure 3is a flow chart of a method for decoding a block of a current frame of a video from a bitstream;
[0151] Figure 4 is a flow chart of a method for encoding a block of a current frame from a bitstream;
[0152] Figure 5 is a block diagram of another encoder for encoding a block of a current frame in a bitstream;
[0153] Figure 6 is a block diagram of another decoder for decoding a block of a current frame of a video from a bitstream;
[0154] Figure 7 It is a schematic diagram of the intra prediction method;
[0155] Figure 8 is a block diagram of an intra prediction unit;
[0156] Fig. 9 is a block diagram of a reference sample preparation unit;
[0157] Fig.10 is the block diagram of the sharpening filter;
[0158] Fig.11 is a flow chart of an interpolation filter selection method when reference sample sharpening is enabled;
[0159] Fig.12 is a flow chart of the prediction mode selection method;
[0160] Fig.13 is a flow chart of a method for conditional parsing of sharpening flags;
[0161] Fig.14 is a flow chart of another method for conditional parsing of sharpening flags;
[0162] Fig.15 is a flow chart of a method for conditional parsing of the PDPC flag; and
[0163] Fig.16 is a flow chart of a method for coordinated boundary smoothing and sharpening. DETAILED DESCRIPTION
[0164] Figure 1 A decoder 100 is shown for decoding blocks of a current frame of video from a bitstream.
[0165] The decoder 100 includes a reference sample selection unit 110 , a filter unit 120 , and a block generation unit 130 .
[0166] The reference sample selection unit 110 is used to select reference samples of the reconstructed part of the current frame. The reference samples can be selected according to a conventional scheme.
[0167] The filter unit 120 is used to filter the reference samples, wherein the filter unit 120 comprises a sharpening filter and / or a de-ringing filter. These filters may be configured based on signaling in the bitstream and / or based on image properties (eg, based on image properties of the reconstructed blocks).
[0168] The block generation unit 130 is configured to generate a prediction of a block based on the filtered reference samples.
[0169] Figure 2 An encoder 200 is shown for encoding a block of a current frame in a bitstream.
[0170] The encoder includes a reference sample selection unit 210 , a filter unit 220 , a block generation unit 230 , and a control unit 240 .
[0171] The reference sample selection unit 210 is used to select reference samples of the reconstructed part of the current frame. The reference samples can be selected according to a conventional scheme.
[0172] The filter unit 220 is used to filter the reference samples, and includes a sharpening filter and / or a de-ringing filter.
[0173] The block generation unit 230 is configured to generate a prediction of a block based on the filtered reference samples.
[0174] The control unit 240 is used to control whether to apply a sharpening filter and / or a de-ringing filter.
[0175] Will refer to Figure 5 and Figure 6 Possible implementations of the decoder 100 and the encoder 200 are described in more detail below.
[0176] Figure 3 A method 300 for obtaining a prediction block of a current frame of a video from a bitstream is shown.
[0177] The method 300 comprises a first step of selecting 310 reference samples of a reconstructed portion of a current frame.
[0178] The method 300 comprises a second step of filtering 320 the reference samples.
[0179] The method comprises a third step of generating 330 a block prediction based on filtered reference samples, wherein filtering the reference samples comprises the steps of sharpening and / or de-ringing the reference samples.
[0180] Figure 4A method 400 for encoding a block of a current frame of a video in a bitstream is shown.
[0181] The method comprises a first step of selecting 410 reference samples of a reconstructed portion of a current frame.
[0182] The method comprises a second step of filtering 420 the reference samples.
[0183] The method comprises a third step of generating 430 a block prediction based on the filtered reference samples, wherein the method further comprises the step of determining whether the reference samples are filtered by sharpening and / or de-ringing.
[0184] Figure 5 An encoder 500 is shown, which includes an input and an output, the input for receiving an input block of frames or pictures of a video stream, and the output for generating a coded video bitstream. In an illustrative practice, the encoder 500 is suitable for applying prediction, transformation, quantization, and entropy coding to the video stream. The transformation, quantization, and entropy coding are performed by a transformation unit 501, a quantization unit 502, and an entropy coding unit 503, respectively, to generate a coded video bitstream as an output.
[0185] The video stream may comprise a plurality of frames, where each frame is divided into blocks of a certain size which are intra-coded or inter-coded. For example, blocks of a first frame of the video stream are intra-coded by an intra-frame prediction unit 509. An intra-frame is coded using only information within the same frame, so it can be decoded independently and it can provide an entry point in the bitstream for random access. Blocks of other frames of the video stream are inter-coded by an inter-frame prediction unit 510: information from coded frames, called reference frames, is used to reduce temporal redundancy, so each block of an inter-coded frame is predicted based on a block in a reference frame. A mode selection unit 508 is adapted to select whether a block in a frame is processed by the intra-frame prediction unit 509 or by the inter-frame prediction unit 510. The mode selection unit 508 also controls the parameters of the intra-frame prediction or the inter-frame prediction.
[0186] The intra prediction unit 509 is a block prediction unit. The intra prediction unit 509 includes a sharpening filter and / or a de-ringing filter ( Figure 5 not shown).
[0187] To perform spatial or temporal prediction, the encoded block may be further processed by an inverse quantization unit 504, an inverse transform unit 505. A loop filter unit 506 is applied after reconstructing the entire frame to obtain a reference frame, which is then stored in a frame buffer 507.
[0188] The inter prediction unit 510 includes as input a block of a current frame or picture to be inter-coded and one or more reference frames or pictures from the frame buffer 507. Motion estimation and motion compensation are applied by the inter prediction unit 510. Motion estimation is used to obtain motion vectors and reference frames based on a specific cost function. Thereafter, motion compensation describes the current block of the current frame according to the transformation from the reference block of the reference frame to the current frame. The inter prediction unit 510 outputs a prediction block for the current block, wherein the prediction block minimizes the difference between the current block to be coded and its prediction block, i.e., minimizes the residual block. The minimization of the residual block is based on, for example, a rate-distortion optimization procedure.
[0189] The intra prediction unit 509 receives as input a block of a current frame or picture to be intra-coded and one or more reference samples from a reconstructed area of the current picture. Thereafter, the intra prediction describes the current block of the current frame according to the transformation from the reference samples of the current frame to the currently coded block. The intra prediction unit 509 outputs a prediction block for the current block, wherein the prediction block minimizes the difference between the current block to be coded and its prediction block, i.e., minimizes the residual block. The minimization of the residual block can be based on, for example, a rate-distortion optimization procedure.
[0190] Afterwards, the difference between the current block and its prediction (ie, the residual block) is transformed by a transform unit 501. The transform coefficients are quantized and entropy encoded by a quantization unit 502 and an entropy encoding unit 503. The encoded video bitstream thus generated includes intra-coded blocks and inter-coded blocks.
[0191] Figure 6 A video decoder 600 is shown. The video decoder 600 comprises in particular a reference picture buffer 607 and an intra prediction unit 609, which is a block prediction unit and comprises a sharpening filter and / or a de-ringing filter. The reference picture buffer 607 is adapted to store at least one reference frame obtained from a coded video bitstream, which reference frame is different from a current frame of the coded video bitstream. The intra prediction unit 609 is used to generate a prediction block, which is an estimate of a block to be decoded. The intra prediction unit 609 is used to generate the prediction based on reference samples obtained from the reference picture buffer 607. The intra prediction unit 609 is used to sharpen and / or de-ring reference samples obtained from the reference picture buffer using a sharpening filter and / or a de-ringing filter.
[0192] The decoder 600 is adapted to decode the encoded video bitstream generated by the video encoder 500, and preferably, both the decoder 600 and the encoder 500 generate the same prediction. The characteristics of the reference picture buffer 607 and the intra prediction unit 609 are similar to those of Figure 5 The features of the reference picture buffer 507 and the intra prediction unit 509 are similar.
[0193] In particular, the video decoder 600 includes other units that are also present in the video encoder 600, for example, an inverse quantization unit 604, an inverse transform unit 605, a loop filtering unit 606, and an intra-frame prediction unit 609, which correspond to the inverse quantization unit 504, the inverse transform unit 505, the loop filtering unit 506, and the intra-frame prediction unit 509 of the video encoder 500, respectively.
[0194] The entropy decoding unit 603 is adapted to decode the received coded video bitstream and obtain the quantized residual transform coefficients and sharpening filter information (if any) accordingly. The quantized residual transform coefficients are fed to the inverse quantization unit 604 and the inverse transform unit 605 to generate a residual block. The residual block is added to the prediction block, and the addition result is fed to the loop filtering unit 606 to obtain the decoded video. The frames of the decoded video can be stored in the reference picture buffer 607 and used as reference frames for inter-frame prediction.
[0195] generally, Figure 5 and Figure 6 The intra prediction unit 509 and the intra prediction unit 609 in the embodiment may use reference samples from the coded region to generate a prediction signal for a block to be coded or a block to be decoded (see Figure 7 ).
[0196] Figure 7 FIG. 7 shows intra prediction of a block 730 based on a plurality of reference samples 720 that are part of an already coded region 710. Figure 7 As indicated by arrow 740 in FIG. 7 , block 730 is predicted based on multiple reference samples. In particular, multiple reference samples may be used to predict a single pixel of block 730. Reference samples 720 may be selected from the coded area, for example, as a vertical column or horizontal row of reference samples surrounding the block to be predicted.
[0197] Figure 8 is a block diagram of the intra-frame prediction unit 800. According to one embodiment of the present invention, the intra-frame prediction unit 800 may be Figure 5 and Figure 6 The intra-frame prediction unit 509 and the intra-frame prediction unit 609 are one embodiment, and the intra-frame prediction unit 800 includes three main modules:
[0198] 1. Reference sample preparation unit 810. In this unit, reference samples are defined and preprocessed according to the selected prediction mode, block size, and sample availability. These steps can be performed by reference sample preparation unit 812, smoothing filter 814, and sharpening filter 816 of reference sample preparation unit 810.
[0199] 2. A prediction block generation unit 820 is used to generate a prediction block. In this unit, the reference sample is extrapolated to the prediction block according to the selected prediction mode (angle, plane, or DC). In order to generate the prediction block, the reference sample should be known at its fractional sampling position. An interpolation filter is used to obtain the value of the reference sample at its fractional sampling position. The interpolation filter can be a simple bilinear filter, or a more complex Gaussian filter or a cubic filter to obtain a more accurate interpolation. These steps can be performed by the bilinear interpolation unit 822, the cubic interpolation unit 824, and the Gaussian interpolation unit 826 of the prediction block generation unit 820.
[0200] 3. A post-prediction filtering unit 830 is used to post-filter the prediction block. In order to further improve the prediction efficiency, the prediction block is additionally filtered before extracting the residual. For example, an intra-frame boundary filter can be used to smooth the difference between the boundary between the prediction block and the encoded adjacent block. Both multi-parameter intra prediction (MPI) and position dependent intra prediction combination (PDPC) are intended to blur different parts of the prediction block with different intensities. These steps can be performed by the boundary smoothing unit 832, the MPI unit 834, and the PDPC unit 836 of the post-prediction filtering unit 830.
[0201] In the reference sample preparation unit 810, sharpened (and / or de-ringing) reference samples are obtained by applying a sharpening (and / or de-ringing) filter to the reference samples, together with the unprocessed reference samples and the smoothed reference samples. For example, the encoder can decide which type of pre-processing should be used during rate-distortion optimization (based on cost or minimum prediction error criteria) or based on some a priori defined rules (e.g., based on block size, prediction mode, etc.).
[0202] Fig. 9 is a block diagram of a reference sample preparation unit 900. The reference sample preparation unit 900 includes a sample selection unit 902, a smoothing filter 904, and a sharpening filter 906. In addition, the reference sample preparation unit 900 includes a switch 908 for switching between the smoothing filter 904, the sharpening filter 906, and a bypass connection, wherein the bypass connection bypasses the filter.
[0203] Preferably, Fig. 9 The sharpening filter 906 is a non-linear sharpening filter based on a distortion.
[0204] Fig.10An example of a warp-based nonlinear sharpening filter 1000 is shown. The filter 1000 includes a first-order derivative unit 1002, an absolute value unit 1004, a clipping unit 1006, a blur filter 1008, a second-order derivative unit 1010, a multiplier 1012, and a warp unit 1014. These units 1002-1014 are connected in sequence. In particular, the units 1002-1014 of the filter 1000 are used to sequentially perform the following steps:
[0205] 1. Obtain the gradient of a row of reference samples by taking the first-order derivative (dx).
[0206] 2. Obtain the edge map by calculating the length of the gradient vector.
[0207] 3. Clip the edge map with a threshold to prevent processing of extremely high and / or low values.
[0208] 4. Blur the clipped edge map using, for example, a Gaussian filter.
[0209] 5. Take the second derivative (d2x).
[0210] 6. Scale the second-order derivative vector d2x by a factor k to obtain a displacement vector (wx). The factor k is considered as a sharpening strength and can be adaptively selected to better fit local image properties, for example, by a rate-distortion optimization process or a minimum prediction error criterion. Optionally, the scaling factor can be indicated by explicit signaling in the bitstream.
[0211] 7. Warping, such as by bilinear interpolation using the obtained displacement vector.
[0212] This warp-based nonlinear sharpening filter can achieve two types of improvements: increasing the sharpness of natural edges and removing ringing artifacts around edges.
[0213] Conventional edge enhancement techniques based on linear sharpening (or deblurring) filters (e.g., "unsharp masking") can improve subjective quality, but are generally unable to suppress the ringing artifacts caused by residual quantization. In many cases, these techniques can even increase the ringing and degrade objective performance characteristics. In many cases, nonlinear filters can provide better results for ringing removal and enhance both subjective and objective edge quality.
[0214] Add a sharpening filter to the processing of the reference sample, e.g. Fig.10 The sharpening filter 1000 makes: Reduce the amount of high-frequency residuals by improving the quality of intra-frame prediction in the presence of natural edges; Reduce edge blurring caused by reference sample quantization; and Improve the subjective and objective quality of edges in the predicted signal, thereby improving the quality of the reconstructed image.
[0215] Adding a sharpening filter of a specific structure (non-linear, warp-based, adaptive) results in: Reduce the ringing effect caused by the quantization of the transform coefficients of the reference samples and increase the sharpness of natural edges; Since the filter in simple adaptive form has only one coefficient for adaptation and transmission, it is performed spatially with limited signaling overhead Adaptive.
[0216] Preferably, the adaptive coefficients are variable for some parts of the content: a sequence, a group of pictures, a coded picture, an arbitrary region, or a regular coded block of arbitrary size (e.g., CTU, CU, PU, TU using HEVC terminology). In order to make the decoder aware of the changed interpolation filter settings, they should be signaled to the decoder or derived at the decoder side without explicit signaling.
[0217] The sharpening (and / or de-ringing) filter may be enabled at all times or switchably turned on / off. For example, by minimizing a prediction error or cost (rate / distortion) criterion, the encoder may make a choice whether to enable or disable the sharpening filter in each specific block (e.g., CTU, CU, PU, TU in H.265 / HEVC terminology) of the coded image and signal this choice in the bitstream with a 1-bit flag.
[0218] The filter 1000 allows to increase the subjective sharpness of natural edges and suppress the ringing effect near the edges caused by the quantization of the reference samples. However, the additional steps of prediction signal generation and post-filtering may cause the natural edges to be blurred again. To avoid this, it is possible to explicitly prohibit the simultaneous operation of sharpening and blurring tools. Such a design reduces the overhead caused by the signaling of each specific prediction enhancement tool and reduces the complexity of the encoder by eliminating contradictory combinations in the processing.
[0219] Preferably, if other steps in obtaining the prediction signal contain tools that cause blurring (e.g. DC and planar prediction modes, Gaussian interpolation filters, bilinear interpolation filters, MPI, PDPC, boundary prediction filters), such tools in combination with sharpening are excluded from processing and signaling.
[0220] Sometimes both sharpening and de-ringing can be achieved with just one filter. However, in some cases where only sharpening or only de-ringing is desired, dedicated filters are provided that serve only one of the two purposes.
[0221] Depending on the specific configuration, enabling the sharpening filter does not always result in disabling all possible blur tools. Some of them may be disabled, while others are not. Such a combination can improve performance.
[0222] Further modifications are made to the subsequent steps of intra prediction - prediction block generation and post-prediction filtering. The prediction block generation uses two interpolation filters. One is a cubic filter that preserves high frequencies, and the second is a low-pass Gaussian filter.
[0223] Fig.11 1 is a flow chart of an exemplary method for determining which filter to use. The method begins at step 1100 and proceeds to step 1102 to determine whether the size of the block matches a standard, for example, whether the size of the block corresponds to a predetermined value. If so, the method proceeds to step 1104 to check whether a sharpening flag is set (e.g., a sharpening flag parsed from a bitstream). If the sharpening flag is true, the method proceeds to step 1108 to use a cubic interpolation filter. Otherwise, the method proceeds to step 1106 to use a Gaussian interpolation filter.
[0224] exist Fig.12 , another exemplary embodiment is shown. Typically, during prediction, a block generation intra prediction mechanism looks at the existing prediction modes and selects the best prediction mode based on minimum prediction error or rate / distortion (cost) criteria. The prediction modes may include planar mode, DC mode, and angular mode.
[0225] Fig.12 The method is preferably applied during encoding, Fig.12 In the method, if the sharp mode is selected in the reference sample preparation step, the DC mode and the planar mode are excluded from the search. Therefore, there is no need to signal the combination of the sharp mode and the DC or planar prediction mode to the decoder.
[0226] Specifically, the method starts at step 1200 and proceeds to step 1204 for each prediction mode 1202 to determine whether the prediction mode is DC or planar. If so, the method determines whether the sharp flag is set in step 1206. If the sharp flag is set, the method proceeds to step 1202 to the next prediction mode. If the sharp flag is not set, the method continues to step 1208 to calculate the prediction distortion / rate. In step 1210, the best prediction mode is saved.
[0227] When all prediction modes have been evaluated, Fig.12 The method ends at step 1212.
[0228] Fig.131 is a flow chart of a corresponding method used in a decoding process. The method starts at step 1300 and proceeds to step 1302 of parsing the intra prediction mode. Subsequently, in step 1304, the method determines whether the prediction mode is DC or planar. If so, the method ends at step 1308, otherwise in step 1306, a sharpening flag is parsed from, for example, a bitstream.
[0229] In the last step of intra prediction - post-prediction filtering can also be a tool that contradicts the sharpening of reference samples. For example, if sharpening of reference samples is chosen, PDPC according to the present invention should also be excluded from the processing. This fact can be used to optimize signaling.
[0230] According to the signaling sequence, when the PDPC flag is written first, the decoder parses as follows Fig.14 When the sharpening flag is written first, the Fig.15 Perform the parsing process. Another predictive post-filtering tool, MPI Prediction, can also be coordinated with the sharpening tool in the same way as PDPC.
[0231] More specifically, Fig.14 is a flow chart of a method used in a decoding process. After initialization 1400, the method proceeds to 1402 to parse the PDPC flag. In step 1404, it is determined whether the PDPC flag is set. If the PDPC flag is set, the method ends at step 1408. Otherwise, the method proceeds to step 1406 to parse the sharpening flag.
[0232] Fig.15 1 is a flow chart of another method used in the decoding process. The method starts at step 1500 and proceeds to 1502 to parse the sharpening flag. If it is determined in step 1504 that the sharpening flag is not set, the method parses the PDPC flag in the next step 1506. The method ends at the last step 1508.
[0233] Fig.16 1 is a flow chart of another exemplary embodiment of the joint operation of reference sample sharpening and boundary smoothing filters. After initialization 1600, the method determines in step 1602 whether boundary smoothing conditions are available. If not, the method ends in the last step 1608. Otherwise, the method determines in step 1604 whether the sharpening flag is set. Only when the sharpening flag is not set, the method proceeds to step 1606 to perform boundary smoothing.
[0234] The above description is only an embodiment of the present invention, and the scope of the present invention is not limited thereto. Those skilled in the art can easily make any changes or substitutions. Therefore, the protection scope of the present invention shall be subject to the protection scope of the attached claims.
Claims
1. A decoder (100; 600) for decoding a block (730) of a current frame of a video from a bitstream, the decoder comprising: - a reference sample selection unit (110) for selecting reference samples (720) for the reconstructed part of the current frame, - a filter unit (120; 1000) for filtering said reference samples; as well as - a block generation unit (130) for generating a prediction of the block based on the filtered reference samples, wherein the prediction of the block is an intra prediction of the block; Wherein, the filter unit comprises a sharpening filter (1000) and / or a de-ringing filter.
2. An encoder (200; 500) for encoding a block of a current frame in a bitstream, the encoder comprising: - a reference sample selection unit (210) for selecting reference samples of the reconstructed part of the current frame, - a filter unit (220; 1000) for filtering the reference samples, wherein the filter unit comprises a sharpening filter and / or a de-ringing filter, - a block generation unit (230) for generating a prediction of the block based on the filtered reference samples, wherein the prediction of the block is an intra prediction of the block, and - a control unit (240) for controlling whether to apply the sharpening filter and / or the de-ringing filter.
3. A method (300) for decoding a block of a current frame of a video from a bitstream, the method comprising: - selecting (310) reference samples (720) of the reconstructed portion (710) of the current frame, - filtering (320) said reference samples, and - generating (330) a prediction of said block (730) based on said filtered reference samples, wherein the prediction of said block is an intra prediction of said block, Wherein, filtering the reference samples comprises the steps of sharpening and / or de-ringing the reference samples.
4. A method (400) for encoding a block of a current frame of a video in a bitstream, the method comprising: - selecting (410) reference samples (720) of the reconstructed portion (710) of the current frame, - filtering (420) said reference samples, - generating (430) a prediction of the block based on the filtered reference samples, wherein the prediction of the block is an intra prediction of the block; The method further comprises the step of determining whether to filter the reference samples by sharpening and / or de-ringing.
5. A computer-readable storage medium storing a program code, the program code comprising instructions for executing the method of one of claims 3 and 4 when executed by a processor.
