Weighted filtering for picture enhancement in video encoding and decoding

Abstract

A video data processing method performed by an encoder is provided, the method comprising the steps of: obtaining a plurality of original pictures from original video data; obtaining a plurality of reference pictures, each corresponding to an original picture from the original video data, wherein the plurality of reference pictures have a lower resolution than the original pictures from the original video data; upsampling the plurality of reference pictures to obtain a plurality of upsampled reference pictures; and determining a weight map for each reference picture using a weight map function, wherein the weight map includes a plurality of weights mapped to the respective spatial positions of the upsampled reference pictures, the upsampled reference pictures being used as input to the weight map function; and determining a filter to be applied to the upsampled reference pictures along with their respective weight maps to filter the upsampled reference pictures. The method includes the steps of obtaining a ring reference picture to obtain a weighted filter that reduces the overall error between multiple upsampled reference pictures and their corresponding original picture by applying the filter with different weights to different spatial locations of the upsampled reference picture; performing interpretation of multiple blocks of a first picture based on multiple reference blocks from one or more of the upsampled reference pictures; and encoding video data encoding information into a bitstream, the encoding information including information about a weighted map function and / or filter used by the decoder, the method further including the steps of applying the weighted filter to the multiple upsampled reference pictures before performing interpretation, or applying the weighted filter to the upsampled reference blocks of the reference picture during interpretation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to the topics of video processing and video encoding and decoding, and more specifically, relates to a weighted filtering method for picture enhancement in video encoding and decoding, a decoder, an encoder, and a computer-readable medium.

BACKGROUND ART

[0002] Current video encoding and decoding schemes such as H.265 / HEVC (High Efficiency Video Coding) and H.266 / VVC (Versatile Video Coding) support spatial scalability of encoded and decoded video streams. This spatial scalability support is included in the second version of HEVC with scalability extension SHVC, while VVC natively supports spatial scalability. Adaptively changing the resolution of encoded and decoded video during encoding and decoding is known from VVC as Reference Picture Resampling (RPR) or Adaptive Resolution Change (ARC). Furthermore, multi-resolution encoding and decoding and multi-layer encoding and decoding enable scalable resolution of encoded and decoded video. Therefore, the spatial resolution at which the video is encoded and decoded may be adaptively changed, and no longer needs to be equivalent to the output or input resolution of the video. The benefit of this additional flexibility is that encoding and decoding lower-resolution video requires a lower bit rate, and may reduce computational complexity at the cost of losing high-frequency information in the downsampling step.

[0003] Encoding and decoding video at a lower resolution than the original resolution requires downsampling and upsampling steps in the signal processing chain. The downsampling step applies an anti-aliasing filter to prevent artifacts caused by high-frequency components in the image. Upsampling applies an interpolation filter to reconstruct intensity values ​​at fractional sample positions.

[0004] In RPR, the resolution of the encoded and decoded video stream may be adaptively changed. As a result, the encoder may encode parts of the video stream at a lower resolution. RPR is applied in interpretation whenever a picture uses a reference picture with a different resolution than the current picture in interpretation. In this step, a resampling operation must be applied so that the referenced picture block is mapped to the same spatial resolution as the current picture.

[0005] In multi-layer coding, video is encoded and decoded at different resolution layers. In the first step, the video is encoded and decoded at the lowest resolution layer. To generate the video stream for the next layer, the video is upsampled, the residuals are encoded and decoded if necessary, and further processing steps are applied. This processing may be applied multiple times depending on the number of layers.

[0006] Finding the optimal high-resolution representation from a low-resolution picture is a crucial part of the encoding and decoding schemes described above. One method is to apply a set of multiphase finite impulse response (FIR) interpolation filters. While these filters provide an approximation of the high-resolution image content, they cannot recover information lost during the downsampling process and suffer from the limitations of linear filtering. As a result, upsampled images are often blurred.

[0007] Image sharpening can improve image quality. However, linear high-pass filters frequently introduce artifacts such as overshoot and ringing. Furthermore, distortions caused by downsampling and upsampling depend on the image content and the encoding / decoding quality of the video (which is affected by the Quantization Parameter (QP) value).

[0008] Filtering video to improve image quality requires the existence of statistical dependencies that can be utilized by the filtering system. Generally, applying in-loop filtering is reasonable if the quality improvement achieved by filtering outweighs the signaling cost at this rate distortion (RD) point. Furthermore, the computation time must be acceptable.

[0009] Many video encoding and decoding systems apply a set of filters to address different types of encoding and decoding errors. For example, there are deblocking filters that can be applied at block boundaries to reduce blocking artifacts. Next, there may be sample adaptive offset (SAO) filters designed primarily to reduce ringing or blur artifacts. Finally, adaptive loop filters (ALF) may be used for objective quality improvements. Note that this is only a partial excerpt and is intended to be an overview of the various applications and types of loop filters.

[0010] Most of these filters only handle a limited range of encoding and decoding errors. Furthermore, VVC / H.266 does not implement filters that explicitly address the problem of blurred image content. However, blurring occurs due to the quantization or removal of high-frequency components. Linear filters are usually insufficient to recover blurred content due to overshoot and ringing problems. In addition, noise amplification is a problem. Linear filtering techniques such as adaptive loop filters attempt to address this problem by introducing class sets to which different filters are applied. However, this increases the encoding and decoding cost. [Overview of the Initiative]

[0011] Embodiments of the present invention provide a video encoding and decoding method, decoder, encoder, and computer-readable medium using a weighted filter, overcoming problems associated with conventional configurations.

[0012] According to the first embodiment, a video data processing method is provided which is performed by an encoder, the method comprising: obtaining a plurality of original pictures from original video data; obtaining a plurality of reference pictures, each corresponding to an original picture from the original video data, wherein the plurality of reference pictures have a lower resolution than the original pictures from the original video data; upsampling the plurality of reference pictures to obtain a plurality of upsampled reference pictures; and determining a weight map for each reference picture using a weight map function, wherein the weight map includes a plurality of weights mapped to the respective spatial positions of the plurality of upsampled reference pictures, the plurality of upsampled reference pictures being used as input to the weight map function; and determining a filter to be applied to the plurality of upsampled reference pictures together with their respective weight maps to filter the upsampled reference pictures. The method includes the steps of: obtaining a weighted filter that reduces the overall error between the upsampling reference pictures and the corresponding original picture by obtaining an upsampling reference picture so that the filter is applied with different weights to different spatial positions of the plurality of upsampling reference pictures; performing interpretation of a plurality of blocks of a first picture based on a plurality of reference blocks from one or more of the plurality of upsampling reference pictures; and encoding the video data encoding information into a bitstream, wherein the encoding information includes information about the weighted map function and / or the filter used by the decoder, the method further including the steps of applying the weighted filter to the plurality of upsampling reference pictures before performing the interpretation, or applying the weighted filter to the upsampling reference blocks of the reference picture during the interpretation.

[0013] In some embodiments, the step of obtaining the weighted filter that reduces the overall error between the plurality of upsampled reference pictures and the corresponding original picture includes the steps of assigning respective importance weighting coefficients to the plurality of reference pictures and / or regions of the plurality of reference pictures, and determining the overall error between the plurality of upsampled reference pictures and the corresponding original picture by weighting the error of each upsampled reference picture based on the importance weighting coefficient of each upsampled reference picture.

[0014] In some embodiments, the step of assigning importance weighting coefficients to the plurality of reference pictures includes the step of assigning higher importance weighting coefficients to reference pictures that are closer to the first picture in time.

[0015] In some embodiments, the step of assigning importance weighting coefficients to the plurality of reference pictures includes the step of assigning importance weighting coefficients based on the quality of each reference picture.

[0016] In some embodiments, the step of assigning importance weighting coefficients to the plurality of reference pictures includes the step of assigning importance weighting coefficients based on local picture features or historical information from encoding the reference pictures or other pictures in the video data.

[0017] In some embodiments, the method further includes the step of adding the acquired weighted filter to a stored set of reference filters.

[0018] In some embodiments, the step of assigning importance weighting coefficients to the plurality of reference pictures and / or regions of the plurality of reference pictures includes the steps of performing trial coding of the first picture, identifying which reference pictures and / or blocks within the reference pictures are to be used in the trial coding for interpretation of one or more blocks of the first picture, and assigning high importance weighting coefficients to the identified reference pictures and / or blocks.

[0019] In some embodiments, the encoded information includes signaled weighted map function parameters, which are configured to enable a decoder to obtain the weighted map using the weighted map function by applying the signaled weighted map function parameters as parameters of the weighted map function and providing the plurality of reference pictures as input to the weighted map function.

[0020] In some embodiments, the encoded information includes signaled filter function parameters, which are configured to enable a decoder to obtain the filter by applying the signaled filter function parameters as parameters of the filter.

[0021] In some embodiments, the weight map and the filter are configured to be applied to the first picture as a step in or post-loop step within the encoding-decoding loop to obtain the filtered first picture.

[0022] In some embodiments, the encoding / decoding loop is an H.266 / VVC encoding / decoding loop.

[0023] In some embodiments, the weighted map and the filter are integrated into an adaptive loop filter, and are configured to be applied to the plurality of reference pictures or derived partitions of the first picture to obtain a filtered picture.

[0024] In some embodiments, the bitstream is rate-distortion (RD) optimized based on the estimated signaling rate and distortion after applying the weighted map and the filter to the plurality of reference pictures or the first picture.

[0025] In some embodiments, the reference picture and / or the first picture includes a luma channel, a chroma channel, or both the luma channel and the chroma channel, and the weighted map and the filter are configured to be applied to the luma channel, the chroma channel, or both thereof.

[0026] In some embodiments, which of the luma channel and the chroma channel the weighted map and the filter are applied to is configured to be predetermined, signaled in the coding information, or inferred from the content of the picture.

[0027] In some embodiments, the method further comprises the step of dividing the plurality of reference pictures and the first picture into a plurality of partitions, wherein the weighted map and the filter are configured to be applied to one or more partitions of the plurality of reference pictures or the first picture, and the partitions are signaled in the coding information.

[0028] In some embodiments, the partitions signaled in the coding information include a block partition signaled in the coding information, a signaled region partition criterion, or a binarized weighted map function in the coding information.

[0029] In some embodiments, the method further comprises determining a plurality of filters applied to the same picture partition.

[0030] In some embodiments, the filter is configured to address the problems of ringing artifacts, blurring and / or blocking artifacts in pictures.

[0031] In some embodiments, the step of determining the weight map using the weight map function comprises applying a weight map function that outputs a scalar weight map, wherein the scalar is binary, integer or floating-point.

[0032] In some embodiments, the step of determining the weight map using the weight map function comprises applying a weight map function that outputs a multi-dimensional weight map, wherein each element is binary, integer or floating-point.

[0033] In some embodiments, the weight map information of one or more channels of the reference picture is calculated using information from one or more channels of the plurality of reference pictures as input.

[0034] In some embodiments, a set of weight map functions is predefined, and the encoded information signals the weight map function to be used.

[0035] In some embodiments, the weight map function is parametric.

[0036] In some embodiments, the encoded information signals a plurality of weight map functions, and the step of obtaining the weight map using the weight map function comprises determining a plurality of weight maps using the plurality of weight map functions, wherein one or more filters are configured to be applied to each signaled weight map.

[0037] In some embodiments, the filtering function and parameters of the filter are configured to be inferred from or from predefined video content signaled by the encoded information.

[0038] In some embodiments, the filter is a linear filter, and the shape of the filter is indicated or predefined by the bitstream.

[0039] In some embodiments, the linear filter is optimized by least-squares optimization or RD optimization.

[0040] In some embodiments, the linear filter is a parametric linear filter.

[0041] In some embodiments, the parametric linear filter is RD-optimized with respect to the minimum error in the output and is derived by least-squares optimization, iterative search, or exhaustive search.

[0042] In some embodiments, the filter is a bilateral filter.

[0043] In some embodiments, the step of obtaining the filter includes the step of obtaining a plurality of filters, each of which is configured to be applied at a position signaled in the bitstream or at a position indicated in the weighted map.

[0044] In some embodiments, the parametric weight map is optimized in conjunction with the filtering function.

[0045] In some embodiments, one or more filters are configured to be applied to the partitions of the reference picture or the first picture based on the block division signaled by the encoded information.

[0046] In some embodiments, one or more filters are configured to be applied to the partitions of the reference picture or the first picture based on the derived region division criteria.

[0047] In some embodiments, the filter and weighted map calculation parameters are encoded by a quantization, prediction, and / or entropy coding / decoding scheme.

[0048] According to the second embodiment, a computer-readable medium is provided, which includes computer-executable instructions stored in the computer-readable medium, and when the computer-executable instructions are executed by a computing device, the computing device causes the computing device to perform any of the methods of the first embodiment.

[0049] According to a third embodiment, an encoder is provided, the encoder comprising one or more processors and a computer-readable medium storing computer-executable instructions, wherein when the computer-executable instructions are executed by the one or more processors, the one or more processors cause the one or more processors to execute any of the methods of the first embodiment.

[0050] A fourth aspect provides a video data processing method performed by a decoder, the method comprising: decoding a bitstream to obtain video data and encoding information; obtaining a plurality of reference pictures from the video data; upsampling the plurality of reference pictures to obtain a plurality of upsampled reference pictures; determining a weight map for each reference picture using a weight map function, wherein the weight map includes a plurality of weights mapped to the respective spatial positions of the plurality of upsampled reference pictures, the plurality of upsampled reference pictures being used as input to the weight map function; and determining a filter to be applied to the plurality of upsampled reference pictures together with each of the weight maps. The method includes the steps of obtaining filtered upsampled reference pictures, thereby obtaining a weighted filter that reduces the overall error between the plurality of upsampled reference pictures and the corresponding original picture by applying the filter with different weights to different spatial locations of the plurality of upsampled reference pictures, and performing interpretation of a plurality of blocks of a first picture based on a plurality of reference blocks from one or more of the plurality of upsampled reference pictures, the method further including the steps of applying the weighted filter to the plurality of upsampled reference pictures before performing the interpretation, or applying the weighted filter to the upsampled reference blocks of the reference picture during the interpretation.

[0051] In some embodiments, the method further includes the step of adding the acquired weighted filter to a stored set of reference filters.

[0052] In some embodiments, the encoded information includes signaled weighted map function parameters, and the step of determining the weighted map using the weighted map function includes the steps of applying the signaled weighted map function parameters as parameters of the weighted map function, and providing the reference picture or the first picture as input to the weighted map function.

[0053] In some embodiments, the encoded information includes signaled filter function parameters, and the step of obtaining the filter includes the step of applying the signaled filter function parameters as parameters of the filter.

[0054] In some embodiments, the step of applying the weight map and the filter to the picture in order to obtain a filtered reference picture or a first picture is performed within the encoding / decoding loop or as a post-loop step.

[0055] In some embodiments, the encoding / decoding loop is an H.266 / VVC encoding / decoding loop.

[0056] In some embodiments, the step of applying the weighted map and the filter to the plurality of reference pictures or the first picture in order to obtain filtered pictures is integrated into an adaptive loop filter and applied to the derived partition.

[0057] In some embodiments, the bitstream is rate-distortion (RD) optimized based on the estimated signaling rate and distortion after applying the weight map and the filter to the plurality of reference pictures or the first picture.

[0058] In some embodiments, the reference picture and / or the picture includes a luma channel, a chroma channel, or both a luma channel and a chroma channel, and the weighting map and the filter are applied to the luma channel, the chroma channel, or both a luma channel and a chroma channel.

[0059] In some embodiments, whether the weighting map and the filter are applied to the luminous channel or the chroma channel is predetermined, signaled in the bitstream, or inferred from the content of the picture.

[0060] In some embodiments, the method further includes the step of dividing the plurality of reference pictures and the first picture into a plurality of partitions, wherein the weighted map and the filter are applied to one or more partitions of the plurality of reference pictures or the first picture, and the partitions are signaled with the encoded information.

[0061] In some embodiments, the partition signaled with the encoded information includes a block partition, a signaled region partitioning criterion, or a binarized weighted map function in the encoded information.

[0062] In some embodiments, the method further includes the step of applying multiple filters to the same picture partition.

[0063] In some embodiments, the filter is configured to address ringing artifacts, blurring, and / or blocking artifacts in the picture.

[0064] In some embodiments, the step of determining the weight map using a weight map function includes the step of applying a weight map function that outputs a scalar weight map, wherein the scalar in the scalar weight map is binary, integer, or floating-point.

[0065] In some embodiments, the step of determining the weight map using a weight map function includes the step of applying a weight map function that outputs a multidimensional weight map, where each element is binary, integer, or floating-point.

[0066] In some embodiments, the weighted map information for one or more channels of the reference picture is calculated using information from one or more channels of the plurality of reference pictures as input.

[0067] In some embodiments, a set of weighted map functions is predefined, and the encoded information signals which weighted map function to be used.

[0068] In some embodiments, the weighted map function is parametric.

[0069] In some embodiments, the encoded information signals a plurality of weighted map functions, the step of determining the weighted map using the weighted map functions includes the step of determining a plurality of weighted maps using the plurality of weighted map functions, and the step of applying the weighted map and the filters includes the step of applying one or more filters to each signaled weighted map.

[0070] In some embodiments, the filtering function and parameters of the filter are predefined, inferred from the video content, or signaled by the encoded information.

[0071] In some embodiments, the filter is a linear filter, and the shape of the filter is indicated or predefined by the bitstream.

[0072] In some embodiments, the linear filter is optimized by least-squares optimization or RD optimization.

[0073] In some embodiments, the linear filter is a parametric linear filter.

[0074] In some embodiments, the parametric linear filter is RD-optimized with respect to the minimum error in the output and is derived by least-squares optimization, iterative search, or exhaustive search.

[0075] In some embodiments, the filter is a bilateral filter.

[0076] In some embodiments, the step of determining the filter includes the step of determining a plurality of filters, and the step of applying the weighting map and the filters to the reference picture or first picture includes the step of applying each of the plurality of filters at a position signaled in the bitstream or at a position indicated in the weighting map.

[0077] In some embodiments, the parametric weight map is optimized in conjunction with the filtering function.

[0078] In some embodiments, the step of applying the weighted map and the filter to the picture includes the step of applying one or more filters to the reference picture or partitions of the first picture based on the block division signaled by the encoded information.

[0079] In some embodiments, the step of applying the weighted map and the filters to the plurality of reference pictures or the first picture includes the step of applying one or more filters to the partitions of the reference pictures or the first picture based on the derived region division criteria.

[0080] In some embodiments, the filter and weighted map calculation parameters are encoded by a quantization, prediction, and / or entropy coding / decoding scheme.

[0081] According to the fifth aspect, a computer-readable medium is provided, which includes computer-executable instructions stored in the computer-readable medium, and when the computer-executable instructions are executed by a computing device, the computing device causes the computing device to perform any of the methods of the fourth aspect.

[0082] According to the sixth aspect, a decoder is provided, the decoder comprising one or more processors and a computer-readable medium storing computer-executable instructions, wherein when the computer-executable instructions are executed by the one or more processors, the one or more processors cause the one or more processors to execute any method of the fourth aspect.

[0083] According to the seventh aspect, a video data processing method performed by an encoder is provided, the method comprising: acquiring original video data; performing trial coding on at least a portion of the original video data to obtain trial coded video data; acquiring a trial first picture based on the trial coded video data by performing interpretation of a plurality of blocks of the trial first picture based on a plurality of reference blocks from one or more reference pictures, wherein the one or more reference pictures have a lower resolution than the trial first picture; and determining a weighted map using a weighted map function, wherein the weighted map includes a plurality of weights mapped to their respective spatial positions in each of the plurality of interpretation blocks of the trial first picture. The process includes the steps of obtaining a weighted filter applied to the multiple inter-prediction blocks of the trial first picture, thereby reducing errors between the trial first picture and the corresponding original first picture in the original video data, by using the multiple inter-prediction blocks of the trial first picture as input to the weighted map function, and determining a filter to be applied to the multiple inter-prediction blocks of the trial first picture together with the weighted map, wherein the filter is applied with different weights to different spatial positions in each of the inter-prediction blocks of the trial first picture, and the filter is configured to obtain a filtered first picture applied to the multiple inter-prediction blocks of the trial first picture together with the weighted map.

[0084] In some embodiments, the method further includes the step of encoding the video data and encoding information, wherein the encoding information includes information about the weighted map function and / or the filter used in the decoder.

[0085] In some embodiments, the method further includes the steps of: determining whether the rate distortion performance of the encoding of the video data and the encoded information is better than the rate distortion performance of the trial encoding; including the encoded video data and encoded information in a bitstream transmitted to a decoder based on the fact that the rate distortion performance of the encoding of the video data and the encoded information is better than the rate distortion performance of the trial encoding; and including the trial encoded video data in a bitstream transmitted to a decoder based on the fact that the rate distortion performance of the encoding of the video data and the encoded information is worse than the rate distortion performance of the trial encoding.

[0086] In some embodiments, the method further includes the step of adding the acquired weighted filter to a stored set of reference filters.

[0087] In some embodiments, the trial coding further includes deriving a trial prediction signal by pre-analysis of the one or more reference pictures, the first picture, the corresponding original pictures in the original video data, and / or coding information.

[0088] In some embodiments, the method further includes the step of repeatedly performing the pre-analysis or trial coding and the step of obtaining a weighted filter until a stopping criterion is met.

[0089] According to the eighth aspect, a computer-readable medium is provided, which includes computer-executable instructions stored in the computer-readable medium, and when the computer-executable instructions are executed by a computing device, the computing device causes the computing device to perform any of the methods of the seventh aspect.

[0090] According to the ninth aspect, an encoder is provided, the encoder comprising one or more processors and a computer-readable medium storing computer-executable instructions thereon, wherein when the computer-executable instructions are executed by the one or more processors, the one or more processors cause the one or more processors to execute any method of the seventh aspect.

[0091] According to a tenth aspect, a video data processing method is provided which is performed by a decoder, the method comprising: decoding a bitstream to obtain video data and encoding information; obtaining one or more reference pictures; performing interpretation of a plurality of blocks of a first picture based on a plurality of reference blocks from the one or more reference pictures, wherein the one or more reference pictures have a lower resolution than the first picture; and determining a weight map using a weight map function, wherein the weight map includes a plurality of weights mapped to each spatial position in each of the plurality of interpretation blocks of the first picture, and the plurality of interpretation blocks of the first picture are inputs to the weight map function The process includes: determining which filters are used and which are applied to the plurality of interprediction blocks of the first picture together with the weighting map, wherein the filters are applied with different weights to different spatial positions in each of the interprediction blocks of the first picture, and the filters are configured to obtain a filtered first picture applied to the plurality of interprediction blocks of the first picture together with the weighting map, thereby reducing errors between the first picture and the corresponding original first picture; and applying the weighting filters to the plurality of interprediction blocks of the first picture.

[0092] In some embodiments, the method further includes the step of adding the acquired weighted filter to a stored set of reference filters.

[0093] Applying filters with local weightings improves encoding and decoding performance and enables a wider range of applications. Weighted filtering may be used to derive the characteristics of the filter at each spatial location using weights, through the use of a weighting function to determine a weighting map. For example, the intensity of a sharpening filter may be increased at picture locations close to edges and decreased in areas far from edges. This may reduce ringing artifacts and overshoot while maintaining sharpening properties.

[0094] These and other aspects of the present application will become more readily apparent from the following description of the embodiments. [Brief explanation of the drawing]

[0095] Embodiments are described below with reference to the accompanying drawings, for illustrative purposes only. [Figure 1] A flowchart illustrating the operation of the decoder according to the first embodiment is shown. [Figure 2] A flowchart illustrating the operation of the encoder according to the first embodiment is shown. [Figure 3] A block diagram illustrating exemplary operation in the first embodiment. [Figure 4] The graph shows multiple filtering methods compared to the ground truth signal. [Figure 5] A block diagram showing exemplary operation in a modified example of the first embodiment. [Figure 6] A flowchart illustrating the operation of the decoder according to the first embodiment is shown. [Figure 7] A flowchart illustrating the operation of the encoder according to the first embodiment is shown. [Figure 8] These are schematic diagrams of decoders according to various embodiments. [Figure 9] These are schematic diagrams of encoders according to various embodiments. [Modes for carrying out the invention]

[0096] The technical solutions in the embodiments of this application are described below clearly and completely with reference to the accompanying drawings.

[0097] The VVC compression standard introduced Reference Picture Resampling (RPR) as a tool for adaptively changing resolution within the encoded and decoded video sequence. Unlike previous standards, the output resolution is not limited to the resolution at which a given frame of the video sequence is encoded and decoded. To achieve this functionality, a set of filters was introduced to facilitate switching between different resolutions.

[0098] The filters used for resolution modification in the VVC standard are currently fixed linear filters. These filters are suitable for a wide range of video content. However, the content within encoded / decoded video sequences is locally correlated and dependent on the specific video sequence. Furthermore, image upscaling is inherently a nonlinear problem. To account for this, the embodiments discussed herein utilize adaptive local weighting filters. This is based on the assumption that optimizing the filters using knowledge of the encoding / decoding and the errors caused by the characteristics of the current frame improves the performance of the encoding / decoding scheme.

[0099] Some configurations support both switching between encoding and decoding resolutions, and temporary encoding and decoding at lower resolutions. Therefore, both downsampling and upsampling procedures are required to switch resolutions and reference pictures of different resolutions in interpretation. Some reference software employs a fixed upsampling procedure with interpolation filters for both of these applications. Upsampling is performed by a finite impulse response filter. The resolution of the output device remains unchanged, but the output itself still needs to be displayed at high resolution. This third embodiment addresses two application scenarios. The first application scenario utilizes enhancement filters only within the encoding / decoding loop; i.e., filters are not used to improve the low-resolution picture upscaled for the output. This involves changing to an upsampling procedure without affecting the downsampling procedure. Consequently, the evaluation of these changes is most effective in the high-resolution region.

[0100] Pictures encoded and decoded at low resolution often lack high-frequency information present in their high-resolution equivalents. As a result, low-resolution encoded and decoded pictures typically appear blurry after upsampling. Consequently, prediction errors are higher than usual when low-resolution frames are referenced. This problem stems from the fact that upsampling processes cannot recover high-frequency information lost during downsampling. However, artifacts caused by reference picture resampling (RPR) are most pronounced in areas that do not conform to the underlying assumptions of the interpolation scheme (i.e., smoothness). In reference software, this is particularly noticeable in content containing high-frequency information such as edges.

[0101] Current reference software employs a fixed set of downsampling and upsampling filters to improve the quality of upsampled pictures or predicted signals, without utilizing any side information from the encoded / decoded video or bitstream. For example, the distribution of frequency components may differ depending on the encoded / decoded video sequence or spatial position, making the most optimal filter dependent on the characteristics of the content. Current filtering methods, such as adaptive loop filters, solve this problem by signaling the filter set. The decision of which filter to apply to a given picture partition can be signaled and further inferred from the picture content. However, in the case of upsampled image content, there are different kinds of errors compared to in-loop filters. Typically, there is some blurring caused by downsampling and upsampling. Therefore, applying sharpening, especially at edges, can be beneficial.

[0102] The embodiments discussed herein address the limitations of upsampling procedures in reference picture resampling (RPR) and picture upsampling. Some current implementations take a frame or block as input and apply a multi-phase interpolation filter set to interpolate subsample positions. The problem is that the problem of image upsampling is inherently a poorly set problem. Therefore, there is no filtering method that can solve this problem for any kind of content. However, side information inferred from encoded / decoded video or encoded in a bitstream can be used to improve this problem. The embodiments discussed herein can be applied by adding another processing step after upsampling, namely an adaptive local weighting filter.

[0103] The underlying assumption is that the optimal strength of the filter depends on the spatial position and can be inferred from the content. The filter is interdependent because it is least-squares optimized based on the applied weights and the upsampled picture. Therefore, the overall performance of the filter depends on the initial selection of the weighting function. For example, consider a least-squares optimized (unweighted) filter such that the results are added to the filtered picture to obtain the output. After this optimization, a particular pixel may have an optimal offset, an offset that is too large, an offset that is too small, or a missigned calculated offset. Here, if it is possible to compute local weights that reduce the offset applied by the filter in areas where the calculated offset is too high or wrongly signed, and increase the offset in areas where the offset is too small, the error will be reduced. Furthermore, since the weights are known beforehand, a better filter can be optimized. However, finding the optimal weighting map function for all kinds of content and all positions within a picture is complex. Nevertheless, approximating the optimal weights with a weighting map can already lead to improved results.

[0104] When RPR upscaling is applied, the content is usually blurred, especially in high-frequency structures such as edges. Linear filters typically exhibit ringing and overshoot artifacts, and filtered edges are often still less sharp compared to ground truth edges. Therefore, increasing the filter offset at the predicted location of the edge and decreasing it around it would improve the filter.

[0105] These technical solutions can be applied to H.265 / HEVC or H.266 / VVC video encoding and decoding systems (for example, in in-loop processing where other filters such as Adaptive Loop Filters (ALF) and Sample Adaptive Offset (SAO) filters are currently applied in such encoding processes). However, it should be understood that these technical solutions can be applied to any other video encoding and decoding system, including video compression. Furthermore, although these principles are mainly described in relation to video processing, they are also applicable to other data formats, including image or audio processing.

[0106] In these embodiments, “video” refers to one or more pictures. In other words, a video may include one or more pictures. A picture may also be called an “image.”

[0107] An "encoder" is a device that can encode data into a bitstream, and a "decoder" is a device that can decode a bitstream to obtain the encoded data or an approximation thereof. A "bitstream" contains a sequence of bits.

[0108] "Intra-prediction" and "inter-prediction" are two prediction operations that can be used within the HEVC and VVC frameworks for a decoder to process a received bitstream to obtain the original signal. In embodiments, "original signal" or "original video" is used to refer to the unencoded data in the encoder. In embodiments, a reference sample may refer to spatially and / or temporally separated picture data used for predicting a picture (or area of ​​a picture). Intra-prediction and inter-prediction operations are also used in the encoder to perform rate distortion determination.

[0109] More specifically, intra-prediction involves predicting data spatially within a single picture, without referencing other (temporarily separated) pictures. In other words, data from the first region of a picture is used to predict data from another region of the same picture, but without dependence on other temporally separated pictures. In this context, the data from the first region of a picture is considered a "reference sample."

[0110] Interpretation involves predicting data across multiple temporally separated pictures. In other words, data from the first region of the first picture is used to predict data from the second region of the second picture. The first and second regions may or may not be spatially separated from each other. In this context, the data from the first region of the first picture is considered a "reference sample." It should be further noted that interpretation may use multiple reference regions from different pictures simultaneously for a single prediction operation.

[0111] In this embodiment, the “residual” may refer to a value obtained based on the original value of the picture region and the predicted value of the picture region (for example, the difference between the original value and the predicted value).

[0112] In the embodiments, "block" may refer to a portion of a picture. For example, a picture may be divided into two or more blocks. However, this is merely an example. If the picture is not divided, "block" can refer to the entire picture.

[0113] In the embodiments, “filter” may refer to a filter that acts to enhance a signal, particularly an upsampled signal. Generally, in the embodiments described, the filter is configured to sharpen blurred content, reduce ringing artifacts, and / or reduce blocking artifacts. However, embodiments are not limited thereto, and the filter may instead be configured to provide alternative or additional enhancement in other embodiments.

[0114] A common optimization problem in video encoding and decoding is minimizing both transmission rate and distortion simultaneously. Lower transmission rates result in stronger and more visible distortion, degrading the viewer's perceived quality. Errors caused by encoding are not random but are caused by processing steps in the encoder and decoder. Two crucial steps in a video encoding and decoding system are prediction and transformation. Quantization of transformation coefficients causes reconstruction errors. Many video encoding and decoding systems employ a hybrid encoding and decoding structure in which the contents of a block are predicted by intra-prediction or inter-prediction. This prediction is usually not perfectly accurate. As a result, the difference with the ground truth signal is calculated, transformed, and encoded to compensate for prediction errors. The signal after summing the residuals is filtered by a so-called in-loop filter.

[0115] The processing steps that cause artifacts are not random. Embodiments of the present invention utilize prior information to address specific types of errors.

[0116] Two useful applications are reducing ringing artifacts and sharpening edges. These two problems can be difficult to address with linear filters. The weighted filters of embodiments of the present invention employ a simple concept to overcome the limitations of conventional linear filters. The idea is to apply two filters. The first filter extracts local information from the decoded picture. The second filter applies filtering that depends on the output of the first filter setting. In some embodiments, the second filter will be RD-optimized depending on the output of the first filter. This can signal a nonlinear filter that adapts to specific image features.

[0117] Furthermore, as discussed above, current video encoding and decoding schemes such as H.265 / HEVC (High Efficiency Video Coding) and H.266 / VVC (Versatile Video Coding) support spatial scalability of encoded and decoded video streams. Therefore, the spatial resolution at which video is encoded and decoded can be adaptively changed and no longer needs to be equivalent to the output or input resolution of the video. The advantage of this additional flexibility is that encoding and decoding lower-resolution video requires lower bitrates and can reduce computational complexity at the cost of losing high-frequency information in the downsampling step.

[0118] For example, in reference picture resampling (RPR), the resolution of the encoded / decoded video stream may be adaptively changed. As a result, the encoder may encode parts of the video stream at a lower resolution. RPR is applied in interpretation whenever a picture uses a reference picture with a different resolution than the current picture in interpretation. In this step, a resampling operation must be applied so that the referenced picture block is mapped to the same spatial resolution as the current picture.

[0119] Finding the optimal high-resolution representation from a low-resolution picture is a crucial part of these encoding and decoding schemes. One approach is to apply a set of multiphase finite impulse response (FIR) interpolation filters to the low-resolution picture. While these filters provide an approximation of the high-resolution image content, they cannot recover information lost during the downsampling process and suffer from the limitations of linear filtering. As a result, the upsampled image is often blurred.

[0120] Image sharpening can improve image quality. However, linear high-pass filters frequently introduce artifacts such as overshoot and ringing. Furthermore, distortions caused by downsampling and upsampling depend on the image content and the encoding / decoding quality of the video (which is affected by the Quantization Parameter (QP) value).

[0121] Figure 1 shows a flowchart of the operation of the decoder 80 according to the first embodiment. Figure 2 shows a flowchart of the operation of the encoder 90 according to this embodiment.

[0122] The flowchart in Figure 1 begins at step 101, in which the decoder 80 decodes the bitstream to obtain video data and encoding information. In this embodiment, the encoding information includes weighted map information.

[0123] In step 102, the decoder 80 obtains (or “reconstructs”) a set of reference pictures based on the video data. The video data includes a compressed version of the original video data. In this embodiment, step 102 includes obtaining a set of temporally separated pictures (or “frames”) according to a prediction scheme specified in H.266 / VVC, which are then stored as reference pictures (for example, for future interpretation of future pictures). However, the embodiment is not limited in this respect, and any other method of obtaining a set of reference pictures from encoded video data may be used instead in other embodiments. Examples include the prediction scheme specified in H.265 / HEVC.

[0124] In step 103, the decoder 80 upsamples the reference picture. Specifically, in this embodiment, the reference picture is encoded in the bitstream at a lower resolution, while the next picture (or "frame") is encoded at a higher resolution. Therefore, the reference picture is upsampled to the resolution of the next picture so that it can be used as a reference picture for interpretation of one or more blocks of the next picture.

[0125] In this embodiment, step 103 includes applying a multiphase FIR interpolation filter set to reconstruct intensity values ​​at fractional sample positions and increase the resolution of each reference picture. However, the embodiment is not limited thereto, and other upsampling methods may be applied instead. In particular, there are many different methods that can be used to perform interpolation. Essentially, upsampling has the problem that fractional sample positions need to be interpolated. These include bilinear interpolation, bicubic interpolation, nearest neighbor interpolation, Lanzos interpolation, and the like.

[0126] In step 104, the decoder 80 uses the weighted map information to obtain a weighted filter that reduces the overall error between multiple upsampled reference pictures (or blocks) and their corresponding original pictures (or blocks). In this embodiment, the weighted map information includes a weighted map function used to compute a weighted map for each reference picture. Each weighted map includes multiple weights mapped to their respective spatial positions in the upsampled reference pictures, and the upsampled reference pictures are used as input to the weighted map function.

[0127] In one such example, when a sharpening filter is used, the weighted map information included in the encoded information is a local gradient calculation function. In such an example, in step 104, the decoder 80 applies the gradient calculation function to each upsampled reference picture to calculate the gradient at each position in the upsampled reference picture, thereby obtaining a scalar weighted map having the same resolution as the upsampled reference picture (having values ​​corresponding to each respective position in the picture). In other words, the reference picture is provided as input to the weighted map function.

[0128] Furthermore, in other embodiments, the weighted map may have a different resolution than the upsampled reference picture and / or may contain vector values ​​instead of scalar values. Moreover, the example picture gradient function is merely illustrative, and it will be understood that in the actual implementation of the embodiments, the selection of an appropriate weighted map function will depend on the context, and in particular, on which filter is used. These factors and possible modifications will be discussed in more detail later.

[0129] Furthermore, while the encoding information in this embodiment has been discussed as including weighted map information, which includes weighted map functions, the embodiments are not limited in this respect. For example, in some embodiments, multiple weighted map functions are stored in the decoder 80. In such cases, the weighted map information instead includes an instruction on which weighted map to use. Furthermore, in some embodiments, the encoding information does not include any explicit weighted map instruction at all. Instead, for example, the encoding information includes filter information, and the decoder 80 then infers which weighted map function to use (for example, if the encoding information indicates that a sharpening filter should be used, the decoder infers that the picture gradient function should be used as the weighted map function). These factors and possible variations will be discussed in more detail later.

[0130] In step 104, the decoder 80 also determines the filter to be used with the weighted map. In this embodiment, the decoder 80 infers the filter to be used from the weighted map information. As discussed above, in one example, the weighted map information included in the encoding information is a local gradient calculation function. In this example, the decoder 80 can infer that a sharpening filter (e.g., pre-stored) should be used in combination with a weighted map that includes picture gradients.

[0131] In step 105, the decoder 80 applies filters and their respective weighting maps (collectively called "weighting filters") to each reference picture to obtain multiple highlighted reference pictures. Thus, step 105 involves using the determined weighting maps to ensure that the filters are applied to different regions of each upsampled reference picture with different intensities. However, as will be discussed further later, step 105 is selective.

[0132] In this embodiment, applying the weighting map and filter to the upsampled reference picture involves providing both the upsampled reference picture and the weighting map as inputs to the filter. This allows the filter to be applied to each value of the upsampled reference picture with different intensities, depending on the weight values ​​in the weighting map. The output of the filter is a map of offset values. In this embodiment, the map of offset values ​​corresponds to the upsampled reference picture in terms of resolution. Once the map of offset values ​​is output, the offset values ​​are added to the values ​​of the upsampled reference picture, resulting in the output (enhanced) reference picture.

[0133] In the above discussion, it is assumed that the filter is a sharpening filter configured to sharpen blurred edges. However, the embodiments are not limited to this, and any suitable filter for enhancing a picture may be used instead.

[0134] In step 106, the decoder 80 performs interpretation of multiple blocks of the first picture based on multiple reference blocks from one or more of the highlighted upsampled reference pictures. Specifically, the decoder 80 performs prediction using the highlighted reference pictures to obtain multiple predicted blocks of the first picture. More specifically, the decoder 80 performs interpretation using the highlighted reference pictures as reference pictures to obtain predicted blocks.

[0135] Following step 106, the predicted first picture may then be used for any desired purpose. In one example, decoder 80 then displays it to the viewer. In another example, decoder 80 stores the first picture for later use. In yet another example, decoder 80 transmits the first picture to an external device for display.

[0136] In some embodiments, the acquired weighted filters are added to a set of reference filters stored for future use.

[0137] As described above, step 105 is optional. This is because, instead of applying the weighted filter to multiple reference pictures, the weighted filter can be applied to the upsampling reference block of the reference picture during interpretation (but before the addition of residuals). Note that both alternatives are equivalent (except for rounding) and are simply different implementations. Therefore, in the modified embodiment, step 105 is not applied, and instead the weighted filter is applied to the first picture after interpretation has been performed (but before the addition of residuals).

[0138] The encoder 90 may perform complementary methods to encode the bitstream provided to the decoder 80. Figure 2 shows a flowchart of the operation of the encoder 90 according to this embodiment.

[0139] In step 201, the encoder 90 obtains multiple original pictures from the original video data. For example, the encoder 90 may receive the original video data from an external server via a communication network (e.g., the Internet). However, there are no limitations on the embodiment regarding how the original video data is obtained.

[0140] In step 202, the encoder 90 obtains multiple reference pictures, each corresponding to an original picture from the original video data, with the multiple reference pictures having a lower resolution than the original pictures from the original video data. For example, the multiple reference pictures may represent previous pictures (or "frames") encoded into a bitstream at a lower resolution than the corresponding original pictures from the original video data (i.e., they may be pictures that are decoded by the decoder based on the encoded bitstream). This is, for example, to account for the network state at the time of encoding. These previous pictures may be stored in the decoder and encoder as a reference frameset for use in predicting future pictures. Thus, each reference picture corresponds to an original picture from the original video data (with differences due to lower resolution and possible prediction errors when obtaining the reference pictures).

[0141] In step 203, the encoder 90 upsamples multiple reference pictures. Step 203 is performed in the same manner as step 103 in Figure 1, and for brevity, a detailed explanation is omitted here.

[0142] In step 204, the encoder 90 obtains a weighted filter that reduces the overall error between multiple upsampled reference pictures (or blocks) and their corresponding original pictures (or blocks). This step involves both determining a weighted map for each reference picture using a weighted map function, where the weighted map contains multiple weights mapped to each spatial location of the upsampled reference picture, and the upsampled reference picture is used as input to the weighted map function, and determining a filter to be applied to the upsampled reference picture along with each weighted map to obtain a filtered upsampled reference picture, thereby ensuring that the filter is applied with different weights to different spatial locations of the upsampled reference picture.

[0143] When performing step 204, the encoder 90 determines a combination of weighted maps and filters that will be used to highlight multiple reference pictures by applying filters with different weights to different regions of the upsampled reference picture.

[0144] To determine a combination of weighting maps and filters to enhance a picture, step 204 may involve a rate distortion (RD) optimization process that involves iteratively applying multiple filters along with multiple weighting maps to the upsampled reference picture. For each application, the average sum difference of values ​​between the resulting upsampled reference picture and the corresponding picture from the original video data is determined. This cycle continues until a stopping criterion is met (e.g., the optimal weighting map for a particular filter has been determined).

[0145] In other words, the encoder performs RD optimization based on the estimated signaling rate and the distortion of the resulting picture.

[0146] A first example of a suitable stopping criterion is that a particular combination of filters and weighting maps results in the mean absolute difference (or mean squared difference) of values ​​between the resulting multiple pictures and the corresponding pictures from the original video data being less than a predetermined threshold difference. A second example of a suitable stopping criterion is that the mean absolute difference (or mean squared difference) of values ​​between the weighting map of the current iteration of the iteration and the weighting map of the previous iteration is less than a second predetermined threshold difference. Because the first example of a suitable stopping criterion directly measures output quality, it can be assumed that it will result in higher final picture quality than the second example. However, the second example ensures that the iteration does not require excessive computation time. In some embodiments, both of these examples are used, and the iteration stops when either of these two stopping criteria is met.

[0147] In the examples discussed above with respect to Figures 1 and 2, a single (sharpening) filter is used (along with a corresponding weighting map). However, while this embodiment has discussed finding a combination of a single filter and a weighting map, the embodiment is not limited in this respect. For example, in some embodiments, the encoder may identify multiple combinations of different types of filters and weighting maps to be used.

[0148] The method in Figure 2 then proceeds to step 205, where the encoder 90 applies the filter and its respective weight map (collectively called the "weighted filter") to each reference picture to obtain multiple weighted reference pictures. Step 205 is performed in the same manner as step 105 in Figure 1, and for brevity, a detailed explanation is omitted here. However, as will be discussed further later, step 205 is selective.

[0149] In step 206, the encoder 90 performs interpretation of multiple blocks of the first picture based on multiple reference blocks from one or more of the enhanced upsampled reference pictures. Similar to step 205, step 206 involves a rate-distortion optimization operation to determine interpretation parameters that are then encoded into a bitstream. However, the embodiments are not limited thereto, and any form of interpretation may be performed instead.

[0150] In step 207, the encoder encodes the video data and encoding information into a bitstream, the encoding information including weight map instruction information that indicates a specific weight map for each reference picture.

[0151] In some embodiments, the acquired weighted filters are added to a set of reference filters stored for future use.

[0152] As described above, step 205 is optional. This is because, instead of applying the weighted filter to multiple reference pictures, the weighted filter can be applied to the upsampling reference block of the reference picture during interpretation (but before the addition of residuals). Note that both alternatives are equivalent (except for rounding) and are simply different implementations. Thus, in the variant of this embodiment, step 105 is not applied, and instead the weighted filter is applied to the first picture after interpretation has been performed (but before the addition of residuals).

[0153] Through the methods discussed with reference to Figures 1 and 2, it can be seen that there is an in-loop filtering method that can be applied within the encoding loop of a video compression system. The in-loop weighted filter employs a function for computing a local weighting / parameter map and a filtering function. The weighting map function computes the weighting map using the input picture and selective encoding information / signaled parameters. The filtering function computes the filtered picture using the input picture, the weighting / parameter map, and selective encoding information / signaled parameters.

[0154] This method improves encoding and decoding performance and enables a wider range of applications by applying filters with local weighting. Weighted filtering may be used to derive the characteristics of the filter at each spatial location using weights, through the use of a weighting function to determine the weighting map. For example, the intensity of a sharpening filter may be increased at picture locations close to edges and decreased in areas far from edges. This may reduce ringing artifacts and overshoot while maintaining sharpening properties.

[0155] In some examples, the optimization discussed with reference to step 204 in Figure 2 involves iterations between filter and weighted map function parameters. For example, the starting parameters for the weighted map are set, and then the filter parameters are optimized based on the current weighted map. Next, the weighted map parameters are optimized based on the found filter parameters, and so on. In such cases, the encoded information will include information about the weighted map function and the filter function (e.g., the parameters used). Of course, this is the basic form of the optimization procedure. In some cases, additional side effect conditions may be set. For example, not only to determine the best filter and weighted map in terms of image quality, but also to keep the encoding rate as low as possible. This can be achieved by introducing these conditions into both of their individual optimizations and selecting the starting point for the next iteration while also considering the rate cost. More generally, it is possible to introduce additional simplifications to limit the computational cost.

[0156] In this embodiment, the weighting map provides linear weighting for the filter. However, the embodiments are not limited thereto, and in other embodiments, the values ​​of the weighting map may instead modify the filtering procedure itself. For example, the filter may be parametric. For instance, the frequency response of an edge-enhancing filter may depend on local weighting map parameters. For example, the sigma value in unsharp masking (a type of sharpening filter) may depend on weighting parameters. In other words, the way the filter works, more specifically the function of the filter, is parametric and does not necessarily depend linearly on the weighting map. Another example is a filter that performs edge thinning (sharpening) by warping a picture. The intensity of the warping may depend on the current weighting map values.

[0157] In this embodiment, the weighted filter is applied in-loop. However, the embodiment is not limited to this particular order, and the weighted filter may be applied additionally or alternatively in other processing steps in other embodiments, such as post-loop.

[0158] As discussed above, in this embodiment, the encoder 90 optimizes the weighting filter for high-resolution pictures based on a set of reference pictures and their corresponding ground truth pictures. That is, the filter is RD-optimized so that the error between the upscaled and filtered low-resolution reference pictures and their ground truth is minimized. This is done under the assumption that a better reference set, i.e., a more emphasized reference picture, leads to an improved prediction of the current picture, resulting in better rate distortion performance for the current picture.

[0159] JPEG2026533907000002.jpg25162

number

[0160] JPEG2026533907000004.jpg31162

number

[0161] JPEG2026533907000006.jpg19163

[0162] Furthermore, in some embodiments, spatial weighting is possible. This allows for the local weighting of the loss function to be found using motion vector data or other encoded information of a reference picture.

[0163] For example, the weighting of the reference picture and the reference picture region may be derived from an analysis of the reference picture using the encoded information and the contents of all available previously encoded and decoded pictures.

[0164] JPEG2026533907000007.jpg71164

[0165] In this regard, trial coding (using potentially limited tools) may be performed to obtain an estimate of which regions are more important for filter optimization. For example, this trial coding may be performed to determine which blocks of which reference picture are most likely to be used in coding the first (predicted) picture, thereby allowing higher importance weights to be applied to those regions.

[0166] Restricted tools refer to simplification in the encoding of pictures (or blocks or slices). This may reduce the quality of the filter (different blocks may be selected), but it can also reduce execution time.

[0167] The restriction / simplification options include the following: - Test fewer intra / intermodes in the prediction (i.e., turn off some tools). For example, turn off all intra tools except directional, planar, and DC modes / check only affine and normal intermode predictions and do not use more complex tools such as decoder-side motion vector derivation (DMVD) or linear illumination compensation (LIC). - Use a set of early termination or restricted partitions in block partitioning. - Trial coding at a lower quality (usually takes less time).

[0168] JPEG2026533907000008.jpg19162

[0169] JPEG2026533907000009.jpg42161

number

[0170] In summary of this method according to the first embodiment, the weighting filter is applied as an enhancement filter for interpretation and upscaled output picture enhancement. One filter may be optimized for each low-resolution picture. Distortion may be given by the encoding cost of the picture and the filter. Distortion may be distortion after the filter has been applied. The filter is selected if the RD cost is low when the filter is applied. This filter may then be used to enhance any upscaled content generated from this picture. Thus, the same filter may be used when the filter is output (i.e., written as an upscaled picture) and when a block of this picture is referenced.

[0171] To illustrate the principles discussed above with reference to embodiments in Figures 1 and 2, an exemplary example is discussed here with reference to Figure 3.

[0172] Figure 3 is a block diagram illustrating the exemplary operation according to the embodiments of Figures 1 and 2 discussed above. In other words, Figure 3 shows a conceptual diagram of how a weighted filter can be implemented.

[0173] As seen in the processing shown in Figure 3, the distorted picture 31 is ultimately converted into the (enhanced) output picture 34.

[0174] The distorted picture corresponds to the upsampled reference picture discussed, for example, with reference to step 103 in Figure 1.

[0175] The input may be a decoded and upscaled picture or block, and side information encoded with an adaptive parameter set in the bitstream. First, the filter parameters are decoded. This may include information about the weighting map function, filter shape, luma flag and chroma flag, and filter coefficients. Next, the weighting map function is applied to obtain a local weighting map. Then, the filter is applied to the upscaled picture or block to generate a filtered map. The filtered map is multiplied by the local weights and added to the upsampled picture.

[0176] For example, referring to step 104 in Figure 1, this distorted picture is then subjected to a weighted map function (f) determined based on the encoding information. w-map ) is provided as input to 3A, resulting in a weighted map 32. In this particular example, as discussed above, the weighted map function 3A is a local gradient calculation function. Thus, the weighted map 32 contains multiple values ​​of the picture gradient at each spatial location within the distorted picture 31.

[0177] Still referring to step 104 in Figure 1, filter 3B is determined. As discussed above, in this embodiment, decoder 80 infers the filter to be used from the weighted map information. As also mentioned above, in this example, the weighted map information included in the encoding information is a local gradient calculation function. Therefore, in this example, decoder 80 infers that a sharpening filter (e.g., pre-stored) should be used in combination with a weighted map containing the picture gradient.

[0178] Referring to step 105 in Figure 1, the decoder 80 then applies the weighting map 32 and filter 3B to the distorted picture 31 to obtain the output picture 34. As seen in Figure 3, this step involves providing the distorted picture 31 and weighting map 32 as input to the (sharpening) filter 3B, thereby yielding a weighted enhancement map 33. This weighted enhancement map 33 contains multiple offset values ​​that are spatially corresponding to the values ​​of the distorted picture 31, respectively.

[0179] Next (as part of step 105 in Figure 1), the decoder 80 adds the offset values ​​from the weighted enhancement map 33 to each of the values ​​in the distorted picture 31 to obtain the output picture 34.

[0180] Through this method, it can be seen that blurring is reduced without significant issues related to ringing artifacts and overshoot that can be caused by sharpening filters. This is achieved by local weighting of the sharpening filter, ensuring that the filter is not applied uniformly to the entire picture, but rather to different spatial parts of the picture with different intensities depending on the characteristics of those spatial parts.

[0181] More specifically, the intensity of the sharpening filter was increased near the edges of the picture and decreased in areas farther from the edges. This reduces ringing artifacts and overshoot while maintaining sharpness.

[0182] It should be noted that in some embodiments, the addition of the offset map is incorporated into the filter function. However, for the purpose of better visualization, these operations are shown as separate steps in Figure 3.

[0183] Furthermore, as will be discussed in more detail later, in some embodiments, both (or either) of the weighted map and the filtering function may be parametric functions that depend on parameters signaled in the bitstream.

[0184] We can see that a weighted filter consists of two main components: a weighted / parameter-map calculation function and a (potentially parametric) filter.

[0185] First, a weight map is calculated. In this embodiment, the weight map is calculated for all points in the picture (however, in other embodiments, it may be calculated at a lower resolution). Next, a filter is applied. The weight / parameter map and the decoded picture are input to the filter. This produces a filtered picture.

[0186] In the embodiments shown in Figures 1 and 2, the method is applied to the luma channel. However, the embodiments are not limited in this respect. In other embodiments, the method is applied to the chroma channel only, or to both the luma and chroma channels. In other words, in some embodiments, both (or either) the weighting / parameter map calculation function and the filter may differ for each channel. Thus, separate information / parameters may be signaled for the luma and chroma components.

[0187] In this embodiment, the weighted filter replaces both the SAO and ALF in the VVC / H.266 system.

[0188] However, embodiments are not limited in this respect. For example, in other embodiments, the weighted filter replaces only one of the SAO or ALF, or is provided in addition to both the SAO and ALF. In another example, the weighted filter may be integrated into the ALF. As a result, each partition of the picture may be filtered by either an optimized linear filter or a weighted filter. This gives the ALF additional flexibility. For maximum flexibility, filters may be added at any point in the chain of in-loop filters.

[0189] While these specific examples have been discussed, it will be understood that the embodiments are not limited to the H.266 / VVC scheme. In other embodiments, the weighted filter may be applied to a completely different encoding / decoding scheme (e.g., H.265 / HEVC or any other suitable encoding / decoding scheme).

[0190] Furthermore, in other embodiments, weighted filters are applied as post-filters to improve the quality of encoded / decoded video. This may be beneficial when back-coupling (from in-loop filtering) leads to worse predictions of subsequent pictures. In such cases, out-loop / post-filtering would be beneficial. Such a decision may, in embodiments, be made when the encoder is performing the encoding.

[0191] Therefore, more generally, in embodiments, the weighted filter may be incorporated into the encoding / decoding loop as an additional processing step of an existing scheme, as a replacement for an existing loop filter, or in a manner that it is integrated into an existing loop filter.

[0192] To further illustrate the concept of weighted filters discussed above, another exemplary implementation is discussed here with reference to Figure 4. Figure 4 shows a graph comparing multiple filtering schemes with a ground truth signal.

[0193] Assume a blurred edge x is the input to the filter. The ground truth signal 41 (shown in Figure 4) is a step function. For ease of simplification and visualization, a one-dimensional signal is shown. Note that this is done solely to illustrate the concept in a simple way. In general, the methods discussed herein may be applied to signals of any dimension.

[0194] The graph in Figure 4 shows four lines: the ground truth signal 41, the blurred signal 42 (e.g., corresponding to the distorted signal 31 in Figure 3), the weighted filtered signal 43 (e.g., corresponding to the output picture 34 in Figure 3), and the unweighted filtered signal 44 (e.g., corresponding to the output picture 34 in Figure 3 when no weighted map was used, i.e., when only a filter was used).

[0195] In unweighted filtering, the blurred signal 42 is filtered by a least-squares optimized linear filter to approximate the original signal as accurately as possible, thereby obtaining the unweighted filtered signal 44. Figure 4 shows that this type of filtering increases edge sharpness, thereby providing a better approximation of the ground truth signal 41, but it causes suboptimal overshoot and ringing.

[0196] Better results are achieved when the filter's high-pass characteristics are stronger at the sharpest parts of the edges, and less so in areas where ringing and overshoot artifacts are caused by the filter. To achieve this, in the weighting signal 42, the offset of the filtered blurred edge compared to the blurred edge is scaled by a larger coefficient at the predicted edge position and by a smaller coefficient where overshoot or ringing artifacts are expected. For this purpose, local weightings are calculated from the picture (i.e., a weighting map is calculated). For example, the magnitude of the gradient approximated by a finite difference (as discussed above with respect to Figures 1 to 3) may be used. To obtain the best results, the filter would then be optimized taking the local weightings into account.

[0197] As shown in Figure 4, the weighted filtered signal 43 has a smaller error than the unweighted filtered signal 44 compared to the ground truth signal 41. Furthermore, in this example, the overshoot of the weighted filtered signal 43 is at a similar level, but the signal sharpness and ringing are not as severe (i.e., it has a similar amplitude but flattens out more quickly).

[0198] This example demonstrates that when signal and error characteristics are known, local adaptability is beneficial for improving filter performance. Adaptive loop filters (ALFs) approach this by applying different filters depending on the picture characteristics. This allows for greater flexibility but comes at the cost of increased bitrate.

[0199] Embodiments of the present invention utilize weighted (and possibly parametric) filters to reduce the need for the use of many different filters. The example shown is one application case in which a single weighted filter can replace a set of filters and achieve similar results. This is particularly applicable when dependencies exist that can be exploited through local parameterization.

[0200] In this embodiment, the coding information has been discussed as including weighted map information, which includes weighted map functions. However, embodiments are not limited in this respect. For example, as discussed above, in some embodiments, multiple weighted map functions are stored in the decoder 80. In such cases, the weighted map information instead includes an instruction on which weighted map to use. Furthermore, in some embodiments, the coding information does not include any explicit weighted map instruction at all. Instead, for example, the coding information includes filter information, and the decoder 80 then infers which weighted map function to use (for example, if the coding information indicates that a sharpening filter should be used, the decoder infers that the picture gradient function should be used as the weighted map function).

[0201] In the embodiments discussed above with reference to Figures 1 and 2, the weighted filter is applied in a loop manner, specifically within the VVC / H.266 system.

[0202] In general, in-loop filters may be applied at all points in the encoding / decoding loop. However, since most loop filters are nonlinear, the order of application may affect the overall performance. An example of a video encoding / decoding system is VVC / H.266. This system has four in-loop filters that are applied sequentially: luma mapping with chroma scaling (LMCS), deblocking, sample-adaptive offset (SAO), and adaptive loop filter (ALF). LMCS addresses errors that are quite different from those of the proposed filter, and inverse mapping should be applied before the proposed filter to avoid artifacts. Furthermore, it makes sense to apply a deblocking filter before the proposed method.

[0203] While specific realizations of the present invention have been discussed above, many variations may be made in other embodiments. These will be discussed here, in particular with respect to the selection of the weighted map function, filtering function, domain partitioning, and signaling.

[0204] As is clear from the above discussion, a weighted map provides one or more weights / parameters to a filter. In other words, a filter is a parametric function that takes a weighted map and a picture (and possibly encoding parameters) as input.

[0205] Regarding the weighting map, in the embodiments shown in Figures 1 and 2, the weighting map is a scalar map. In this scalar map, exactly one value is assigned to each spatial position. However, in other embodiments, the weighting map is a multidimensional map. In such embodiments, there is a vector of values ​​at each spatial position. It should also be noted that in embodiments of the present invention, the spatial size of the weighting map is not limited to the resolution of the picture. Depending on the requirements, in some embodiments, a smaller resolution may be used to reduce computational complexity. The optimal choice of weighting map function depends largely on the type of errors to be addressed by the filters and the type of filters applied to address those errors.

[0206] As discussed above, in the embodiments of Figures 1 and 2, applying the weighted map and filter to the reference picture / interpretation picture (step 105 in Figure 1) involves providing both the picture and the weighted map as input to the filter. This allows the filter to be applied to each value of the picture with different intensity, depending on the value of each weight in the weighted map. The output of the filter is a map of offset values, which corresponds to the picture in resolution. Once the map of offset values ​​is output, the offset values ​​are added to the values ​​of the picture, resulting in the output (enhanced) picture. An example of this is shown, for example, in Figure 3 discussed above.

[0207] However, the embodiments are not limited to this particular implementation. In an alternative and simpler implementation, the intensity values ​​(i.e., the weights of the weighting map) are used by the filter to scale the offset generated by the filter. This is done by multiplying the intensity values ​​(i.e., weights) by the difference between the filter output and the acquired picture. Adding this scaled difference to the acquired picture modifies the offset generated by the filter according to the calculated weights. As a result, the effect of the applied filter differs depending on the spatial position.

[0208] An example of this configuration is shown in Figure 5. Figure 5 is a block diagram illustrating the exemplary operation of this alternative method.

[0209] In the method shown in Figure 5, the distorted picture 51 is provided as input to a weighting map and separately as input to a filter. The resulting intensity values ​​(i.e., weights) of the weighting map 52 are then multiplied by the difference between the filter output 55 and the distorted picture 51. Adding this scaled difference to the distorted picture 51 modifies the offset generated by the filter according to the calculated weights.

[0210] In other words, scalar weightings are calculated to weight the filter's output by sample-by-sample multiplication. The calculated offsets are added to the input picture to obtain the output.

[0211] A comparison with the example in Figure 3 shows that the weighted map 52 is not applied as an input to the filter, but rather its value is simply multiplied by the output from the filter (specifically, the difference between the filter output 55 and the distorted picture 51).

[0212] Figure 6 shows a flowchart of the operation of the decoder 80 according to the second embodiment. Figure 7 shows a flowchart of the operation of the encoder 90 according to this second embodiment.

[0213] The flowchart in Figure 1 begins at step 601, in which the decoder 80 decodes the bitstream to obtain video data and encoding information. In this embodiment, the encoding information includes weighted map information.

[0214] In step 602, the decoder 80 acquires one or more reference pictures. Step 602 is performed in the same manner as step 102 in Figure 1, and for brevity, a detailed explanation is omitted here.

[0215] In step 603, the decoder performs interpretation of multiple blocks of the first picture based on multiple reference blocks from one or more reference pictures. Specifically, decoder 80 performs prediction using the reference pictures to obtain multiple predicted blocks of the first picture. More specifically, decoder 80 performs interpretation using the reference pictures as reference pictures to obtain predicted blocks.

[0216] In this embodiment, one or more reference pictures have a lower resolution than the first picture. This is because, for example, multiple reference pictures may represent previous pictures (or "frames") encoded into a bitstream at a lower resolution than the corresponding original pictures from the original video data (i.e., they may be pictures that are decoded by the decoder based on the encoded bitstream). This is to take into account, for example, the network state at the time of encoding. These previous pictures may be stored in the decoder and encoder as a reference frameset for use in predicting future pictures. Thus, each reference picture corresponds to an original picture from the original video data (with differences due to lower resolution and possible prediction errors when obtaining the reference picture).

[0217] Therefore, the interpretation operation in step 604 involves upsampling the reference block (or the entire reference picture) of the reference picture in order to predict the block of the first picture.

[0218] In step 604, the decoder 80 obtains a weighted filter applied to the multiple inter-prediction blocks of the first picture based on the encoding information, by determining a weighted map using a weighted map function, the weighted map including a plurality of weights mapped to each spatial position in each of the inter-prediction blocks of the first picture, the inter-prediction blocks of the first picture being used as input to the weighted map function, and by determining a filter to be applied to the plurality of inter-prediction blocks of the first picture together with the weighted map, the filter being applied to each of the inter-prediction blocks of the first picture with different weights, the filter being applied to the plurality of inter-prediction blocks of the first picture together with the weighted map to obtain a filtered first picture, thereby reducing errors between the first picture and the corresponding original first picture.

[0219] Here, the decoder 80 determines the weight map using the weight map information. In this embodiment, the decoder 80 determines the weight map using the weight map information. In this embodiment, the weight map information includes a weight map function used to calculate the weight map.

[0220] In one such example, when a sharpening filter is used, the weighted map information included in the encoded information is a local gradient calculation function. In such an example, in step 103, the decoder 80 applies the gradient calculation function to calculate the gradient at each position in the interprediction block of the first picture, thereby obtaining a scalar weighted map having the same resolution as the first picture (having values ​​corresponding to each respective position in the picture). In other words, the first picture is provided as input to the weighted map function.

[0221] However, it should be understood that this is merely one example. In other embodiments, the weighted map may have a different resolution than the first picture and / or may contain vector values ​​instead of scalar values. Furthermore, it should be understood that the example of the picture gradient function is merely an example, and in the actual implementation of the embodiments, the choice of an appropriate weighted map function will depend on the situation, and in particular on which filter is used. These factors and possible variations will be discussed in more detail later.

[0222] Furthermore, while the embodiments have been discussed in this embodiment as including weighted map information containing weighted map functions, the embodiments are not limited in this respect. For example, in some embodiments, multiple weighted map functions are stored in the decoder 80. In such cases, the weighted map information instead includes instructions on which weighted map to use. Furthermore, in some embodiments, the encoding information does not include any explicit weighted map instructions. Instead, for example, the encoding information includes filter information, and the decoder 80 then infers which weighted map function to use (for example, if the encoding information indicates that a sharpening filter should be used, the decoder infers that a picture gradient function should be used as the weighted map function). These factors and possible variations will be discussed in more detail later.

[0223] In step 604, the decoder 80 further determines the filter to be used. In this embodiment, the decoder 80 infers the filter to be used from the weighted map information. As discussed above, in one example, the weighted map information included in the encoding information is a local gradient calculation function. In this example, the decoder 80 can infer that a sharpening filter (e.g., pre-stored) should be used in combination with a weighted map that includes the picture gradient.

[0224] In step 606, the decoder 80 applies the weighted map and filter to the interprediction block of the first picture to obtain the filtered picture. Thus, step 606 involves using the determined weighted map to ensure that the filter is applied to different regions of the interprediction block of the first picture with different intensities.

[0225] In this embodiment, applying the weighting map and filter to the first picture involves providing both the first picture and the weighting map as inputs to the filter. This allows the filter to be applied to each value of the picture with different intensity, depending on the value of each weight in the weighting map. The output of the filter is a map of offset values. In this embodiment, the map of offset values ​​corresponds to the first picture in terms of resolution. Once the map of offset values ​​is output, the offset values ​​are added to the values ​​of the first picture to produce the output (enhanced) picture.

[0226] In the above discussion, it is assumed that the filter is a sharpening filter configured to sharpen blurred edges. However, the embodiments are not limited to this, and any suitable filter for enhancing a picture may be used instead.

[0227] Following step 605, the output picture may then be used for any desired purpose. In one example, the decoder 80 then displays the output picture to the viewer. In another example, the decoder 80 stores the picture for later use. In yet another example, the decoder 80 sends the picture to an external device for display.

[0228] In this embodiment, weighted filters may be added to the reference set for future filter optimization.

[0229] The encoder 90 may perform complementary methods to encode the bitstream provided to the decoder 80. Figure 7 shows a flowchart of the operation of the encoder 90 according to this embodiment.

[0230] In step 701, the encoder 90 acquires the original video data. For example, the encoder 90 may receive the original video data from an external server via a communication network (e.g., the Internet). However, there are no limitations on the embodiments regarding how the original video data is acquired.

[0231] In step 702, the encoder 90 performs trial coding on trial coded video data, which is at least a portion of the original video data. For example, this trial coding may involve coding interprediction parameters for obtaining a trial first picture from a reference picture. At least a portion of the original video data may refer to, for example, a picture (or "frame") or a slice.

[0232] In step 703, the decoder 90 obtains a trial first picture based on trial encoded video data. In this embodiment, this step involves performing interpretation of multiple blocks of the trial first picture based on multiple reference blocks from one or more reference pictures, where one or more reference pictures have a lower resolution than the trial first picture. Therefore, this step involves upsampling the referenced reference blocks from at least one or more reference pictures.

[0233] Step 704 involves determining a weight map using a weight map function, the weight map comprising a plurality of weights mapped to each spatial position in each interprediction block of the trial first picture, the interprediction blocks of the trial first picture being used as input to the weight map function, and determining a filter to be applied to the plurality of interprediction blocks of the trial first picture together with the weight map, the filter being applied with different weights to different spatial positions in each of the interprediction blocks of the trial first picture, the filter being configured to obtain a first picture that has been applied to and filtered by the plurality of interprediction blocks of the trial first picture together with the weight map, thereby obtaining a weight filter applied to the plurality of interprediction blocks of the trial first picture to reduce errors between the trial first picture and the corresponding original first picture in the original video data.

[0234] In performing these steps, the encoder 90 determines a combination of weighting maps and filters to be used to highlight the first picture by applying filters with different weights to different regions of the interpretation block of the first picture.

[0235] To determine a combination of weighting maps and filters that enhance a picture, step 704 may include a rate-distortion (RD) optimization process that involves iteratively applying multiple weighting maps and filters to the first trial picture. For each application, the mean difference in values ​​between the resulting picture and the corresponding picture from the original video data is determined. This cycle continues until a stopping criterion is met (e.g., the optimal weighting map for a particular filter has been determined).

[0236] In other words, the encoder performs RD optimization based on the estimated signaling rate and distortion of the obtained picture.

[0237] A first example of a suitable stopping criterion is that a particular weighting map results in the mean absolute difference (or mean squared difference) of values ​​between the resulting picture and the corresponding picture from the original video data being less than a predetermined threshold difference. A second example of a suitable stopping criterion is that the mean absolute difference (or mean squared difference) of values ​​between the weighting map of the current iteration of the iteration and the weighting map of the previous iteration is less than a second predetermined threshold difference. Because the first example of a suitable stopping criterion directly measures output quality, it can be assumed that it will result in a higher final picture quality than the second example. However, the second example ensures that the iteration does not require excessive computation time. In some embodiments, both of these examples are used, and the iteration stops when either of these two stopping criteria is met.

[0238] In the example discussed above with respect to Figure 6, a single (sharpening) filter is used (along with a corresponding weighting map). However, while this embodiment has discussed finding a combination of a single filter and a weighting map, the embodiment is not limited in this respect. For example, in some embodiments, the encoder may identify multiple combinations of different types of filters and weighting maps to be used.

[0239] The method in Figure 7 then proceeds to step 705, in which the encoder 90 encodes the video data and encoded information into a bitstream, the encoded information including weighted map information.

[0240] Through the methods discussed with reference to Figures 6 and 7, it can be seen that there is an in-loop filtering method that can be applied within the encoding loop of a video compression system. The in-loop weighted filter employs a function for computing a local weighting / parameter map and a filtering function. The weighting map function computes the weighting map using the input picture and selective encoding information / signaled parameters. The filtering function computes the filtered picture using the input picture, the weighting / parameter map, and selective encoding information / signaled parameters.

[0241] This method improves encoding and decoding performance and enables a wider range of applications by applying filters with local weighting. Weighted filtering may be used to derive the characteristics of the filter at each spatial location using weights, through the use of a weighting function to determine the weighting map. For example, the intensity of a sharpening filter may be increased at picture locations close to edges and decreased in areas far from edges. This may reduce ringing artifacts and overshoot while maintaining sharpening properties.

[0242] In some examples, the optimization discussed with reference to step 704 in Figure 7 involves iterations between filter and weighted map function parameters. For example, the starting parameters for the weighted map are set, and then the filter parameters are optimized based on the current weighted map. Next, the weighted map parameters are optimized based on the found filter parameters, and so on. In such cases, the encoded information will include information about the weighted map function and the filter function (e.g., the parameters used). Of course, this is the basic form of the optimization procedure. In some cases, additional side effect conditions may be set. For example, not only to determine the best filter and weighted map in terms of image quality, but also to keep the encoding rate as low as possible. This can be achieved by introducing these conditions into both of their individual optimizations and selecting the starting point for the next iteration while also considering the rate cost. More generally, it is possible to introduce additional simplifications to limit the computational cost.

[0243] In this embodiment, the weighting map provides linear weighting for the filter. However, the embodiments are not limited thereto, and in other embodiments, the values ​​of the weighting map may instead modify the filtering procedure itself. For example, the filter may be parametric. For instance, the frequency response of an edge-enhancing filter may depend on local weighting map parameters. For example, the sigma value in unsharp masking (a type of sharpening filter) may depend on weighting parameters. In other words, the way the filter works, more specifically the function of the filter, is parametric and does not necessarily depend linearly on the weighting map. Another example is a filter that performs edge thinning (sharpening) by warping a picture. The intensity of the warping may depend on the current weighting map values.

[0244] In this embodiment, weighted filters may be added to the reference set for future filter optimization.

[0245] In this embodiment, the weighted filter is applied in the loop. However, the embodiment is not limited to this particular order, and the weighted filter may be applied additionally or alternatively in other processing steps, such as post-loop, in other embodiments.

[0246] In a modified version of this embodiment, before step 705 in Figure 7, the encoder 90 makes a decision on whether to include any information about the weighted filter in the encoded information (i.e., whether the decoder should obtain and apply the weighted filter). In particular, as discussed above, steps 701-704 involve trial encoding of the picture (or slice) to find the interpredicted motion vector. This is used to find the low-resolution block referenced by the current picture. Given this information, the filter is optimized so as to minimize the error between the referenced block and the ground truth of the current (high-resolution) picture.

[0247] In this modified version, the next optimization round is performed after step 704, using a filter to highlight all low-resolution blocks (of the reference picture) in the motion compensation optimization so that an optimized motion vector that takes the filtered low-resolution picture into account can be found. Then, if the RD performance is better than the unfiltered version (i.e., the version without the weighted filter), it is determined that the decoder should use the weighted filter.

[0248] In summary, this modified version involves the following series of steps: 1. Perform trial encoding on the picture (or slice). 2. Find the predicted data (from the low-resolution reference picture) and the corresponding ground truth in the first (current) picture. 3. Optimize the weighted filter to minimize prediction errors. 4. Encode the picture (or slice) using a weighted filter. 5. If the RD performance in Step 4 is better than in Step 1: Adopt the result from Step 4 and add the filter to the reference set for future filter optimization. Otherwise, use the output from Step 1.

[0249] As discussed, this second embodiment involves optimizing filters based on a set of blocks (which may be transformed, e.g., shifted) such that the error between those blocks and the ground truth of the current picture is minimized. The set of blocks and their predicted signals may be derived by trial coding (or pre-analysis of the picture content) of the picture or slice. The filter or set of filters is optimized by minimizing the error between the predicted signals and the ground truth of the current block based on the potentially estimated signals in the picture (e.g., of the current frame).

[0250] In some embodiments, this method is applied iteratively because the predicted signal changes. This can lead to different sets of blocks being selected by the prior estimation method.

[0251] It will be further understood that the first and second embodiments discussed above can be combined by applying joint optimization. Thereafter, the weighted filter may be optimized to minimize the weighted sum of loss terms, which is the sum of the loss of the error terms generated by the method of the first embodiment or the method of the second embodiment.

[0252] In some embodiments, the weighted map function is an edge detector that assigns higher weights to positions on edges and lower values ​​to areas around edges and flat regions. Thus, in such embodiments, the resulting weighted map is scalar. This may allow edge sharpening to be achieved with fewer artifacts. In such embodiments, the encoder may encode, for example, information identifying the edge detector function in a bitstream.

[0253] In other embodiments, the weighted map function is a detector for ringing artifacts. This assigns, for example, the probability that ringing is present at a given location. Thus, in such embodiments, the resulting weighted map is a scalar. Through the application of this weighted map, the filter strength is then set based on this probability. In such embodiments, the filter may be, for example, a simple linear low-pass filter. However, often ringing is near edges that should ideally be preserved. Therefore, the filtering solution should ideally preserve the edges. An example of a suitable filter would be a bilateral filter. In embodiments utilizing such a filter, the parameters may be optimized in an encoder and transmitted in a bitstream.

[0254] Furthermore, in some embodiments, these two optional weighting maps may be combined into a two-dimensional weighting map, or both may be applied sequentially. Combining them provides greater flexibility with respect to filtering. Knowing the estimated probability of ringing and the presence of edges helps the encoder find the optimal filtering in its rate-distortion optimization process. For example, edge amplification should be applied more carefully when it is very close to this location, as it may be amplified and may be optimized by the encoder in the encoding process.

[0255] Examples of applying bilateral filters also illustrate how, in some embodiments, multidimensional weighting maps, rather than simple scalars, can be applied. For example, in some embodiments, the encoder determines different parameters when there are areas of high contrast compared to areas of low contrast. In this case, the bilateral filter parameters are estimated from the acquired picture and signaled to the decoder.

[0256] Another use in other embodiments is to estimate the parameters of an edge sharpening filter from a picture. For example, if the weighting map estimates whether there is very sharp content, such as text, at a particular location within a picture / block, different sharpening may be applied compared to other types of content. Note that in such embodiments, the weighting map may be derived from picture content, encoded information, or signaled parameters. In these embodiments, the weighting map may be one-dimensional. The weighting map may be binary, integer, or floating-point. In these embodiments, the weighting map may alternatively be multidimensional, with each element being binary, integer, or floating-point. The data type of each element of the weighting map depends on the requirements of the filtering system.

[0257] In some embodiments, weighted map calculation parameters are signaled in a bitstream to determine the type of weighted map function. In other embodiments, the signaled weighted map calculation parameters are parameters to the function itself. For example, an edge map may have a steepness scaling parameter that determines how much the weighting is increased based on steepness. Note that this may be a nonlinear scaling. For example, scaling by raising the value to the power of n may be used.

[0258] In some embodiments, a set of (possibly parametric) weighted map functions is predefined. Therefore, the bitstream signals only the weighted map functions / weighted maps used (and potentially specific parameters used in these weighted map functions), rather than the entire set of weighted map functions.

[0259] In summary regarding weighted maps, in some embodiments, a weighted map function is applied that outputs a scalar weighted map, where the scalar can be binary, integer, or floating-point. In other embodiments, a weighted map function is applied that outputs a multidimensional weighted map, where each element can be binary, integer, or floating-point. In these embodiments, weighted map information for one or more channels of the acquired picture is computed using information from one or more channels of the reconstructed picture as input. In some embodiments, a set of (possibly parametric) weighted map functions is predefined, and the bitstream signals which weighted map function / weighted map to use.

[0260] With respect to the filters, the filter functions used in some embodiments of the present invention are generally multidimensional parametric functions that take a weighting map, filter parameters, one or more channels of the acquired picture, and optionally encoding information as input. The output is one or more (weighted) filtered channels of the acquired picture.

[0261] While embodiments have generally been discussed in relation to the application of a single filter, the present invention is not limited in this respect. For example, in other embodiments, a series of filters having different parameters and possibly different weighting maps may be applied. Different filtering functions may be most appropriate depending on the type of artifact, complexity requirements, and RD determination.

[0262] For example, in some embodiments, a linear weighted filter may be used. In such embodiments, the filter may be weighted by multiplying the filter's output by a local weight and adding the result to the acquired picture. The advantage of such a system is that the optimal parameters can be found by least-squares optimization. As a result, parameter search is not required to find the optimal solution.

[0263] However, in such embodiments, a linear filter would require a relatively large number of filter coefficients (which need to be transmitted) depending on the characteristics of the picture. To address this, in some embodiments, a parametric description of the filter is used to reduce the encoding and decoding costs (although this comes at the expense of reduced flexibility).

[0264] One example of a parametric description is modeling a high-pass filter as a Gaussian difference filter. In this case, only the sigma value needs to be transmitted, rather than the entire set of filter coefficients. This approach is useful when the frequency response of the parametric filter is sufficiently close to the filter distribution that would be obtained by least-squares optimization. However, depending on the side effects of the parametric representation, a closed-form solution may not be found, in which case iterative optimization will be necessary.

[0265] Another type of filter that can be used in other embodiments is a (parametric) nonlinear filter. Examples include bilateral filters, median filters, or other filters. The parameters of these filters may be signaled in the bitstream or given by a weighting map. Note that it is also optional to switch the type of filtering function based on the weighting map parameters.

[0266] In other words, in some embodiments, the weighted map values ​​at a particular scalar position can indicate the type of filtering function used at that spatial position of the picture.

[0267] In summary regarding filters, in some embodiments, one or more filters may be applied to each signaled weight map. The filtering function and filter parameters may be signaled in the bitstream, predefined, or inferred from the content or encoding information of the video sequence. In some embodiments, a linear filter is applied as the filter of the filtering function. The shape of the filter may be indicated in the bitstream or predefined. In some embodiments, the linear filter is optimized by least-squares optimization or RD optimization. In some embodiments, a parametric linear filter is used, and parameters may be used to generate the corresponding linear filter. In some embodiments, the parametric linear filter is RD optimized with respect to the minimum error in the output, or RD optimized. The optimal filter may be derived by least-squares optimization, iterative search, or exhaustive search. In some embodiments, a parametric or non-parametric nonlinear filter is applied to the filtering function. In some embodiments, a bilateral filter is applied to the filtering function. In some embodiments, a combination of the described filtering methods is applied, and the filtering method used at each position is signaled or indicated by the weight map. In some embodiments, the encoder optimizes the parametric weight map together with the filtering function.

[0268] In some embodiments described herein, the weighting filter has been described as being applied to the entire picture (e.g., the entire reference picture). However, embodiments are not limited in this respect. In variations of these embodiments, different filter settings may be used for different partitions of the picture. Furthermore, overlapping application of filters is possible. Two exemplary implementations for region partitioning are discussed here.

[0269] The first is a block division of the picture, where each filter is applied to one or more blocks. Alignment to coding tree unit (CTU) and coding (CU) boundaries may be considered in some embodiments. The picture division in question will then be signaled in the bitstream.

[0270] The second is to partition the picture based on picture characteristics. In some embodiments, this partitioning may be derived on both the decoder and encoder sides without requiring signaling in the bitstream. This may be implemented as a binarized or non-binary weighted map. Each resulting partition is then processed individually, or the partitions are processed in groups in embodiments.

[0271] When region partitioning is used, the filters applied to one partition or group of partitions may be optimized on the encoder side, and the parameters may then be signaled in the bitstream.

[0272] Therefore, in some embodiments, the picture may be divided into multiple partitions. The partitions may be defined by signaled block partitioning, signaled region partitioning criteria, and / or binarized weighted map functions. Furthermore, in some embodiments, multiple filters may be applied to the same picture partition.

[0273] The use of partitions in these methods may be particularly useful when dealing with large or highly diverse pictures. Such pictures may contain very different types of content, and error characteristics may vary significantly across different partitions of the picture. Consequently, optimizing two or more filters for different partitions of a picture can lead to superior performance.

[0274] As discussed in the embodiments described, the encoding information may be included in the bitstream, for example, with respect to the weighted map function and / or filter applied. More generally, in embodiments, the encoding information may include, but is not limited to, filter coefficients, weighted map function parameters, on / off flags, filter encoding parameters, or region parameters.

[0275] In some embodiments, this information (e.g., all parameters) is encoded and signaled to reduce the transmission rate and improve the efficiency of the overall filtering process. This is done by utilizing redundancy with respect to the transmitted parameters. This redundancy is utilized by predicting and entropy coding of the filter parameters. Furthermore, in some embodiments, the parameters are quantized to reduce the number of possible representations. A detailed description of parameter coding and decoding schemes that can be used to encode the parameters of this type of filter can be found in PCT / CN2023 / 105596, which is incorporated herein by reference in its entirety.

[0276] In the embodiments described herein, the filter used (i.e., the sharpening filter) is based on the concept of a Wiener filter. In other embodiments, the filter is a linear filter optimized by the encoder using a least-squares optimization procedure (i.e., a linear filter that minimizes the squared error between the filtered signal and the ground truth signal). Of course, in some embodiments, additional side effect conditions are set in determining the signal enhancement filter, such as the filter shape and the filter coefficients that must be equal.

[0277] However, while embodiments have been discussed with reference to filters based on the concept of Wiener filters, embodiments are not limited in this respect, and other types of filters may be used instead. For example, filters based on Sobel filters or unsharp masking filters as sharpening filters. Other nonlinear options include bilateral filters and diffuse filters, as well as adaptive loop filters (ALFs).

[0278] For example, in some embodiments, the weighted filter may be integrated into an adaptive loop filter (ALF) of an existing encoding / decoding scheme (e.g., H.265 / HEVC and H.266 / VVC) and applied to partitions derived by ALF optimization as an alternative to the linear filter.

[0279] Given the similarities in determining the weighted filter in the first and second embodiments (e.g., steps 204 and 704 in Figure 2), the specific implementation details and variations described with respect to the first embodiment will not be repeated here for brevity. However, it will be understood that they are applicable to this second embodiment.

[0280] As an overview of this method in the second embodiment, suppose we have a video sequence encoded with RPR. The sequence is encoded in high resolution for a certain number of pictures, then switches to a lower encoding resolution, and then switches back to a higher resolution. RD-optimizing one filter for each low-resolution picture and applying it to inter-prediction blocks from this picture is not necessarily efficient, because the filter is also used to improve the upscaled low-resolution output picture, and therefore it is not necessary to upscale and refine the entire picture. Firstly, not all low-resolution pictures are used as references for high-resolution pictures. Furthermore, not all picture regions are used as references. This first embodiment optimizes the filter for blocks that are estimated to be used by high-resolution frames. Trial encoding of slices may be performed to find the referenced blocks. Then, all referenced low-resolution blocks are stored along with the ground truth data of the motion-compensated blocks in the current picture. If the referenced low-resolution region is below a predefined threshold, the optimization may be skipped. Otherwise, the filter may be optimized so that the summation of errors across all blocks is least-squares-minimized. Next, the filter is assigned to the high-resolution picture, and the picture is encoded again. If the RD cost of encoding the picture with the filter is lower than the cost of encoding it without the filter, the filter may be applied. Otherwise, the picture may be encoded without the filter.

[0281] For example, an optimization algorithm can be summarized as follows: 1. Encode the slice (or block or picture). 2. Test whether the reference is from a low-resolution picture in at least 2% of the picture area. a. If true: Proceed to step 3. b. If false: Proceed to step 7 3. RD optimize the emphasis based on interpredicted blocks and their corresponding ground truth. 4. Assign the filter to the current slice (or picture or block) as an enhancement filter, and perform trial encoding on the slice (or picture or block) 5. Test whether the RD performance when using the filter is better than that without the filter a. If true: proceed to step 7 b. If false: proceed to step 6 6. Delete the assigned filter, and re-encode the slice without the filter 7. End of slice compression

[0282] In summary, in some configurations, picture upscaling in RPR may be performed by a polyphase interpolation filter. Upscaling is used for inter prediction and to rescale the picture before the picture is written to the output stream. In these embodiments, this is modified such that the weighted edge enhancement filter is applied to the inter prediction process, or the edge enhancement filter is applied to the inter prediction to upscale the low-resolution picture after upscaling.

[0283] As discussed, the first embodiment and the second embodiment provide a weighted edge enhancement filtering method in the context of reference picture resampling. The method may be applied as an enhancement filter for inter prediction blocks of a low-resolution picture, and in scenarios where the same filter is applied as an enhancement filter for inter prediction blocks of a low-resolution image and as an enhancement filter for a low-resolution picture that has been upscaled before output.

[0284] Figure 8 shows a schematic diagram of a decoder 80 according to an embodiment. Specifically, Figure 8 shows a schematic diagram of a decoder 80 configured to perform any of the decoder methods discussed herein. Such detailed descriptions are omitted here for the sake of brevity.

[0285] As shown in Figure 8, the decoder 80 comprises a processor 81 and a computer-readable medium 82. The processor 81 and the computer-readable medium 82 may be connected via a bus system. The computer-readable medium is configured to store programs, instructions, or code. The processor 81 is configured to execute the programs, instructions, or code in the computer-readable medium 82 to complete the operation in the decoder method embodiment described herein.

[0286] Accordingly, in the embodiment, the computer-readable medium 82 is configured to store a computer program executable by the processor 81, and the processor 81 is configured to execute the computer program to perform steps in any decoder method discussed herein.

[0287] Figure 9 shows a schematic diagram of an encoder 90 according to an embodiment. Specifically, Figure 9 shows a schematic diagram of an encoder 90 configured to perform any of the encoding methods discussed herein. Such a detailed description is omitted here for brevity.

[0288] As shown in Figure 9, the encoder 90 comprises a processor 91 and a computer-readable medium 92. The processor 91 and the computer-readable medium 92 may be connected via a bus system. The computer-readable medium is configured to store programs, instructions, or code. The processor 91 is configured to execute the programs, instructions, or code in the computer-readable medium 92 to complete the operation in the decoder method embodiment described herein.

[0289] Accordingly, in the embodiment, the computer-readable medium 92 is configured to store a computer program executable by the processor 91, and the processor 91 is configured to execute the computer program to perform steps in any decoder method discussed herein.

[0290] As discussed, in the embodiments, the weighted filter is a parametric filter. Depending on the implementation, in some embodiments it is also content-adaptive. The two main objectives of the filters discussed in the embodiments are sharpening blurred content and reducing ringing artifacts. However, other objectives of the filtering process are also possible, such as reducing blocking artifacts.

[0291] In some embodiments, the weighted filter is a locally adaptive filter. Therefore, it may be used to address nonlinear filtering problems. Two useful applications would be reducing ringing artifacts and sharpening edges. These two problems can be difficult to address with linear filters. The weighted filter in embodiments of the present invention employs a simple concept to overcome the limitations of conventional linear filters. The idea is to apply two filters: A first filter extracts local information from the decoded picture. A second filter applies filtering that depends on the output of the first filter setting. In some embodiments, the second filter is RD-optimized depending on the output of the first filter. This may signal a nonlinear filter that adapts to specific image features.

[0292] To further contextualize the features of the embodiments of the present invention, we hereby provide a discussion of the relationship between the weighted filtering of these embodiments and existing in-loop filters.

[0293] In-loop filtering is a part of modern video coding and decoding systems. Typically, different sets of filters are applied sequentially. These filters may be parametric; that is, a set of filter parameters is sent that changes their behavior depending on the RD decision. They may also be non-parametric. Furthermore, filters may be content-adaptive (i.e., filters may have different behavior depending on their spatial position within the picture). Sample-adaptive offset (SAO) filters and adaptive loop filters (ALF) perform local classification of content and apply different behavior depending on the class. Typically, the classification behavior applied is signaled.

[0294] In detail, the in-loop ALF method optimizes a set of linear filters. Each linear filter is applied to a partition of the picture. The partitions are derived from the local characteristics of the picture. Furthermore, the partitions may be merged. Such information is signaled in the bitstream. However, in embodiments of the present invention, locally weighted / parameterized filters are applied. This reduces the need for picture partitioning and optimization of multiple filters, improving encoding and decoding efficiency.

[0295] In SAO, each sample (or "pixel") is assigned to a class based on the local characteristics of the picture. For each class, an (intensity) offset is calculated and signaled by the encoder. Thus, class determination is performed, and different actions are taken for each class. However, in embodiments of the present invention, locally weighted / parameterized filters are applied. This reduces the need for picture segmentation and optimization of multiple filters, improving encoding and decoding efficiency.

[0296] As discussed, the embodiments provide a method and device for encoding and decoding parameters of an in-loop (or post-loop) filtering scheme. This enables efficient encoding, taking into account the transmission requirements and characteristics of the encoded information in an adaptive parameter set. The embodiments employ weighted / parametric filtering. The weighted / local parameters are calculated from the decoded video picture (e.g., picture block). In some embodiments, the calculation function is parametric and / or signaled by the encoder.

[0297] In the following, locally parameterized weights will be referred to as "weight maps."

[0298] The term "weighting map" has been used, but this term is used for readability and is not intended to have a restrictive meaning. For example, in some embodiments, the weighting map may include local parameters or vectors that simultaneously contain parameters and weight values. In addition to calculating the weights, a filter is applied. In some embodiments, this filter may be parametric with respect to the weighting map, or it may be locally weighted according to the weighting map.

[0299] As discussed, the embodiment relates to a method for optimizing weighted filters, such as weighted edge enhancement filters. This method may also be used to optimize other types of filters intended for enhancing reference pictures.

[0300] The optimization methods may be used to optimize weighted filters for application as in-loop filters. These optimization techniques offer advantages over optimizing separate filters for each low-resolution picture when the filter is applied only as an in-loop filter for emphasizing interpredictive blocks originating from the low-resolution picture. A first use case makes sense in reference picture resampling / adaptive resolution coding and decoding scenarios, where the filter is applied to improve the quality of the low-resolution picture before it is displayed to the viewer and as a reference picture. However, post-filtering of upscaled pictures may be undesirable in some applications where the upscaling behavior is defined according to the viewer's needs. In other words, it is not always desirable to apply both in all applications.

[0301] If upscaling post-filtering is not applied, the optimization target changes. Therefore, the optimization procedure and the thus optimized filter will be different. Furthermore, it makes sense to assign the optimized filter to high-resolution pictures in the encoding / decoding video sequence rather than low-resolution pictures. Typically, some pictures are encoded / decoded at high resolution. The resolution is then changed to a lower encoding / decoding resolution based on bandwidth and / or content. At some point, the resolution may be switched back to high resolution. At this point, there is the option that the low-resolution picture is referenced by the high-resolution picture.

[0302] In this case, enhancement of the referenced low-resolution block may be used. In such a scenario, the first high resolution refers to a region of the low-resolution picture. However, not all reference pictures in the reference set are used as references, and not all portions of a reference picture are used for prediction. Accordingly, it is proposed to optimize the filter of the high-resolution picture so that it can reasonably and well apply to all content that may be referenced / is expected to be referenced by the high-resolution picture. The motivation for using this optimization and filter allocation scheme is to achieve more efficient filter coding. In general, not all pictures are used for inter prediction, and even when they are used, only a small portion of the referenced picture may be used, so it is not efficient to optimize a separate filter for each low-resolution picture. In such a scenario, optimizing a separate filter is too costly. In a given scenario, the filter is used to enhance blocks from the low-resolution picture when referenced by the high-resolution picture. Typically, different pictures are referenced more or less frequently depending on motion, temporal distance, and other factors within the video sequence. Furthermore, not all regions of a picture are referenced with equal probability. As a result, it makes sense to optimize the filter to take those statistical biases into account. Methods for doing so have been proposed herein.

[0303] In particular, the described embodiments provide a method for estimating filter parameters of an in-loop filtering scheme, comprising the step of deriving filter coefficients based on a set of potentially overlapping blocks or partitions. The blocks may originate from the current picture and / or a temporally spaced set of pictures. The filter may be applied to the result of the sub-block interpretation, or may be applied to enhance the partitions of the assigned picture. The described embodiments include optimizing the filter for application as an in-loop filter (i.e., the output of the filter may be reused by the prediction system to encode subsequent pictures).

[0304] The described weighted filter is applied to interpretation to enhance the output of the interpretation based on criteria such as encoded information or any results of pre-analysis. These criteria may be derived from the video content or signaled in the bitstream. Furthermore, the weighted filter may also be applied to the entire set of reference frames to generate enhanced references. Aside from rounding issues, this would yield equivalent results.

[0305] As discussed, the embodiment provides an in-loop filtering method applicable within the encoding / decoding loop of a video compression system. The in-loop filter employs a function for computing a local parameter map and a filtering function. The weighting map function may compute a weighted map using input pictures / blocks, encoding information, and / or signaled parameters. The filtering function may compute a filtered picture using input pictures / blocks, encoding information, signaled parameters, and / or parameter maps. Optimization may be performed based on blocks or partitions of potentially overlapping pictures. Optimization may find an optimized filter for enhancing the encoded signal using ground truth and predicted signals of the current picture and all reference pictures.

[0306] In some embodiments, weighted filters are optimized based on a set of reference pictures and their ground truth representations. This minimizes errors between the reference pictures and their corresponding ground truth representations through weighted co-optimization, resulting in one or more filters being optimized such that applying them to the set of reference pictures minimizes the weighting error.

[0307] In some embodiments, the errors of reference frames are weighted based on their temporal distance from the current frame.

[0308] In some embodiments, the error in the reference frame is weighted based on local image features or knowledge gained from encoding this frame or other frames of the video.

[0309] In some embodiments, the picture is optimized based on the prediction signal and the error of the current picture to ground truth.

[0310] In some embodiments, the prediction signal is derived by trial coding of a picture or slice.

[0311] In some embodiments, the predicted signal is derived by a prior analysis of a reference picture, the current picture, their corresponding ground truth signals, and encoded information.

[0312] In some embodiments, the pre-analysis or trial coding and filter optimization steps are performed repeatedly until some stopping criterion is met.

[0313] In some embodiments, the filtering of blocks within the reference frame and the weighted loss with respect to the predicted signal are jointly optimized.

[0314] In some embodiments, the method is incorporated into the coding loop as an additional processing step, as a replacement for an existing loop filter, or integrated into an existing loop filter.

[0315] In some embodiments, the method is RD-optimized based on the estimated signaling rate and strain after applying the method.

[0316] In some embodiments, the method applies to a luma channel, a chroma channel, or both luma and chroma channels. The channel to be processed may be pre-configured, signaled by the bitstream, or inferred from its content.

[0317] While some embodiments have been described as applicable to pictures, embodiments are not limited in this respect. For example, in some embodiments, the method is applied to one or more partitions of a picture. Partitions may be defined by signaled block partitioning, potentially signaled region partitioning criteria, or a binarized weighted map function.

[0318] In some embodiments, multiple filters are applied to the same picture partition.

[0319] In some embodiments, this method is used to address ringing artifacts, blurring, or blocking artifacts.

[0320] In some embodiments, the method includes the step of applying a weighted map function that outputs a scalar weighted map, wherein the scalar is binary, integer, or floating-point.

[0321] In some embodiments, the method includes the step of applying a weighted map function that outputs an n-dimensional weighted map, where each element is binary, integer, or floating-point.

[0322] In some embodiments, weighted map information is calculated for one or more channels of the reconstructed picture and used as input from one or more channels of the reconstructed picture.

[0323] In some embodiments, a set of potentially parametric weighting map functions is predefined, and the encoder signals the weighting map and weighting map function to be used.

[0324] In some embodiments, one or more filters are applied to each signaled weight map. The filtering function and filter parameters may be predefined, signaled in the bitstream, or inferred from the content or encoding information of the video sequence.

[0325] In some embodiments, the method involves applying a linear filter as the filter of the filtering function. The shape of the filter may be indicated by a bitstream or predefined.

[0326] In some embodiments, the linear filter is optimized by least-squares optimization or RD optimization.

[0327] In some embodiments, a parametric linear filter is applied. Parameters may be used to generate the corresponding linear filter.

[0328] In some embodiments, the parametric linear filter is RD-optimized with respect to the minimum error in the output. The optimal filter may be derived by least-squares optimization, iterative search, or exhaustive search.

[0329] In some embodiments, parametric or non-parametric nonlinear filters are applied to the filtering function.

[0330] In some embodiments, a bilateral filter is applied to the filtering function.

[0331] In some embodiments, a combination of the described filtering methods is applied, and the filtering method used at each location is signaled or indicated by a weighted map.

[0332] In some embodiments, the encoder optimizes the parametric weight map along with a filtering function.

[0333] In some embodiments, one or more filters are applied to the partitions of a picture based on signaled block divisions.

[0334] In some embodiments, one or more filters are applied to the partitions of the picture based on the derived region division criteria.

[0335] In some embodiments, the weighted filter is integrated into an adaptive loop filter (ALF) and applied to partitions derived by ALF optimization as an alternative to the linear filter.

[0336] In some embodiments, the encoder encodes the filter and weighted map calculation parameters by quantization, prediction, or entropy coding.

[0337] In some embodiments, the weighted filter is applied as a post-filter.

[0338] Embodiments of the present invention can also provide a computer-readable medium having computer-executable instructions for causing one or more processors of a computing device to perform a method of any embodiment of the present invention.

[0339] Examples of computer-readable media include, but are not limited to, volatile and non-volatile media, removable and non-removable media, and include solid-state memory, removable disks, hard disk drives, magnetic media, and optical disks. Generally, computer-readable media include any type of media suitable for storing, encoding, or carrying a set of instructions that can be executed by one or more computers to perform one or more processes and features described herein.

[0340] It will be understood that the functions of each component discussed can be combined in many ways other than those discussed in the above description. For example, in some embodiments, multiple functions of the discussed devices can be incorporated into a single device. In other embodiments, at least one function of the discussed devices can be divided among multiple separate (or distributed) devices.

[0341] Conditional language such as "may be" is generally used to indicate that a feature / step is used in a particular embodiment, but an alternative embodiment may include an alternative feature or omit such a feature entirely.

[0342] Furthermore, it will be understood that the method steps are not limited to the specific order described, and they can be combined in any other suitable order. In some embodiments, this may result in several method steps being performed in parallel. Moreover, in some embodiments, certain method steps may be omitted entirely.

[0343] While specific embodiments have been discussed, these are used to illustrate the overall teachings of the invention, and it will be understood that various modifications can be made without departing from the scope of the invention. The scope of the invention should be construed in accordance with the appended claims and any equivalents thereof.

[0344] Referring to the exemplary embodiments described above, many further variations and modifications will come to mind for those skilled in the art. These are given only as examples and are not intended to limit the scope of the invention as determined by the appended claims.

Claims

1. A video data processing method performed by an encoder, wherein the method is Steps to obtain multiple original pictures from the original video data, A step of obtaining a plurality of reference pictures, each corresponding to an original picture from the original video data, wherein the plurality of reference pictures have a lower resolution than the original picture from the original video data. The steps include: obtaining multiple upsampled reference pictures by upsampling the aforementioned multiple reference pictures; A weight map is determined for each reference picture using a weight map function, wherein the weight map includes a plurality of weights mapped to the respective spatial positions of the plurality of upsampled reference pictures, and the plurality of upsampled reference pictures are used as input to the weight map function. By determining the filter to be applied to the upsampling reference picture along with each weight map and obtaining the filtered upsampling reference picture, the filter is applied with different weights to different spatial positions of the multiple upsampling reference pictures, The steps include obtaining a weighted filter that reduces the overall error between the multiple upsampled reference pictures and the corresponding original pictures, The steps include performing an interpretation of multiple blocks of a first picture based on multiple reference blocks from one or more of the multiple upsampled reference pictures, The steps include encoding the video data encoding information into a bitstream, wherein the encoding information includes information about the weighted map function and / or the filter used in the decoder, The aforementioned method, Before performing the interpretation, the weighted filter is applied to the multiple upsampled reference pictures, or The interpretation further includes the step of applying the weighted filter to the upsampling reference block of the reference picture, method.

2. The step of obtaining the weighted filter that reduces the overall error between the plurality of upsampled reference pictures and the corresponding original picture is: The steps include assigning importance weighting coefficients to the respective regions of the plurality of reference pictures and / or the plurality of reference pictures, The process includes determining the overall error between the plurality of upsampled reference pictures and their corresponding original pictures by weighting the error of each upsampled reference picture based on the importance weighting coefficient of each upsampled reference picture. The method according to claim 1.

3. The step of assigning importance weighting coefficients to each of the aforementioned multiple reference pictures is: The step includes assigning a higher importance weighting coefficient to reference pictures that are temporally closer to the first picture. The method according to claim 2.

4. The step of assigning importance weighting coefficients to each of the aforementioned multiple reference pictures is: This includes the step of assigning importance weighting coefficients based on the quality of each reference picture, The method according to claim 2 or 3.

5. The step of assigning importance weighting coefficients to each of the aforementioned multiple reference pictures is: The step of assigning importance weighting coefficients based on local picture features or historical information derived from encoding the reference picture or other pictures in the video data, The method according to any one of claims 2 to 4.

6. The method further includes the step of adding the acquired weighted filter to a stored set of reference filters. The method according to any one of claims 1 to 5.

7. The step of assigning importance weighting coefficients to the regions of the plurality of reference pictures and / or the plurality of reference pictures is: The steps include performing trial encoding of the first picture, The steps include identifying which reference picture and / or blocks within the reference picture are used in the trial coding for interpretation of one or more blocks of the first picture, The process includes the step of assigning a high importance weighting coefficient to identified reference pictures and / or blocks, The method according to any one of claims 1 to 6.

8. The encoded information includes signaled weighted map function parameters, and the signaled weighted map function parameters are Applying the signaled weighted map function parameters as parameters of the weighted map function, By providing the aforementioned multiple reference pictures as input to the weighted map function, The decoder is configured to allow the weight map to be obtained using the weight map function, The method according to any one of claims 1 to 7.

9. The encoded information includes signaled filter function parameters, and the signaled filter function parameters are By applying the signaled filter function parameters as the parameters of the filter, The decoder is configured to enable the acquisition of the filter, The method according to any one of claims 1 to 8.

10. The weight map and the filter are configured to be applied to the first picture as a step or post-loop step within the encoding / decoding loop to obtain the filtered first picture. The method according to any one of claims 1 to 9.

11. The aforementioned coding / decoding loop is an H.266 / VVC coding / decoding loop. The method according to claim 10.

12. The weight map and the filter are integrated into an adaptive loop filter and applied to the multiple reference pictures or the derived partitions of the first picture to obtain the filtered pictures. The method according to claim 10 or 11.

13. The bitstream is rate-distortion (RD) optimized based on the estimated signaling rate and distortion after applying the weight map and the filter to the plurality of reference pictures or the first picture. The method according to any one of claims 1 to 12.

14. The aforementioned reference picture and / or the first picture includes a luma channel, a chroma channel, or both a luma channel and a chroma channel. The weighting map and the filter are configured to be applied to the luma channel, the chroma channel, or both the luma channel and the chroma channel. The method according to any one of claims 1 to 13.

15. Whether the weighting map and the filter are applied to the chroma channel or the chroma channel is predetermined, signaled by the encoded information, or inferred from the content of the picture. The method according to claim 14.

16. The method described above is a step of dividing the plurality of reference pictures and the first picture into a plurality of partitions, The weighted map and the filter are configured to be applied to the plurality of reference pictures or to one or more partitions of the first picture. The partition further includes the step of being signaled with the encoded information, The method according to any one of claims 1 to 15.

17. The partition signaled by the encoded information includes a block partition signaled by the encoded information, a signaled region partitioning criterion, or a binarized weighted map function in the encoded information. The method according to claim 16.

18. The method further includes the step of determining a number of filters to be applied to the same picture partition. The method according to claim 16 or 17.

19. The filter is configured to address ringing artifacts, blurring, and / or blocking artifacts in the picture. The method according to any one of claims 1 to 18.

20. The step of determining the weight map using a weight map function is: A step of applying a weight map function that outputs a scalar weight map, wherein the scalars in the color weight map are binary, integers, or floating-point numbers. The method according to any one of claims 1 to 19.

21. The step of determining the weight map using a weight map function is: A step of applying a weight map function that outputs a multidimensional weight map, wherein each element is binary, integer, or floating-point, The method according to any one of claims 1 to 20.

22. The weighted map information for one or more channels of the aforementioned reference picture is calculated using information from one or more channels of the plurality of reference pictures as input. The method according to any one of claims 1 to 21.

23. A set of weighted map functions is predefined, The encoded information signals the weighted map function to be used. The method according to any one of claims 1 to 22.

24. The aforementioned weighted map function is parametric. The method according to claim 23.

25. The aforementioned encoded information signals multiple weighted map functions, The step of obtaining the weight map using the weight map function includes the step of determining a plurality of weight map functions using the plurality of weight map functions, One or more filters are configured to be applied to each signaled weight map. The method according to any one of claims 1 to 24.

26. The filtering function and parameters of the filter are configured to be inferred from the content of the video, which is signaled by the encoded information, or to be inferred from the encoded information. The method according to claim 25.

27. The filter is a linear filter, and the shape of the filter is indicated by the bitstream or is predefined. The method according to any one of claims 1 to 26.

28. The linear filter is optimized by least-squares optimization or RD optimization. The method according to claim 27.

29. The aforementioned linear filter is a parametric linear filter. The method according to claim 27 or 28.

30. The parametric linear filter is RD-optimized with respect to the minimum error in the output and is derived by least-squares optimization, iterative search, or exhaustive search. The method according to claim 29.

31. The aforementioned filter is a bilateral filter. The method according to any one of claims 1 to 26.

32. The step of obtaining the aforementioned filter includes the step of obtaining multiple filters, Each of the plurality of filters is configured to be applied at a position signaled in the bitstream or at a position indicated in the weight map. The method according to any one of claims 1 to 31.

33. The parametric weight map is optimized together with the filtering function. The method according to any one of claims 1 to 32.

34. One or more filters are configured to be applied to the partitions of the reference picture or the first picture based on the block division signaled by the encoded information. The method according to any one of claims 1 to 32.

35. One or more filters are configured to be applied to the reference picture or partitions of the first picture based on the derived region division criteria. The method according to any one of claims 1 to 34.

36. The filter and weight map calculation parameters are encoded by a quantization, prediction, and / or entropy coding / decoding scheme. The method according to any one of claims 1 to 35.

37. Computer-readable medium, The computer executable instructions are stored in the computer-readable medium, and when executed by a computing device, the computer executable instructions cause the computing device to perform the method according to any one of claims 1 to 36. Computer-readable media.

38. One or more processors, A computer-readable medium comprising a computer-readable medium including computer-executable instructions stored therein, wherein when the computer-executable instructions are executed by one or more processors, the one or more processors cause the one or more processors to perform the method according to any one of claims 1 to 36. Encoder.

39. A video data processing method performed by a decoder, wherein the method is The steps include: decoding the bitstream to obtain video data and encoding information; The steps include obtaining multiple reference pictures from the aforementioned video data, The steps include: obtaining multiple upsampled reference pictures by upsampling the aforementioned multiple reference pictures; A weight map is determined for each reference picture using a weight map function, wherein the weight map includes a plurality of weights mapped to the respective spatial positions of the plurality of upsampled reference pictures, and the plurality of upsampled reference pictures are used as input to the weight map function. By determining the filter to be applied to each of the multiple upsampling reference pictures along with the respective weight map, and obtaining the filtered upsampling reference pictures, the filter is applied with different weights to different spatial positions of the multiple upsampling reference pictures, The steps include obtaining a weighted filter that reduces the overall error between the multiple upsampled reference pictures and the corresponding original pictures, The step of performing an interpretation of multiple blocks of a first picture based on multiple reference blocks from one or more of the multiple upsampled reference pictures, The aforementioned method, Before performing the interpretation, the weighted filter is applied to the multiple upsampled reference pictures, or The interpretation further includes the step of applying the weighted filter to the upsampling reference block of the reference picture, method.

40. The method further includes the step of adding the acquired weighted filter to a stored set of reference filters. The method according to claim 39.

41. The encoded information includes signaled weighted map function parameters, The step of determining the weight map using the weight map function is: The steps include applying the signaled weighted map function parameters as parameters of the weighted map function, The step of providing the reference picture or the first picture as input to the weighted map function, The method according to claim 39.

42. The aforementioned coded information includes signaled filter function parameters, The step of obtaining the aforementioned filter is: The step includes applying the signaled filter function parameters as parameters of the filter, The method according to claim 40 or 41.

43. The step of applying the weight map and the filter to the picture in order to obtain a filtered reference picture or a first picture is performed within the encoding / decoding loop or as a post-loop step. The method according to any one of claims 40 to 42.

44. The aforementioned coding / decoding loop is an H.266 / VVC coding / decoding loop. The method according to claim 43.

45. To obtain filtered pictures, the step of applying the weight map and the filter to the reference picture or first picture is integrated into an adaptive loop filter and applied to the derived partition. The method according to claim 43 or 44.

46. The bitstream is rate-distortion (RD) optimized based on the estimated signaling rate and distortion after applying the weight map and the filter to the plurality of reference pictures or the first picture. The method according to any one of claims 39 to 45.

47. The reference picture and / or the picture includes a luma channel, a chroma channel, or both a luma channel and a chroma channel, and the weighted map and the filter are applied to the luma channel, the chroma channel, or both a luma channel and a chroma channel. The method according to any one of claims 39 to 46.

48. Whether the weight map and the filter are applied to the chroma channel or the chroma channel is predetermined, signaled in the bitstream, or inferred from the content of the picture. The method according to claim 47.

49. The method described above is a step of dividing the plurality of reference pictures and the first picture into a plurality of partitions, The weighted map and the filter are applied to the plurality of reference pictures or to one or more partitions of the first picture. The partition further includes the step of being signaled with the encoded information, The method according to any one of claims 39 to 48.

50. The partition signaled by the encoded information includes a block partition signaled by the encoded information, a signaled region partitioning criterion, or a binarized weighted map function in the encoded information. The method according to claim 49.

51. The method further includes the step of applying multiple filters to the same picture partition. The method according to claim 49 or 50.

52. The filter is configured to address ringing artifacts, blurring, and / or blocking artifacts in the picture. The method according to any one of claims 39 to 51.

53. The step of determining the weight map using a weight map function is: A step of applying a weight map function that outputs a scalar weight map, wherein the scalars in the scalar weight map are binary, integers, or floating-point numbers. The method according to any one of claims 39 to 52.

54. The step of determining the weight map using a weight map function is: A step of applying a weight map function that outputs a multidimensional weight map, wherein each element is binary, integer, or floating-point, The method according to any one of claims 39 to 53.

55. The weighted map information for one or more channels of the aforementioned reference picture is calculated using information from one or more channels of the plurality of reference pictures as input. The method according to any one of claims 39 to 54.

56. A set of weighted map functions is predefined, and the encoded information signals the weighted map function to be used. The method according to any one of claims 39 to 55.

57. The aforementioned weighted map function is parametric. The method according to claim 56.

58. The aforementioned encoded information signals multiple weighted map functions, The step of determining the weight map using the weight map function includes the step of determining a plurality of weight map functions using the plurality of weight map functions, The step of applying the weighted map and the filter includes the step of applying one or more filters to each signaled weighted map. The method according to any one of claims 39 to 57.

59. The filtering function and parameters of the filter are predefined, inferred from the video content, or signaled by the encoded information. The method according to claim 58.

60. The filter is a linear filter, and the shape of the filter is indicated by the bitstream or is predefined. The method according to any one of claims 39 to 59.

61. The linear filter is optimized by least-squares optimization or RD optimization. The method according to claim 60.

62. The aforementioned linear filter is a parametric linear filter. The method according to any one of claims 58 to 61.

63. The parametric linear filter is RD-optimized with respect to the minimum error in the output and is derived by least-squares optimization, iterative search, or exhaustive search. The method according to claim 62.

64. The aforementioned filter is a bilateral filter. The method according to any one of claims 39 to 59.

65. The step of determining the aforementioned filter includes the step of determining a plurality of filters, The step of applying the weight map and the filters to the plurality of reference pictures or the first picture includes the step of applying each of the plurality of filters at a position signaled in the bitstream or at a position indicated in the weight map. The method according to any one of claims 39 to 64.

66. The parametric weight map is optimized together with the filtering function. The method according to any one of claims 39 to 65.

67. The step of applying the weighted map and the filter to the picture includes the step of applying one or more filters to the reference picture or partitions of the first picture based on the block division signaled by the encoded information, The method according to any one of claims 1 to 66.

68. The step of applying the weighted map and the filter to the plurality of reference pictures or the first picture includes the step of applying one or more filters to the partitions of the reference pictures or the first picture based on the derived region division criteria. The method according to any one of claims 39 to 67.

69. The filter and weight map calculation parameters are encoded by a quantization, prediction, and / or entropy coding / decoding scheme. The method according to any one of claims 39 to 68.

70. Computer-readable medium, The computer executable instructions are stored in the computer-readable medium, and when executed by a computing device, the computer executable instructions cause the computing device to perform the method according to any one of claims 1 to 69. Computer-readable media.

71. One or more processors, A computer-readable medium comprising a computer-readable medium including computer-executable instructions stored therein, wherein when the computer-executable instructions are executed by one or more processors, the one or more processors cause the one or more processors to perform the method according to any one of claims 1 to 70. decoder.

72. A video data processing method performed by an encoder, wherein the method is Steps to obtain the original video data, The steps include: performing trial coding on at least a portion of the original video data to obtain trial coded video data; A step of obtaining a trial coded video data by performing interpretation of multiple blocks of the trial first picture based on multiple reference blocks from one or more reference pictures, wherein the one or more reference pictures have a lower resolution than the trial first picture. A weight map is determined using a weight map function, wherein the weight map includes a plurality of weights mapped to each spatial position in each of the plurality of interprediction blocks of the trial first picture, and the plurality of interprediction blocks of the trial first picture are used as inputs to the weight map function. By determining a filter to be applied to the plurality of interprediction blocks of the trial first picture together with the weighting map, the filter is applied with different weights to different spatial positions in each of the interprediction blocks of the trial first picture, and the filter is configured to obtain a first picture that has been applied to the plurality of interprediction blocks of the trial first picture together with the weighting map and filtered, The process includes the step of obtaining a weighted filter applied to the plurality of interpretation blocks of the trial first picture to reduce the error between the trial first picture and the corresponding original first picture in the original video data, method.

73. A step of encoding the video data and encoding information, further comprising a step of including information about the weighted map function and / or the filter used in the decoder, The method according to claim 72.

74. The aforementioned method, A step of determining whether the rate distortion performance of the encoding of the video data and the encoded information is better than the rate distortion performance of the trial encoding, The steps include including the encoded video data and encoded information in the bitstream transmitted to the decoder, based on the fact that the rate distortion performance of the encoding of the video data and encoded information is better than the rate distortion performance of the trial encoding, The further step includes including the trial-encoded video data in a bitstream transmitted to a decoder, based on the fact that the rate-distortion performance of encoding the video data and the encoding information is not better than the rate-distortion performance of the trial encoding. The method according to claim 73.

75. The aforementioned method, The process further includes the step of adding the acquired weighted filter to the stored reference filter set, The method according to any one of claims 72 to 74.

76. The trial coding further includes deriving a trial prediction signal by pre-analysis of one or more reference pictures, the first picture, the corresponding original picture in the original video data, and / or coding information. The method according to any one of claims 72 to 75.

77. The method further includes the step of repeatedly performing the pre-analysis or trial coding and the step of obtaining a weighted filter until the stopping criteria are met. The method according to claim 76.

78. A computer-readable medium including computer-executable instructions stored therein, wherein, when executed by a computing device, the computer-executable instructions cause the computing device to perform the method according to any one of claims 72 to 77. Computer-readable media.

79. One or more processors, A computer-readable medium comprising a computer-readable medium containing computer-executable instructions stored therein, wherein when the computer-executable instructions are executed by one or more processors, the one or more processors cause the one or more processors to execute the method according to any one of claims 72 to 77. Encoder.

80. A video data processing method performed by a decoder, wherein the method is The steps include: decoding the bitstream to obtain video data and encoding information; Steps to obtain one or more reference pictures, A step of performing an interpretation of multiple blocks of a first picture based on multiple reference blocks from one or more reference pictures, wherein the one or more reference pictures have a lower resolution than the first picture. A weight map is determined using a weight map function, wherein the weight map includes a plurality of weights mapped to each spatial position in each of the plurality of interprediction blocks of the first picture, and the plurality of interprediction blocks of the first picture are used as inputs to the weight map function. By determining a filter to be applied to the plurality of interprediction blocks of the first picture together with the weighting map, the filter is applied with different weights to different spatial positions in each of the interprediction blocks of the first picture, and the filter is configured to obtain a filtered first picture applied to the plurality of interprediction blocks of the first picture together with the weighting map, A step of obtaining a weighted filter applied to the plurality of interprediction blocks of the first picture based on the encoding information, thereby reducing the error between the first picture and the corresponding original first picture, The step of applying the weighted filter to the plurality of interprediction blocks of the first picture is included. method.

81. The method further includes the step of adding the acquired weighted filter to a stored set of reference filters. The method according to claim 80.

Images