Weighted filtering for image enhancement in video coding

Abstract

A method of processing video data performed by an encoder is provided. The method comprises the following steps: acquiring a plurality of original images from original video data; a plurality of reference images are obtained, each reference image in the plurality of reference images corresponds to an original image in original video data, and the resolution of the plurality of reference images is lower than the resolution of the plurality of original images in the original video data; performing up-sampling on the plurality of reference images to obtain a plurality of up-sampled reference images; a weighted filter is obtained to reduce overall error between the plurality of up-sampled reference images and corresponding original images by determining a weighted mapping for each reference image using a weighted mapping function, the weighted mapping comprising a plurality of weights mapped to respective spatial locations of the plurality of up-sampled reference images, and determining a filter that is applied to the plurality of up-sampled reference images by respective weighted mappings to obtain a filtered up-sampled reference image such that the filter is applied to different spatial positions of the plurality of up-sampled reference images with different weights; performing inter prediction on a plurality of blocks of the first image based on a plurality of reference blocks in one or more of the plurality of up-sampled reference images; and encoding the video data encoding information into a code stream, the encoding information comprising information on the weight mapping function and / or the filter to be used at a decoder, where the method further comprises applying the weight filter to the plurality of upsampled reference pictures prior to performing the inter prediction, and applying the weight filter to the plurality of upsampled reference pictures prior to performing the inter prediction. Or during the inter prediction, applying the weighting filter to an up-sampled reference block of the plurality of reference pictures.

Description

Technical Field

[0001] This application relates to the field of computer vision, specifically to the subject of video processing and video encoding and decoding, and more specifically to methods, decoders, encoders, and computer-readable media for weighted filtering of image enhancement in video encoding and decoding. Background Technology

[0002] Current video codec schemes, such as H.265 / HEVC (High Efficiency Video Coding) and H.266 / VVC (Versatile Video Coding), support spatial scalability of the encoded video stream. This support for spatial scalability is included in version 2 of HEVC with Scalable High Efficiency Video Coding (SHVC), while VVC natively supports spatial scalability. Adaptively changing the resolution of the encoded video during encoding and decoding is called reference picture resampling (RPR) or adaptive resolution change (ARC) in VVC. Furthermore, multi-resolution encoding and decoding, and multi-layer encoding and decoding, allow for scalable resolution encoding and decoding of video. Given this, the spatial resolution of the video being encoded and decoded can be adaptively changed, and it no longer needs to be equivalent to the output or input resolution of the video. The advantage of this additional flexibility is that encoding and decoding lower-resolution video requires a lower bit rate and can reduce computational complexity, at the cost of losing high-frequency information in the downsampling step.

[0003] Encoding and decoding video at a resolution lower than its original resolution requires downsampling and upsampling steps in the signal processing chain. In the downsampling step, an anti-aliasing filter is applied to prevent artifacts caused by high-frequency components in the image. The upsampling process applies an interpolation filter to reconstruct the intensity values ​​at fractional sample locations.

[0004] In RPR, the resolution of the encoded and decoded video stream can be adaptively changed. Therefore, the encoder can encode portions of the video stream at a lower resolution. RPR is applied to inter-frame prediction whenever a reference image with a different resolution than the current image is used in the inter-frame prediction. In this step, a resampling operation is applied to map the reference image patch to the same spatial resolution as the current image.

[0005] In multi-layer encoding and decoding, video is encoded and decoded at different resolution layers. In the first step, the video is encoded and decoded at the lowest resolution layer. To generate the video stream for the next layer, the video is upsampled, and the residual may be encoded and decoded, with further processing steps applied. This process can be applied multiple times based on the number of layers.

[0006] Finding the optimal high-resolution representation from a low-resolution image is a crucial part of the aforementioned encoding / decoding scheme. One approach is to apply a set of multi-phase finite impulse response (FIR) interpolation filters. While these filters do provide an approximation of the high-resolution image content, they cannot recover the information lost during downsampling and are limited by linear filtering operations. Consequently, the upsampled image is often blurry.

[0007] Image sharpening operations can increase image quality. However, linear high-pass filters often introduce artifacts such as overshoot and ringing. Furthermore, distortion caused by downsampling and upsampling depends on the image content and the video's encoding / decoding quality (affected by the quantization parameter (QP) value).

[0008] Filtering video to improve image quality requires the existence of statistical dependencies that can be utilized by the filtering system. Typically, applying in-loop filtering is meaningful if the quality improvement achieved by filtering outweighs the signaling cost at the rate distortion (RD) point. Furthermore, computation time needs to be acceptable.

[0009] In many video codec systems, a range of filters are used to address different types of codec errors. For example, there is a deblocking filter that can be applied to block boundaries to reduce block artifacts. Next, there are sample adaptive offset (SAO) filters, primarily used to reduce ringing or blur artifacts. Finally, adaptive loop filters (ALFs) can be used for objective quality enhancement. Note that this is just a brief excerpt, intended to outline the different applications and types of loop filters.

[0010] Most of these filters only handle a limited range of encoding / decoding errors. Furthermore, no filters specifically designed for blurred image content are implemented in VVC / H.266. However, blurring does occur due to quantization or removal of high-frequency components. Typically, linear filters are insufficient to recover blurred content due to overshoot and ringing issues. Additionally, noise amplification is a problem. Linear filtering methods, such as adaptive loop filters, attempt to address this by introducing a set of categories for which different filters are applied. However, this increases encoding / decoding costs. Summary of the Invention

[0011] Embodiments of this application provide a method, decoder, encoder, and computer-readable medium that uses weighted filters for video encoding and decoding to overcome problems associated with conventional layouts.

[0012] According to a first aspect, a method for processing video data is provided, performed by an encoder, the method comprising: acquiring a plurality of original images from raw video data; acquiring a plurality of reference images, each of the plurality of reference images corresponding to an original image in the raw video data, wherein the resolution of the plurality of reference images is lower than the resolution of the plurality of original images in the raw video data; upsampling the plurality of reference images to obtain a plurality of upsampled reference images; and obtaining a weighted filter to reduce the overall error between the plurality of upsampled reference images and their corresponding original images by: determining a weighted mapping for each reference image using a weighted mapping function, the weighted mapping including a plurality of weights mapped to corresponding spatial locations of the plurality of upsampled reference images, wherein the plurality of upsampled reference images are used as the weighted mapping function. The method involves taking input data and determining a filter, which is applied to the plurality of upsampled reference images through a corresponding weighted mapping to obtain a filtered upsampled reference image, such that the filter is applied to different spatial locations of the plurality of upsampled reference images with different weights; performing inter-frame prediction on multiple blocks of a first image based on multiple reference blocks in one or more of the plurality of upsampled reference images; and encoding the video data encoding information into a bitstream, the encoding information including information about the weighted mapping function and / or the filter to be used at the decoder, wherein the method further includes: applying the weighted filter to the plurality of upsampled reference images before performing the inter-frame prediction, or applying the weighted filter to the upsampled reference blocks of the plurality of reference images during the inter-frame prediction.

[0013] In some embodiments, obtaining the weighted filter to reduce the overall error between the plurality of upsampled reference images and the corresponding original image includes: assigning corresponding importance weighting factors to the plurality of reference images and / or regions of the plurality of reference images, and determining the overall error between the plurality of upsampled reference images and the corresponding original image by weighting the error of the upsampled reference images based on the importance weighting factor of each upsampled reference image.

[0014] In some embodiments, assigning corresponding importance weighting factors to the plurality of reference images includes assigning a higher importance weighting factor to reference images that are temporally closer to the first image.

[0015] In some embodiments, assigning corresponding importance weighting factors to the plurality of reference images includes: assigning importance weighting factors based on the quality of each reference image.

[0016] In some embodiments, assigning corresponding importance weighting factors to the plurality of reference images includes assigning importance weighting factors based on local image features or historical information in the encoding of the plurality of reference images or other images of the video data.

[0017] In some embodiments, the method further includes adding the acquired weighted filter to a stored set of reference filters.

[0018] In some embodiments, assigning appropriate importance weighting factors to the plurality of reference images and / or regions of the plurality of reference images includes: performing trial coding on the first image; identifying which reference images and / or blocks in the reference images are used in the trial coding of inter-frame prediction of one or more blocks of the first image; and assigning high importance weighting factors to the identified reference images and / or blocks.

[0019] In some embodiments, the encoded information includes signaling-transmitted weighted mapping function parameters, which are configured to allow the decoder to obtain the weighted mapping by: applying the signaling-transmitted weighted mapping function parameters as parameters of the weighted mapping function; and providing the plurality of reference images as input to the weighted mapping function.

[0020] In some embodiments, the encoded information includes filter function parameters transmitted by signaling, the signaling-transmitted filter function parameters being configured to allow the decoder to obtain the filter by applying the signaling-transmitted filter function parameters as parameters of the filter.

[0021] In some embodiments, as a step within the encoding / decoding loop or as a step after the loop, the weighted mapping and the filter are configured to be applied to the first image to obtain a filtered first image.

[0022] In some embodiments, the codec loop is an H.266 / VVC codec loop.

[0023] In some embodiments, the weighted mapping and the filter are configured to be integrated into an adaptive loop filter and applied to the plurality of reference images or a derived partition of the first image to obtain a filtered image.

[0024] In some embodiments, after the weighted mapping and the filter are applied to the plurality of reference images or the first image, the bitstream is rate-distortion (RD) optimized based on the estimated signaling rate and distortion.

[0025] In some embodiments, the plurality of reference images and / or the first image includes a luminance channel, a chrominance channel, or both a luminance channel and a chrominance channel, and wherein the weighted mapping and the filter are configured to be applied to the luminance channel, the chrominance channel, or both the luminance channel and the chrominance channel.

[0026] In some embodiments, the weighting mapping and the filter to be applied to which of the luminance and chrominance channels are predetermined, notified by signaling in the encoded information, or configured to be inferred from the image content.

[0027] In some embodiments, the method further includes: partitioning the plurality of reference images and the first image into a plurality of partitions, wherein the weighted mapping and the filter are configured to be applied to one or more partitions of the plurality of reference images or the first image, wherein the partitions are notified by signaling in the encoded information.

[0028] In some embodiments, the partitions notified by signaling in the encoded information include block partitions notified by signaling in the encoded information, region partitioning criteria notified by signaling, or binarized weighted mapping functions.

[0029] In some embodiments, the method further includes: determining a plurality of filters to be applied to the same image partition.

[0030] In some embodiments, the filter is configured to resolve ringing, blurring, and / or blockiness issues in an image.

[0031] In some embodiments, determining the weighted mapping using a weighted mapping function includes: applying a weighted mapping function to an output scalar weighted mapping, wherein the scalar in the scalar weighted mapping is binary, integer, or floating-point.

[0032] In some embodiments, determining the weighted mapping using a weighted mapping function includes: applying a weighted mapping function that outputs a multidimensional weighted mapping, wherein each element in the multidimensional weighted mapping is binary, integer, or floating-point.

[0033] In some embodiments, the weighted mapping information of one or more channels of the plurality of reference images is calculated using information from one or more channels of the plurality of reference images as input.

[0034] In some embodiments, a set of weighted mapping functions is predefined, and the encoded information is signaled to the weighted mapping functions to be used.

[0035] In some embodiments, the weighted mapping function is parameterized.

[0036] In some embodiments, the encoded information is signaled to a plurality of weighted mapping functions, wherein obtaining the weighted mapping using the weighted mapping functions includes: determining a plurality of weighted mappings using the plurality of weighted mapping functions, and wherein one or more filters are configured to be applied for each weighted mapping signaled to the plurality of weighted mappings.

[0037] In some embodiments, the filter function and parameters of the filter are notified by signaling in the encoded information, are predefined, are configured to be inferred from the video content, or are configured to be inferred from the encoded information.

[0038] In some embodiments, the filter is a linear filter, and the shape of the filter is indicated in the bitstream or is predefined.

[0039] In some embodiments, the linear filter is optimized through least squares optimization or RD optimization.

[0040] In some embodiments, the linear filter is a parameterized linear filter.

[0041] In some embodiments, the parameterized linear filter is RD optimized at the output for a minimum error, which is derived through least squares optimization, iterative search, or exhaustive search.

[0042] In some embodiments, the filter is a bilateral filter.

[0043] In some embodiments, obtaining the filter includes obtaining a plurality of filters, wherein each of the plurality of filters is configured to be applied at a location in the bitstream that is signaled or at a location indicated in the weighted mapping.

[0044] In some embodiments, the parameterized weighted mapping is optimized together with the filtering function.

[0045] In some embodiments, one or more filters are configured to be applied to partitions of the plurality of reference images or the first image based on block partitions notified by signaling in the encoded information.

[0046] In some embodiments, one or more filters are configured to be applied to partitions of the plurality of reference images or the first image based on derived region partitioning criteria.

[0047] In some embodiments, the filter and weighted mapping computation parameters are encoded using quantization, prediction, and / or entropy encoding / decoding schemes.

[0048] According to a second aspect, a computer-readable medium is provided, including computer-executable instructions stored on the computer-readable medium, wherein, when executed by a computing device, the computer-executable instructions cause the computing device to perform any of the methods in the first aspect.

[0049] According to a third aspect, an encoder is provided, comprising: one or more processors; and a computer-readable medium storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the methods in the first aspect.

[0050] According to a fourth aspect, a method for processing video data is provided, executed by a decoder, the method comprising: decoding a bitstream to obtain video data and encoding information; obtaining a plurality of reference images from the video data; upsampling the plurality of reference images to obtain a plurality of upsampled reference images; and obtaining a weighted filter to reduce the overall error between the plurality of upsampled reference images and their corresponding original images by: determining a weighted mapping for each reference image using a weighted mapping function, the weighted mapping including a plurality of weights mapped to corresponding spatial locations of the plurality of upsampled reference images, wherein the plurality of upsampled reference images are used as inputs to the weighted mapping function. The method further includes determining a filter, which is applied to the plurality of upsampled reference images through a corresponding weighted mapping to obtain a filtered upsampled reference image, such that the filter is applied to different spatial locations of the plurality of upsampled reference images with different weights; and performing inter-frame prediction on a plurality of blocks of a first image based on a plurality of reference blocks in one or more of the plurality of upsampled reference images, wherein the method further includes: applying the weighted filter to the plurality of upsampled reference images before performing the inter-frame prediction, or applying the weighted filter to the upsampled reference blocks of the plurality of reference images during the inter-frame prediction.

[0051] In some embodiments, the method further includes adding the acquired weighted filter to a stored set of reference filters.

[0052] In some embodiments, the encoded information includes weighted mapping function parameters transmitted via signaling, and wherein determining the weighted mapping using the weighted mapping function includes: applying the weighted mapping function parameters transmitted via signaling as parameters of the weighted mapping function; and providing the plurality of reference images or the first image as input to the weighted mapping function.

[0053] In some embodiments, the encoded information includes filter function parameters transmitted by signaling, and wherein obtaining the filter includes: applying the filter function parameters transmitted by signaling as parameters of the filter.

[0054] In some embodiments, applying the weighted mapping and the filter to the image to obtain a filtered reference image or a filtered first image occurs within the encoding / decoding loop or as a step after the loop.

[0055] In some embodiments, the codec loop is an H.266 / VVC codec loop.

[0056] In some embodiments, the step of applying the weighted mapping and the filter to the plurality of reference images or the first image to obtain a filtered image is integrated into an adaptive loop filter and applied to the derived partition.

[0057] In some embodiments, after the weighted mapping and the filter are applied to the plurality of reference images or the first image, the bitstream is rate-distortion (RD) optimized based on the estimated signaling rate and distortion.

[0058] In some embodiments, the plurality of reference images and / or the images include a luminance channel, a chrominance channel, or both a luminance channel and a chrominance channel, and wherein the weighted mapping and the filter are applied to the luminance channel, the chrominance channel, or both the luminance channel and the chrominance channel.

[0059] In some embodiments, the weighting mapping and the filter to be applied to which of the luminance and chroma channels are predetermined, notified by signaling in the bitstream, or inferred from the image content.

[0060] In some embodiments, the method further includes: partitioning the plurality of reference images and the first image into a plurality of partitions, wherein the weighted mapping and the filter are applied to one or more partitions of the plurality of reference images or the first image, wherein the partitions are notified by signaling in the encoded information.

[0061] In some embodiments, the partitions notified by signaling in the encoded information include block partitions notified by signaling in the encoded information, region partitioning criteria notified by signaling, or binarized weighted mapping functions.

[0062] In some embodiments, the method further includes applying multiple filters to the same image partition.

[0063] In some embodiments, the filter is configured to resolve ringing, blurring, and / or blockiness issues in an image.

[0064] In some embodiments, determining the weighted mapping using a weighted mapping function includes: applying a weighted mapping function to an output scalar weighted mapping, wherein the scalar in the scalar weighted mapping is binary, integer, or floating-point.

[0065] In some embodiments, determining the weighted mapping using a weighted mapping function includes: applying a weighted mapping function that outputs a multidimensional weighted mapping, wherein each element in the multidimensional weighted mapping is binary, integer, or floating-point.

[0066] In some embodiments, the weighted mapping information of one or more channels of the plurality of reference images is calculated using information from one or more channels of the plurality of reference images as input.

[0067] In some embodiments, a set of weighted mapping functions is predefined, and the encoded information is signaled to the weighted mapping functions to be used.

[0068] In some embodiments, the weighted mapping function is parameterized.

[0069] In some embodiments, the encoded information is signaled to a plurality of weighted mapping functions, wherein determining the weighted mapping using the weighted mapping functions includes: determining a plurality of weighted mappings using the plurality of weighted mapping functions, and wherein applying the weighted mappings and the filters includes: applying one or more filters for each weighted mapping signaled to the system.

[0070] In some embodiments, the filter function and parameters of the filter are notified by signaling in the encoded information, are predefined, are inferred from the video content, or are inferred from the encoded information.

[0071] In some embodiments, the filter is a linear filter, and the shape of the filter is indicated in the bitstream or is predefined.

[0072] In some embodiments, the linear filter is optimized through least squares optimization or RD optimization.

[0073] In some embodiments, the linear filter is a parameterized linear filter.

[0074] In some embodiments, the parameterized linear filter is RD optimized at the output for a minimum error, which is derived through least squares optimization, iterative search, or exhaustive search.

[0075] In some embodiments, the filter is a bilateral filter.

[0076] In some embodiments, determining the filter includes: determining a plurality of filters, wherein applying the weighted mapping and the filter to the plurality of reference images or the first image includes: applying each of the plurality of filters at a location notified by signaling in the bitstream or at a location indicated in the weighted mapping.

[0077] In some embodiments, the parameterized weighted mapping is optimized together with the filtering function.

[0078] In some embodiments, applying the weighted mapping and the filter to the image includes: applying one or more filters to a partition of the plurality of reference images or the first image based on block partitions notified by signaling in the encoded information.

[0079] In some embodiments, applying the weighted mapping and the filter to the plurality of reference images or the first image includes: applying one or more filters to partitions of the plurality of reference images or the first image based on derived region partitioning criteria.

[0080] In some embodiments, the filter and weighted mapping computation parameters are encoded using quantization, prediction, and / or entropy encoding / decoding schemes.

[0081] According to a fifth aspect, a computer-readable medium is provided, on which computer-executable instructions are stored, wherein, when executed by a computing device, the computer-executable instructions cause the computing device to perform any of the methods in the fourth aspect.

[0082] According to a sixth aspect, a decoder is provided, comprising: one or more processors; and a computer-readable medium including computer-executable instructions stored on the computer-readable medium, wherein, when executed by the one or more processors, the computer-executable instructions cause the one or more processors to perform any method of the fourth aspect.

[0083] According to a seventh aspect, a method for processing video data is provided, executed by an encoder, the method comprising: acquiring raw video data; performing trial coding on at least a portion of the raw video data to obtain trial-coded video data; acquiring a trial first image based on the trial-coded video data; performing inter-frame prediction on multiple blocks of the trial first image based on multiple reference blocks in one or more reference images, the one or more reference images having a lower resolution than the trial first image; and acquiring a weighted filter to be applied to the multiple inter-frame prediction blocks of the trial first image to reduce the error between the trial first image and a corresponding raw first image in the raw video data by using: A weighted mapping function determines a weighted mapping, which includes multiple weights mapped to corresponding spatial locations in each of the plurality of inter-frame prediction blocks of the experimental first image, wherein the plurality of inter-frame prediction blocks of the experimental first image are used as inputs to the weighted mapping function, and determines a filter that is applied to the plurality of inter-frame prediction blocks of the experimental first image via the weighted mapping, wherein the filter is configured to be applied to the plurality of inter-frame prediction blocks of the experimental first image via the weighted mapping to obtain a filtered first image, such that the filter is applied with different weights to different spatial locations in each of the plurality of inter-frame prediction blocks of the experimental first image.

[0084] In some embodiments, the video data and encoding information are encoded, the encoding information including information about the weighted mapping function and / or the filter to be used at the decoder.

[0085] In some embodiments, the method further includes: determining whether the rate-distortion performance of encoding the video data and the encoded information is better than the rate-distortion performance of the test encoding; including the encoded video data and encoded information in a bitstream transmitted to the decoder based on the fact that the rate-distortion performance of encoding the video data and the encoded information is better than the rate-distortion performance of the test encoding; and including the test-encoded video data in a bitstream transmitted to the decoder based on the fact that the rate-distortion performance of encoding the video data and the encoded information is not better than the rate-distortion performance of the test encoding.

[0086] In some embodiments, the method further includes adding the acquired weighted filter to a stored set of reference filters.

[0087] In some embodiments, the test coding further includes deriving a test prediction signal by pre-analyzing the one or more reference images, the first image, the corresponding original image in the original video data, and / or coding information.

[0088] In some embodiments, the method further includes: iteratively performing the pre-analysis or the trial coding, and obtaining the weighted filter, until the stopping criterion is met.

[0089] According to an eighth aspect, a computer-readable medium is provided, including computer-executable instructions stored on the computer-readable medium, wherein, when executed by a computing device, the computer-executable instructions cause the computing device to perform any of the methods in the seventh aspect.

[0090] According to a ninth aspect, an encoder is provided, comprising: one or more processors; and a computer-readable medium including computer-executable instructions stored on the computer-readable medium, wherein, when executed by the one or more processors, the computer-executable instructions cause the one or more processors to perform any of the methods in the seventh aspect.

[0091] According to a tenth aspect, a method for processing video data is provided, executed by a decoder, the method comprising: decoding a bitstream to obtain video data and encoding information; acquiring one or more reference images; performing inter-frame prediction on multiple blocks of a first image based on multiple reference blocks in the one or more reference images, wherein the resolution of the one or more reference images is lower than the resolution of the first image; and, based on the encoding information, obtaining a weighted filter to be applied to the multiple inter-frame prediction blocks of the first image to reduce the error between the first image and a corresponding original first image by: determining a weighted mapping using a weighted mapping function, the weighted mapping including mappings to the multiple reference blocks of the first image. Multiple weights for corresponding spatial locations in each of the inter-frame prediction blocks in a plurality of inter-frame prediction blocks, wherein the plurality of inter-frame prediction blocks of the first image are used as input to the weighted mapping function, and a filter is determined, the filter being applied to the plurality of inter-frame prediction blocks of the first image via the weighted mapping, wherein the filter is configured to be applied to the plurality of inter-frame prediction blocks of the first image via the weighted mapping to obtain a filtered first image, such that the filter is applied with different weights to different spatial locations in each of the plurality of inter-frame prediction blocks of the first image; and the weighted filter is applied to the plurality of inter-frame prediction blocks of the first image.

[0092] In some embodiments, the method further includes adding the acquired weighted filter to a stored set of reference filters.

[0093] Applying locally weighted filters improves encoding / decoding performance and allows for a wider range of applications. Weighted filtering, determined by a weighting function, can be used in conjunction with weights to guide the filter's properties at each spatial location. For example, the intensity of a sharpening filter can be increased near edges in an image and decreased in regions far from edges. This reduces ringing and overshoot while preserving sharpening properties.

[0094] These and other aspects of this application will become more apparent from the description of the following embodiments. Attached Figure Description

[0095] Embodiments will now be described by way of example only, with reference to the accompanying drawings.

[0096] Figure 1 A flowchart illustrating the operation of the decoder according to the first embodiment is shown;

[0097] Figure 2 A flowchart illustrating the operation of the encoder according to the first embodiment is shown;

[0098] Figure 3 A block diagram illustrating example operations in the first embodiment is shown;

[0099] Figure 4 A graph comparing multiple filtering schemes to the ground truth signal is shown.

[0100] Figure 5 A block diagram illustrating example operations in a variation of the first embodiment is shown;

[0101] Figure 6 A flowchart illustrating the operation of the decoder according to the first embodiment is shown;

[0102] Figure 7 A flowchart illustrating the operation of the encoder according to the first embodiment is shown;

[0103] Figure 8 Schematic diagrams of decoders according to various embodiments are shown; and

[0104] Figure 9 A schematic diagram of an encoder according to various embodiments is shown. Detailed Implementation

[0105] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0106] The VVC compression standard introduces Reference Image Resampling (RPR) as a tool to adaptively change the resolution within the video sequence at the codec layer. Unlike previous standards, the output resolution is not limited by the resolution used to encode and decode a given frame of the video sequence. To achieve this, a set of filters is introduced to facilitate switching between different resolutions.

[0107] The filters used for resolution changes in the VVC standard are currently fixed linear filters. These filters are well-suited for a wide range of video content. However, the content within a encoded video sequence is locally correlated and depends on the specific video sequence. Furthermore, image upsampling presents inherent nonlinearity issues. To address this problem, the embodiments discussed in this paper employ an adaptive locally weighted filter. This is based on the assumption that optimizing the filter using knowledge of the errors caused by encoding and decoding and the characteristics of the current frame improves the performance of the encoding and decoding scheme.

[0108] Some arrangements support both switching between encoding / decoding resolutions and temporarily encoding / decoding at a lower resolution. Therefore, this requires both downsampling and upsampling processes to switch resolutions during inter-frame prediction and use reference images of different resolutions. Some reference software employs a fixed upsampling process using interpolation filters for both applications. Upsampling is performed using finite impulse response filters. Although the output device resolution remains unchanged, the output itself still needs to be displayed at a high resolution. The third embodiment addresses two application scenarios. The first application scenario uses enhancement filters only within the encoding / decoding loop; that is, these filters are not used to improve the upsampled low-resolution image used for the output. This involves modifying the upsampling process while keeping the downsampling process unaffected. Therefore, evaluating these modifications is most effective in the high-resolution domain.

[0109] Images encoded at lower resolutions typically lack the high-frequency information present in their high-resolution equivalents. Therefore, images encoded at low resolutions often appear blurry after upsampling. Consequently, prediction errors are typically higher when referencing a low-resolution frame. This problem stems from the fact that the upsampling process cannot recover the high-frequency information lost during downsampling. However, artifacts caused by reference image resampling (RPR) are most pronounced in regions that do not align with the fundamental assumptions of the interpolation scheme (i.e., smoothness). In reference software, this is particularly noticeable in content that includes high-frequency information, such as edges.

[0110] Current reference software employs a fixed set of downsampling and upsampling filters without utilizing any side information from the encoded video or bitstream to enhance the quality of the upsampled image or predicted signal. For example, the distribution of frequency components may vary depending on the encoded video sequence or spatial location, making the optimal filter dependent on the characteristics of the content. Current filtering methods (e.g., adaptive loop filters) address this issue by signaling the set of filters. The decision of which filter to apply to a given image partition is communicated via signaling and can also be inferred from the image content. However, in the case of upsampled image content, different types of errors exist compared to filters within each loop. Typically, some blurring occurs due to both downsampling and upsampling. Therefore, applying sharpening (especially at edges) can be beneficial.

[0111] The embodiments discussed in this paper address the limitations of upsampling in Reference Image Resampling (RPR) and image upsampling. Some current implementations take frames or blocks as input and apply a set of multi-phase interpolation filters to interpolate the subsample locations. The problem is that the image upsampling problem is inherently ill-conditioned. Therefore, no single filtering method can solve this problem for arbitrary types of content. However, side information inferred from the encoded or decoded video or encoded in the bitstream can be used to correct this problem. The embodiments discussed in this paper can be applied by adding another processing step (i.e., an adaptive local weighted filter) after upsampling.

[0112] The basic assumption is that the optimal strength of the filter depends on its spatial location and can be inferred from the content. Since the filter is least-squares optimized based on the applied weights and upsampled image, there is an interdependence. Therefore, the overall performance of the filter depends on the initial choice of the weighting function. For example, consider a least-squares optimized (unweighted) filter, the result of which is added to the filtered image to obtain the output. After this optimization, some pixels may have optimal, excessively large, excessively small, or incorrectly signed computational offsets. Now, if local weighting can be computed, which reduces the offset applied by the filter in regions with excessively high or incorrectly signed offsets and increases the offset in regions with excessively small offsets, the error will be reduced. Furthermore, since the weights are known in advance, a better filter can be optimized. However, finding the optimal weighting mapping function for every type of content and every location in the image is complex. Nevertheless, approximating the optimal weighting through weighted mapping can already yield improved results.

[0113] In RPR upsampling applications, content is often blurred, especially in high-frequency structures such as edges. Linear filters typically exhibit ringing and overshoot effects, and filtered edges often remain less steep than true edges. Therefore, increasing the filter offset at the predicted location of the edge and decreasing the offset around it will improve the filter.

[0114] These techniques can be applied to H.265 / HEVC or H.266 / VVC video codec systems (e.g., in in-loop processing, where other filters, such as Adaptive Loop Filter (ALF) and Sample Adaptive Offset Filter (SAO), are currently used in such codecs). However, it should be understood that these techniques can also be applied to any other video codec system involving video compression. Furthermore, while these principles are primarily illustrated with reference to video processing, they are also applicable to other data formats, including image processing and even audio processing.

[0115] In this embodiment, "video" refers to one or more images. In other words, a video may include one or more images. An image may also be referred to as a "picture".

[0116] An encoder is a device that encodes data into a bitstream, while a decoder is a device that decodes the bitstream to obtain the encoded data or an approximation thereof. A bitstream consists of a sequence of bits.

[0117] Intra-frame prediction and inter-frame prediction are two prediction operations used within the HEVC and VVC frameworks by the decoder to process the received bitstream to obtain the raw signal. In the embodiments, "raw signal" or "raw video" refers to the data before the encoder performs encoding. The reference sample in the embodiments may refer to spatially and / or temporally spaced image data used for image (or image region) prediction. Intra-frame and inter-frame prediction operations are also used by the encoder to make rate-distortion decisions.

[0118] More specifically, intra-frame prediction involves predicting data spatially within a single image, without referencing other (temporally spaced) images. In other words, data from a first region of an image is used to predict data from another region of the same image, independent of another temporally spaced image. In this context, data from the first region of the image is considered a "reference sample."

[0119] Inter-frame prediction involves predicting data across multiple temporally spaced images. In other words, data from a first region of a first image is used to predict data from a second region of a second image. The first and second regions may or may not be spatially separated. In this context, the data from the first region of the first image is considered a "reference sample." It should be noted that inter-frame prediction may sometimes use multiple reference regions from different images simultaneously, i.e., for a single prediction operation.

[0120] In an embodiment, "residual" may refer to the value obtained based on the original value of a region of an image and the predicted value of that region (e.g., the difference between the original value and the predicted value).

[0121] In this embodiment, "block" may refer to a portion of an image. For example, an image may be divided into two or more blocks. However, this is merely an example. If the image is not divided, then "block" may refer to the entire image.

[0122] In the embodiments, "filter" may refer to a filter used to enhance a signal (especially an upsampled signal). Typically, in the described embodiments, the filter is configured to sharpen blurred content, reduce ringing effects, and / or reduce blocky artifacts. However, the embodiments are not limited thereto, and in other embodiments, the filter may be configured to provide alternative or additional enhancements.

[0123] A general optimization problem in video encoding and decoding is to simultaneously minimize transmission rate and distortion. Lower transmission rates lead to stronger and more visible distortion, thus reducing the viewer's perceived quality. Errors caused by encoding are not random but are caused by processing steps in the encoder and decoder. Two important steps in a video encoding and decoding system are prediction and transform. Quantization of transform coefficients introduces reconstruction errors. Many video encoding and decoding systems employ a hybrid encoding and decoding architecture, where the content of a block is predicted via intra-frame prediction or inter-frame prediction. This prediction is often not perfectly accurate. Therefore, the difference from the true signal needs to be calculated, transformed, and encoded to compensate for the prediction error. The signal after adding the residual is then filtered by a so-called in-loop filter.

[0124] The processing steps that cause artifacts are not random. Embodiments of the present invention utilize prior information to address specific types of errors.

[0125] Two useful applications are reducing ringing artifacts and sharpening edges. These two problems can be difficult to solve with linear filters. The weighted filter of this invention uses a direct concept to overcome the limitations of conventional linear filters. The idea is to apply two filters. The first filter extracts local information from the decoded image. The second filter applies filtering based on the output of the first filter settings. In some embodiments, the second filter will be rate-distortion optimized based on the output of the first filter. Thus, signaling can be provided for nonlinear filters adapted to specific image features.

[0126] Furthermore, as discussed above, current video codec schemes (such as H.265 / High Efficiency Video Codec (HEVC) and H.266 / Various Universal Video Codec (VVC)) support spatial scalability of the encoded and decoded video streams. Therefore, the spatial resolution used for encoding and decoding video can be adaptively changed, no longer needing to be equivalent to the video's output or input resolution. The advantage of this added flexibility is that encoding and decoding lower-resolution video requires a lower bit rate and reduces computational complexity, at the cost of losing high-frequency information during the downsampling step.

[0127] For example, in Reference Image Resampling (RPR), the resolution of the encoded and decoded video stream can be adaptively changed. Therefore, the encoder can encode portions of the video stream at a lower resolution. RPR is applied to inter-frame prediction whenever a reference image with a different resolution than the current image is used in inter-frame prediction. In this step, a resampling operation is applied so that the reference image patch is mapped to the same spatial resolution as the current image.

[0128] Finding the optimal high-resolution representation from a low-resolution image is a crucial part of these encoding / decoding schemes. One approach is to apply a set of multi-phase finite impulse response (FIR) interpolation filters to the low-resolution image. While these filters do provide an approximation of the high-resolution image content, they cannot recover the information lost during downsampling and are limited by linear filtering operations. Consequently, the upsampled image is often blurry.

[0129] Image sharpening operations can increase image quality. However, linear high-pass filters often introduce artifacts such as overshoot and ringing. Furthermore, the distortion caused by downsampling and upsampling depends on the image content and the video's encoding / decoding quality (affected by the quantization parameter (QP) value).

[0130] Figure 1 A flowchart illustrating the operation of the decoder 80 according to the first embodiment is shown. Figure 2 A flowchart illustrating the operation of encoder 90 according to this embodiment is shown.

[0131] Figure 1The flowchart begins at step 101, where decoder 80 decodes the bitstream to obtain video data and encoding information. In this embodiment, the encoding information includes weighted mapping information.

[0132] In step 102, decoder 80 acquires (or “reconstructs”) a set of reference images based on the video data. The video data includes a compressed version of the original video data. In this embodiment, step 102 involves acquiring a set of temporally spaced images (or “frames”) according to a prediction scheme specified in H.266 / Voice Universal Coding (VVC) and then storing them as reference images (e.g., for future inter-frame prediction of future images). However, the embodiment is not limited to this, and in other embodiments, any other method for acquiring a set of reference images from coded video data may be used. Examples include prediction schemes specified in H.265 / HEVC.

[0133] In step 103, the decoder 80 upsamples the reference image. Specifically, in this embodiment, the reference image has already been encoded in the bitstream at a lower resolution, while the next image (or "frame") has been encoded at a higher resolution. Therefore, the reference image is upsampled to the resolution of the next image to be used as a reference image for inter-frame prediction of one or more blocks of that next image.

[0134] In this embodiment, step 103 involves applying a set of multiphase finite impulse response (FIR) interpolation filters to reconstruct the intensity values ​​of the fractional sample locations, thereby increasing the resolution of each reference image. However, the embodiment is not limited to this and other upsampling methods can be applied instead. Specifically, there are many different methods available for performing interpolation. Essentially, the problem we encounter in upsampling is that the fractional sample locations need to be interpolated. These methods include bilinear interpolation, bicubic interpolation, nearest neighbor interpolation, and Lanczos interpolation, among others.

[0135] In step 104, the decoder 80 acquires a weighted filter to use weighted mapping information to reduce the overall error between multiple upsampled reference images (or blocks) and their corresponding original images (or blocks). In this embodiment, the weighted mapping information includes a weighted mapping function for calculating the weighted mapping of each reference image. Each weighted mapping includes multiple weights mapped to a corresponding spatial location in the upsampled reference image, wherein the upsampled reference image is used as the input to the weighted mapping function.

[0136] One such example is when the sharpening filter is to be used, the weighted mapping information included in the encoded information is a local gradient calculation function. In this example, in step 104, for each upsampled reference image, the decoder 80 applies the gradient calculation function to compute the gradient at each location in the upsampled reference image, thereby obtaining a scalar weighted mapping with the same resolution as the upsampled reference image (whose value corresponds to each corresponding location in the image). In other words, the reference image is provided as input to the weighted mapping function.

[0137] Furthermore, in other embodiments, the weighted mapping may have a different resolution than the upsampled reference image, and / or may include vector values ​​instead of scalar values. It is also understood that the examples of image gradient functions are merely illustrative, and in actual implementations of the embodiments, the choice of a suitable weighted mapping function will depend on the circumstances, particularly which filter is to be used. These factors and possible variations will be discussed in more detail later.

[0138] Furthermore, while it has been discussed in this embodiment that the encoded information includes weighted mapping information, which in turn includes weighted mapping functions, the embodiments are not limited in this respect. For example, in some embodiments, multiple weighted mapping functions are stored in the decoder 80. In such cases, the weighted mapping information instead includes an indication of which weighted mappings should be used. Furthermore, in some embodiments, the encoded information does not include any explicit weighted mapping indication. Instead, for example, the encoded information may include filter information, which is then inferred by the decoder 80 as the weighted mapping function to be used (e.g., if the encoded information indicates that a sharpening filter should be used, the decoder infers that an image gradient function should be used as the weighted mapping function). These factors and possible variations will be discussed in more detail later.

[0139] In step 104, decoder 80 further determines the filter to be used with the weighted mapping. In this embodiment, decoder 80 infers the filter to be used from the weighted mapping information. As discussed above, in one example, the weighted mapping information included in the encoded information is a local gradient calculation function. In this example, decoder 80 may combine a weighted mapping that includes image gradients to infer the sharpening filter to be used (e.g., a pre-stored one).

[0140] In step 105, the decoder 80 applies filters and corresponding weighted maps (collectively referred to as “weighted filters”) to each reference image to obtain multiple enhanced reference images. Therefore, step 105 involves using the determined weighted maps such that filters are applied to different regions of each upsampled reference image with different intensities. However, step 105 is optional, as discussed further below.

[0141] In this embodiment, applying a weighted map and filter to the upsampled reference image involves providing both the upsampled reference image and the weighted map as inputs to the filter. This results in the filter being applied with varying intensities to each value of the upsampled reference image, depending on the values ​​of the corresponding weights in the weighted map. The output of the filter is an offset map. In this embodiment, the offset map corresponds to the upsampled reference image in resolution. Once the offset map is output, the offset values ​​are then added to the values ​​of the upsampled reference image to obtain the output (enhanced) reference image.

[0142] In the above discussion, it is assumed that the filter is a sharpening filter configured to sharpen blurred edges. However, the embodiments are not limited to this, and any suitable filter for enhancing an image can be used instead.

[0143] In step 106, decoder 80 performs inter-frame prediction on multiple blocks of the first image based on multiple reference blocks in one or more enhanced upsampled reference images. Specifically, decoder 80 performs prediction operations using the enhanced reference images to obtain multiple prediction blocks of the first image. More specifically, decoder 80 uses the enhanced reference images as reference images to perform inter-frame prediction to obtain prediction blocks.

[0144] After step 106, the predicted first image can then be used for any desired purpose. In one example, decoder 80 then displays it to a viewer. In another example, decoder 80 stores the first image for later use. In yet another example, decoder 80 transmits the first image to an external device for display.

[0145] In some embodiments, the acquired weighted filters are added to a stored set of reference filters for future use.

[0146] As described above, step 105 is optional. This is because, instead of applying the weighted filter to multiple reference images, the weighted filter can instead be applied to an upsampled reference block of the reference image during inter-frame prediction (but before adding the residual). Note that these two alternatives are equivalent (except for rounding) and are simply different implementations. Therefore, in a variant of this embodiment, step 105 is not applied; instead, the weighted filter is applied to the first image after inter-frame prediction is performed (but before adding the residual).

[0147] Encoder 90 can perform a complementary method to encode the bitstream provided to decoder 80. Figure 2 A flowchart illustrating the operation of encoder 90 according to this embodiment is shown.

[0148] In step 201, encoder 90 acquires multiple raw images from the raw video data. For example, encoder 90 can receive raw video data from an external server via a communication network (e.g., the Internet). However, in this embodiment, there are no restrictions on how the raw video data is acquired.

[0149] In step 202, encoder 90 acquires a plurality of reference images, each corresponding to an original image in the original video data, wherein the resolution of the plurality of reference images is lower than that of the original image in the original video data. For example, the plurality of reference images may represent previous images (or “frames”) that, for example, were encoded into the bitstream at a lower resolution than the corresponding original image in the original video data, taking into account network conditions at the time of encoding (i.e., they may be images that the decoder decodes based on the encoded bitstream). These previous images may be stored in the decoder and encoder as a set of reference frames for use in inter-frame prediction of future images. Therefore, each reference image corresponds to an original image in the original video data (although there is a difference due to the lower resolution and possible prediction errors when acquiring the reference images).

[0150] In step 203, encoder 90 upsamples multiple reference images. Step 203 is related to... Figure 1 The steps corresponding to step 103 are performed in the same way, and for the sake of brevity, detailed descriptions are omitted here.

[0151] In step 204, encoder 90 acquires a weighted filter to reduce the overall error between multiple upsampled reference images (or blocks) and their corresponding original images (or blocks). This step involves both determining a weighted mapping for each reference image using a weighted mapping function, which includes multiple weights mapped to corresponding spatial locations in the upsampled reference image, wherein the upsampled reference image is used as input to the weighted mapping function; and determining a filter to be applied to the upsampled reference image using the corresponding weighted mapping to acquire a filtered upsampled reference image, such that the filter is applied with different weights to different spatial locations in the upsampled reference image.

[0152] During step 204, encoder 90 determines a combination of weighted mapping and filters to enhance multiple reference images by applying filters with different weights to different regions of the upsampled reference images.

[0153] To determine the combination of weighted mappings and filters for the enhanced image, step 204 may involve a rate-distortion (RD) optimization process, which involves iteratively applying multiple filters and multiple weighted mappings to the upsampled reference image. For each application, the average total difference between the resulting upsampled reference image and the corresponding image from the original video data is determined. This loop continues until a stopping criterion is met (e.g., an optimal weighted mapping has been determined for a particular filter).

[0154] In other words, an RD optimization process was performed at the encoder based on the estimated signaling rate and the distortion of the acquired image.

[0155] A first example of a suitable stopping criterion is that a specific combination of filters and weighted mappings results in the mean absolute difference (or mean squared difference) of values ​​between the resulting multiple images and their corresponding images from the original video data being less than a predetermined threshold difference. A second example of a suitable stopping criterion is that the mean absolute difference (or mean squared difference) of values ​​between the weighted mapping of the current iteration of the iterative process and the weighted mapping of the previous iteration of the iterative process is less than a second predetermined threshold difference. The first example of a suitable stopping criterion directly measures the output quality, and therefore can be assumed to produce a higher final image quality than the second example. However, the second example ensures that the iterative processing does not require excessive computation time. In some embodiments, both examples are used, and the iterative process stops when either of these stopping criteria is met.

[0156] In the above about Figure 1 and Figure 2 In the examples discussed, a single (sharpening) filter (and its corresponding weighted mapping) was used. However, although this embodiment has been discussed in terms of finding a single combination of filter and weighted mapping, the embodiment is not limited in this respect. For example, in some embodiments, the encoder can identify multiple combinations of weighted mapping to be used with different types of filters.

[0157] Then, Figure 2 The method continues to step 205, where encoder 90 applies filters and corresponding weighted maps (collectively referred to as "weighted filters") to each reference image to obtain multiple weighted reference images. Step 205 is consistent with... Figure 1 Step 105 is performed in the manner described, and for the sake of brevity, a detailed description is omitted here. However, step 205 is optional, as will be discussed further later.

[0158] In step 206, encoder 90 performs inter-frame prediction on multiple blocks of the first image based on multiple reference blocks in one or more enhanced upsampled reference images. Similar to step 205, step 206 involves rate-distortion optimization to determine inter-frame prediction parameters, which are then encoded into the bitstream. However, the embodiment is not limited to this and any form of inter-frame prediction can be performed instead.

[0159] In step 207, the encoder encodes video data and encoding information into the bitstream. The encoding information includes weighted mapping indication information, which indicates a specific weighted mapping for each reference image.

[0160] In some embodiments, the acquired weighted filters are added to a stored set of reference filters for future use.

[0161] As described above, step 205 is optional. This is because, instead of applying the weighted filter to multiple reference images, the weighted filter can instead be applied to an upsampled reference block of the reference image during inter-frame prediction (but before adding the residual). Note that these two alternatives are equivalent (except for rounding) and are simply different implementations. Therefore, in a variant of this embodiment, step 105 is not applied; instead, the weighted filter is applied to the first image after inter-frame prediction is performed (but before adding the residual).

[0162] By reference Figure 1 and Figure 2 The discussed method reveals an in-loop filtering approach applied within the coding loop of a video compression system. The in-loop weighted filter employs a function for calculating a local weighted mapping / parameter map and a filtering function. The weighted mapping function uses the input image and optional coding information / parameters communicated via signaling to calculate the weighted mapping. The filtering function uses the input image, the weighted / parameter map, and optional coding information / parameters communicated via signaling to calculate the filtered image.

[0163] According to this method, encoding / decoding performance can be increased and a wider range of applications can be allowed by applying locally weighted filters. By determining the weighted mapping using a weighting function, weighted filtering can be applied with weights to guide the filter's properties at each spatial location. For example, the intensity of a sharpening filter can be increased near the edges of an image and decreased in regions far from the edges. In this way, ringing and overshoot can be reduced while maintaining sharpening properties.

[0164] In some examples, combined Figure 2The optimization discussed in step 204 involves iterating between filter and weighted mapping function parameters. For example, initial parameters for the weighted mapping are set, and then filter parameters are optimized based on the current weighted mapping. Then, weighted mapping parameters are optimized based on the found filter parameters, and so on. In such cases, the encoding information will include information about the weighted mapping function and the filter function (e.g., the parameters to be used). This is, of course, the basic form of the optimization process. In some cases, additional lateral constraints can be set, for example, not only to determine the optimal filter and weighted mapping in terms of image quality, but also to keep the bit rate as low as possible. This can be achieved by introducing those conditions into those two optimizations and also considering the bit rate cost when choosing the starting point for the next iteration. More generally, simplifications that limit computational cost can be additionally introduced.

[0165] In this embodiment, the weighting map provides linear weights to the filter. However, the embodiment is not limited to this, and in other embodiments, the value of the weighting map can instead modify the filtering process itself. For example, the filter can be parameterized. For example, the frequency response of an edge enhancement filter can depend on local weighting map parameters. For example, the sigma value in an unsharpened mask (a type of sharpening filter) can depend on the weighting parameters. This means that how the filter works (or more specifically, the function of the filter) is parameterized and does not necessarily depend linearly on the weighting map. Another example is a filter that performs edge thinning (sharpening) by distorting an image. The intensity of the distortion can depend on the current weighting map value.

[0166] In this embodiment, the weighted filter is applied within the loop. However, the embodiment is not limited to this particular order, and in other embodiments, the weighted filter may be applied in other processing steps (e.g., after the loop).

[0167] As discussed above, in this embodiment, encoder 90 optimizes the weighted filter for high-resolution images based on a set of reference images and their corresponding ground truth images. This means that the filter is RD-optimized to minimize the error between the upsampled and filtered low-resolution reference image and its ground truth. This is accomplished under the premise of a better reference set, i.e., the enhanced reference images improve the prediction of the current image, which in turn results in better rate-distortion performance for the current image.

[0168] In this embodiment, Figure 2 Step 204 involves optimizing a filter to minimize the error when applied to a set of low-resolution reference images. Thus, This is a reference image. These are the real images corresponding to those images, as shown in equation (1):

[0169] (1)

[0170] However, the embodiments are not limited in this respect, and other factors can also be considered. For example, in a variation of this embodiment, temporal proximity is also considered. Thus, the loss of each reference image is weighted by an importance factor. Weighting is applied so that images that are temporally closer to the first image (i.e., the images to be predicted between frames) are weighted more than images that are further apart in time. This is shown in Equation (2):

[0171] (2)

[0172] In other variations, when the importance weighting factor is determined At the same time, other factors (including other encoding / decoding factors, such as the encoding / decoding quality of the reference image) are also taken into consideration.

[0173] Furthermore, in some embodiments, spatial weighting is possible. Thus, motion vector data or other encoded information from the reference image can be used to find the local weighting of the loss function.

[0174] For example, the weighting of the reference image and its regions can be derived by analyzing the reference image using encoded information and the contents of all available images and previously encoded / decoded images.

[0175] Therefore, it is possible to determine which reference images (and which parts of these reference images) should be assigned the highest importance weighting factor. For example, if the video data involves a basketball game, importance weighting factors can be assigned primarily based on the temporal location of the images. (That is, assigning higher weight to the most recent temporal image). Because of the fast movement in a basketball game, the most recent image is likely to be the most relevant image for performing inter-frame predictions for the next image. However, if the video data involves objects that move very slowly (e.g., showing the sky and the clouds that hardly move), the temporal location of the frame is less important. Instead, higher importance can be given to a specific reference image (or a portion of a reference image) that shows the sky with high quality (e.g., with almost no blur), because this is the most likely reference image (or a portion of a reference image) to be used for inter-frame predictions.

[0176] Relatedly, experimental coding (through potentially restrictive tools) can be performed to estimate which regions are more important for filter optimization. For example, this experimental coding can be performed to determine which blocks of which reference images are most likely to be used to encode the first (to be predicted) image, allowing high importance weights to be applied to those regions.

[0177] Restrictive tools refer to simplifications in the encoding of an image (or block or strip). While this may reduce the quality of the filter (the selected block may differ), it reduces runtime.

[0178] The options for limiting / simplifying include:

[0179] - Test a smaller number of intra / inter-frame modes in the prediction (i.e., turn off some tools). Examples include turning off all intra-frame tools except for directional, planar, and DC modes / checking only affine and regular inter-frame predictions, but not using more sophisticated tools such as decoder-side motion vector derivation (DMVD) or linear illumination compensation (LIC).

[0180] - Use early-stopped or restricted sets of partitions in block partitioning; and

[0181] - Perform trial coding at a lower quality (which usually takes less time).

[0182] More generally, it can be seen that the importance-weighted factor A reference image (or a portion thereof) can be assigned to a reference image based on the likelihood that the reference image will be used in inter-frame prediction.

[0183] To summarize this variation: Given a set of reference images (from previous images) Corresponding real image and reference filter set The weighted filter is optimized through the following steps:

[0184] 1. Calculate the importance weighting of the reference image;

[0185] 2. Calculate the RD optimization filter to minimize the weighted loss between the filtered image and its corresponding ground truth value while considering the bit rate:

[0186]

[0187] 3. Either assign the filter to the current (high-resolution) image and apply it to all inter-frame prediction blocks before adding residuals, or apply it to all low-resolution images to generate a high-resolution reference. Note that these two alternatives are equivalent (except for rounding) and are simply different implementations.

[0188] 4. Add the filter to the reference filter set.

[0189] As an overview of this method in the first embodiment, the weighted filter is applied as an enhancement filter for inter-frame prediction and upsampling output image enhancement. One filter can be optimized for each low-resolution image. Distortion can be given by the coding cost of the image and the filter. Distortion can be the distortion after the filter is applied. If the rate-distortion cost is low when applying a filter, that filter is selected. The filter can then be used to enhance any upsampled content generated from the image. Therefore, if the filter is output (i.e., written as an upsampled image) and if patches of that image are referenced, the same filter can be used.

[0190] For illustrative purposes, please refer to the following. Figure 1 and Figure 2 The principles described in the embodiments will now be referred to Figure 3 Explanatory examples.

[0191] Figure 3 The above discussion illustrates the following. Figure 1 and Figure 2 The embodiments illustrate the block diagram of example operations. In other words, Figure 3 A conceptual diagram illustrating how a weighted filter can be implemented is shown.

[0192] like Figure 3 As shown in the process, the distorted image 31 is eventually converted into the (enhanced) output image 34.

[0193] The distorted image corresponds to, for example, a reference. Figure 1 The upsampled reference image discussed in step 103.

[0194] The input can be a decoded upsampled image or patch, along with side information encoded in an adaptive parameter set within the bitstream. First, the filter parameters are decoded. This can include information about the weighted mapping function, filter shape, luminance and chrominance flags, and filter coefficients. Next, the weighted mapping function is applied to obtain a locally weighted mapping. Then, the filter is applied to the upsampled image or patch to generate a filtered map. The filtered map is multiplied by the locally weighted map and added to the upsampled image.

[0195] For example, refer to Figure 1 Step 104, then the distorted image is provided as input to the weighted mapping function (f) determined based on the encoded information. w-map 3A is used to produce a weighted map 32. In this particular example, as discussed above, the weighted map function 3A is a local gradient calculation function. Therefore, the weighted map 32 includes multiple values ​​of the image gradient at each spatial location in the distorted image 31.

[0196] Still referencing Figure 1Step 104 determines filter 3B. As discussed above, in this embodiment, decoder 80 infers the filter to be used from the weighted mapping information. Also as mentioned above, in this example, the weighted mapping information included in the encoded information is a local gradient calculation function. Therefore, in this example, decoder 80 infers that a sharpening filter (e.g., a pre-stored one) should be used in combination with a weighted mapping that includes the image gradient.

[0197] refer to Figure 1 In step 105, the decoder 80 then applies the weighted map 32 and filter 3B to the distorted image 31 to obtain the output image 34. (As...) Figure 3 As shown, this step involves providing the distorted image 31 and the weighted map 32 as inputs to the (sharpening) filter 3B to obtain a weighted enhancement map 33. The weighted enhancement map 33 includes multiple offset values, which spatially correspond to the values ​​of the distorted image 31.

[0198] Next (as) Figure 1 As part of step 105, decoder 80 adds the offset value in weighted enhancement image 33 to the corresponding value in distorted image 31 to obtain output image 34.

[0199] This method demonstrates that blur is reduced without the significant ringing and overshoot issues that can occur with sharpening filters. This is achieved through local weighting of the sharpening filter, ensuring that it is not applied uniformly across the entire image, but rather applied with varying intensities to different spatial regions of the image, depending on their properties.

[0200] More specifically, the intensity of the sharpening filter is increased near the edges of the image and decreased in areas farther from the edges. This reduces ringing and overshoot while maintaining sharpening properties.

[0201] Note that in some embodiments, the addition of the offset map is incorporated into the filter function. However, for better visualization purposes, these operations are performed separately. Figure 3 The steps are shown separately.

[0202] Furthermore, as will be discussed in more detail later, in some embodiments, both (or one of) the weighted mapping and the filtering function can be parameterized functions that depend on parameters notified by signaling in the bitstream.

[0203] As can be seen, the weighted filter consists of two main components: the weighted / parameter graph computation function and the (potentially parameterized) filter.

[0204] First, a weighted map is computed. In this embodiment, the weighted map is computed at every point in the image (although in other embodiments, it may be computed at a lower resolution). Next, a filter is applied. Thus, the weighted / parameter map and the decoded image are input to the filter. This generates a filtered image.

[0205] exist Figure 1 and Figure 2 In one embodiment, the method is applied to the luma channel. However, the embodiments are not limited to this. In other embodiments, the method is applied only to the chroma channel, or to both the luma and chroma channels. In other words, in some embodiments, the weighted / parameter graph calculation function and the filter (or one of them) may be different for different channels. Therefore, separate information / parameters can be signaled for the luma and chroma components.

[0206] In this embodiment, the weighted filter replaces the SAO and ALF in the VVC / H.266 system.

[0207] However, the embodiments are not limited to this aspect. For example, in other embodiments, the weighted filter replaces only one of the SAO or ALF, or is provided separately from the SAO and ALF. In another example, the weighted filter can be integrated into the ALF, such that each partition of the image can be filtered by an optimized linear filter or by a weighted filter. This gives the ALF additional flexibility. For maximum flexibility, the filter can be added at any point in the filter chain within the loop.

[0208] Although these specific examples have been discussed, it is understood that the embodiments are not limited to the H.266 / VVC scheme. In other embodiments, the weighting filter is applied to entirely different codec schemes (e.g., H.265 / HEVC or any other suitable codec scheme).

[0209] Furthermore, in other embodiments, weighted filters are used as post-filters to enhance the quality of the encoded / decoded video. This is beneficial if post-coupling (of in-loop filtering) leads to worse predictions of subsequent images. In such cases, out-of-loop / post-filtering would be advantageous. In embodiments, this determination can be made by the encoder during encoding.

[0210] Therefore, more generally, in embodiments, weighted filters can be integrated into the encoding / decoding loop as an additional processing step in existing schemes, as an alternative to existing loop filters, or integrated into existing loop filters.

[0211] To further explain the concept of weighted filters discussed above, we will now refer to... Figure 4 Discuss another example implementation. Figure 4A graph comparing several filtering schemes to a real signal is shown.

[0212] Assume there exists a blurred edge "x" as the input to the filter. The real signal 41 (e.g.) Figure 4 The signal shown (indicated by the figure) is a step function. For simplicity and visualization, a one-dimensional signal is displayed. Note that this is done merely to explain the concept in a simple way. Generally, the methods discussed in this article can be applied to signals of any dimension.

[0213] Figure 4 The chart shows four lines. These represent the true signal 41, the ambiguous signal 42 (for example, corresponding to...), and so on. Figure 3 Distortion 31), weighted filtered signal 43 (for example, corresponding to Figure 3 The output image 34) and the unweighted filtered signal 44 (e.g., if weighted mapping has not been used, i.e., only a filter was used, corresponding to Figure 3 Output image 34).

[0214] In unweighted filtering, the ambiguous signal 42 is filtered by a least-squares optimized linear filter to approximate the original signal as accurately as possible, thus obtaining the unweighted filtered signal 44. Figure 4 As can be seen, this type of filtering does increase the steepness of the edges, thus providing a better approximation of the true signal 41, but it causes overshoot and ringing effects, which is non-optimal.

[0215] Better results can be obtained if the high-pass characteristic of the filter is stronger at the steepest parts of the edges and weaker in areas where the filter causes ringing and overshoot artifacts. To achieve this, in the weighted signal 42, the offset of the filtered blurred edge relative to the blurred edge is scaled by a larger factor at the predicted location of the edge, while a smaller factor is used if overshoot or ringing effects are expected. For this purpose, local weighting (i.e., calculating a weighted mapping) is computed from the image. For example, (see reference above) Figures 1 to 3 As discussed, the gradient magnitude can be approximated by finite differences. To obtain the optimal result, the filter will then be optimized considering local weighting.

[0216] like Figure 4 As shown, the weighted filtered signal 43 has a smaller error than the unweighted filtered signal 44 compared to the real signal 41. Furthermore, it can be seen in this example that the overshoot of the weighted filtered signal 43 is at a similar level, but the steepness and ringing of the signal are less severe (i.e., it has a similar amplitude but flattens out earlier).

[0217] This example demonstrates that local adaptability can improve filter performance if the characteristics and errors of the signal are known. Adaptive Loop Filters (ALFs) approach this by applying different filters depending on the image characteristics. This allows for greater flexibility, but at the cost of increased bit rate.

[0218] Embodiments of the present invention utilize weighted (potentially parameterized) filters to reduce the need for using multiple different filters. The example shown is an application scenario where a single weighted filter can replace a set of filters while achieving similar results. This is particularly applicable to situations where there are dependencies that can be exploited through local parameterization.

[0219] In this embodiment, it has been discussed that the encoded information includes weighted mapping information, which includes weighted mapping functions. However, the embodiments are not limited to this aspect. For example, as discussed above, in some embodiments, multiple weighted mapping functions are stored in the decoder 80. In this case, the weighted mapping information instead includes an indication of which weighted mapping to use. Furthermore, in some embodiments, the encoded information does not include any explicit weighted mapping indication. Instead, for example, the encoded information may include filter information, which is then inferred by the decoder 80 as the weighted mapping function to be used (e.g., if the encoded information indicates that a sharpening filter should be used, the decoder infers that an image gradient function should be used as the weighted mapping function).

[0220] References above Figure 1 and Figure 2 In the described embodiments, the weighted filter is specifically applied in an in-loop manner within the VVC / H.266 system.

[0221] Typically, in-loop filters can be applied at every point within the codec loop. However, the order in which they are applied can impact overall performance because most in-loop filters are non-linear. An example of a video codec system is VVC / H.266. In this system, four in-loop filters are applied sequentially: luma mapping with chroma scaling (LMCS), deblocking, sample adaptive offset (SAO), and adaptive loop filter (ALF). LMCS addresses errors very different from those of the proposed filter, and the inverse mapping should be applied before the proposed filter to avoid artifacts. Furthermore, it makes sense to apply the deblocking filter before the proposed method.

[0222] Although specific implementations of the invention have been discussed above, several variations can be made in other embodiments, which will now be discussed, particularly regarding the selection of the weighted mapping function, the filtering function, the regional partitioning, and the signaling notification.

[0223] As can be seen from the above discussion, the weighted map provides one or more weights / parameters to the filter. In other words, the filter is a parameterized function that takes the weighted map and the image (and possibly the encoded parameters) as input.

[0224] Regarding weighted mapping, Figure 1 and Figure 2 In one embodiment, the weighted mapping is a scalar graph. In this scalar graph, each spatial location is assigned exactly one value. However, in other embodiments, the weighted mapping is a multidimensional graph. In such embodiments, each spatial location has a value vector. It should also be noted that in embodiments of the invention, the spatial size of the weighted mapping is not limited to the resolution of the image. Depending on the requirements, in some embodiments, the weighted mapping may have a smaller resolution to reduce computational complexity. The optimal choice of the weighted mapping function depends largely on the type of error the filter addresses and the type of filter used to process these errors.

[0225] As discussed above, in Figure 1 and Figure 2 In the embodiments, weighted mapping and filters are applied to the reference image / inter-frame prediction image ( Figure 1 Step 105 involves providing both the image and the weighted map as inputs to the filter. This causes the filter to be applied with varying intensities to each value of the image, depending on the values ​​of the corresponding weights in the weighted map. The output of the filter is an offset map, and the offset map corresponds to the image at a specific resolution. Once the offset map is output, the offset values ​​are then added to the values ​​of the image to produce the output (enhanced) image. For example, Figure 3 This example is shown in the figure, which has been discussed above.

[0226] However, the embodiments are not limited to this particular implementation. In an alternative, simpler implementation, the filter uses intensity values ​​(i.e., the weights of the weighted mapping) to scale the offset generated by the filter. This is achieved by multiplying the intensity values ​​(i.e., the weights) by the difference between the filter's output and the acquired image. This scaled difference is added to the acquired image, and then the offset generated by the filter is varied depending on the calculated weights. Therefore, the effect of the applied filter varies depending on the spatial location.

[0227] Examples of this setting are as follows: Figure 5 As shown, Figure 5 A block diagram illustrating an example operation of this alternative method is shown.

[0228] It can be seen that, Figure 5The method involves feeding the distorted image 51 as input to a weighted map and also as input to a filter. The intensity values ​​(i.e., weights) of the resulting weighted map 52 are then multiplied by the difference between the output of the filter 55 and the distorted image 51. This scaled difference is added to the distorted image 51, and the offset generated by the filter is then varied depending on the calculated weights.

[0229] In other words, a scalar weight is computed to weight the filter output through sample-by-sample multiplication. The computed offset is then added to the input image to obtain the output.

[0230] Through with Figure 3 The example comparison shows that the weighted mapping 52 is not applied as input to the filter, but its value is simply multiplied by the output of the filter (specifically, the difference between the output of filter 55 and the distorted image 51).

[0231] Figure 6 A flowchart illustrating the operation of the decoder 80 according to the second embodiment is shown. Figure 7 An operation flowchart of the encoder 90 according to the second embodiment is shown.

[0232] Figure 1 The flowchart begins at step 601, where decoder 80 decodes the bitstream to obtain video data and encoding information. In this embodiment, the encoding information includes weighted mapping information.

[0233] In step 602, decoder 80 acquires one or more reference images. Step 602 is related to... Figure 1 Step 102 is performed in the corresponding manner, and for the sake of brevity, a detailed description is omitted here.

[0234] In step 603, the decoder performs inter-frame prediction on multiple blocks of the first image based on multiple reference blocks in one or more reference images. Specifically, the decoder 80 uses the reference images to perform prediction operations to obtain multiple prediction blocks of the first image. More specifically, the decoder 80 uses the reference images as reference images to perform inter-frame prediction to obtain prediction blocks.

[0235] In this embodiment, the resolution of the one or more reference images is lower than that of the first image. This is because, for example, the multiple reference images may represent previous images (or "frames") that have been encoded into a bitstream with a resolution lower than that of the corresponding original images in the original video data (i.e., they may be images that the decoder decodes based on the encoded bitstream), for example, considering the network conditions at the time of the encoding. These previous images may be stored in the decoder and encoder as a set of reference frames for use in inter-frame prediction of future images. Therefore, each reference image corresponds to an original image in the original video data (although there are differences due to the lower resolution when the reference images are acquired and possible prediction errors).

[0236] Therefore, the inter-frame prediction operation in step 604 involves upsampling a reference block (or the entire reference image) of the reference image to predict a block of the first image.

[0237] In step 604, the decoder 80 obtains a weighted filter based on the encoded information and applies it to multiple inter-frame prediction blocks of the first image to reduce the error between the first image and the corresponding original first image by: determining a weighted mapping using a weighted mapping function, the weighted mapping including multiple weights that are mapped to corresponding spatial locations in each inter-frame prediction block of the first image, wherein the inter-frame prediction blocks of the first image are used as input to the weighted mapping function; and determining a filter to be applied to the inter-frame prediction blocks of the first image through the weighted mapping, wherein the filter is configured to be applied to the inter-frame prediction blocks of the first image through the weighted mapping to obtain a filtered first image, such that the filter is applied with different weights to different spatial locations in each inter-frame prediction block of the first image.

[0238] Here, decoder 80 uses weighted mapping information to determine the weighted mapping. In this embodiment, decoder 80 uses weighted mapping information to determine the weighted mapping. In this embodiment, the weighted mapping information includes a weighted mapping function used to calculate the weighted mapping.

[0239] One such example is when the sharpening filter is to be used, the weighted mapping information included in the encoded information is a local gradient calculation function. In this example, in step 103, the decoder 80 applies the gradient calculation function to calculate the gradient at each location in the inter-frame prediction block of the first image, thereby obtaining a scalar weighted map with the same resolution as the first image (having values ​​corresponding to each corresponding location in the image). In other words, the first image is provided as input to the weighted mapping function.

[0240] However, it should be understood that this is merely an example. In other embodiments, the weighted mapping may have a different resolution than the first image, and / or may include vector values ​​instead of scalar values. Furthermore, it is understood that the example of the image gradient function is merely illustrative, and in the actual implementation of the embodiment, the choice of a suitable weighted mapping function will depend on the circumstances, particularly which filter is to be used. These factors and possible variations will be discussed in more detail later.

[0241] Furthermore, while this embodiment has discussed that the encoded information includes weighted mapping information, and that the weighted mapping information includes weighted mapping functions, the embodiments are not limited in this respect. For example, in some embodiments, multiple weighted mapping functions are stored in the decoder 80. In this case, the weighted mapping information instead includes an indication of which weighted mapping to use. Furthermore, in some embodiments, the encoded information does not include any explicit weighted mapping indication. Instead, for example, the encoded information may include filter information, which is then inferred by the decoder 80 to use the weighted mapping function (e.g., if the encoded information indicates that a sharpening filter should be used, the decoder infers that an image gradient function should be used as the weighted mapping function). These factors and possible variations will be discussed in more detail later.

[0242] In step 604, decoder 80 further determines the filter to be used. In this embodiment, decoder 80 infers the filter to be used from the weighted mapping information. As discussed above, in one example, the weighted mapping information included in the encoded information is a local gradient calculation function. In this example, decoder 80 can infer that a sharpening filter should be used (e.g., a pre-stored one) in combination with a weighted mapping that includes the image gradient.

[0243] In step 606, the decoder 80 applies a weighted map and a filter to the inter-frame prediction block of the first image to obtain a filtered image. Therefore, step 606 involves using the determined weighted map such that the filter is applied with different intensities to different regions of the inter-frame prediction block of the first image.

[0244] In this embodiment, applying a weighted map and filter to the first image involves providing both the first image and the weighted map as inputs to the filter. This results in the filter being applied to each value of the image with varying intensity depending on the values ​​of the corresponding weights in the weighted map. The output of the filter is an offset value map. In this embodiment, the offset value map corresponds to the first image in resolution. Once the offset value map is output, the offset values ​​are then added to the values ​​of the first image to produce an output (enhanced) image.

[0245] In the above discussion, it is assumed that the filter is a sharpening filter configured to sharpen blurred edges. However, the embodiments are not limited to this, and any suitable filter for enhancing an image can be used instead.

[0246] After step 605, the output image can then be used for any desired purpose. In one example, decoder 80 then displays the output image to a viewer. In another example, decoder 80 stores the image for later use. In yet another example, decoder 80 transmits the image to an external device for display.

[0247] In this embodiment, a weighted filter can be added to the reference set for future filter optimization.

[0248] Encoder 90 can perform a complementary method to encode the bitstream provided to decoder 80. Figure 7 A flowchart illustrating the operation of the encoder 90 according to this embodiment is shown.

[0249] In step 701, encoder 90 acquires the raw video data. For example, encoder 90 can receive the raw video data from an external server via a communication network (e.g., the Internet). However, in this embodiment, there are no restrictions on how the raw video data is acquired.

[0250] In step 702, encoder 90 performs trial encoding of at least a portion of the raw video data into trial-encoded video data. For example, the trial encoding may involve encoding inter-frame prediction parameters for obtaining a first trial image from a reference image. The at least portion of the raw video data may refer to, for example, an image (or "frame") or a strip.

[0251] In step 703, decoder 90 acquires a test first image based on the test-encoded video data. In this embodiment, this step involves performing inter-frame prediction on multiple blocks of the test first image based on multiple reference blocks in one or more reference images, the resolution of which is lower than that of the test first image. Therefore, this step involves upsampling at least the referenced blocks in the one or more reference images.

[0252] In step 704, a weighted filter for multiple inter-frame prediction blocks to be applied to the experimental first image is obtained in the following manner to reduce the error between the experimental first image and the corresponding original first image in the original video data: a weighted mapping function is used to determine a weighted mapping, which includes multiple weights mapped to corresponding spatial locations in each inter-frame prediction block of the experimental first image, wherein the inter-frame prediction blocks of the experimental first image are used as inputs to the weighted mapping function; and a filter is determined to be applied to the inter-frame prediction blocks of the experimental first image through the weighted mapping, wherein the filter is configured to be applied to the inter-frame prediction blocks of the experimental first image through the weighted mapping to obtain a filtered first image, such that the filter is applied with different weights to different spatial locations in each inter-frame prediction block of the experimental first image.

[0253] In performing these steps, encoder 90 determines a combination of weighted mapping and filters to enhance the trial first image by applying filters with different weights to different regions of the inter-frame prediction block of the first image.

[0254] To determine the combination of weighted mappings and filters for the enhanced image, step 704 may involve a rate-distortion (RD) optimization process, which involves iteratively applying multiple filters and multiple weighted mappings to a trial first image. For each application, the average difference between the resulting image and the corresponding image from the original video data is determined. This loop continues until a stopping criterion is met (e.g., an optimal weighted mapping has been determined for a particular filter).

[0255] In other words, an RD optimization process is performed at the encoder based on the estimated signaling rate and the distortion of the acquired image.

[0256] A first example of a suitable stopping criterion is: a specific weighted mapping such that the mean absolute difference (or mean squared difference) of the values ​​between the resulting image and the corresponding image from the original video data is less than a predetermined threshold difference. A second example of a suitable stopping criterion is: the mean absolute difference (or mean squared difference) of the values ​​between the weighted mapping of the current iteration of the iterative process and the weighted mapping of the previous iteration of the iterative process is less than a second predetermined threshold difference. The first example of a suitable stopping criterion directly measures the output quality, and therefore can be assumed to produce a higher final image quality than the second example. However, the second example ensures that the iterative processing does not require excessive computation time. In some embodiments, both examples are used, and the iterative process stops when either of these stopping criteria is met.

[0257] The above about Figure 6The examples presented use a single (sharpening) filter (and its corresponding weighted mapping). However, while this embodiment has been discussed in terms of finding a single combination of filter and weighted mapping, the embodiment is not limited in this respect. For example, in some embodiments, the encoder can identify multiple combinations of weighted mapping to be used with different types of filters.

[0258] Figure 7 The method then proceeds to step 705, in which encoder 90 encodes video data and encoding information, including weighted mapping information, into the bitstream.

[0259] By reference Figure 6 and Figure 7 The discussed method demonstrates the existence of an in-loop filtering approach applied within the coding loop of a video compression system. The in-loop weighted filter employs a function for calculating a local weighted mapping / parameter map and a filtering function. The weighted mapping function calculates the weighted mapping using the input image and optional coding information / parameters communicated via signaling. The filtering function calculates the filtered image using the input image, the weighted mapping / parameter map, and optional coding information / parameters communicated via signaling.

[0260] According to this method, encoding / decoding performance can be increased and a wider range of applications can be allowed by applying locally weighted filters. By determining the weighted mapping using a weighting function, weighted filtering can be applied with weights to guide the filter's properties at each spatial location. For example, the intensity of a sharpening filter can be increased near the edges of an image and decreased in regions far from the edges. In this way, ringing and overshoot can be reduced while maintaining sharpening properties.

[0261] In some examples, reference Figure 7 Step 704 discusses optimization involving iteration between filter and weighted mapping function parameters. For example, initial parameters for the weighted mapping are set, and then filter parameters are optimized based on the current weighted mapping. Then, weighted mapping parameters are optimized based on the found filter parameters, and so on. In such cases, the encoding information will include information about the weighted mapping function and the filter function (e.g., the parameters to be used). This is, of course, the basic form of the optimization process. In some cases, additional lateral constraints can be set, for example, not only to determine the optimal filter and weighted mapping in terms of image quality, but also to keep the bit rate as low as possible. This can be achieved by introducing those conditions into those two optimizations and also considering the bit rate cost when choosing the starting point for the next iteration. More generally, simplifications that limit computational cost can be additionally introduced.

[0262] In this embodiment, the weighting map provides linear weights to the filter. However, the embodiment is not limited to this, and in other embodiments, the value of the weighting map can instead modify the filtering process itself. For example, the filter can be parameterized. For example, the frequency response of an edge enhancement filter can depend on local weighting map parameters. For example, the sigma value in an unsharpened mask (a type of sharpening filter) can depend on the weighting parameters. This means that how the filter works (or more specifically, the function of the filter) is parameterized and does not necessarily depend linearly on the weighting map. Another example is a filter that performs edge thinning (sharpening) by distorting an image. The intensity of the distortion can depend on the current weighting map value.

[0263] In this embodiment, a weighted filter can be added to the reference set for future filter optimization.

[0264] In this embodiment, the weighted filter is applied within the loop. However, the embodiment is not limited to this particular order, and in other embodiments, the weighted filter may be applied in other processing steps (e.g., after the loop).

[0265] In a variation of this embodiment, in Figure 7 Prior to step 705, encoder 90 determines whether to include any information about the weighted filter in the encoded information (i.e., whether the decoder should acquire and apply the weighted filter). Specifically, as discussed above, steps 701 through 704 involve trial encoding of the image (or stripe) to find the motion vectors predicted between frames. This is used to find the low-resolution block referenced by the current image. Given this information, the filter is optimized such that the error between the referenced block and the ground truth of the current (high-resolution) image is minimized.

[0266] In this variant, a next round of optimization is performed after step 704, in which filters are used to enhance all low-resolution blocks (of the reference image) in the motion compensation optimization, making it possible to find optimized motion vectors that take into account the filtered low-resolution image. Then, if the RD performance is higher than the unfiltered version (i.e., the version without weighted filters), it is determined that weighted filters should be used at the decoding end.

[0267] In summary, this variation involves a set of steps:

[0268] 1. Experimentally encode the image (or strip).

[0269] 2. Locate the predicted data (from the low-resolution reference image) and its corresponding ground truth value in the first (current) image.

[0270] 3. Optimize the weighted filter to minimize the prediction error.

[0271] 4. Encode the image (or strip) using this weighted filter.

[0272] 5. If the RD performance of step 4 is better than that of step 1: then take the result of step 4 and add the filter to the reference set for future filter optimization. Otherwise, use the output of step 1.

[0273] As discussed above, this second embodiment may involve optimizing a filter based on a set of blocks (possibly transformed, such as translated) such that the error between these blocks and the ground truth of the current image is minimized. Thus, this set of blocks and its predicted signal can be derived by trial-and-error encoding of the image or strip (or by pre-analyzing the image content). The filter or filter set is optimized based on the (possibly estimated) signal in the image (i.e., the current frame) by minimizing the error between the predicted signal and the ground truth of the current block.

[0274] In some embodiments, the prediction signal may change, which may cause the prediction method to select a different set of blocks, so the method is applied iteratively.

[0275] It will be further understood that the first and second embodiments described above are combined by applying joint optimization. Therefore, the weighted filter can be optimized to minimize the weighted sum of loss terms, which is a combination of the losses of the error terms generated by the method of the first embodiment or the method of the second embodiment.

[0276] In some embodiments, the weighted mapping function is an edge detector that assigns higher weights to locations on edges and lower values ​​to areas surrounding the edges and flat regions. Therefore, in such embodiments, the resulting weighted mapping is a scalar. This allows for edge sharpening with fewer artifacts. In such embodiments, the encoder may, for example, encode information identifying the edge detector function into the bitstream.

[0277] In other embodiments, the weighted mapping function is for a ringing detector. For example, this assigns a probability that ringing is present at a given location. Therefore, in such embodiments, the resulting weighted mapping is a scalar. By applying this weighted mapping, the filter strength is set based on this probability. In such embodiments, the filter can be, for example, a simple linear low-pass filter. However, in many cases, ringing is close to the edge that should ideally be preserved. Therefore, the filtering scheme should ideally preserve the edge. An example of a suitable filter is a bilateral filter. In embodiments utilizing such filters, the parameters can be optimized at the encoder and transmitted in the bitstream.

[0278] Furthermore, in some embodiments, these two weighted mapping options can be combined into a two-dimensional weighted mapping, or they can be applied sequentially. Combining them provides more flexibility in filtering. Knowing the estimated probability of ringing and the presence of edges helps the encoder find the optimal filter during its rate-distortion optimization process. For example, edge amplification should be applied more cautiously if ringing is very close to this location, as ringing may be amplified, which can be optimized by the encoder during encoding.

[0279] Examples of applying bilateral filters also illustrate how multidimensional weighted mappings, rather than just scalars, can be applied in some embodiments. For instance, in some embodiments, the encoder determines different parameters if there are regions of greater contrast compared to regions of lower contrast. In this case, the parameters of the bilateral filter are estimated from the acquired image and signaled to the decoder.

[0280] Another application in other embodiments is to estimate parameters for edge sharpening filters from an image. For example, if the weighted map estimates whether there is very sharp content (such as text) at a certain location in an image / block, a different sharpening can be applied than for other types of content. Note that in such embodiments, the weighted map can be derived from image content, encoding information, or parameters notified by signaling. In these embodiments, the weighted map can be one-dimensional. The weighted map can be binary, integer, or floating-point. In these embodiments, the weighted map can alternatively be multi-dimensional, with each element in the weighted map being binary, integer, or floating-point. The data type of each element in the weighted map depends on the requirements of the filtering system.

[0281] In some embodiments, the weighted mapping computation parameters are signaled in the bitstream, and the type of the weighted mapping function is determined. In other embodiments, the weighted mapping computation parameters signaled are parameters of the function itself. For example, the edge map may have a steepness scaling parameter that determines how much weight is added based on steepness. Note that this may be non-linear scaling. For example, scaling may be performed using a power of n of the value.

[0282] In some embodiments, a predefined (possibly parameterized) set of weighted mapping functions is used. Therefore, the bitstream only notifies the weighted mapping functions / weighted mappings used (and possibly specific parameters to be used in these weighted mapping functions) via signaling, rather than notifying all weighted mapping functions via signaling.

[0283] In summary, regarding weighted mapping, in some embodiments, a weighted mapping function that outputs a scalar weighted mapping is applied, where the scalar is binary, integer, or floating-point. In other embodiments, a weighted mapping function that outputs a multidimensional weighted mapping is applied, where each element is binary, integer, or floating-point. In these embodiments, using information from one or more channels of the reconstructed image as input, weighted mapping information for one or more channels of the acquired image is calculated. In some embodiments, a predefined (possibly parameterized) set of weighted mapping functions is provided, where the bitstream is signaled to use the weighted mapping function / weighted mapping used.

[0284] Regarding filters, the filter functions used in some embodiments of this invention are typically multidimensional parameterized functions that take a weighted mapping, filter parameters, one or more channels of the acquired image, and possible encoding information as input. The output is one or more (weighted) filtered channels of the acquired image.

[0285] Although these embodiments have been generally discussed for applications involving a single filter, the invention is not limited in this respect. For example, in other embodiments, a series of filters with different parameters and possibly different weighted mappings can be applied. Depending on the type of artifact, complexity requirements, and RD decisions, different filtering functions may be most suitable.

[0286] For example, in some embodiments, a linearly weighted filter can be used. In such embodiments, the filter is weighted by multiplying its output by a local weight, and then the result is added to the acquired image. The advantage of such systems is that the optimal parameters can be found through least-squares optimization. Therefore, a parameter search is not required to find the optimal solution.

[0287] However, in such embodiments, depending on the characteristics of the image, a relatively large number of filter coefficients (which will need to be transmitted) may require linear filters. To offset this, in some embodiments, parameterized descriptions of the filters are used to reduce encoding and decoding costs (although at the cost of reduced flexibility).

[0288] An example of parameterization is modeling a high-pass filter as a Gaussian difference filter. In this case, only the sigma value needs to be transmitted, not the entire set of filter coefficients. This approach is useful if the frequency response of the parameterized filter is sufficiently close to the distribution of the filter obtained through least-squares optimization. However, depending on the lateral constraints of the parameterized representation, a closed-form solution may not be found, in which case iterative optimization will be required.

[0289] In another embodiment, another type of filter that can be used is a (parameterized) nonlinear filter. Examples include bilateral filters, median filters, or other filters. The parameters of these filters can be notified in the bitstream via signaling or given by a weighted mapping. Note that switching the type of filter function based on the weighted mapping parameters is also an option.

[0290] In other words, in some embodiments, the value of the weighted mapping at a particular scalar location can indicate the type of filtering function to be used at that spatial location in the image.

[0291] In summary, regarding filters, in some embodiments, one or more filters may be applied for each weighted map notified by signaling. The parameters of the filtering function and filters may be notified by signaling in the bitstream, are predefined, or inferred from the content or encoding information of the video sequence. In some embodiments, linear filters are applied as filters in the filtering function. The shape of the filter may be indicated in the bitstream or predefined. In some embodiments, the linear filter is optimized by least-squares optimization or rate-distortion optimization. In some embodiments, parameterized linear filters are used, where these parameters can be used to generate the corresponding linear filter. In some embodiments, the parameterized linear filter is RD optimized for minimum error at the output, or RD optimization is performed. The optimal filter can be derived by least-squares optimization, iterative search, or exhaustive search. In some embodiments, parameterized or non-parameterized nonlinear filters are applied in the filtering function. In some embodiments, bilateral filters are used in the filtering function. In some embodiments, a combination of the discussed filtering methods is applied, where the filtering method used at each location is notified by signaling or indicated by the weighted map. In some embodiments, the encoder optimizes the parameterized weighted map and the filtering function together.

[0292] In some embodiments described herein, the weighted filter is described as being applied to the entire image (e.g., the entire reference image). However, the embodiments are not limited to this. In variations of these embodiments, different filtering settings may be applied to different regions of the image. Furthermore, overlapping application of filters is possible. Two example implementations for region partitioning will now be discussed.

[0293] The first method is block-by-block partitioning of the image, where each filter is applied to one or more blocks. In some embodiments, alignment of coding tree units (CTUs) and coding units (CUs) boundaries may be considered. Subsequently, the applicable image partitions are signaled in the bitstream.

[0294] The second approach is to partition the image based on its characteristics. In some embodiments, this partitioning can be derived at both the decoding and encoding ends without signaling notification within the bitstream. This can be implemented as a binary weighted mapping or a non-binary weighted mapping. In embodiments, each resulting partition can then be processed individually, or partitions can be processed in groups.

[0295] When using region partitioning, the filter to be applied to a region or group of regions can be optimized at the encoding end, and then the parameters can be signaled in the bitstream.

[0296] Therefore, in some embodiments, an image can be divided into multiple partitions. Partitions can be defined by block partitions notified by signaling, region partitioning criteria notified by signaling, and / or binarized weighted mapping functions. Furthermore, in some embodiments, multiple filters can be applied to the same image partition.

[0297] Using partitioning in these ways is particularly useful when processing large or highly diverse images. Such images may contain very different types of content, and the error characteristics can vary significantly across different partitions of the image. Therefore, optimizing two or more filters for different partitions of the image can yield high performance.

[0298] As discussed in the described embodiments, encoding information (e.g., information about the weighted mapping function and / or filter to be applied) may be included in the bitstream. More generally, in embodiments, the encoding information may include (but is not limited to) filter coefficients, weighted mapping function parameters, on / off flags, filter coding parameters, or region parameters, etc.

[0299] In some embodiments, this information (e.g., all parameters) is encoded and signaled to reduce the transmission rate and improve the efficiency of the overall filtering process. This is achieved by utilizing redundancy regarding the transmitted parameters. This redundancy is utilized through prediction and entropy coding of the filter parameters. Furthermore, in some embodiments, the parameters are quantized to reduce the number of possible representations. For a detailed description of parameter encoding / decoding schemes that can be used to encode such filter parameters, see PCT / CN2023 / 105596, which is incorporated herein by reference in its entirety.

[0300] In the embodiments described herein, the filter used (i.e., the sharpening filter) is based on the concept of a Wiener filter. In other embodiments, the filter is a linear filter that has been optimized at the encoder through a least-squares optimization process (i.e., a linear filter that minimizes the squared error between the filtered signal and the true signal). Of course, in some embodiments, additional lateral constraints are set when determining the signal enhancement filter, such as filter shape and filter coefficients that must be equal.

[0301] However, although the embodiments have been discussed with reference to filters based on the Wiener filter concept, the embodiments are not limited in this respect and other types of filters can be used instead, such as filters based on Sobel filters or unsharpened mask filters as sharpening filters. Other nonlinear options include bilateral filters and diffused filters, as well as adaptive loop filters (ALF).

[0302] For example, in some embodiments, as an alternative to linear filters, weighted filters can be integrated into the adaptive loop filter (ALF) of existing codec schemes (e.g., H.265 / HEVC and H.266 / VVC) and applied to partitions derived through ALF optimization.

[0303] Considering the similarity between the determination of the weighted filter in the first and second embodiments (e.g., Figure 2 For the sake of brevity, the specific implementation details and variations described for the first embodiment will not be repeated herein. However, it should be understood that they apply to this second embodiment.

[0304] As an overview of this method in the second embodiment, it is assumed that there is a video sequence encoded via RPR. This sequence encodes a certain number of images at high resolution, then switches to a lower encoding resolution, and then switches back to a higher resolution. RD optimization of a filter for each low-resolution image and its application to inter-frame prediction blocks within that image is not necessarily efficient because it is not a case where the entire image needs to be upsampled and thinned, as the filter is also used to improve the upsampled low-resolution output image. First, not all low-resolution images can be used as references for high-resolution images. Furthermore, not all image regions can be used as references. The first embodiment optimizes the filter for estimating the blocks to be used by the high-resolution frame. To find the reference blocks, trial coding of the strips can be performed. Then, all referenced low-resolution blocks are stored along with ground truth data for motion-compensated blocks in the current image. If the reference low-resolution region is below a predefined threshold, optimization can be skipped. Otherwise, the filter can be optimized such that the sum of errors for all blocks is minimized by least squares. This filter can then be assigned to the high-resolution image, and that image is encoded again. If encoding the image using this filter results in a lower rate-distortion cost than not using it, then the filter can be set. Otherwise, the image can be encoded without using this filter.

[0305] As an example, the optimization algorithm can be summarized as follows:

[0306] 1. Encode strips (or blocks or images);

[0307] 2. Test whether the reference image comes from at least 2% of the low-resolution image of the image region.

[0308] a. If true: Proceed to step 3.

[0309] b. If false: Proceed to step 7;

[0310] 3. Based on inter-frame prediction blocks and their corresponding ground truth values, rate-distortion optimization is performed on the enhancement.

[0311] 4. Assign the filter as an enhancement filter to the current strip (or image or block) and perform experimental encoding on the strip (or image or block);

[0312] 5. Test whether the rate-distortion performance with this filter is better than without it.

[0313] a. If true: Proceed to step 7.

[0314] b. If false: Proceed to step 6;

[0315] 6. Remove the assigned filter and re-encode the stripe without using the filter;

[0316] 7. Strip compression ends.

[0317] In summary, in some arrangements, image upsampling in RPR can be accomplished using a multi-phase interpolation filter. Upsampling is used for inter-frame prediction to rescale the image before writing it to the output stream. In these embodiments, this is modified by applying a weighted edge enhancement filter during inter-frame prediction, or by applying an edge enhancement filter during inter-frame prediction to upsample the low-resolution image after upsampling.

[0318] As discussed above, the first and second embodiments provide weighted edge enhancement filtering methods in the context of reference image resampling. These methods can be applied as enhancement filters for inter-frame prediction blocks of low-resolution images, and can be applied in scenarios where the same filter is used as an enhancement filter for inter-frame prediction blocks of low-resolution images, and as an enhancement filter for upsampled low-resolution images before their output.

[0319] Figure 8 A schematic diagram of a decoder 80 according to an embodiment is shown. Specifically, Figure 8 A schematic diagram of decoder 80 is shown, which is configured to perform any of the decoder methods discussed herein. For the sake of brevity, such detailed descriptions are omitted here.

[0320] like Figure 8 As shown, decoder 80 includes processor 81 and computer-readable medium 82. Processor 81 and computer-readable medium 82 can be connected via a bus system. The computer-readable medium is configured to store programs, instructions, or code. Processor 81 is configured to execute the programs, instructions, or code in computer-readable medium 82 to perform the operations described in the decoder method embodiment herein.

[0321] Therefore, in this embodiment, the computer-readable medium 82 is configured to store a computer program that can run in the processor 81, and the processor 81 is configured to run the computer program to perform the steps in any decoder method discussed herein.

[0322] Figure 9 A schematic diagram of an encoder 90 according to an embodiment is shown. Specifically, Figure 9 A schematic diagram of encoder 90 is shown, which is configured to perform any encoder method discussed herein. For the sake of brevity, such detailed descriptions are omitted here.

[0323] like Figure 9 As shown, encoder 90 includes processor 91 and computer-readable medium 92. Processor 91 and computer-readable medium 92 can be connected via a bus system. The computer-readable medium is configured to store programs, instructions, or code. Processor 91 is configured to execute the programs, instructions, or code in computer-readable medium 92 to perform operations in the decoder method embodiment herein.

[0324] Therefore, in this embodiment, the computer-readable medium 92 is configured to store a computer program that can run in the processor 91, and the processor 91 is configured to run the computer program to perform the steps in any decoder method discussed herein.

[0325] As discussed earlier, in the embodiments, the weighted filter is a parametric filter. Depending on the implementation, in some embodiments, the weighted filter is also content-adaptive. The two main objectives of the filter discussed in the embodiments are to sharpen blurred content and reduce ringing effects. However, other objectives of the filtering process are possible, such as reducing blockiness.

[0326] In some embodiments, the weighted filter is a locally adaptive filter. Therefore, this weighted filter can be used to handle nonlinear filtering problems. Two useful applications would be the reduction of ringing artifacts and edge sharpening. These two problems can be difficult to solve with linear filters. The weighted filter of this embodiment employs a direct concept to overcome the limitations of conventional linear filters. The idea is to apply two filters. The first filter extracts local information from the decoded image. The second filter applies a filter that depends on the output of the first filter. In some embodiments, the second filter is RD optimized based on the output of the first filter. Thus, a nonlinear filter adapted to specific image features can be signaled.

[0327] In order to place the features of the embodiments of the present invention in a further context, a discussion of the weighted filtering of these embodiments relative to existing in-loop filters will now be provided.

[0328] In-loop filtering is a component of modern video encoding and decoding systems. Typically, a set of different filters are applied sequentially. These filters can be parametric, meaning a set of filter parameters is sent that change behavior depending on the RD (Resolution-Oriented) decision. They can also be non-parametric. Furthermore, filters can be content-adaptive (i.e., filters can behave differently depending on their spatial location within the image). Sample Adaptive Offset (SAO) filters and Adaptive Loop Filters (ALF) perform local content classification and apply different operations depending on the category. Typically, the classification operation to be applied is communicated via signaling.

[0329] In detail, the in-loop ALF method optimizes a set of linear filters. Each linear filter is applied to a partition of the image. This partition is derived from the local characteristics of the image. Furthermore, partitions can be merged. This information is communicated in the bitstream via signaling. However, in this embodiment of the invention, locally weighted / parameterized filters are applied. Therefore, the need for image partitioning and optimization of multiple filters is reduced, thereby improving encoding and decoding efficiency.

[0330] In SAO, each sample (or "pixel") is assigned to a category based on the local characteristics of the image. For each category, an (intensity) offset is calculated and signaled at the encoder. Therefore, a category decision is performed, and different operations are executed for each category. However, in this embodiment of the invention, a locally weighted / parameterized filter is applied. This reduces the need for image partitioning and optimization of multiple filters, thereby improving encoding and decoding efficiency.

[0331] As discussed above, embodiments provide methods and apparatus for encoding and decoding parameters for in-loop (or post-loop) filtering schemes. This allows for consideration of transmission requirements within an adaptive parameter set and the characteristics of the encoded information, enabling efficient encoding. Embodiments employ weighted / parameterized filtering. The weighted / local parameters are calculated from the decoded video image (e.g., image patches). In some embodiments, the calculation function may be parameterized and / or notified by the encoder via signaling.

[0332] In the following text, we will refer to the locally parameterized weights as “weighted mappings”.

[0333] Although the term "weighted mapping" is used, it is for readability purposes and is not intended to be restrictive. For example, in some embodiments, a weighted mapping may include local parameters or vectors that include both parameter and weight values. In addition to the weighted calculation, a filter is applied. In some embodiments, the filter may be parameterized with respect to the weighted mapping, or it may be locally weighted according to the weighted mapping.

[0334] As discussed above, the embodiments relate to methods for optimizing weighted filters (e.g., weighted edge enhancement filters). These methods can also be used to optimize other types of filters designed to enhance a reference image.

[0335] These optimization methods can be used to optimize weighted filters for applications as in-loop filters. These optimization techniques are advantageous if the filter is applied solely as an in-loop filter to inter-prediction blocks derived from enhancing low-resolution images, compared to optimizing a separate filter for each low-resolution image. The first application makes sense in reference image resampling / adaptive resolution encoding / decoding scenarios, where filters are applied to improve the quality of both the low-resolution and reference images before presenting the low-resolution image to the viewer. However, post-filtering the upsampled image may not be desirable in some applications where the upsampling operation is defined by the viewer as needed. That is, applying both is not always desirable in all applications.

[0336] If upsampling filtering is not applied, the optimization objective changes. Therefore, the optimization process and the filters optimized in this way will differ. Furthermore, it makes sense to assign the optimized filters to high-resolution images in the encoded video sequence, rather than low-resolution images. Typically, some images are encoded at high resolution. Then, based on bandwidth and / or content, the resolution is changed to a lower encoding / decoding resolution. Sometimes, the resolution may switch back to high resolution. In this case, the option exists where the low-resolution image is referenced to the high-resolution image.

[0337] In this scenario, enhancement of the referenced low-resolution blocks can be used. In such cases, a first high-resolution image references a region of the low-resolution image. However, not all reference images in the reference set are used as references, and not all portions of the referenced images are used for prediction. Therefore, a filter optimized for the high-resolution image is proposed, making it well-suited to all that the high-resolution image might / expect to reference. The motivation for using this optimization and filter allocation scheme is to achieve more efficient filter coding. Typically, optimizing a separate filter for each low-resolution image is inefficient because not all images are used for inter-frame prediction, and even when used, sometimes only a small portion of the image is referenced. In such scenarios, optimizing a separate filter is prohibitively expensive. In a given scenario, when blocks in a low-resolution image are referenced by a high-resolution image, filters are used to enhance these blocks. Typically, the frequency with which different images are referenced depends more or less on motion, temporal distance, and other factors in the video sequence. Furthermore, not all regions of an image are referenced with equal probability. Therefore, it makes sense to optimize the filter in a way that takes these statistical biases into account. This paper proposes a method to achieve this.

[0338] Specifically, the described embodiments provide a method for estimating filter parameters for an in-loop filtering scheme, including deriving filter coefficients based on a set of potentially overlapping blocks or partitions. These blocks may originate from the current image and / or a set of temporally spaced images. The filter may be applied to the interpolation results of sub-blocks or to partitions of the assigned image for enhancement. The described embodiments relate to optimizing filters applied as in-loop filters (i.e., the output of which can be multiplexed by a prediction system to encode subsequent images).

[0339] The described weighted filter is applied to inter-frame prediction to enhance the output of the prediction based on criteria such as coded information or any results of pre-analysis. These criteria can be derived from the video content or informed by signaling in the bitstream. Furthermore, the weighted filter can also be applied to the entire set of reference frames to generate an enhanced reference. Aside from the rounding issue, this will produce equivalent results.

[0340] As discussed above, the embodiments provide an in-loop filtering method applicable within the coding loop of a video compression system. This in-loop filter employs a function for calculating a local parameter map and a filtering function. The weighted mapping function can calculate a weighted map using the input image / block, coding information, and / or parameters communicated via signaling. The filtering function can calculate a filtered image using the input image / block, coding information, parameters communicated via signaling, and / or the parameter map. Optimization can be performed based on potentially overlapping blocks or partitions of the image. Optimization can use the ground truth and predicted signals of the current image and all reference images to find an optimized filter for enhancing the coded signal.

[0341] In some embodiments, the weighted filters are optimized based on a set of reference images and their ground truth representations. Thus, by performing weighted joint optimization to minimize the error between the reference images and their corresponding ground truth representations, one or more filters are optimized such that applying these filters to the set of reference images minimizes the weighted error.

[0342] In some embodiments, the error of the reference frame is weighted based on its temporal distance from the current frame.

[0343] In some embodiments, the error of the reference frame is weighted based on local image features or knowledge derived from encoding this frame or other frames of the video.

[0344] In some embodiments, the image is optimized based on the ground truth error between the predicted signal and the current image.

[0345] In some embodiments, the prediction signal is derived by experimentally encoding the image or strip.

[0346] In some embodiments, the predicted signal is derived by pre-analyzing a reference image, the current image, their corresponding real signals, and coded information.

[0347] In some embodiments, the pre-analysis or experimental coding and filter optimization steps are performed iteratively until a certain stopping criterion is met.

[0348] In some embodiments, the filtering and weighted loss of the block in the reference frame and the predicted signal are jointly optimized.

[0349] In some embodiments, the method is integrated into the codec loop as an additional processing step, as an alternative to an existing loop filter, or integrated into an existing loop filter.

[0350] In some embodiments, the method performs RD optimization based on the signaling rate and distortion estimated after the method is applied.

[0351] In some embodiments, the method is applied to the luminance channel, the chroma channel, or both the luminance and chroma channels. The processed channel can be preset, signaled in the bitstream, or inferred from the content.

[0352] Although some embodiments are described as being applied to images, the embodiments are not limited in this respect. For example, in some embodiments, the method is applied to one or more partitions of an image. Partitions may be defined by block partitions notified by signaling, possible region partitioning criteria notified by signaling, or binarized weighted mapping functions.

[0353] In some embodiments, multiple filters are applied to the same image partition.

[0354] In some embodiments, the method is used to address problems of ringing effects, blurring, or blockiness.

[0355] In some embodiments, the method involves applying a weighted mapping function that outputs a scalar-weighted mapping, where the scalars are binary, integer, or floating-point numbers.

[0356] In some embodiments, the method involves applying a weighted mapping function that outputs an n-dimensional weighted mapping, where each element is binary, integer, or floating-point.

[0357] In some embodiments, weighted mapping information is calculated for one or more channels of the reconstructed image, while using information from one or more channels of the reconstructed image as input.

[0358] In some embodiments, a set of possible parameterized weighted mapping functions is predefined, and the encoder notifies the use of the weighted mapping and weighted mapping functions via signaling.

[0359] In some embodiments, one or more filters are applied for each weighted mapping notified by signaling. The parameters of the filtering function and the filters may be notified by signaling in the bitstream, are predefined, or inferred from the content or encoding information of the video sequence.

[0360] In some embodiments, the method involves applying a linear filter as a filter in a filtering function. The shape of the filter can be indicated in the bitstream or predefined.

[0361] In some embodiments, the linear filter is optimized using least squares optimization or RD optimization.

[0362] In some embodiments, a parameterized linear filter is applied. Parameters can be used to generate the corresponding linear filter.

[0363] In some embodiments, the parameterized linear filter is optimized at the output for minimum error using an RD (Regression-Oriented Optimization) method. The optimal filter can be derived through least-squares optimization, iterative search, or exhaustive search.

[0364] In some embodiments, a parameterized or non-parameterized nonlinear filter is applied to the filtering function.

[0365] In some embodiments, a bilateral filter is applied in the filtering function.

[0366] In some embodiments, a combination of the described filtering methods is applied, wherein the filtering method used at each location is indicated by signaling or by a weighted mapping.

[0367] In some embodiments, the encoder optimizes the parameterized weighted mapping and the filtering function together.

[0368] In some embodiments, one or more filters are applied to partitions of an image based on block partitions notified by signaling.

[0369] In some embodiments, one or more filters are applied to partitions of an image based on derived region partitioning criteria.

[0370] In some embodiments, as an alternative to linear filters, weighted filters are integrated into an adaptive loop filter (ALF) and applied to partitions derived by optimization of the ALF.

[0371] In some embodiments, the encoder encodes the filter and weighted mapping computation parameters using quantization, prediction, or entropy coding schemes.

[0372] In some embodiments, the weighted filter is applied as a post-filter.

[0373] Embodiments of the present invention may also provide a computer-readable medium having computer-executable instructions to cause one or more processors of a computing device to perform the methods of any embodiment of the present invention.

[0374] Examples of computer-readable media include volatile and non-volatile media, removable and non-removable media, and include, but are not limited to: solid-state storage; removable disks; hard disk drives; magnetic media; and optical disks. Generally, computer-readable media includes any type of media suitable for storing, encoding, or carrying a set of instructions executable by one or more computers to perform one or more of any of the processes or features described herein.

[0375] It should be understood that the functionality of each component discussed can be combined in a variety of ways besides those described above. For example, in some embodiments, the functionality of multiple discussed devices can be incorporated into a single device. In other embodiments, the functionality of at least one discussed device can be divided among multiple separate (or distributed) devices.

[0376] Conditional language (e.g., “may / can”) is typically used to indicate the use of features / steps in a particular embodiment, but alternative embodiments may include alternative features or omit these features entirely.

[0377] Furthermore, the method steps are not limited to the specific order described, and it should be understood that these steps can be combined in any other suitable order. In some embodiments, this may result in some method steps being executed in parallel. Additionally, in some embodiments, certain method steps may be omitted entirely.

[0378] Although certain embodiments have been discussed, it should be understood that these embodiments are used to illustrate the overall teachings of the invention, and various modifications can be made without departing from the scope of the invention. The scope of the invention should be interpreted in accordance with the appended claims and any equivalents thereof.

[0379] Many further variations and modifications will be suggested to those skilled in the art on their own. These embodiments are given by way of example only when referenced to the foregoing illustrative embodiments and are not intended to limit the scope of the invention, which is determined by the appended claims.

Claims

1. A method for processing video data, performed by an encoder, the method comprising: Obtain multiple raw images from the raw video data; Acquire multiple reference images, each of which corresponds to an original image in the original video data, wherein the resolution of the multiple reference images is lower than the resolution of the multiple original images in the original video data; Upsampling is performed on the plurality of reference images to obtain a plurality of upsampled reference images; The weighted filter is obtained in the following way to reduce the overall error between the multiple upsampled reference images and the corresponding original images: A weighted mapping function is used to determine a weighted mapping for each reference image, the weighted mapping including multiple weights mapped to corresponding spatial locations of the plurality of upsampled reference images, wherein the plurality of upsampled reference images are used as inputs to the weighted mapping function; A filter is determined, and the filter is applied to the plurality of upsampled reference images through a corresponding weighted mapping to obtain a filtered upsampled reference image, such that the filter is applied to different spatial locations of the plurality of upsampled reference images with different weights; Based on multiple reference blocks in one or more of the multiple upsampled reference images, inter-frame prediction is performed on multiple blocks of the first image; and The video data encoding information is encoded into the bitstream, the encoding information including information about the weighted mapping function and / or the filter to be used at the decoder. The method further includes: Before performing the inter-frame prediction, the weighted filter is applied to the plurality of upsampled reference images, or During the inter-frame prediction, the weighted filter is applied to the upsampled reference blocks of the plurality of reference images.

2. The method according to claim 1, wherein, Obtaining the weighted filter to reduce the overall error between the plurality of upsampled reference images and the corresponding original images includes: Assign corresponding importance weighting factors to the plurality of reference images and / or regions of the plurality of reference images; The overall error between the plurality of upsampled reference images and the corresponding original images is determined by weighting the error of the upsampled reference images based on the importance weighting factor of each upsampled reference image.

3. The method according to claim 2, wherein, Assigning corresponding importance weighting factors to the plurality of reference images, including: A higher importance weighting factor is assigned to reference images that are closer in time to the first image.

4. The method according to claim 2 or claim 3, wherein, Assigning corresponding importance weighting factors to the plurality of reference images, including: Importance weighting factors are assigned based on the quality of each reference image.

5. The method according to any one of claims 2 to 4, wherein, Assigning corresponding importance weighting factors to the plurality of reference images, including: Importance weighting factors are assigned based on local image features or historical information in the encoding of the plurality of reference images or other images of the video data.

6. The method according to any one of claims 1 to 5, further comprising: Add the acquired weighted filter to the stored set of reference filters.

7. The method according to any one of claims 1 to 6, wherein, Assigning corresponding importance weighting factors to the plurality of reference images and / or regions of the plurality of reference images, including: Perform test coding on the first image; Identify which reference images and / or blocks in the reference images are used in the trial coding of inter-frame prediction for one or more blocks of the first image; and Assign high importance weighting factors to the identified reference images and / or blocks.

8. The method according to any one of claims 1 to 7, wherein, The encoded information includes weighted mapping function parameters transmitted via signaling, which are configured to allow the decoder to obtain the weighted mapping using the weighted mapping function in the following manner: The parameters of the weighted mapping function transmitted via signaling are applied as parameters of the weighted mapping function; and The multiple reference images are provided as input to the weighted mapping function.

9. The method according to any one of claims 1 to 8, wherein, The encoded information includes filter function parameters transmitted via signaling, which are configured to allow the decoder to obtain the filter in the following manner: The filter function parameters transmitted via signaling are applied as parameters of the filter.

10. The method according to any one of claims 1 to 9, wherein, As a step within the encoding / decoding loop or as a step after the loop, the weighted mapping and the filter are configured to be applied to the first image to obtain a filtered first image.

11. The method according to claim 10, wherein, The encoding / decoding loop is an H.266 / VVC encoding / decoding loop.

12. The method according to claim 10 or 11, wherein, The weighted mapping and the filter are configured to be integrated into an adaptive loop filter and applied to the plurality of reference images or a derived partition of the first image to obtain a filtered image.

13. The method according to any one of claims 1 to 12, wherein, After applying the weighted mapping and the filter to the plurality of reference images or the first image, the bitstream is rate-distortion (RD) optimized based on the estimated signaling rate and distortion.

14. The method according to any one of claims 1 to 13, wherein, The plurality of reference images and / or the first image include a luminance channel, a chrominance channel, or both a luminance channel and a chrominance channel; The weighted mapping and the filter are configured to be applied to the luminance channel, the chrominance channel, or both the luminance channel and the chrominance channel.

15. The method according to claim 14, wherein, The weighted mapping and the filter to be applied to which of the luminance and chrominance channels are predetermined, notified by signaling in the encoded information, or configured to be inferred from the image content.

16. The method according to any one of claims 1 to 15, further comprising: The plurality of reference images and the first image are divided into multiple partitions. The weighted mapping and the filter are configured to be applied to one or more partitions of the plurality of reference images or the first image. The partition is notified by signaling in the encoded information.

17. The method according to claim 16, wherein, The partitions notified by signaling in the encoded information include block partitions notified by signaling in the encoded information, region partitioning criteria notified by signaling, or binarized weighted mapping functions.

18. The method according to claim 16 or 17, further comprising: Determine the multiple filters to be applied to the same image partition.

19. The method according to any one of claims 1 to 18, wherein, The filter is configured to resolve ringing, blurring, and / or blockiness issues in the image.

20. The method according to any one of claims 1 to 19, wherein, Determining the weighted mapping using a weighted mapping function includes: Apply a weighted mapping function to the output scalar weighted mapping, where the scalars in the scalar weighted mapping are binary, integer, or floating-point.

21. The method according to any one of claims 1 to 20, wherein, Determining the weighted mapping using a weighted mapping function includes: The application outputs a weighted mapping function for a multidimensional weighted mapping, where each element in the multidimensional weighted mapping is binary, integer, or floating-point.

22. The method according to any one of claims 1 to 21, wherein, The weighted mapping information of one or more channels of the plurality of reference images is calculated using information from one or more channels of the plurality of reference images as input.

23. The method according to any one of claims 1 to 22, wherein, The set of weighted mapping functions is predefined; The encoded information is used to notify the weighted mapping function to be used via signaling.

24. The method according to any one of claims 23, wherein, The weighted mapping function is parameterized.

25. The method according to any one of claims 1 to 24, wherein, The encoded information is communicated to multiple weighted mapping functions via signaling. The step of obtaining the weighted mapping using the weighted mapping function includes: determining multiple weighted mappings using the multiple weighted mapping functions; One or more filters are configured to be applied to each weighted mapping of signaling notifications.

26. The method according to claim 25, wherein, The filter function and parameters of the filter are notified by signaling in the encoded information, are predefined, are configured to be inferred from the video content, or are configured to be inferred from the encoded information.

27. The method according to any one of claims 1 to 26, wherein, The filter is a linear filter, and the shape of the filter is indicated in the bitstream or is predefined.

28. The method according to claim 27, wherein, The linear filter is optimized through least squares optimization or RD optimization.

29. The method according to any one of claims 27 to 28, wherein, The linear filter is a parameterized linear filter.

30. The method according to claim 29, wherein, The parameterized linear filter is optimized at the output for minimum error, which is derived through least squares optimization, iterative search, or exhaustive search.

31. The method according to any one of claims 1 to 26, wherein, The filter is a bilateral filter.

32. The method according to any one of claims 1 to 31, wherein, Obtaining the filters includes: obtaining multiple filters. Each of the plurality of filters is configured to be applied at a location in the bitstream that is signaled or at a location indicated in the weighted mapping.

33. The method according to any one of claims 1 to 32, wherein, The parameterized weighted mapping is optimized together with the filtering function.

34. The method according to any one of claims 1 to 32, wherein, One or more filters are configured to be applied to partitions of the plurality of reference images or the first image based on block partitions notified by signaling in the encoded information.

35. The method according to any one of claims 1 to 34, wherein, One or more filters are configured to be applied to partitions of the plurality of reference images or the first image based on derived region partitioning criteria.

36. The method according to any one of claims 1 to 35, wherein, The filter and weighted mapping calculation parameters are encoded using quantization, prediction, and / or entropy encoding / decoding schemes.

37. A computer-readable medium comprising computer-executable instructions stored thereon, wherein, When executed by a computing device, the computer-executable instructions cause the computing device to perform the method according to any one of claims 1 to 36.

38. An encoder, comprising: One or more processors; as well as A computer-readable medium includes computer-executable instructions stored on the computer-readable medium, wherein, when executed by the one or more processors, the computer-executable instructions cause the one or more processors to perform the method according to any one of claims 1 to 36.

39. A method for processing video data, performed by a decoder, the method comprising: Decode the bitstream to obtain video data and encoding information; Multiple reference images are obtained from the video data; Upsampling is performed on the plurality of reference images to obtain a plurality of upsampled reference images; The weighted filter is obtained in the following way to reduce the overall error between the multiple upsampled reference images and the corresponding original images: A weighted mapping function is used to determine a weighted mapping for each reference image, the weighted mapping including multiple weights mapped to corresponding spatial locations of the plurality of upsampled reference images, wherein the plurality of upsampled reference images are used as inputs to the weighted mapping function; A filter is determined, which is applied to the plurality of upsampled reference images through a corresponding weighted mapping to obtain a filtered upsampled reference image, such that the filter is applied with different weights to different spatial locations in the plurality of upsampled reference images; and Based on multiple reference blocks in one or more upsampled reference images, inter-frame prediction is performed on multiple blocks of the first image. The method further includes: Before performing the inter-frame prediction, the weighted filter is applied to the plurality of upsampled reference images, or During the inter-frame prediction, the weighted filter is applied to the upsampled reference blocks of the plurality of reference images.

40. The method of claim 39, further comprising: Add the acquired weighted filter to the stored set of reference filters.

41. The method according to claim 39, wherein, The encoded information includes weighted mapping function parameters transmitted via signaling; Specifically, determining the weighted mapping using the weighted mapping function includes: The parameters of the weighted mapping function transmitted via signaling are applied as parameters of the weighted mapping function; and The plurality of reference images or the first image are provided as input to the weighted mapping function.

42. The method according to claim 40 or claim 41, wherein, The encoded information includes filter function parameters transmitted via signaling; The process of obtaining the filter includes: The filter function parameters transmitted via signaling are applied as parameters of the filter.

43. The method according to any one of claims 40 to 42, wherein, Applying the weighted mapping and the filter to the image to obtain a filtered reference image or a filtered first image occurs within the encoding / decoding loop or as a step after the loop.

44. The method according to claim 43, wherein, The encoding / decoding loop is an H.266 / VVC encoding / decoding loop.

45. The method according to claim 43 or 44, wherein, The steps of applying the weighted mapping and the filter to the plurality of reference images or the first image to obtain the filtered image are integrated into an adaptive loop filter and applied to the derived partition.

46. ​​The method according to any one of claims 39 to 45, wherein, After applying the weighted mapping and the filter to the plurality of reference images or the first image, the bitstream is rate-distortion (RD) optimized based on the estimated signaling rate and distortion.

47. The method according to any one of claims 39 to 46, wherein, The plurality of reference images and / or the images include a luminance channel, a chrominance channel, or both a luminance channel and a chrominance channel; The weighted mapping and the filter are applied to the luminance channel, the chrominance channel, or both the luminance channel and the chrominance channel.

48. The method according to claim 47, wherein, The weighted mapping and the filter to be applied to which of the luminance and chroma channels are predetermined, notified by signaling in the bitstream, or inferred from the image content.

49. The method according to any one of claims 39 to 48, further comprising: The plurality of reference images and the first image are divided into multiple partitions. The weighted mapping and the filter are applied to one or more partitions of the plurality of reference images or the first image. The partition is notified by signaling in the encoded information.

50. The method according to claim 49, wherein, The partitions notified by signaling in the encoded information include block partitions notified by signaling in the encoded information, region partitioning criteria notified by signaling, or binarized weighted mapping functions.

51. The method according to claim 49 or 50, further comprising: Apply multiple filters to the same image partition.

52. The method according to any one of claims 39 to 51, wherein, The filter is configured to resolve ringing, blurring, and / or blockiness issues in the image.

53. The method according to any one of claims 39 to 52, wherein, Determining the weighted mapping using a weighted mapping function includes: Apply a weighted mapping function to the output scalar weighted mapping, where the scalars in the scalar weighted mapping are binary, integer, or floating-point.

54. The method according to any one of claims 39 to 53, wherein, Determining the weighted mapping using a weighted mapping function includes: The application outputs a weighted mapping function for a multidimensional weighted mapping, where each element in the multidimensional weighted mapping is binary, integer, or floating-point.

55. The method according to any one of claims 39 to 54, wherein, The weighted mapping information of one or more channels of the plurality of reference images is calculated using information from one or more channels of the plurality of reference images as input.

56. The method according to any one of claims 39 to 55, wherein, The set of weighted mapping functions is predefined; The encoded information is used to notify the weighted mapping function to be used via signaling.

57. The method according to claim 56, wherein, The weighted mapping function is parameterized.

58. The method according to any one of claims 39 to 57, wherein, The encoded information is communicated to multiple weighted mapping functions via signaling. Wherein, determining the weighted mapping using the weighted mapping function includes: determining multiple weighted mappings using the multiple weighted mapping functions; The application of the weighted mapping and the filter includes: applying one or more filters for each weighted mapping notified by signaling.

59. The method according to claim 58, wherein, The filter function and parameters of the filter are notified by signaling in the encoded information, are predefined, inferred from the video content, or inferred from the encoded information.

60. The method according to any one of claims 39 to 59, wherein, The filter is a linear filter, and the shape of the filter is indicated in the bitstream or is predefined.

61. The method according to claim 60, wherein, The linear filter is optimized through least squares optimization or RD optimization.

62. The method according to any one of claims 58 to 61, wherein, The linear filter is a parameterized linear filter.

63. The method according to claim 62, wherein, The parameterized linear filter is optimized at the output for minimum error, which is derived through least squares optimization, iterative search, or exhaustive search.

64. The method according to any one of claims 39 to 59, wherein, The filter is a bilateral filter.

65. The method according to any one of claims 39 to 64, wherein, Determining the filter includes: determining multiple filters. Applying the weighted mapping and the filter to the plurality of reference images or the first image includes applying each of the plurality of filters at a location notified by signaling in the bitstream or at a location indicated in the weighted mapping.

66. The method according to any one of claims 39 to 65, wherein, The parameterized weighted mapping is optimized together with the filtering function.

67. The method according to any one of claims 1 to 66, wherein, Applying the weighted mapping and the filter to the image includes: applying one or more filters to a partition of the plurality of reference images or the first image based on block partitions notified by signaling in the encoded information.

68. The method according to any one of claims 39 to 67, wherein, Applying the weighted mapping and the filter to the plurality of reference images or the first image includes: applying one or more filters to partitions of the plurality of reference images or the first image based on derived region partitioning criteria.

69. The method according to any one of claims 39 to 68, wherein, The filter and weighted mapping calculation parameters are encoded using quantization, prediction, and / or entropy encoding / decoding schemes.

70. A computer-readable medium comprising computer-executable instructions stored thereon, wherein, When executed by a computing device, the computer-executable instructions cause the computing device to perform the method according to any one of claims 1 to 69.

71. A decoder, comprising: One or more processors; as well as A computer-readable medium includes computer-executable instructions stored on the computer-readable medium, wherein, when executed by the one or more processors, the computer-executable instructions cause the one or more processors to perform the method according to any one of claims 1 to 70.

72. A method for processing video data, performed by an encoder, the method comprising: Obtain raw video data; At least a portion of the original video data is subjected to experimental coding to obtain experimentally coded video data; A test first image is obtained based on the test-encoded video data, and inter-frame prediction is performed on multiple blocks of the test first image based on multiple reference blocks in one or more reference images, wherein the resolution of the one or more reference images is lower than the resolution of the test first image; as well as The weighted filters for the multiple inter-frame prediction blocks to be applied to the test first image are obtained in the following manner to reduce the error between the test first image and the corresponding original first image in the original video data: A weighted mapping is determined using a weighted mapping function, the weighted mapping comprising multiple weights mapped to corresponding spatial locations in each of the plurality of inter-frame prediction blocks of the first experimental image, wherein the plurality of inter-frame prediction blocks of the first experimental image are used as inputs to the weighted mapping function; A filter is determined and applied to the plurality of inter-frame prediction blocks of the test first image via the weighted mapping, wherein the filter is configured to be applied to the plurality of inter-frame prediction blocks of the test first image via the weighted mapping to obtain a filtered first image, such that the filter is applied with different weights to different spatial locations in each of the plurality of inter-frame prediction blocks of the test first image.

73. The method of claim 72, further comprising: The video data and encoding information are encoded, the encoding information including information about the weighted mapping function and / or the filter to be used at the decoder.

74. The method of claim 73, further comprising: Determine whether the rate-distortion performance of encoding the video data and the encoded information is better than the rate-distortion performance of the experimental encoding; Based on the fact that the rate-distortion performance of encoding the video data and the encoded information is better than that of the experimental encoding, the encoded video data and encoded information are included in the bitstream transmitted to the decoder. Based on the fact that the rate-distortion performance of encoding the video data and the encoded information is not better than the rate-distortion performance of the experimental encoding, the experimentally encoded video data is included in the bitstream transmitted to the decoder.

75. The method according to any one of claims 72 to 74, further comprising: Add the acquired weighted filter to the stored set of reference filters.

76. The method according to any one of claims 72 to 75, wherein, The experimental coding also includes deriving experimental prediction signals by pre-analyzing the one or more reference images, the first image, the corresponding original image in the original video data, and / or coding information.

77. The method of claim 76, further comprising: The steps of pre-analysis or experimental coding and obtaining weighted filters are performed iteratively until the stopping criterion is met.

78. A computer-readable medium comprising computer-executable instructions stored thereon, wherein, When executed by a computing device, the computer-executable instructions cause the computing device to perform the method according to any one of claims 72 to 77.

79. An encoder, comprising: One or more processors; as well as A computer-readable medium includes computer-executable instructions stored on the computer-readable medium, wherein, when executed by the one or more processors, the computer-executable instructions cause the one or more processors to perform the method according to any one of claims 72 to 77.

80. A method for processing video data, performed by a decoder, the method comprising: Decode the bitstream to obtain video data and encoding information; Acquire one or more reference images; Inter-frame prediction is performed on multiple blocks of the first image based on multiple reference blocks in the one or more reference images, wherein the resolution of the one or more reference images is lower than the resolution of the first image; Based on the encoded information, a weighted filter for the plurality of inter-frame prediction blocks to be applied to the first image is obtained in the following manner to reduce the error between the first image and the corresponding original first image: A weighted mapping function is used to determine a weighted mapping, which includes multiple weights mapped to corresponding spatial locations in each of the plurality of inter-frame prediction blocks of the first image, wherein the plurality of inter-frame prediction blocks of the first image are used as inputs to the weighted mapping function. A filter is determined, which is applied to the plurality of inter-frame prediction blocks of the first image via the weighted mapping, wherein the filter is configured to be applied to the plurality of inter-frame prediction blocks of the first image via the weighted mapping to obtain a filtered first image, such that the filter is applied with different weights to different spatial locations in each of the plurality of inter-frame prediction blocks of the first image; and The weighted filter is applied to the plurality of inter-frame prediction blocks of the first image.

81. The method of claim 80, further comprising: Add the acquired weighted filter to the stored set of reference filters.

Images