Weighted filtering for image enhancement in video coding

Abstract

A method for processing video data performed by a decoder is provided. The method includes decoding a bitstream to obtain video data and encoding information; obtaining an image based on the video data; determining a weighted graph using a weighted graph function, the weighted graph including multiple weights mapped to corresponding spatial locations in the image, wherein the image and / or encoding information are used as input to the weighted graph function; determining a filter; and applying the weighted graph and the filter to the image to obtain a filtered image.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more particularly to the subject of video processing and video encoding and decoding, and more specifically to methods, decoders, encoders and computer-readable media for image enhancement in video encoding and decoding. Background Technology

[0002] Current video codec schemes such as H.265 / HEVC and H.266 / VVC apply so-called in-loop filters to the encoded video content within the codec loop. These filters are designed to hide certain types of artifacts, such as blockiness, or to improve overall quality. This processing step not only improves the output image quality. Specifically, the filtered image is often then used in most codec settings to predict the next image (e.g., inter-frame prediction can be performed). Therefore, the quality of the subsequently encoded image can also be improved.

[0003] Filtering video to improve image quality requires the existence of statistical correlations that can be utilized by the filtering system. Generally, applying in-loop filtering is meaningful if the quality improvement achieved by the pass-through exceeds the signaling cost at the rate distortion (RD) point. Furthermore, computation time needs to be acceptable.

[0004] In many video codec systems, a series of filters are applied to handle different types of codec errors. For example, there are deblocking filters that can be applied at block boundaries to reduce block artifacts. Next, there are sample adaptive offset (SAO) filters, which are primarily designed to reduce ringing or blur artifacts. Finally, adaptive loop filters (ALFs) can be used for objective quality enhancement. It should be noted that this is only a small excerpt, intended as an overview of the different applications and types of loop filters.

[0005] Most of these filters only handle a limited range of encoding / decoding errors. Furthermore, no filters specifically addressing blurred image content are implemented in VVC / H.266. However, blurring does occur due to the quantization or removal of high-frequency components. Typically, linear filters are insufficient to recover blurred content due to overshoot and ringing issues. Additionally, noise amplification is a problem. Linear filtering methods like adaptive loop filters attempt to address this by introducing a class of filters that apply different methods. However, this increases encoding / decoding costs. Summary of the Invention

[0006] Embodiments of this application provide a method, decoder, encoder, and computer-readable medium for video encoding and decoding using weighted filters, which overcomes problems associated with conventional arrangements.

[0007] According to a first aspect, a method for processing video data performed by a decoder, the method comprising: decoding a bitstream to obtain video data and encoding information; obtaining an image based on the video data; determining a weighted graph using a weighted graph function, the weighted graph including multiple weights mapped to corresponding spatial locations of the image, wherein the image and / or encoding information are used as input to the weighted graph function; determining a filter; and applying the weighted graph and the filter to the image to obtain a filtered image, such that the filter is applied with different weights to different spatial locations of the image.

[0008] In some embodiments, the encoded information includes weighted graph function parameters notified by a signal, and determining the weighted graph using the weighted graph function includes: applying the weighted graph function parameters notified by a signal as parameters of the weighted graph function; and providing an image as input to the weighted graph function.

[0009] In some embodiments, the encoded information includes filter function parameters notified by a signal, and determining the filter includes applying the filter function parameters notified by the signal as parameters of the filter.

[0010] In some embodiments, applying a weighted graph and a filter to an image to obtain a filtered image occurs either within the coding loop or as a post-loop step.

[0011] In some embodiments, the coding loop is an H.266 / VVC coding loop.

[0012] In some embodiments, the step of applying a weighted graph and a filter to an image to obtain a filtered image is integrated into an adaptive loop filter and applied to the derived partitions.

[0013] In some embodiments, the bitstream is rate-distortion (RD) optimized based on estimated signaling rate and distortion after a weighted graph and filters are applied to the image.

[0014] In some embodiments, the image includes a luminance channel, a chrominance channel, or both a luminance channel and a chrominance channel, wherein a weighted graph and a filter are applied to the luminance channel, the chrominance channel, or both a luminance channel and a chrominance channel.

[0015] In some embodiments, the weighted graph and filter will be applied to which of the luminance and chroma channels is predetermined, indicated by a signal in the bitstream, or inferred from the content of the image.

[0016] In some embodiments, the method further includes segmenting the image into multiple partitions, wherein a weighted graph and a filter are applied to one or more partitions of the image, and wherein the partitions are signaled in the encoded information.

[0017] In some embodiments, the partition is signaled in the encoded information, including block segmentation signaled in the encoded information, region segmentation criteria signaled, or binarized weighted graph function.

[0018] In some embodiments, the method further includes applying multiple filters to the same image partition.

[0019] In some embodiments, the filter is configured to resolve issues of ringing artifacts, blurring, and / or block artifacts in an image.

[0020] In some embodiments, determining a weighted graph using a weighted graph function includes applying a weighted graph function to the output scalar weighted graph, wherein the scalar is binary, integer, or floating-point.

[0021] In some embodiments, determining a weighted graph using a weighted graph function includes applying a weighted graph function that outputs a multidimensional weighted graph, in which each element is binary, integer, or floating-point.

[0022] In some embodiments, the weighted graph information of one or more channels of the obtained image is calculated using information from one or more channels of the obtained image as input.

[0023] In some embodiments, a set of weighted graph functions is predefined, and the encoded information signals the weighted graph functions to be used.

[0024] In some embodiments, the weighted graph function is parameterized.

[0025] In some embodiments, encoded information is signaled to a plurality of weighted graph functions, wherein determining a weighted graph using the weighted graph functions includes determining a plurality of weighted graphs using the plurality of weighted graph functions, and wherein applying weighted graphs and filters includes applying one or more filters for each weighted graph signaled.

[0026] In some embodiments, the filter function and parameters are indicated by signals in the encoded information, predefined, inferred from the content of the video, or inferred from the encoded information.

[0027] In some embodiments, the filter is a linear filter, and the shape of the filter is indicated or predefined in the bitstream.

[0028] In some embodiments, the linear filter is optimized through least squares optimization or RD optimization.

[0029] In some embodiments, the linear filter is a parameterized linear filter.

[0030] In some embodiments, the parameterized linear filter is RD optimized relative to the minimum error at the output derived through least squares optimization, iterative search, or exhaustive search.

[0031] In some embodiments, the filter is a bilateral filter.

[0032] In some embodiments, determining a filter includes determining a plurality of filters, wherein applying a weighted graph and filters to an image includes signaling in the bitstream or applying each of the plurality of filters at a location indicated in the weighted graph.

[0033] In some embodiments, the parameterized weighted graph is optimized together with the filtering function.

[0034] In some embodiments, applying weighted graphs and filters to an image includes applying one or more filters to partitions of the image based on block segmentation notified by signals in the encoded information.

[0035] In some embodiments, applying weighted graphs and filters to an image includes applying one or more filters to partitions of the image based on derived region segmentation criteria.

[0036] In some embodiments, the filter and weighted graph computation parameters are encoded using quantization, prediction, and / or entropy coding schemes.

[0037] According to a second aspect, a computer-readable medium is provided, comprising computer-executable instructions stored thereon, which, when executed by a computing device, cause the computing device to perform any of the methods of the first aspect.

[0038] According to a third aspect, a decoder is provided, including one or more processors; and a computer-readable medium including computer-executable instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform any of the methods in the first aspect.

[0039] According to a fourth aspect, a method for processing video data performed by an encoder is provided, the method comprising: obtaining raw video data; compressing the raw video data into compressed video data; obtaining an image based on the compressed video data; determining a weighted graph using a weighted graph function, the weighted graph including multiple weights mapped to corresponding spatial locations of the image, wherein the image is used as input to the weighted graph function; determining a filter to be applied to the image using the weighted graph, wherein the filter is configured to be applied to the image with the weighted graph to obtain a filtered image, such that the filter is applied with different weights to different spatial locations of the image at a decoder; and encoding the compressed video data and encoding information into a bitstream, the encoding information including information about the weighted graph function and / or the filter to be used at the decoder.

[0040] In some embodiments, the encoded information includes signal-notified weighted graph function parameters, which are configured to enable the decoder to determine the weighted graph by: applying the signal-notified weighted graph function parameters as parameters of the weighted graph function; and providing an image as input to the weighted graph function.

[0041] In some embodiments, the encoded information includes signal-informed filter function parameters, which are configured such that the decoder can determine the filter by applying the signal-informed filter function parameters as parameters of the filter.

[0042] In some embodiments, the weighted graph and filter are configured to be applied to the image as a step within the coding loop or as a post-loop step to obtain a filtered image.

[0043] In some embodiments, the coding loop is an H.266 / VVC coding loop.

[0044] In some embodiments, the weighted graph and filter are configured to be integrated into an adaptive loop filter and applied to the derived partition of the image to obtain a filtered image.

[0045] In some embodiments, the bitstream is rate-distortion (RD) optimized based on estimated signaling rate and distortion after a weighted graph and filters are applied to the image.

[0046] In some embodiments, the image includes a luminance channel, a chrominance channel, or both a luminance channel and a chrominance channel, and a weighted graph and a filter are configured to be applied to the luminance channel, the chrominance channel, or both a luminance channel and a chrominance channel.

[0047] In some embodiments, the weighted graph and filters will be applied to which of the luminance and chrominance channels is predetermined, indicated by a signal in the encoded information, or configured to be inferred from the content of the image.

[0048] In some embodiments, the method further includes segmenting the image into multiple partitions, wherein weighted graphs and filters are configured to be applied to one or more partitions of the image, wherein the partitions are signaled in encoded information.

[0049] In some embodiments, the partition is signaled in the encoded information, including block segmentation signaled in the encoded information, region segmentation criteria signaled, or binarized weighted graph function.

[0050] In some embodiments, the method further includes determining multiple filters to be applied to the same image partition.

[0051] In some embodiments, the filter is configured to resolve issues of ringing artifacts, blurring, and / or block artifacts in an image.

[0052] In some embodiments, determining a weighted graph using a weighted graph function includes applying a weighted graph function to the output scalar weighted graph, wherein the scalar is binary, integer, or floating-point.

[0053] In some embodiments, determining a weighted graph using a weighted graph function includes applying a weighted graph function that outputs a multidimensional weighted graph, in which each element is binary, integer, or floating-point.

[0054] In some embodiments, the weighted graph information of one or more channels of the obtained image is calculated using information from one or more channels of the obtained image as input.

[0055] In some embodiments, a set of weighted graph functions is predefined, and encoded information is signaled to indicate which weighted graph function to use.

[0056] In some embodiments, the weighted graph function is parameterized.

[0057] In some embodiments, encoded information signals multiple weighted graph functions, wherein determining a weighted graph using the weighted graph functions includes determining multiple weighted graphs using multiple weighted graph functions, and wherein one or more filters are configured to be applied for each weighted graph signaled.

[0058] In some embodiments, the filter's filtering function and parameters are signaled in the encoded information, predefined, configured to be inferred from the video content, or configured to be inferred from the encoded information.

[0059] In some embodiments, the filter is a linear filter, and the shape of the filter is indicated or predefined in the bitstream.

[0060] In some embodiments, the linear filter is optimized through least squares optimization or RD optimization.

[0061] In some embodiments, the linear filter is a parameterized linear filter.

[0062] In some embodiments, the parameterized linear filter is RD optimized relative to the minimum error at the output derived through least squares optimization, iterative search, or exhaustive search.

[0063] In some embodiments, the filter is a bilateral filter.

[0064] In some embodiments, determining a filter includes determining a plurality of filters, wherein each of the plurality of filters is configured to be applied at a location indicated by a signal in the bitstream or at a location indicated in a weighted graph.

[0065] In some embodiments, the parameterized weighted graph is optimized together with the filtering function.

[0066] In some embodiments, one or more filters are configured to be applied to partitions of an image based on block segmentation notified by signals in the encoded information.

[0067] In some embodiments, one or more filters are configured to be applied to partitions of an image based on derived region segmentation criteria.

[0068] In some embodiments, the filter and weighted graph computation parameters are encoded using quantization, prediction, and / or entropy coding schemes.

[0069] According to the fifth aspect, a computer-readable medium is provided, comprising computer-executable instructions stored thereon, which, when executed by a computing device, cause the computing device to perform any of the methods of the fourth aspect.

[0070] According to the sixth aspect, one or more processors are provided; and a computer-readable medium including computer-executable instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform any method of the fourth aspect.

[0071] The application of locally weighted filters improves coding performance and allows for a wider range of applications. By using a weighting function to determine the weighted graph, weighted filtering can be used in conjunction with weights to guide the properties of the filter at each spatial location. For example, the intensity of a sharpening filter can be increased near the edges of the image and decreased in regions farther away from the edges. This reduces ringing artifacts and overshoot while maintaining sharpening properties.

[0072] These and other aspects of this application will become more apparent from the following description of the embodiments. Attached Figure Description

[0073] Embodiments will now be described by way of example only with reference to the accompanying drawings, in which: Figure 1 A flowchart illustrating the operation of the decoder according to an embodiment is shown; Figure 2 A flowchart illustrating the operation of the encoder according to an embodiment is shown; Figure 3 A block diagram illustrating example operations in an embodiment is shown; Figure 4 A graph comparing several filtering schemes with real ground signals is shown.

[0074] Figure 5 A block diagram illustrating example operations in an embodiment is shown; Figure 6 Schematic diagrams of decoders according to various embodiments are shown; and Figure 7 A schematic diagram of an encoder according to various embodiments is shown. Detailed Implementation

[0075] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0076] These technical solutions can be applied to H.265 / HEVC or H.266 / VVC video coding systems (e.g., in the in-loop processes where other filters such as the Adaptive Loop Filter (ALF) and Sample Adaptive Offset Filter (SAO) are currently applied in such coding processes). However, it should be understood that these technical solutions can be applied to any other video coding system involving video compression. Furthermore, while these principles are primarily illustrated with reference to video processing, they are also applicable to other data formats, including image processing or even audio processing.

[0077] In this embodiment, "video" refers to one or more images. In other words, a video may include one or more images. An image may also be referred to as a "picture".

[0078] An encoder is a device that encodes data into a bitstream, while a decoder is a device that decodes the bitstream to obtain the encoded data or an approximation of the encoded data. A bitstream consists of a sequence of bits.

[0079] Intra-frame prediction and inter-frame prediction are two prediction operations that can be used within the HEVC and VVC frameworks for the decoder to process the received bitstream in order to obtain the original signal. In the embodiments, "original signal" or "original video" refers to the data prior to encoding at the encoder. The reference sample in the embodiments may refer to image data used to predict the spatial and / or time intervals of a picture (or a region of a picture). Intra-frame and inter-frame prediction operations are also used at the encoder to make rate-distortion determination.

[0080] More specifically, intra-frame prediction involves predicting data spatially within a single image, without referencing other images (at different time intervals). In other words, data from a first region of an image is used to predict data from another region of the same image, but does not depend on images from another time interval. In this context, the data from the first region of the image is considered a "reference sample."

[0081] Inter-frame prediction involves predicting data across images spanning multiple time intervals. In other words, data from a first region of a first image is used to predict data from a second region of a second image. The first and second regions may or may not be spatially separated. In this context, the data from the first region of the first image is considered a "reference sample." Further note that inter-frame prediction can sometimes use multiple reference regions from different images simultaneously, i.e., for a single prediction operation.

[0082] In the embodiments, "residual" may refer to the value obtained based on the original value of a region of the image and the predicted value of a region of the image (e.g., the difference between the original value and the predicted value).

[0083] In this embodiment, "block" can refer to a portion of an image. For example, an image can be divided into two or more blocks. However, this is just an example. If the image is not divided, then "block" can refer to the entire image.

[0084] In the embodiments, "filter" may refer to a filter used to enhance a signal.

[0085] Typically, in the described embodiments, the filter is configured to sharpen blurred content, reduce ringing artifacts, and / or reduce block artifacts. However, embodiments are not limited thereto, and the filter may alternatively be configured to provide alternative or additional enhancements in other embodiments.

[0086] A common optimization problem in video coding is simultaneously minimizing transmission rate and distortion. Lower transmission rates lead to stronger and more noticeable distortion, which reduces the viewer's perceived quality. Errors caused by coding are not random but are caused by processing steps in the encoder and decoder. Two important steps in a video coding system are prediction and transform. Quantization of transform coefficients introduces reconstruction errors. Many video coding systems employ a hybrid coding structure, where the content of a block is predicted via intra-frame prediction or inter-frame prediction. This prediction is often not perfectly accurate. Therefore, the difference between the ground truth signal and the prediction error is calculated, transformed, and encoded. The signal after incorporating the residual is filtered by a so-called in-loop filter.

[0087] The processing steps result in artifacts Ih that are not random. Embodiments of the present invention utilize prior information that can be used to address specific types of errors.

[0088] Two useful applications are reducing ringing artifacts and sharpening edges. These two problems are difficult to solve with linear filters. The weighted filter in embodiments of the present invention uses a simple concept to overcome the limitations of conventional linear filters. The idea is to apply two filters. The first filter extracts local information from the decoded image. The second filter applies filtering based on the output of the first filter. In some embodiments, the second filter will be RD-optimized based on the output of the first filter. Thus, a nonlinear filter that adapts to certain image features can be signaled.

[0089] Figure 1 A flowchart illustrating the operation of decoder 60 according to an embodiment is shown. Figure 2 A flowchart illustrating the operation of encoder 70 according to this embodiment is shown.

[0090] Figure 1 The flowchart begins in step 101, where decoder 60 decodes the bitstream to obtain video data and encoding information. In this embodiment, the encoding information includes weighted graph information.

[0091] In step 102, decoder 60 obtains (or “reconstructs”) an image based on the video data. The video data includes a compressed version of the original video data. In this embodiment, step 102 involves obtaining a prediction block using the video data via an intra-frame prediction operation, as specified in H.266 / VVC. However, embodiments are not limited in this respect, and in other embodiments, any other method for obtaining an image from compressed video data may be used alternatively. Examples include inter-frame prediction as specified in H.266 / VVC or intra-frame or inter-frame prediction as specified in H.265 / HEVC.

[0092] In step 103, decoder 60 uses weighted graph information to determine the weighted graph. In this embodiment, decoder 60 uses weighted graph information to determine the weighted graph. In this embodiment, the weighted graph information includes a weighted graph function for calculating the weighted graph.

[0093] One such example is when a sharpening filter is to be used, the weighted map information included in the encoded information is a local gradient calculation function. In such an example, at step 103, the decoder 60 applies the gradient calculation function to calculate the gradient at each location in the image, thereby deriving a scalar weighted map with the same resolution as the obtained image (having values ​​corresponding to each detail location in the image). In other words, the image is provided as input to the weighted map function.

[0094] However, it should be understood that this is merely an example. In other embodiments, the weighted graph may have a different resolution than the image and / or may include vector values ​​instead of scalar values. Furthermore, it should be understood that the example of the image gradient function is merely illustrative, and in the actual implementation of the embodiment, the choice of an appropriate weighted graph function will depend on the environment, particularly which filter is to be used. These factors and possible variations will be discussed in more detail later.

[0095] Furthermore, while it has been discussed in this embodiment that the encoded information includes weighted graph information, which includes weighted graph functions, the embodiments are not limited in this respect. For example, in some embodiments, multiple weighted graph functions are stored at decoder 60. In this case, the weighted graph information alternatively includes an indication of which weighted graph to use. Furthermore, in some embodiments, the encoded information does not include any explicit weighted graph indication. Alternatively, for example, the encoded information may include filter information, and then decoder 60 infers the weighted graph function to use (e.g., if the encoded information indicates that a sharpening filter should be used, the decoder infers that an image gradient function should be used as the weighted graph function). These factors and possible variations will be discussed in more detail later.

[0096] In step 104, decoder 60 determines the filter to be used. In this embodiment, decoder 60 infers the filter to be used based on weighted map information. As described above, in one example, the weighted map information included in the encoded information is a local gradient calculation function. In this example, decoder 60 can infer (e.g., pre-stored) that a sharpening filter should be used in conjunction with a weighted map that includes the image gradient.

[0097] In step 105, the decoder 60 applies a weighted map and filters to the image to obtain a filtered image. Therefore, step 105 involves using the determined weighted map such that filters are applied to different regions of the image with varying intensities.

[0098] In this embodiment, applying a weighted graph and a filter to an image involves providing both the image and the weighted graph as inputs to the filter. This results in the filter being applied to each value of the image with varying intensity, depending on the values ​​of the corresponding weights in the weighted graph. The output of the filter is a mapping of offset values. In this embodiment, the mapping of offset values ​​corresponds to the image at resolution. Once the mapping of offset values ​​is output, the offset values ​​are added to the values ​​of the image to produce the output (enhanced) image.

[0099] In the discussion above, it was assumed that the filter was a sharpening filter configured to sharpen blurred edges. However, the embodiments are not limited to this, and any suitable filter for enhancing an image can be used instead.

[0100] After step 105, the output image can then be used for any desired purpose. In one example, decoder 60 then displays the output image to a viewer. In another example, decoder 60 stores the image for later use. In yet another example, decoder 60 sends the image to an external device for display.

[0101] Encoder 70 can perform a complementary method to encode the bitstream provided to decoder 60. Figure 2 A flowchart illustrating the operation of encoder 70 according to this embodiment is shown.

[0102] In step 201, encoder 70 obtains the raw video data. For example, encoder 70 may receive the raw video data from an external server via a communication network (e.g., the Internet). However, there are no limitations on how the raw video data is obtained in this embodiment.

[0103] In step 202, encoder 70 compresses the raw video data into a form to be transmitted to decoder 60.

[0104] In step 203, encoder 70 obtains an image based on the compressed video data. In this embodiment, step 203 is compared with... Figure 1 Step 102 occurs in the same manner. In other words, in this embodiment, step 203 involves obtaining a prediction block using compressed video data via an intra-frame prediction operation, as specified in H.265 / HEVC. However, as stated above, the embodiment is not limited in this respect, and in other embodiments, any other method for obtaining images from compressed video data may be used alternatively. Examples include inter-frame prediction as specified in H.265 / HEVC or intra-frame or inter-frame prediction as specified in H.266 / VVC.

[0105] In step 204, encoder 70 determines a weighted map, and in step 205, encoder 70 determines the filters to be used. In performing these steps, encoder 70 determines the combination of weighted maps and filters to be used to enhance the image by applying filters with different weights to different regions of the image.

[0106] To determine the combination of weighted maps and filters for enhancing the image, steps 204-205 may involve a rate-distortion (RD) optimization process, which involves iteratively applying multiple filters to the image with multiple weighted maps. For each application, the average difference between the values ​​of the resulting image and the corresponding image from the original video data is determined. This loop continues until a stopping criterion is met (e.g., an optimal weighted map has been determined for a particular filter).

[0107] In other words, an RD optimization process occurs at the encoder based on the estimated signaling rate and the distortion of the acquired image.

[0108] A first example of a suitable stopping criterion is that a particular weighted graph results in the mean absolute difference (or root mean square error) of the values ​​between the resulting image and the corresponding image from the original video data being less than a predetermined threshold difference. A second example of a suitable stopping criterion is that the mean absolute difference (or root mean square error) of the values ​​between the weighted graph of the current iteration of the iteration process and the weighted graph of the previous iteration of the iteration process is less than a second predetermined threshold difference. The first example of a suitable stopping criterion directly measures the output quality and can therefore be assumed to result in a higher final image quality than the second example. However, the second example ensures that the iteration process does not require excessive computation time. In some embodiments, both examples are used, and the iteration process stops when either of these two stopping criteria is met.

[0109] The above about Figure 1 In the example discussed in A, a single (sharpening) filter is used (with a corresponding weighted graph). However, while this embodiment has been discussed regarding finding a single combination of filter and weighted graph, the embodiment is not limited in this respect. For example, in some embodiments, the encoder can identify multiple combinations of weighted graphs and different types of filters to be used.

[0110] Then Figure 2 The method continues to step 206, in which encoder 70 encodes the compressed video data and encoding information into a bitstream, the encoding information including weighted graph information.

[0111] By reference Figure 1 and Figure 2The discussed methods reveal the existence of loop filtering methods applied within the coding loop of video compression systems. The in-loop weighted filter employs functions for calculating local weighted graphs / parameter graphs and a filtering function. The weighted graph function calculates the weighted graph using the input image and optionally encoded information / parameters provided by a signal. The filtering function calculates the filtered image using the input image, the weighted graph / parameter graph, and optionally encoded information / parameters provided by a signal.

[0112] According to this method, coding performance can be improved and a wider range of applications can be allowed by applying filters with local weights. Weighted filtering can be used with weights to guide the properties of the filter at each spatial location by using a weighting function to determine the weighted graph. For example, the intensity of a sharpening filter can be increased near the edges of the image and decreased in regions farther from the edges. This reduces ringing artifacts and overshoot while maintaining sharpening properties.

[0113] In some examples, reference Figure 2 Steps 204-205 discuss optimizations involving iterations between filter and weighted graph function parameters. For example, initial parameters for the weighted graph are set, and then filter parameters are optimized based on the current weighted graph. Then, the weighted graph parameters are optimized based on the found filter parameters, and so on. In this case, the encoding information will include information about the weighted graph function and the filter function (e.g., the parameters to be used). This is, of course, a basic form of the optimization procedure. In some cases, additional lateral constraints can be set, for example, to determine the optimal filter and weighted graph not only in terms of image quality but also with the lowest possible encoding rate. This can be achieved by introducing these conditions into these separate optimizations and choosing the starting point for the next iteration while considering the rate cost. More generally, simplifications that limit computational cost can be further introduced.

[0114] In this embodiment, the weighted map provides linear weights to the filter. However, the embodiment is not limited to this, and in other embodiments, the values ​​of the weighted map can alternatively modify the filtering process itself. For example, the filter can be parameterized. For example, the frequency response of an edge enhancement filter can depend on local weighted map parameters. For example, the sigma value in an anti-sharpening mask (a type of sharpening filter) can depend on the weighting parameters. This means that the way the filter works, or more specifically, the function of the filter is parameterized and does not necessarily depend linearly on the weighted map. Another example is a filter that performs edge thinning (sharpening) by distorting an image. The intensity of the distortion can depend on the current weighted map values.

[0115] In this embodiment, the weighted filter is applied within the loop. However, the embodiment is not limited to this particular order, and the weighted filter may be applied additionally or alternatively at other processing steps in other embodiments, such as after the loop.

[0116] To illustrate the above reference Figure 1 and Figure 2 The principles discussed in the embodiments will now be referenced. Figure 3 Discuss illustrative examples.

[0117] Figure 3 The above discussion illustrates the following. Figure 1 and Figure 2 A block diagram illustrating example operations of an embodiment. In other words, Figure 3 A conceptual diagram illustrating how to implement a weighted filter is shown.

[0118] It can be seen that, from Figure 3 During the process, the distorted image 31 eventually becomes the (enhanced) output image 34.

[0119] Distorted image corresponds to reference, for example Figure 1 The image obtained in step 102 is discussed.

[0120] refer to Figure 1 Step 103, for example, then the distorted image is provided as input to a weighted graph function (f) determined based on the encoded information. w-map )3A is used to obtain weighted graph 32. In this particular example, as described above, the weighted graph function 3A is a local gradient calculation function. Thus, weighted graph 32 includes multiple values ​​of the image gradient at each spatial location in the distorted image 31.

[0121] refer to Figure 1 Step 104 determines filter 3B. As described above, in this embodiment, decoder 60 infers the filter to be used based on the weighted map information. Also as described above, in this example, the weighted map information included in the encoded information is a local gradient calculation function. Therefore, in this example, decoder 60 infers (e.g., pre-stored) that the sharpening filter should be used in conjunction with a weighted map that includes the image gradient.

[0122] refer to Figure 1 In step 105, decoder 60 then applies weighted image 32 and filter 3B to distorted image 31 to obtain output image 34. (As from...) Figure 3 As can be seen, this step involves providing the distorted image 31 and the weighted image 32 as inputs to the (sharpening) filter 3B, thereby producing a weighted enhancement image 33. The weighted enhancement image 33 includes multiple offset values ​​that spatially correspond to the values ​​of the distorted image 31.

[0123] Next (as) Figure 1 As part of step 105, decoder 60 adds the offset value from weighted enhanced image 33 to the corresponding value from distorted image 31 to obtain output image 34.

[0124] This method demonstrates that blurring has been reduced without any significant issues regarding ringing artifacts or overshoot that might be caused by the sharpening filter. This is achieved through local weighting of the sharpening filter, ensuring that it is not applied uniformly across the entire image, but rather with varying intensities to different spatial regions of the image based on their properties.

[0125] More specifically, the intensity of the sharpening filter is increased near the edges of the image and decreased in areas further away from the edges. This reduces ringing artifacts and overshoot while maintaining sharpening properties.

[0126] Note that in some embodiments, the offset map is incorporated into the filter function. However, for better visualization, these operations are performed separately. Figure 3 The steps are shown separately.

[0127] Furthermore, as will be discussed in more detail later, in some embodiments, both the weighted graph and the filtering function (or one of them) may be dependent on a parameterized function notified by signals in the bitstream.

[0128] As can be seen, the weighted filter consists of two main components: the weighted graph / parameter graph computation function and (possibly a parameterized) filter.

[0129] First, a weighted image is computed. In this embodiment, the weighted image can be computed at each point in the image (although in other embodiments, the weighted image can be computed at a lower resolution). Next, a filter is applied. Thus, the weighted image / parameter image and the decoded image are input into the filter. This generates a filtered image.

[0130] exist Figure 1 and Figure 2 In one embodiment, the method is applied to the luma channel. However, the embodiments are not limited to this. In other embodiments, the method is applied only to the chroma channel, or to both the luma and chroma channels. In other words, in some embodiments, the weighted graph / parameter graph calculation function and the filter (or either one) can be different for different channels. Therefore, separate information / parameters for the luma and chroma components can be communicated using signals.

[0131] In this embodiment, a weighted filter replaces both the SAO and ALF of the VVC / H.266 system.

[0132] However, the embodiments are not limited to this aspect. For example, in other embodiments, the weighted filter replaces only one of the SAO and ALF, or the weighted filter is set in addition to the SAO and ALF. In another example, the weighted filter can be integrated into the ALF, so that each partition of the image can be filtered by an optimized linear filter or by a weighted filter. In this way, the ALF will gain additional flexibility. For maximum flexibility, the filter can be added at any point in the filter chain within the loop.

[0133] While these specific examples have been discussed, it should be understood that the embodiments are not limited to the H.266 / VVC scheme in this manner. In other embodiments, the weighting filter is used for entirely different coding schemes (e.g., H.265 / HEVC or any other suitable coding scheme).

[0134] Furthermore, in other embodiments, weighted filters are applied as post-filters to enhance the quality of the encoded video. This is beneficial where back-coupling (from in-loop filtering) would lead to worse predictions for subsequent images. In this case, out-of-loop / post-filtering would be advantageous. In embodiments, such a determination can be made by the encoder during encoding.

[0135] Therefore, more generally, in embodiments, weighted filters can be integrated into the coding loop as an additional processing step in existing schemes, as a replacement for existing loop filters, or integrated into existing loop filters.

[0136] To further explain the concept of weighted filters discussed above, please refer to... Figure 4 Let's discuss another example implementation. Figure 4 A graph comparing several filtering schemes with real ground signals is shown.

[0137] Let there be a blurred edge As input to the filter. Ground real signal 41 (e.g.) Figure 4 The signal shown is a step function. For simplicity and ease of visualization, a one-dimensional signal is presented. It should be noted that this is done merely to explain the concept in a simple way. Generally, the methods discussed in this paper can be applied to signals of any dimension.

[0138] Figure 4 The diagram shows four lines. There is a ground-based true signal 41 and an ambiguous signal 42 (e.g., corresponding to...). Figure 3 Distortion 31), weighted filtered signal 43 (e.g., corresponding to) Figure 3 The output image 34) and the unweighted filtered signal 44 (e.g., corresponding to the case without using a weighted image, i.e., using only a filter). Figure 3 Output image 34).

[0139] In unweighted filtering, the ambiguous signal 42 is filtered by a least-squares optimized linear filter to approximate the original signal as accurately as possible, thus obtaining the unweighted filtered signal 44. Figure 4 As can be seen, this type of filtering does increase the steepness of the edges, thus providing a better approximation of the true ground signal 41, but it causes suboptimal overshoot and ringing.

[0140] Better results are obtained if the filter's high-pass characteristic is stronger at the steepest parts of the edges and weaker in areas where the filter causes ringing and overshoot artifacts. To achieve this, in the weighted signal 42, the offset of the filtered blurred edge compared to the blurred edge is scaled by a larger factor at the predicted location of the edge, and by a smaller factor if expected overshoot or ringing artifacts are present. For this purpose, local weighting is calculated based on the image (i.e., a weighted map is calculated). For example (and as mentioned above regarding...) Figures 1 to 3 The magnitude of the gradient (as discussed) can be approximated by finite differences. To obtain the best results, local weighting will then be considered to optimize the filter.

[0141] like Figure 4 As shown, the weighted filtered signal 43 has a smaller error relative to the ground truth signal 41 compared to the unweighted filtered signal 44. Furthermore, in this example, it can be seen that for the weighted filtered signal 43, the overshoot is at a similar level, but the steepness and ringing of the signal are not as severe (i.e., it has a similar amplitude, but it flattens out earlier).

[0142] This example demonstrates that local adaptation can improve filter performance if the error and signal characteristics are known. Adaptive Loop Filters (ALFs) achieve this by applying different filters based on the characteristics of the image. This allows for greater flexibility, but at the cost of increased bit rate.

[0143] Embodiments of the present invention utilize weighted (potentially parameterized) filters to reduce the need for using many different filters. The example shown is an application case where a single weighted filter can replace a set of filters while achieving similar results. This is particularly applicable when dependencies exist that can be exploited through local parameterization.

[0144] In this embodiment, it has been discussed that the encoded information includes weighted graph information, which includes weighted graph functions. However, the embodiments are not limited to this aspect. For example, as described above, in some embodiments, multiple weighted graph functions are stored at decoder 60. In this case, the weighted graph information alternatively includes an indication of which weighted graph to use. Furthermore, in some embodiments, the encoded information does not include any explicit weighted graph indication. Alternatively, for example, the encoded information may include filter information, and then decoder 60 infers the weighted graph function to use (e.g., if the encoded information indicates that a sharpening filter should be used, the decoder infers that an image gradient function should be used as the weighted graph function).

[0145] Refer to the above Figure 1 and Figure 2 In the embodiments discussed, the weighted filter is applied in an in-loop manner, particularly in VVC / H.266 systems.

[0146] Typically, in-loop filters can be applied at every point within the coding loop. However, because most loop filters are nonlinear, the order in which they are applied can impact overall performance. An example of a video coding system is VVC / H.266. In this system, four in-loop filters are applied sequentially: luma mapping with chroma scaling (LMCS), deblocking, sample adaptive offset (SAO), and adaptive loop filter (ALF). LMCS addresses errors very different from those of the proposed filter and should be applied before the proposed filter to avoid artifacts. Furthermore, applying the deblocking filter before the proposed method makes sense.

[0147] While specific implementations of the invention have been discussed above, many variations can be made in other embodiments that will now be discussed, particularly regarding the selection of weighted graph functions, filtering functions, region segmentation, and signaling.

[0148] As is evident from the above discussion, the weighted graph provides one or more weights / parameters to the filter. In other words, the filter is a parameterized function that takes the weighted graph and the image (and possibly the encoded parameters) as input.

[0149] Regarding weighted graphs, Figure 1 and Figure 2In one embodiment, the weighted graph is a scalar graph. In this scalar graph, each spatial location is assigned a value. However, in other embodiments, the weighted graph is a multidimensional graph. In such embodiments, there exists a vector of values ​​at each spatial location. It should also be noted that in embodiments of the invention, the spatial size of the weighted graph is not limited to the resolution of the image. Depending on requirements, in some embodiments, the spatial size of the weighted graph may have a smaller resolution to reduce computational complexity. The optimal choice of the weighted graph function depends largely on the type of error processed by the filter and the type of filter applied to process these errors.

[0150] As mentioned above, in Figure 1 and Figure 2 In one embodiment, a weighted graph and a filter are applied to the image ( Figure 1 Step 105 in the above section: 1) This involves providing both the image and the weighted graph as inputs to the filter. This results in the filter being applied with varying intensities to each value of the image, depending on the values ​​of the corresponding weights in the weighted graph. The output of the filter is a graph of offset values, and this graph of offset values ​​corresponds to the image at a specific resolution. Once the graph of offset values ​​is output, the offset values ​​are added to the values ​​of the image to produce the output (enhanced) image. An example of this is as discussed above. Figure 3 As shown.

[0151] However, the embodiments are not limited to this particular implementation. In an alternative, simpler implementation, the intensity values ​​(i.e., the weights of the weighted image) are used by a filter to scale the offset generated by the filter. This is done by multiplying the intensity values ​​(i.e., the weights) by the difference between the filter's output and the resulting image. This scaled difference is then added to the resulting image to modify the offset generated by the filter according to the calculated weights. Therefore, the effect of the applied filter varies depending on the spatial location.

[0152] An example of this setting is as follows: Figure 5 As shown, a block diagram illustrating an example operation of this alternative method is presented.

[0153] It can be seen that, Figure 5 The method involves providing the distorted image 51 as input to a weighted image and separately as input to a filter. The intensity values ​​(i.e., weights) of the resulting weighted image 52 are then multiplied by the difference between the output of filter 55 and the distorted image 51. This scaled difference is then added to the distorted image 51, altering the offset generated by the filter according to the calculated weights.

[0154] In other words, a scalar weight is computed to weight the filter's output through sample-by-sample multiplication. The computed offset is then added to the input image to obtain the output.

[0155] From and Figure 3A comparison of the examples shows that instead of applying the weighted image 52 as input to the filter, the value of the weighted image 52 is simply multiplied by the output from the filter (specifically, the difference between the output of filter 55 and the distorted image 51).

[0156] In some embodiments, the weighted graph function is an edge detector that assigns higher weights to locations on edges and lower values ​​to locations around edges and in flat regions. Therefore, in such embodiments, the resulting weighted graph is a scalar. This allows for edge sharpening with fewer artifacts. In such embodiments, the encoder may, for example, encode information used to identify the edge detector function in the bitstream.

[0157] In other embodiments, the weighted graph function is a detector for ringing artifacts. For example, this assigns a probability that ringing is present at a given location. Therefore, in such embodiments, the resulting weighted graph is a scalar. This weighted graph is applied, and the filter strength is then set based on this probability. In such embodiments, the filter can be, for example, a simple linear low-pass filter. However, in many cases, ringing is close to the edge that should ideally be preserved. Therefore, ideally, the filtering solution should preserve the edge. An example of a filter suitable for this is a bilateral filter. In embodiments utilizing such a filter, the parameters can be optimized at the encoder and transmitted in the bitstream.

[0158] Furthermore, in some embodiments, these two options for the weighted graph can be combined into a two-dimensional weighted graph, or they can be applied sequentially. Combining them in filtering provides greater flexibility. Understanding the estimated ringing probabilities and the presence of edges helps the encoder find the optimal filtering during its rate-distortion optimization process (e.g., Figure 2 (Steps 204-205). For example, if the ringing is very close to the position, edge amplification should be applied more carefully because the ringing will be amplified, which can be optimized by the encoder during the encoding process.

[0159] Previous examples of applying bilateral filters also illustrate how multidimensional weighted graphs, rather than just scalars, can be applied in some embodiments. For example, in some embodiments, the encoder determines different parameters if a region with high contrast exists compared to a region with low contrast. In this case, the parameters of the bilateral filter are estimated based on the acquired image and the decoder is notified with a signal.

[0160] Another application in other embodiments is to estimate parameters for edge sharpening filters from an image. For example, if the weighted graph estimates whether there is very sharp content, such as text, at a certain location in the image, different sharpening can be applied compared to other types of content. Note that in such embodiments, the weighted graph can be derived from image content, encoded information, or parameters signaled by a signal. In these embodiments, the weighted graph can be one-dimensional. The weighted graph can be binary, integer, or floating-point. In these embodiments, the weighted graph can alternatively be multi-dimensional, where each element is binary, integer, or floating-point. The data type of each element of the weighted graph depends on the requirements of the filtering system.

[0161] In some embodiments, the weighted graph computation parameters are signaled in the bitstream and determine the type of the weighted graph function. In other embodiments, the signaled weighted graph computation parameters are parameters of the function itself. For example, the edge graph may have a steepness scaling parameter that determines how much weight is added based on steepness. Note that this may be non-linear scaling. For example, scaling by powers of the value may be used.

[0162] In some embodiments, a set of (potentially parameterized) weighted graph functions are predefined. Therefore, the bitstream only signals one or more of the weighted graph functions / one or more weighted graphs used (and specific parameters that may be used in these weighted graph functions), rather than the entire weighted graph function.

[0163] In summary, regarding weighted graphs, in some embodiments, a weighted graph function is applied that outputs a scalar weighted graph, where the scalar is binary, integer, or floating-point. In other embodiments, a weighted graph function is applied that outputs a multidimensional weighted graph, where each element is binary, integer, or floating-point. In these embodiments, weighted graph information for one or more channels of the obtained image is calculated while using information from one or more channels of the reconstructed image as input. In some embodiments, a predefined set of (potentially parameterized) weighted graph functions is defined, where the bitstream signals the use of one or more weighted graph functions / one or more weighted graphs.

[0164] Regarding filters, the one or more filter functions used in some embodiments of the present invention are typically multidimensionally parameterized functions that take a weighted graph, filter parameters, one or more channels of the obtained image, and possible encoding information as input. The output is one or more (weighted) filtered channels of the obtained image.

[0165] While embodiments have been generally discussed regarding the application of a single filter, the invention is not limited in this respect. For example, in other embodiments, a series of filters with different parameters and possibly different weighted graphs can be applied. Depending on the type of artifact, complexity requirements, and RD decisions, different filtering functions may be most suitable.

[0166] For example, in some embodiments, a linearly weighted filter can be used. In such embodiments, the filter can be weighted by multiplying its output by a local weight and then adding the result to the obtained image. The advantage of this system is that the optimal parameters can be found through least-squares optimization. Therefore, parameter search is not required to find the optimal solution.

[0167] However, in such embodiments, depending on the characteristics of the image, linear filters will require a relatively large number of filter coefficients (which will need to be transmitted). To offset this, in some embodiments, parameterized descriptions of the filters are used to reduce coding costs (although this comes at the cost of reduced flexibility).

[0168] An example of parameterization is modeling a high-pass filter as a difference of a Gaussian filter. Then, only the σ value needs to be sent instead of the entire set of filter coefficients. This method is useful if the frequency response of the parameterized filter is sufficiently close to the distribution of the filter obtained through least-squares optimization. However, depending on the lateral constraints of the parameterized representation n, a solution in a closed-form may not be found, in which case iterative optimization will be required.

[0169] Another type of filter that can be used in another embodiment is a (parametric) nonlinear filter. Examples include bilateral filters, median filters, or other filters. The parameters of these filters can be indicated by the signal in the bitstream or given by a weighted graph. Note that switching the type of filter function based on the weighted graph parameters is also an option.

[0170] In other words, in some embodiments, the value of the weighted graph at a particular scalar location can indicate the type of filter function to be used at that spatial location in the image.

[0171] In summary, regarding filters, in some embodiments, one or more filters may be applied to each weighted graph notified by a signal. The parameters of the filtering function and filters may be notified by a signal in the bitstream, predefined, or inferred from the content of the video sequence or encoded information. In some embodiments, a linear filter is applied as a filter in the filtering function. The shape of the filter may be indicated or predefined in the bitstream. In some embodiments, the linear filter is optimized by least-squares optimization or RD optimization. In some embodiments, a parameterized linear filter is used, where parameters are available to generate the corresponding linear filter. In some embodiments, the minimum error of the parameterized linear filter with respect to the output is RD optimized or RD optimized. The optimal filter may be derived by least-squares optimization, iterative search, or exhaustive search. In some embodiments, a parameterized or non-parameterized nonlinear filter is applied in the filtering function. In some embodiments, a bilateral filter is used in the filtering function. In some embodiments, a combination of the filtering methods discussed is applied, where the filtering method used at each location is notified or indicated by a signal in the weighted graph. In some embodiments, the encoder optimizes the parameterized weighted graph together with the filtering function.

[0172] In the embodiments described herein, the weighted filter has been described as being applied to the entire image. However, the embodiments are not limited to this. In variations of these embodiments, different filtering settings may exist for different regions of the image. Furthermore, overlapping application of filters is possible. Two example implementations for region segmentation will now be discussed.

[0173] The first example is block-by-block segmentation of an image, where each filter is applied to one or more blocks. In some embodiments, alignment with coding tree unit (CTU) and coding (CU) boundaries may be considered. The applicable image segmentation is then signaled in the bitstream.

[0174] The second example is image segmentation based on image features. In some embodiments, this segmentation can be derived on both the decoder and encoder sides without needing to be signaled in the bitstream. This can be implemented as a binarized or non-binary weighted map. Each resulting partition can then be processed individually, or, in some embodiments, multiple partitions can be processed in groups.

[0175] When using region segmentation, the filter to be applied to a region or group of regions can be optimized on the encoder side, and then the parameters can be signaled in the bitstream.

[0176] Therefore, in some embodiments, an image can be segmented into multiple partitions. Partitions can be defined by signaled block segmentation, signaled region segmentation criteria, and / or by a binarized weighted graph function. Furthermore, in some embodiments, multiple filters can be applied to the same image partition.

[0177] Using partitioning in these ways can be useful, especially when dealing with large or very different images. Such images may contain very different types of content, and the error characteristics at different partitions of the image can be very different. Therefore, optimizing two or more filters for different partitions of the image can lead to excellent performance.

[0178] As discussed in the described embodiments, encoding information may be included in the bitstream, such as information about the weighted graph function and / or filter to be applied to the image. More generally, in embodiments, the encoding information may include (but is not limited to) filter coefficients, weighted graph function parameters, on / off flags, filter coding parameters, or region parameters, etc.

[0179] In some embodiments, this information (e.g., all parameters) is encoded and signaled to reduce the transmission rate and improve the efficiency of the overall filtering process. This is done by utilizing redundancy regarding the transmitted parameters. This redundancy is utilized through prediction and entropy encoding of the filter parameters. Furthermore, in some embodiments, the parameters are quantized to reduce the number of possible representations.

[0180] In the embodiments described herein, the filters used (i.e., sharpening filters) are based on the concept of Wiener filters. In other embodiments, the filters are linear filters that have already been optimized at the encoder through a least-squares optimization process (i.e., linear filters that minimize the squared error between the filtered signal and the true signal). Of course, in some embodiments, additional lateral constraints are set when determining the signal enhancement filter, such as filter shape and filter coefficients that must be equal.

[0181] However, while embodiments have been discussed with reference to filters based on the Wiener filter concept, these embodiments are not limited in this respect, and other types of filters, such as Sobel-based filters or desharpening masking filters as sharpening filters, can be used alternatively. Other nonlinear options include bilateral filters and diffused filters, as well as adaptive loop filters (ALF).

[0182] For example, in some embodiments, weighted filters can be integrated into the adaptive loop filter (ALF) of existing coding schemes (e.g., H.265 / HEVC and H.266 / VVC) and applied as an alternative to linear filters to partitions derived through ALF optimization.

[0183] Figure 6 A schematic diagram of a decoder 60 according to an embodiment is shown. Specifically, Figure 6 A schematic diagram of a decoder 60 configured to perform any of the decoder methods discussed herein is shown. For the sake of brevity, such a detailed description is omitted here.

[0184] like Figure 6 As shown, decoder 60 includes processor 61 and computer-readable medium 62. Processor 61 and computer-readable medium 62 may be connected via a bus system. The computer-readable medium is configured to store programs, instructions, or code. Processor 61 is configured to execute the programs, instructions, or code in computer-readable medium 62 to perform operations in the decoder method embodiments of this document.

[0185] Therefore, in this embodiment, the computer-readable medium 62 is configured to store a computer program that can run in the processor 61, and the processor 61 is configured to run the computer program to perform the steps in any decoder method discussed herein.

[0186] Figure 7 A schematic diagram of an encoder 70 according to an embodiment is shown. Specifically, Figure 7 A schematic diagram of an encoder 70 configured to perform any of the encoder methods discussed herein is shown. For the sake of brevity, such a detailed description is omitted here.

[0187] like Figure 7 As shown, encoder 70 includes processor 71 and computer-readable medium 72. Processor 71 and computer-readable medium 72 may be connected via a bus system. The computer-readable medium is configured to store programs, instructions, or code. Processor 71 is configured to execute the programs, instructions, or code in computer-readable medium 72 to perform operations in the decoder method embodiments of this document.

[0188] Therefore, in this embodiment, the computer-readable medium 72 is configured to store a computer program that can run in the processor 71, and the processor 71 is configured to run the computer program to perform the steps in any decoder method discussed herein.

[0189] As discussed, in the embodiments, the weighted filter is a parametric filter. Depending on the implementation, in some embodiments, the weighted filter is also content-adaptive. The two main objectives of the filters discussed in the embodiments are sharpening of blurred content and reduction of ringing artifacts. However, other objectives of the filtering process are also possible, such as reducing block artifacts.

[0190] In some embodiments, the weighted filter is a locally adaptive filter. Therefore, it can be used to handle nonlinear filtering problems. Two useful applications are reducing ringing artifacts and sharpening edges. These two problems are difficult to solve with linear filters. The weighted filter of embodiments of the present invention uses a simple concept to overcome the limitations of conventional linear filters. The idea is to apply two filters. The first filter extracts local information from the decoded image. The second filter applies filtering based on the output of the first filter. In some embodiments, the second filter is RD-optimized based on the output of the first filter. Thus, a nonlinear filter that adapts to certain image features can be signaled.

[0191] In order to place the features of the embodiments of the present invention in a further context, a discussion of weighted filtering of these embodiments will now be provided relative to existing loop filters.

[0192] Loop filtering is a component of modern video coding systems. Typically, a set of different filters are applied sequentially. These filters can be parametric, meaning a set of filter parameters is sent, and their behavior changes based on RD (Resolution-Oriented) decisions. They can also be non-parametric. Furthermore, filters can be content-adaptive (i.e., filters can behave differently depending on their spatial location within an image). Sample Adaptive Shift (SAO) filters and Adaptive Loop Filters (ALF) perform local classification of the content and apply different operations based on the category. Typically, the classification operation to be applied is signaled.

[0193] In detail, the in-loop ALF method optimizes a set of linear filters. Each linear filter is applied to a partition of the image. Segmentation is derived from local attributes of the image. Furthermore, partitions may be merged. This information is signaled in the bitstream. However, in embodiments of the present invention, locally weighted / parameterized filters are applied. This reduces the need for image segmentation and optimization of multiple filters, thereby improving coding efficiency.

[0194] In SAO, each sample (or "pixel") is assigned to a category based on the local features of the image. For each category, an (intensity) offset is calculated and signaled at the encoder. Thus, a category decision is made, and different operations are performed for each category. However, in embodiments of the present invention, locally weighted / parameterized filters are applied. This reduces the need for image segmentation and optimization of multiple filters, thereby improving encoding efficiency.

[0195] As discussed, embodiments provide methods and apparatus for encoding parameters of in-loop (or post-loop) filtering schemes. Therefore, the requirements for transmission of adaptive parameter sets and the characteristics of the encoded information can be considered to allow for efficient encoding. Embodiments employ weighted / parameterized filtering. The weighted / local parameters are calculated based on decoded video images (e.g., image patches). In some embodiments, the calculation function may be parameterized and / or transmitted by the encoder via signals.

[0196] In the following text, we will refer to the locally parameterized weights as a “weighting-map”.

[0197] Although the term "weighted graph" has been used, it is for readability purposes and is not intended to be restrictive. For example, in some embodiments, the weighted graph may contain local parameters or a vector containing both parameters and weights. In addition to the calculation of weights, filters are applied. In some embodiments, the filter may be parameterized with respect to the weighted graph, or it may be locally weighted according to the weighted graph.

[0198] As discussed, the embodiments provide an in-loop filtering method applicable within the coding loop of a video compression system. The in-loop filter employs a function for calculating local parameter mappings and a filtering function. The weighted graph function can use the input image, encoding information, and / or parameters notified by signals to calculate a weighted graph. The filtering function can use the input image, encoding information, parameters notified by signals, and / or a parameter graph to calculate a filtered image.

[0199] In some embodiments, the method is integrated into the coding loop as an additional processing step, either as a replacement for an existing loop filter or as an integral part of an existing loop filter.

[0200] In some embodiments, after applying the method, the method is optimized by RD based on the estimated signaling rate and distortion.

[0201] In some embodiments, the method is applied to the luminance channel, the chroma channel, or both the luminance and chroma channels. The processed channels can be preset, signaled in the bitstream, or inferred from the content.

[0202] In some embodiments, the method is applied to one or more partitions of an image. Partitions can be defined by block segmentation notified by signals, region segmentation criteria possibly notified by signals, or by a binarized weighted graph function.

[0203] In some embodiments, multiple filters are applied to the same image partition.

[0204] In some embodiments, the method is used to address the problems of ringing artifacts, blurring, or block artifacts.

[0205] In some embodiments, the method involves applying a weighted graph function that outputs a scalar-weighted graph, where the scalars are binary, integer, or floating-point numbers.

[0206] In some embodiments, the method involves applying a weighted graph function that outputs an n-dimensional weighted graph, where each element is binary, integer, or floating-point.

[0207] In some embodiments, weighted graph information is calculated for one or more channels of the reconstructed image while using information from one or more channels of the reconstructed image as input.

[0208] In some embodiments, a set of possible parameterized weighted graph functions is predefined, and the encoder signals the weighted graph and weighted graph functions used.

[0209] In some embodiments, one or more filters are applied to each weighted graph notified by a signal. The parameters of the filtering function and the filters can be notified by a signal in the bitstream, predefined, or inferred from the content of the video sequence or encoded information.

[0210] In some embodiments, the method involves applying a linear filter as a filter in a filtering function. The shape of the filter can be indicated in the bitstream or predefined.

[0211] In some embodiments, the linear filter is optimized through least squares optimization or RD optimization.

[0212] In some embodiments, a parameterized linear filter is applied. Parameters can be used to generate the corresponding linear filter.

[0213] In some embodiments, the minimum error of the parameterized linear filter with respect to the output is RD-optimized or RD-optimized. The optimal filter can be derived through least-squares optimization, iterative search, or exhaustive search.

[0214] In some embodiments, a parameterized or non-parameterized nonlinear filter is applied to the filtering function.

[0215] In some embodiments, a bilateral filter is applied in the filtering function.

[0216] In some embodiments, a combination of the described filtering methods is applied, wherein the filtering method used at each location is notified or indicated by a signal from the weighted graph.

[0217] In some embodiments, the encoder, together with the filtering function, optimizes the parameterized weighted graph.

[0218] In some embodiments, one or more filters are applied to partitions of an image based on block segmentation notified by signals.

[0219] In some embodiments, one or more filters are applied to partitions of an image based on a derived region segmentation criterion.

[0220] In some embodiments, the weighted filter is integrated into the adaptive loop filter (ALF) and applied to partitions derived through ALF optimization as an alternative to the linear filter.

[0221] In some embodiments, the encoder encodes the filter and weighted graph computation parameters using quantization, prediction, or entropy coding schemes.

[0222] In some embodiments, the weighted filter is applied as a post-filter.

[0223] Embodiments of the present invention may also provide a computer-readable medium having computer-executable instructions that cause one or more processors of a computing device to perform the methods of any embodiment of the present invention.

[0224] Examples of computer-readable media include volatile and non-volatile media, removable and non-removable media, and include, but are not limited to: solid-state storage, removable disks, hard disk drives, magnetic media, and optical disks. Generally, computer-readable media includes any type of media suitable for storing, encoding, or carrying a series of instructions executable by one or more computers to perform any or more of the processes and features described herein.

[0225] It should be understood that, unlike those discussed in the foregoing description, the functionality of each of the components discussed can be combined in various ways. For example, in some embodiments, more than one function of the devices discussed can be incorporated into a single device. In other embodiments, the functionality of at least one of the devices discussed can be split into multiple separate (or distributed) devices.

[0226] Conditional language such as “may” is often used to indicate the use of features / steps in a particular embodiment, but alternative embodiments may include alternative features or omit these features entirely.

[0227] Furthermore, the method steps are not limited to the specific order described, and it should be understood that these steps can be combined in any other suitable order. In some embodiments, this can result in the parallel execution of some method steps. Additionally, in some embodiments, specific method steps may be omitted entirely.

[0228] While certain embodiments have been discussed, it should be understood that these embodiments are intended to illustrate the general teachings of the invention, and various modifications may be made without departing from the scope of the invention. The scope of the invention should be interpreted in accordance with the appended claims and any equivalents thereof.

[0229] Many further variations and modifications will arise in those skilled in the art when referring to the foregoing illustrative embodiments, which are given by way of example only and are not intended to limit the scope of the invention as defined by the appended claims.

Claims

1. A method for processing video data performed by a decoder, the method comprising: Decode the bitstream to obtain video data and encoding information; Images are obtained based on the video data; A weighted graph is determined using a weighted graph function, the weighted graph including multiple weights mapped to corresponding spatial locations of the image, wherein the image and / or the encoded information are used as input to the weighted graph function; Determine the filter to be applied to the weighted graph; and The weighted graph and the filter are applied to the image to obtain a filtered image, such that the filter is applied to different spatial locations in the image with different weights.

2. The method of claim 1, wherein, The encoded information includes weighted graph function parameters notified by signals, and The determination of the weighted graph using the weighted graph function includes: The parameters of the weighted graph function notified by the signal are applied as parameters of the weighted graph function; and The image is provided as input to the weighted graph function.

3. The method of claim 1 or 2, wherein, The encoded information includes filter function parameters communicated via signals, and Determining the filter includes: The filter function parameters, which are notified by the signal, are used as the parameters of the filter.

4. The method of any one of claims 1 to 3, wherein, The weighted image and the filter are applied to the image to obtain the filtered image, either within the coding loop or as a post-loop step.

5. The method of claim 4, wherein, The encoding loop is an H.266 / VVC encoding loop.

6. The method of claim 4 or 5, wherein, The step of applying the weighted graph and the filter to the image to obtain the filtered image is integrated into an adaptive loop filter and applied to the derived partition.

7. The method of any one of claims 1 to 6, wherein, The bitstream is rate-distortion (RD) optimized based on estimated signaling rate and distortion after the weighted graph and the filter are applied to the image.

8. The method according to any one of claims 1 to 7, wherein, The image includes a luminance channel, a chroma channel, or both the luminance channel and the chroma channel, and The weighted graph and the filter are applied to the luminance channel, the chrominance channel, or both the luminance channel and the chrominance channel.

9. The method according to claim 8, wherein, The weighted graph and the filter will be applied to which of the luminance and chroma channels is predetermined, indicated by a signal in the bitstream, or inferred from the content of the image.

10. The method according to any one of claims 1 to 9, further comprising dividing the image into multiple partitions, in, The weighted graph and the filter are applied to one or more partitions of the image. The partition is indicated by a signal in the encoded information.

11. The method according to claim 10, wherein, The partition is signaled in the encoded information, including block segmentation, region segmentation criteria, or binarized weighted graph function signaled in the encoded information.

12. The method of claim 10 or 11 further comprises applying multiple filters to the same image partition.

13. The method according to any one of claims 1 to 12, wherein, The filter is configured to resolve issues of ringing artifacts, blurring, and / or block artifacts in the image.

14. The method according to any one of claims 1 to 13, wherein, Determining the weighted graph using the weighted graph function includes: Apply a weighted graph function to output a scalar weighted graph, where the scalar is binary, integer, or floating-point.

15. The method according to any one of claims 1 to 13, wherein, Determining the weighted graph using the weighted graph function includes: The application outputs a weighted graph function for a multidimensional weighted graph, where each element in the multidimensional weighted graph is binary, integer, or floating-point.

16. The method according to any one of claims 1 to 15, wherein, The weighted graph information of one or more channels of the obtained image is calculated using information from one or more channels of the obtained image as input.

17. The method according to any one of claims 1 to 16, wherein, A predefined set of weighted graph functions, and The encoded information is used to signal the weighted graph function to be used.

18. The method according to claim 17, wherein, The weighted graph function is parameterized.

19. The method according to any one of claims 1 to 18, wherein, The encoded information is used to notify multiple weighted graph functions via signals. Specifically, determining the weighted graph using the weighted graph function includes determining multiple weighted graphs using the plurality of weighted graph functions, and The application of the weighted graph and the filter includes applying one or more filters to each weighted graph notified by the signal.

20. The method according to claim 19, wherein, The parameters and filtering functions of the filter are indicated by signals in the encoded information, predefined, inferred from the content of the video, or inferred from the encoded information.

21. The method according to any one of claims 1 to 20, wherein, The filter is a linear filter, and the shape of the filter is indicated or predefined in the bitstream.

22. The method according to claim 21, wherein, The linear filter is optimized using least squares optimization or rate-distortion (RD) optimization.

23. The method according to any one of claims 19 to 22, wherein, The linear filter is a parameterized linear filter.

24. The method according to claim 23, wherein, The parameterized linear filter is rate-distortion (RD) optimized relative to the minimum error at the output derived through least-squares optimization, iterative search, or exhaustive search.

25. The method according to any one of claims 1 to 20, wherein, The filter is a bilateral filter.

26. The method according to any one of claims 1 to 25, wherein, Determining the filter includes determining multiple filters. Applying the weighted graph and the filter to the image includes applying each of the plurality of filters at a location indicated by a signal in the bitstream or at a location indicated in the weighted graph.

27. The method according to any one of claims 1 to 26, wherein, The parameterized weighted graph is optimized together with the filtering function.

28. The method according to any one of claims 1 to 27, wherein, Applying the weighted graph and the filter to the image includes: applying one or more filters to partitions of the image based on block segmentation notified by signals in the encoded information.

29. The method according to any one of claims 1 to 27, wherein, Applying the weighted graph and the filter to the image includes: applying one or more filters to partitions of the image based on the derived region segmentation criteria.

30. The method according to any one of claims 1 to 29, wherein, The filter and weighted graph calculation parameters are encoded using quantization, prediction, and / or entropy coding schemes.

31. A computer-readable medium comprising computer-executable instructions stored thereon, which, when executed by a computing device, cause the computing device to perform the method according to any one of claims 1 to 30.

32. A decoder, comprising: One or more processors; and A computer-readable medium, including computer-executable instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 30.

33. A method for processing video data performed by an encoder, the method comprising: Obtain raw video data; The original video data is compressed into compressed video data; Images are obtained based on the compressed video data; A weighted graph is determined using a weighted graph function, the weighted graph comprising multiple weights mapped to corresponding spatial locations in the image, wherein the image is used as input to the weighted graph function; Determine the filter to be applied to the image using the weighted graph, wherein the filter is configured to be applied to the image using the weighted graph to obtain a filtered image, such that at the decoder, the filter is applied with different weights to different spatial locations in the image; and The compressed video data and encoding information are encoded into a bitstream, the encoding information including information about the weighted graph function and / or filter to be used at the decoder.

34. The method according to claim 33, wherein, The encoded information includes weighted graph function parameters signaled by a signal, the weighted graph function parameters being configured such that the decoder can use the weighted graph function to determine the weighted graph by: The parameters of the weighted graph function notified by the signal are applied as parameters of the weighted graph function; and The image is provided as input to the weighted graph function.

35. The method according to claim 33 or claim 34, wherein, The encoded information includes filter function parameters signaled by a signal, which are configured to enable the decoder to determine the filter by: The filter function parameters, which are notified by the signal, are used as the parameters of the filter.

36. The method according to any one of claims 33 to 35, wherein, The weighted graph and the filter are configured to be applied to the image as a step within the encoding loop or as a post-loop step to obtain the filtered image.

37. The method of claim 36, wherein, The encoding loop is an H.266 / VVC encoding loop.

38. The method according to claim 36 or 37, wherein, The weighted graph and the filter are configured to be integrated into an adaptive loop filter and applied to the derived partitions of the image to obtain the filtered image.

39. The method according to any one of claims 33 to 39, wherein, The bitstream is rate-distortion (RD) optimized based on estimated signaling rate and distortion after the weighted graph and the filter are applied to the image.

40. The method according to any one of claims 33 to 40, wherein, The image includes a luminance channel, a chroma channel, or both the luminance channel and the chroma channel, and The weighted graph and the filter are configured to be applied to the luminance channel, the chrominance channel, or both the luminance channel and the chrominance channel.

41. The method according to claim 40, wherein, The weighted graph and the filter will be applied to which of the luminance and chroma channels is predetermined, indicated by a signal in the encoded information, or configured to be inferred from the content of the image.

42. The method according to any one of claims 33 to 41, further comprising dividing the image into multiple partitions, in, The weighted graph and the filter are configured to be applied to one or more partitions of the image. The partition is indicated by a signal in the encoded information.

43. The method according to claim 42, wherein, The partition is signaled in the encoded information, including block segmentation, region segmentation criteria, or binarized weighted graph function signaled in the encoded information.

44. The method of claim 42 or 43 further includes determining a plurality of filters to be applied to the same image partition.

45. The method according to any one of claims 33 to 44, wherein, The filter is configured to resolve issues of ringing artifacts, blurring, and / or block artifacts in the image.

46. ​​The method according to any one of claims 33 to 45, wherein, Determining the weighted graph using the weighted graph function includes: Apply a weighted graph function to output a scalar weighted graph, where the scalar is binary, integer, or floating-point.

47. The method according to any one of claims 33 to 45, wherein, Determining the weighted graph using the weighted graph function includes: The application outputs a weighted graph function for a multidimensional weighted graph, where each element in the multidimensional weighted graph is binary, integer, or floating-point.

48. The method according to any one of claims 33 to 47, wherein, The weighted graph information of one or more channels of the obtained image is calculated using information from one or more channels of the obtained image as input.

49. The method according to any one of claims 33 to 48, wherein, A predefined set of weighted graph functions, and The encoded information is used to signal the weighted graph function to be used.

50. The method according to claim 49, wherein, The weighted graph function is parameterized.

51. The method according to any one of claims 33 to 50, wherein, The encoded information is used to notify multiple weighted graph functions via signals. Specifically, determining the weighted graph using the weighted graph function includes determining multiple weighted graphs using the plurality of weighted graph functions, and One or more filters are configured to be applied for each weighted graph notified by the signal.

52. The method according to claim 51, wherein, The parameters and filtering functions of the filter are signaled in the encoded information, predefined, configured to be inferred from the content of the video, or configured to be inferred from the encoded information.

53. The method according to any one of claims 33 to 52, wherein, The filter is a linear filter, and the shape of the filter is indicated or predefined in the bitstream.

54. The method according to claim 53, wherein, The linear filter is optimized using least squares optimization or rate-distortion (RD) optimization.

55. The method according to any one of claims 53 to 54, wherein, The linear filter is a parameterized linear filter.

56. The method according to claim 55, wherein, The parameterized linear filter is rate-distortion (RD) optimized relative to the minimum error at the output derived through least-squares optimization, iterative search, or exhaustive search.

57. The method according to any one of claims 33 to 52, wherein, The filter is a bilateral filter.

58. The method according to any one of claims 33 to 57, wherein, Determining the filter includes determining multiple filters. Each of the plurality of filters is configured to be applied in the bitstream by signaling or at a location indicated in the weighted graph.

59. The method according to any one of claims 33 to 58, wherein, The parameterized weighted graph is optimized together with the filtering function.

60. The method according to any one of claims 33 to 59, wherein, One or more filters are configured to be applied to partitions of the image based on block segmentation notified by signals in the encoded information.

61. The method according to any one of claims 33 to 59, wherein, One or more filters are configured to be applied to partitions of the image based on a derived region segmentation criterion.

62. The method according to any one of claims 33 to 61, wherein, The filter and weighted graph calculation parameters are encoded using quantization, prediction, and / or entropy coding schemes.

63. A computer-readable medium comprising computer-executable instructions stored thereon, which, when executed by a computing device, cause the computing device to perform the method according to any one of claims 33 to 62.

64. An encoder, comprising: One or more processors; and A computer-readable medium, including computer-executable instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of claims 33 to 62.

Images