Video coding using signal enhancement filtering.

Abstract

A video data processing method, performed by a decoder, is provided, comprising: decoding a bitstream to obtain video data and coding information, the coding information including weight map indication information; obtaining a picture block based on the video data; up-sampling the picture block; determining a weight map using the weight map indication information; and applying a signal enhancement filter to the up-sampled picture block together with the weight map, whereby the signal enhancement filter is applied with different weights to different regions of the picture block to obtain an enhanced picture block.

Description

[Technical Field]

[0001] The present application relates to the field of computer vision, in particular to the topic of video processing and video coding, and more particularly to a method, decoder, encoder, and computer-readable medium for video coding with signal enhancement filtering. [Background technology]

[0002] Current video coding standards, such as H.265 / HEVC (High Efficiency Video Coding) and H.266 / VVC (Versatile Video Coding), support spatial scalability of coded video streams. While this support is included in the scalable second version of HEVC (SHVC), VVC natively supports spatial scalability. Adaptively changing the resolution of coded video during the coding process is known as reference picture resampling (RPR) or adaptive resolution change (ARC) in VVC. Furthermore, multi-resolution coding and layered coding enable the resolution of coded video to be scalable. Therefore, the spatial resolution at which video is coded can be adaptively changed and no longer needs to be equal to the output or input resolution of the video. The benefit of this additional flexibility is that coding lower-resolution video can potentially result in lower bitrates and reduced computational complexity, but at the cost of losing high-frequency information in the downsampling step.

[0003] Encoding video at a resolution lower than the original requires downsampling and upsampling steps in the signal processing chain. During the downsampling step, anti-aliasing filters are applied to prevent artifacts due to high frequency components in the image. During the upsampling process, interpolation filters are applied to reconstruct intensity values ​​at fractional sample positions.

[0004] In RPR, the resolution of the coded video stream may change adaptively. As a result, the encoder can code parts of the video stream at a lower resolution. RPR is applied in inter prediction whenever a picture uses a reference picture with a different resolution than the current picture. This step requires applying a resampling operation so that the reference picture blocks are mapped to the same spatial resolution as the current picture.

[0005] In multi-layer coding, video is coded at different resolution layers. In the first step, the video is coded at the lowest resolution layer. To generate the video stream for the next layer, the video is upsampled, potentially with residual coding, and further processing steps applied. This process may be applied multiple times based on the number of layers.

[0006] Finding the optimal high-resolution representation from a low-resolution picture is a key part of these coding schemes. One method is to apply a set of polyphase Finite Impulse Response (FIR) interpolation filters. While these filters provide an approximation of the high-resolution image content, they cannot recover information lost in the downsampling process and are limited to linear filtering operations. As a result, upsampled images are often blurry.

[0007] Image sharpening operations can improve image quality. However, linear high-pass filters often introduce artifacts such as overshoot and ringing. Furthermore, the distortion caused by downsampling and upsampling depends on the image content and the video coding quality (which is affected by the Quantization Parameter (QP) value). Summary of the Invention

[0008] SUMMARY OF THE INVENTION Embodiments of the present application provide methods, decoders, encoders, and computer-readable media for video encoding with signal enhancement filtering that overcome problems associated with conventional arrangements.

[0009] According to a first aspect, there is provided a computer-implemented method for processing video data, to be performed by a decoder, the method comprising: decoding a bitstream to obtain video data and encoding information, the encoding information including weighting map indication information; obtaining a picture block based on the video data; up-sampling the picture block; determining a weighting map using the weighting map indication information; and applying a signal enhancement filter to the up-sampled picture block together with the weighting map, whereby the signal enhancement filter is applied with different weights to different regions of the picture block to obtain an enhanced picture block.

[0010] In some embodiments, the signal enhancement filter comprises a linear filter optimized by a least squares optimization procedure.

[0011] In some embodiments, the weighting map comprises a plurality of weight values ​​each corresponding to a value of the upsampled picture block.

[0012] In some embodiments, the encoded information further comprises signal enhancement filter indication information, and the method further comprises decoding the bitstream to determine the signal enhancement filter.

[0013] In some embodiments, the filter parameters of the signal enhancement filter are explicitly signaled in the bitstream or are derived by the decoder from the video data of the bitstream.

[0014] In some embodiments, the signal enhancement filter instruction information instructs the signal enhancement filter to reuse one or more filter parameters stored in a filter buffer of the decoder.

[0015] In some embodiments, determining the weight map using the weight map indication comprises determining a weight map function using the weight map indication, and calculating the weight map by applying the weight map function to the upsampled picture block.

[0016] In some embodiments, the weight map indication includes a weight map identifier that identifies one of a plurality of pre-defined weight map functions.

[0017] In some embodiments, the weighting map indication comprises parameters of the weighting map function.

[0018] In some embodiments, the picture block is a predictive block, and obtaining the picture block based on the video data comprises obtaining the predictive block by performing a prediction operation using video data.

[0019] In some embodiments, the prediction operation is inter-prediction or intra-prediction.

[0020] In some embodiments, a residual is coded into the bitstream at the resolution of the upsampled picture block, and the method further comprises decoding the bitstream to determine the residual, and applying the residual to the enhanced prediction block.

[0021] In some embodiments, the picture block is a reference sample, and the method further comprises obtaining a prediction block by performing a prediction operation using the enhanced reference sample.

[0022] In some embodiments, the prediction operation includes inter prediction, the reference sample corresponds to a first picture of video data encoded in the bitstream, and the predictive block corresponds to a second picture of the video data encoded in the bitstream, the second picture being temporally separated from the first picture, and the first picture being encoded in the bitstream at a lower resolution than the second picture.

[0023] In some embodiments, the encoding information indicates application of multiple filters and multiple respective weighting maps to the picture block.

[0024] In some embodiments, the encoding information indicates the use of different weighting maps and / or signal enhancement filters for different picture blocks of a picture.

[0025] According to a second aspect, there is provided a computer-readable medium including computer-executable instructions stored thereon that, when executed by a computing device, cause the computing device to perform any of the methods described in the first aspect.

[0026] According to a third aspect, there is provided a decoder comprising one or more processors and a computer-readable medium comprising computer-executable instructions stored on the computer-readable medium that, when executed by the one or more processors, cause the one or more processors to perform any of the methods of the first aspect.

[0027] According to a fourth aspect, there is provided a video data processing method to be performed by an encoder, the method comprising the steps of obtaining original video data, and obtaining a downsampled version of the original video data. Video Data and obtaining the downsampled Tabi obtaining a picture block based on the original video data; upsampling the picture block; applying a signal enhancement filter together with a weighting map to the upsampled picture block, whereby the signal enhancement filter is applied with different weights to different regions of the picture block to obtain an enhanced picture block, thereby recovering the loss due to downsampling and upsampling of the original video data; Tabi encoding the video data and encoding information into a bitstream, the encoding information including weighting map indication information indicating the weighting map.

[0028] In some embodiments, the signal enhancement filter comprises a linear filter optimized by a least squares optimization procedure.

[0029] In some embodiments, the weighting map includes multiple weight values ​​each corresponding to a value of the upsampled picture block.

[0030] In some embodiments, obtaining the enhanced picture block comprises performing a rate-distortion optimization operation to determine the weighting map.

[0031] In some embodiments, performing the rate-distortion optimization operation includes iteratively obtaining an enhanced picture block by applying the signal enhancement filter with different weighting maps to the upsampled picture block until a particular weighting map obtains an enhanced picture block that is within a threshold similarity to a corresponding original picture block obtained from the original video data.

[0032] In some embodiments, the method further comprises performing the rate-distortion optimization operation to determine the signal enhancement filter.

[0033] In some embodiments, performing the rate-distortion optimization operation comprises iteratively obtaining enhanced picture blocks by applying different signal enhancement filters with different weighting maps to the upsampled picture block until a particular combination of at least one signal enhancement filter and weighting map obtains an enhanced picture block that is within a threshold similarity to a corresponding original picture block obtained from the original video data.

[0034] In some embodiments, the encoded information further comprises signal enhancement filter indication information.

[0035] In some embodiments, the filter parameters of the signal enhancement filter are explicitly signaled in the bitstream or are derived by a decoder from the video data in the bitstream.

[0036] In some embodiments, the signal enhancement filter instruction information instructs the signal enhancement filter to reuse one or more filter parameters stored in a filter buffer of a decoder.

[0037] In some embodiments, the weighting map is determined by determining a weighting map function using weighting map indication information and calculating a weighting map by applying the weighting map function to the upsampled picture block.

[0038] In some embodiments, the weight map indication includes a weight map identifier that identifies one of a plurality of pre-defined weight map functions.

[0039] In some embodiments, the weighting map indication comprises parameters of the weighting map function.

[0040] In some embodiments, the picture block is a predictive block, and obtaining the picture block based on the downsampled video data comprises obtaining the predictive block by performing a prediction operation using the original video data.

[0041] In some embodiments, the prediction operation is inter-prediction or intra-prediction.

[0042] In some embodiments, the method further comprises encoding a residual into the bitstream at the resolution of the upsampled picture block, and the method further comprises applying the residual to the enhanced prediction block.

[0043] In some embodiments, the picture block is a reference sample, and the method further comprises obtaining a prediction block by performing a prediction operation using the enhanced reference sample.

[0044] In some embodiments, the prediction operation includes inter-prediction, the reference sample corresponds to a first picture of video data encoded in the bitstream, and the predictive block corresponds to a second picture of the video data encoded in the bitstream, the second picture being temporally separated from the first picture, and the first picture being encoded in the bitstream at a lower resolution than the second picture.

[0045] In some embodiments, the encoding information indicates application of multiple filters and multiple respective weighting maps to the picture block.

[0046] In some embodiments, the encoding information indicates the use of different weighting maps and / or signal enhancement filters for different picture blocks of a picture.

[0047] According to a fifth aspect, there is provided a computer-readable medium including computer-executable instructions stored on the computer-readable medium, the computer-executable instructions, when executed by a computing device, causing the computing device to perform any of the methods of the fourth aspect.

[0048] According to a sixth aspect, there is provided an encoder comprising one or more processors and a computer-readable medium comprising computer-executable instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform any of the methods described in the fourth aspect.

[0049] Applying a signal enhancement filter with local weighting improves coding performance. Local weighting allows for the filter strength to be smoothly increased or decreased in localized regions. For example, weighting can be used to increase filter strength in edge regions while decreasing filter strength in regions where signal enhancement filters typically exhibit ringing. In such a configuration, an optimized filter may amplify high-frequency components without introducing significant ringing. For example, in image upsampling scenarios, the amplification of high-frequency components can be used to sharpen blurred edges. In some instances, adaptive filters can be used to address coding artifacts and different characteristics of video content.

[0050] These and other aspects of the present application will become more readily apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0051] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1A] 4 shows a flowchart of the operation of the decoder according to the first embodiment. [Figure 1B] 4 shows a flowchart of the operation of the encoder according to the first embodiment. [Figure 2A] FIG. 10 is a block diagram showing an exemplary operation of a decoder according to a modification of the first embodiment. [Figure 2B] FIG. 10 is a block diagram showing an exemplary operation of an encoder according to a modification of the first embodiment. [Figure 3] FIG. 10 is a block diagram showing an exemplary operation of a decoder according to a modification of the first embodiment. [Figure 4A] FIG. 10 is a block diagram illustrating an exemplary operation of an encoder and a decoder according to an exemplary implementation of the second embodiment. [Figure 4B] 10 shows a flowchart of the operation of a decoder according to the second embodiment. [Figure 4C]10 shows a flowchart of the operation of the encoder according to the second embodiment. [Figure 5A] FIG. 10 is a block diagram illustrating an exemplary operation of a decoder according to an exemplary implementation of the third embodiment. [Figure 5B] FIG. 10 is a block diagram illustrating an exemplary operation of an encoder according to an exemplary implementation of the third embodiment. [Figure 6A] 11 shows a flowchart of the operation of a decoder according to the third embodiment. [Figure 6B] 10 shows a flowchart of the operation of the encoder according to the third embodiment. [Figure 7] 1 shows a schematic diagram of a decoder according to various embodiments; [Figure 8] 1 shows a schematic diagram of an encoder according to various embodiments; DETAILED DESCRIPTION OF THE INVENTION

[0052] DETAILED DESCRIPTION OF THE INVENTION The technical solutions in the embodiments of the present application are described clearly and completely below with reference to the accompanying drawings.

[0053] These technical solutions are applicable to H.265 / HEVC or H.266 / VVC video coding systems, and in particular to the performance of RPR, ARC, multi-resolution coding, and multi-layer coding. However, it should be understood that these technical solutions are also applicable to any other video coding system that includes upsampling. Furthermore, although these principles are primarily described with reference to video processing, they are also applicable to other data formats, such as image processing and audio processing.

[0054] In the embodiments, "video" refers to one or more pictures. In other words, a video may include one picture or multiple pictures. A picture may also be called an "image."

[0055] An "encoder" is a device that can encode data into a bitstream, and a "decoder" is a device that can decode a bitstream to obtain the encoded data or an approximation of the encoded data. A "bitstream" consists of a sequence of bits.

[0056] "Intra-prediction" and "inter-prediction" are two prediction operations that a decoder can use within the HEVC and VVC frameworks to process a received bitstream to obtain an original signal. In embodiments, "original signal" or "original video" is used to refer to data before it is encoded by encoder 20. In embodiments, reference samples may refer to spatially and / or temporally separated picture data used to predict a picture (or region of a picture). Intra-prediction and inter-prediction operations are also used by encoders to make rate-distortion decisions.

[0057] In particular, intra-prediction involves spatially predicting data within a single picture without reference to other (temporally distant) pictures. In other words, data from a first region of a picture is used to predict data from other regions of the same picture, but without relying on other temporally distant pictures. In this specification, the data from the first region of a picture is considered a "reference sample."

[0058] Inter prediction involves predicting data between multiple temporally separated pictures. In other words, data from a first region of a first picture is used to predict data from a second region of a second picture. The first and second regions may or may not be spatially separated. In this specification, the data from the first region of the first picture is considered a "reference sample." It is further noted that inter prediction may use multiple reference regions from different pictures at once for a single prediction operation.

[0059] In embodiments, a "residual" may refer to a value obtained based on an original value of a region of a picture and a predicted value of that region (eg, the difference between the original value and the predicted value).

[0060] In the embodiments, a "block" may refer to a portion of a picture. For example, a picture may be divided into two or more blocks. However, this is by way of example only. If the picture is not divided, a "block" may refer to the entire picture.

[0061] A "signal enhancement filter" may refer to a filter that functions to enhance a signal, particularly an upsampled signal. Generally, in the described embodiments, the signal enhancement filter is a filter configured to reduce edge blurring (i.e., a filter that sharpens picture blocks). However, embodiments are not limited in this regard, and in other embodiments, the signal enhancement filter may be configured to provide alternative or additional signal enhancements, such as removal of blocking and / or ringing artifacts.

[0062] Fig. 1A shows a flowchart of the operation of a decoder 10 according to the first embodiment. Fig. 1B shows a flowchart of the operation of an encoder 20 according to the first embodiment.

[0063] The flowchart of FIG. 1A begins at step 101, where the decoder 10 decodes the bitstream to obtain video data and encoding information, the encoding information including weighting map indication information.

[0064] In step 102, the decoder 10 obtains a picture block based on the video data. In this embodiment, the video data is a downsampled version of the original video data. Video DataIn other words, the video data includes a low-resolution version of the original video data. Therefore, step 102 includes obtaining data corresponding to a picture block in the video data.

[0065] However, embodiments are not limited thereto, and the video data may include any data that the decoder 10 can use to obtain a picture block (e.g., by performing a prediction operation, such as intra-prediction or inter-prediction, using the video data).

[0066] In step 103, the decoder 10 upsamples the picture block. In this embodiment, step 103 involves applying a set of polyphase FIR interpolation filters to reconstruct intensity values ​​at fractional sample positions, thereby increasing the resolution of the picture block. However, embodiments are not limited to this, and other upsampling methods can be applied instead. In particular, there are many different methods that can be used to perform the interpolation. Essentially, upsampling presents a problem in that fractional sample positions need to be interpolated. This includes, but is not limited to, bilinear interpolation, bicubic interpolation, nearest neighbor interpolation, Lanczos interpolation, etc.

[0067] In this embodiment, step 103 includes upsampling the picture block to the resolution of the original video data, although the embodiment is not limited thereto and in other embodiments, the picture block may be upsampled to a resolution lower than the resolution of the original video data.

[0068] In step 104, the decoder 10 determines a weighting map using the weighting map indication information. In this embodiment, the weighting map indication information explicitly signals values ​​of a weighting map having a resolution corresponding to the resolution of the upsampled picture block. In one example, if the size of the upsampled picture block is 5x5, the weighting map indication information includes 25 values, one for each position of the 5x5 block. In this example, in step 104, the decoder 10 determines a weighting map having these 25 values.

[0069] However, embodiments of the present application are not limited in this respect. For example, in some embodiments, the resolution of the weighting map may be smaller than the resolution of the upsampled picture block. In such cases, a single value in the weighting map may correspond to multiple values ​​of the picture block. Furthermore, in other embodiments, the weighting map may be determined in other manners, as described below.

[0070] In step 105, the decoder 10 applies a signal enhancement filter (SEF) together with a weighting map to the upsampled picture block, whereby the signal enhancement filter is applied with different weights to different regions of the picture block to obtain an enhanced picture block.

[0071] Thus, step 105 includes using a weighting map so that the signal enhancement filter is applied with different strengths to different regions of the upsampled picture block.

[0072] In this embodiment, the signal enhancement filter is a pre-set filter for enhancing the upsampled picture, although embodiments are not limited in this respect. For example, in other embodiments, the encoding information further indicates the particular signal enhancement filter to use and / or particular parameters of the signal enhancement filter.

[0073] In this embodiment, the signal enhancement filter is a sharpening filter configured to sharpen blurry edges, although embodiments are not so limited and any suitable signal enhancement filter may be used instead.

[0074] After step 105, the enhanced picture blocks may be used for any desired purpose. In one example, the decoder 10 then displays the enhanced picture blocks to a viewer. In another example, the decoder 10 stores the picture blocks for later use. In another example, the decoder 10 transmits the picture blocks to an external device for display.

[0075] A complementary method may be performed by the encoder 20 to encode the bitstream provided to the decoder 10. Figure 1B shows a flow chart of the operation of the encoder 20 according to the first embodiment.

[0076] In step 201, the encoder 20 acquires original video data. For example, the encoder 20 may receive the original video data from an external server via a communication network (e.g., the Internet). However, there is no limitation on the method of acquiring the original video data in the embodiment.

[0077] In step 202, the encoder 20 generates a downsampled version of the original video data. Video Data In this embodiment, step 202 includes encoder 20 downsampling the original video data to obtain low-resolution video data. However, embodiments are not limited in this respect. For example, in some embodiments, encoder 20 may instead receive both the original video data and the downsampled video data from an external source, e.g., an external server, via a communications network (e.g., the Internet).

[0078] In step 203, the encoder 20 Tabi1A. In this embodiment, step 203 is performed in the same manner as step 102 in FIG. 1A. In other words, in this embodiment, the downsampled video data comprises a lower-resolution version of the original video data. Therefore, step 203 includes a step of obtaining data corresponding to a picture block in the video data.

[0079] However, embodiments are not limited thereto. For example, in other embodiments, step 203 may instead include performing a prediction operation using the downsampled video data, such as intra-prediction or inter-prediction, to obtain the picture block.

[0080] In step 204, encoder 20 upsamples the picture block. Step 204 is performed in a manner corresponding to step 103 of Figure 1A, and will not be described in detail here for the sake of brevity.

[0081] In step 205, the encoder 20 applies a signal enhancement filter together with a weighting map to the upsampled picture block, whereby the signal enhancement filter is applied with different weights to different regions of the picture block to obtain an enhanced picture block.

[0082] As described above in connection with FIG. 1A, in this embodiment, a single pre-set signal enhancement filter is used. To determine a weighting map to enhance a picture block, step 205 performs a rate distortion (R) calculation, which involves iteratively applying the signal enhancement filter to the upsampled picture block with different weighting maps. D ) optimization process. For each application, the average difference in values ​​between the resulting picture block and the corresponding picture block from the original video data is determined. This cycle continues until a stopping criterion is met.

[0083] A first example of a suitable stopping criterion is that a particular weighting map causes the mean absolute difference (or mean squared difference) of values ​​between the resulting picture block and the corresponding picture block from the original video data to be less than a predetermined threshold difference. A second example of a suitable stopping criterion is that the mean absolute difference (or mean squared difference) of values ​​between the weighting map of a current iteration of the iterative process and the weighting map of a previous iteration of the iterative process to be less than a second predetermined threshold difference. Because the first example of a suitable stopping criterion directly measures output quality, it can be assumed to result in a higher final image quality than the second example. However, the second example ensures that the iterative process does not require excessive computation time. In some embodiments, both of these examples are used, and the iterative process stops when either one of these two stopping criteria is met.

[0084] In this embodiment, the signal enhancement filter is determined at the encoder based on the concept of a Wiener filter. Other signal enhancement filters are linear filters optimized at the encoder by a least-squares optimization procedure (i.e., a linear filter that minimizes the squared error between the filtered signal and the ground truth signal). Of course, in some embodiments, additional boundary constraints are placed on the determination of the signal enhancement filter, such as filter shape or filter coefficients that must be equal.

[0085] However, although the present embodiments are described with reference to a signal enhancement filter based on the Wiener filter concept, the embodiments are not limited in this respect and other types of filters may instead be used as sharpening filters, such as filters based on Sobel filters or unsharp masking filters. Other non-linear options include bilateral filters, diffusion filters, and adaptive loop filters (ALFs).

[0086] The method of FIG. 1B then proceeds to step 206, where encoder 20 generates a downsampled Tabi The video data and the coding information are encoded into a bitstream, and the coding information includes weight map indication information that indicates a particular weight map.

[0087] This method improves the quality of the upsampled picture, especially by reducing the distortions caused by low-resolution coding: the loss of high-frequency information and distortion caused by video coding.

[0088] In particular, local weighting provided by a weighting map can be applied to smoothly increase or decrease the strength of the filter in local regions. In some instances, the weighting map can provide weighting that increases the filter strength in edge regions but decreases it in areas where ringing typically occurs. In such a setting, the optimized filter can amplify high-frequency components without introducing significant ringing. This is particularly useful in image upsampling scenarios, where amplification of high-frequency components must be utilized to sharpen blurry edges.

[0089] In some cases, a signal enhancement filter can be applied to reduce edge blurring caused by the upsampling operation, and an appropriate weighting map can be extracted, e.g., to identify areas where ringing may occur and assign a lower weight to those areas.

[0090] Ringing is typically produced by the quantization of high-frequency components during the encoding process. Therefore, it can be assumed that ringing occurs most frequently around strong edges and corners, since these typically lead to frequency responses that contain high-frequency components. In one example, an edge detector can be used to find the strongest edges in a picture. All samples that are a certain distance from an edge and within the same block can be considered candidates for ringing. However, it should be understood that this is only one example of how ringing can be identified. Other methods can be used in addition or instead.

[0091] Similarly, the filter can be applied to other types of errors besides edge blurring, making the approach very flexible. Examples include blocking and ringing artifacts.

[0092] In some examples, the optimization described with reference to step 205 of FIG. 1B involves iterating between signal enhancement filter and weighting map function parameters. For example, setting starting weighting map parameters, then optimizing filter parameters based on the current weighting map. Next, optimizing weighting map parameters based on the found filter parameters, and so on. Of course, this is a basic form of optimization procedure. In some cases, additional bounding constraints may be set, e.g., to determine the best filter and weighting map in terms of picture quality, but also to achieve the lowest possible coding rate. This can be achieved by introducing these conditions into both of these individual optimizations and selecting the starting point for the next iteration while also considering rate costs. More generally, additional simplifications that limit computational costs may be introduced.

[0093] It should be understood that there are many ways to perform the rate-distortion optimization described with reference to step 205 of FIG. 1B in variations of this embodiment. One example involves optimizing the weighting maps of each filter according to the following case description: whereby the residual is recalculated based on the results of the previous filter. A second example is the joint optimization of the weighting maps and the signal enhancement filter. In this case, the optimization procedure depends heavily on the weighting map function. However, the most common optimization involves optimizing the signal enhancement filter over all possible weighting maps and selecting the best option in terms of RD cost. To solve this efficiently, simplifications can be applied to find a sufficiently good solution. In the case of a linearly optimizable parametric weighting map, it is possible to perform an iterative method to optimize the filter parameters based on the current weighting map. Then, the weighting map parameters are optimized based on the current filter parameters, and so on. In other words, the signal enhancement filter is a filter with one or more parameters that depend on the weighting map. This can be beneficial because jointly optimizing both would be computationally complex. Assuming one of the components is fixed at each optimization step simplifies the optimization of the remaining parameters.

[0094] In this embodiment, the weighting map provides linear weighting for the signal enhancement filter. However, embodiments are not limited in this respect, and in other embodiments, the values ​​of the weighting map can instead modify the filtering procedure itself. For example, the signal enhancement filter may be parametric. In this case, the frequency response of the edge enhancement filter may depend on the local weighting map parameters. For example, the sigma value in an unsharp masking filter (a type of sharpening filter) may depend on the weighting parameters. This means that the way the filter operates, or more specifically, the function of the filter, is parametric and does not necessarily depend linearly on the weighting map. Another example is a filter that thins (sharpens) edges by distorting an image. The strength of the distortion may depend on the current weighting map values.

[0095] In this embodiment, the weighted signal enhancement filter is applied after upsampling and before any other operations. In particular, for example, the signal enhancement filter is applied before the addition of the residual signal. This position in the processing chain has been shown to be effective. However, embodiments are not limited to this particular order, and in other embodiments, the weighted signal enhancement filter may additionally or alternatively be applied at other processing steps.

[0096] In this embodiment, the signal enhancement filter is a pre-set filter for enhancing the upsampled picture, although embodiments are not limited in this respect. For example, in other embodiments, the encoding information may further indicate the particular signal enhancement filter to use and / or particular parameters of the signal enhancement filter.

[0097] Next, a first modification of the first embodiment will be described, in which the weighting map indication information and the weighting map are determined in a manner different from that described above.

[0098] As mentioned above, in the first embodiment, the weighting map indication information explicitly signals the value of a weighting map having a resolution corresponding to the resolution of the upsampled picture block. However, the embodiment is not limited to this. In a first variant of this embodiment, the weighting map function is pre-defined (e.g., stored in a memory of the decoder 10).

[0099] In this first variant, the weighting map indication indicates the parameters (or coefficients) of a preset weighting map to use. In this first variant, step 104 of Figure 1A includes the decoder 10 applying a weighting map function (with parameters encoded in the weighting map indication in the bitstream) to the upsampled picture block to determine the weighting map.

[0100] An example of a weighting map function is the Sobel Magnitude Map given by equation (1).

number

[0101] Another example of a weighting map function is the inverse of this function as shown in equation (2).

number

[0102] However, it should be understood that these are merely examples and that other weighting map functions may be used in addition or instead.

[0103] This method of applying a single weighting map function to a picture block (containing multiple samples) applies the weighting map function multiple times to multiple samples of the picture block, resulting in a weighting map that is calculated in the decoder 10 and that depends on the values ​​of the picture block.

[0104] As can be seen, in this variant, the calculation of the weighting maps is performed in the decoder and there is no need to code the weighting maps into the bitstream, thereby reducing coding costs. Furthermore, since the most appropriate weighting map depends on the picture content, calculating the weighting maps by applying a function to the picture blocks themselves ensures that the decoder can calculate the most appropriate weighting map.

[0105] Next, a second modification of the first embodiment will be described, in which the encoded information further includes parameters used for the signal enhancement filter.

[0106] In a second variant of the first embodiment (which can optionally be combined with the first variant described above), the encoding information further includes filter parameters of a signal enhancement filter. In other words, this second variant still involves the use of a pre-set signal enhancement filter as described above in relation to the first embodiment, but the encoder 20 can indicate which parameters (or coefficients) to use when applying the signal enhancement filter. In other words, the signal enhancement filter is adaptive.

[0107] In this variation, the filter parameters may be explicitly signaled and derived from the video data, or encoder 20 may be instructed to reuse previously signaled coefficients.

[0108] Regarding the point that the filter parameters are derived from the video data, if high-resolution and low-resolution video information is available, it is possible to estimate a filter that is approximately appropriate for the given data. This is the case for a picture immediately after its resolution has been changed to low. However, in this case, it is useful to restrict the filter to a region where it can be assumed that the motion between the high-resolution and low-resolution pictures can be compensated for and that the shape and orientation of objects will not change significantly.

[0109] Additionally, previously decoded video / picture information may be reused to find more optimized filtering parameters. This refers to the idea of ​​using information from multiple pictures to get a more detailed estimate of the location of edges or artifacts. For example, if an edge in a previous picture is discovered and ringing artifacts appear in the next picture that were not previously present, this information can be incorporated into the filter to optimize it so that these artifacts are not emphasized, or in the best case, eliminated. The use of temporal information can aid this estimation and lead to a more accurate estimate.

[0110] One option is to signal that the parameters of the previous filter are fully reused. A second option is to partially reuse information from the previous filter. This could be, for example, weight map parameters or a subset of the filter coefficients.

[0111] Coding costs can be reduced because only the filter coefficients (and / or instructions to reuse previous coefficients) need to be signaled, rather than the details of the signal enhancement filter function.

[0112] In a third variant, the decoder 10 stores a filter buffer that stores previously used signal enhancement filter parameters. Based on this, the encoder 20 has the option of simply including in the encoding information an indication to use one or more previously used filter parameters, without having to include the specific parameters themselves, thereby reducing encoding costs.

[0113] Similarly, in a fourth variant, a weighting map buffer is stored in the decoder 10 to store previously used weighting map function parameters. Based on this, the encoder 20 has the option of simply including in the encoding information an indication to use one or more previously used weighting map function parameters, without having to include the particular parameters themselves.

[0114] Of course, although these variations are described individually above, this is for ease of explanation only, and any or all of these variations may be combined.

[0115] To illustrate this, an exemplary embodiment of a combination of the first, second, and third variants of the first embodiment will be described with reference to Figures 2A and 2B, where Figure 2A shows a block diagram illustrating an exemplary operation of decoder 10 and Figure 2B shows a block diagram illustrating an exemplary operation of encoder 20.

[0116] As shown in Figure 2A, the decoder 10 obtains coded information 1001 and an upsampled picture block 1002. The upsampled picture block 1002 is obtained in the manner described with reference to steps 102-103 in Figure 1A.

[0117] As described above in relation to the first modification, the coded information 1001 is coded by a preset weighting map function f w-map 1003. When such parameters are applied, the weighting map function f w-map1003 is applied to the upsampled picture block 1002 to obtain a weighting map with a resolution corresponding to the resolution of the upsampled picture block 1002 .

[0118] As described above in relation to the second and third variants, the coding information is transmitted to the filter buffer 1005 via parameters and / or preset signal enhancement filters f filter1 1004. The signal enhancement filter f filter1 1004 is then applied to the upsampled picture block 1002 along with the weighting map to form a signal enhancement filter f filter1 1004 is applied with different weights to different regions of the upsampled picture block 1002, thereby obtaining an enhanced picture block 1006.

[0119] In this example, a complementary method is performed by the encoder 20, as shown in Figure 2B. In a complementary manner to the block diagram of Figure 2A, it can be seen that the original video (or picture block) 1007a and the upsampled picture block 1002a are used as inputs. This is followed by the weighting map function f w-map 1003a, a filter buffer 1005a, and an optimizer 1008a are performed to generate coded information 1001a. In particular, the encoder 20 takes as input the original video data 1007a (or simply the original picture block) and the upsampled picture block 1002a, and performs rate-distortion optimization using different weighting map function parameters and / or signal enhancement filter parameters in the optimizer 1008a process. This process is similar to that described with reference to step 205 of FIG. 1B, except that the variables selected by the encoder 20 are the weighting map parameters and the signal enhancement filter parameters. Thus, an iterative process is performed in which different weighting map parameters and signal enhancement filter parameters are applied until a stopping criterion is met.

[0120] A first example of a suitable stopping criterion is that the mean absolute difference (or mean squared difference) of values ​​between the resulting picture block and the corresponding picture block from the original video is less than a first predetermined threshold difference. A second example of a suitable stopping criterion is that the mean absolute difference (or mean squared difference) of parameters (i.e., weighting map function parameters and / or signal enhancement function parameters) between the current iteration of the iterative process and the previous iteration of the iterative process is less than a second predetermined threshold difference. Because the first example of a suitable stopping criterion directly measures output quality, it can be assumed that the final image quality will be higher than the second example. However, the second example ensures that the iterative process does not require excessive computation time. In some embodiments, both of these examples are used, and the iterative process stops when either one of these two stopping criteria is met.

[0121] Once the specific parameter combinations have been determined, they are output to the coded information 1001 of the bitstream.

[0122] The freedom to vary both the weight parameters and the filter parameters improves the encoder's ability to enhance image quality. For example, the signal enhancement filter may be a linear filter. Generally, such linear filters cannot be used to solve nonlinear problems. For example, edge sharpening and super-resolution problems are generally nonlinear problems, which limits the ability of such signal enhancement filters to solve these problems. However, the additional use of the weighting map overcomes this problem. In particular, even if the weighting map function used to calculate the weighting map is also linear, this combined use of two linear functions (i.e., the weighting map function and the signal enhancement filter function) makes it possible to solve these nonlinear problems.

[0123] It can be seen that these variations include the use of weighting map calculation parameters and filter parameters of an adaptive signal enhancement filter, whereby the weighting map is estimated at the decoder side and applied in the filtering process together with the filter to enhance the current picture block, thus making use of weighting maps calculated at the decoder side using image processing.

[0124] In addition to what is described above with respect to the first embodiment variant, the embodiments are not limited to the use of a single preset weighting map function and / or a single preset function. In some embodiments, multiple preset weighting map functions and / or signal enhancement filters are available (e.g., stored in memory of decoder 10). In such variants, in addition to the information described above with respect to the first and / or second variants, the encoding information further includes one or more identifiers that identify which weighting map function and / or signal enhancement filter to apply.

[0125] Thus, in some embodiments, multiple functions for the calculation of weighting maps are pre-defined, and only the calculation parameters and weighting map identifiers need to be signaled in the bitstream.Furthermore, in some embodiments, multiple signal enhancement filters are pre-defined, and only the calculation parameters and signal enhancement filter identifiers need to be signaled in the bitstream.

[0126] Although the first embodiment has been described with reference to a single signal enhancement filter and a single weighting map, embodiments are not so limited, and in some variations of the first embodiment, multiple signal enhancement filters and weighting maps are used instead.

[0127] In one such variation, multiple signal enhancement filters with multiple respective weighting maps are applied to a single picture block (i.e., multiple signal enhancement filters with multiple respective weighting maps are applied to the same region of a picture), and in this variation, signal enhancement filter identifiers are coded into the bitstream along with weighting map indication information for each identified signal enhancement filter.

[0128] 3 shows a block diagram illustrating an exemplary operation of the decoder 10 according to this variant of the first embodiment. As shown in FIG. 3, compared with the block diagram of FIG. 2A, based on the coding information 1001′ and the filter buffer 1005′, multiple weighted signal enhancement filters are determined and applied sequentially to the upsampled picture block 1002′ to obtain an enhanced picture block 1006′.

[0129] In the configuration of FIG. 3, the weighted signal enhancement filter is applied to the function f in the same manner as described with reference to FIG. 2A. w-map1 1003'~f w-mapN 1009' and f filter1 1004'~f filterN 1010', although this is by way of example only and embodiments are not limited in this respect. For example, in an alternative variation, the values ​​of each weighting map may instead be directly encoded into the bitstream in a manner corresponding to that described above with reference to FIGS. 1A and 1B.

[0130] In some embodiments, different signal enhancement filters (and corresponding weighting maps) are applied to different picture blocks. Thus, in some embodiments, each filter may be limited to a specific block, which also needs to be taken into account in the optimization. Thus, filters may be applied to picture blocks (or "partitions") according to rate-distortion criteria to take into account different local image distortion characteristics.

[0131] In this regard, the error statistics are assumed to depend on their spatial location. For example, if we consider a simple picture with a scene in the bottom half and sky in the top half, dividing the picture into these very different regions and optimizing individual filters for those regions will lead to better overall results in terms of rate-distortion.

[0132] To indicate which signal enhancement filters to use for a particular picture block, in some embodiments the coding information includes local on / off flags for different filters, and optionally weighting map functions (or local on / off flags for those functions) and parameters for those functions.

[0133] 3, a series of weighted signal enhancement filters (e.g., adaptive loop filters) each calculate an offset map, and then the calculated offsets are added to the upsampled picture block to obtain an enhanced picture block.

[0134] However, in other variants, other implementations are possible. For example, instead of applying it independently, a weighted adaptive loop filter can be applied to the output of the previous filter. If this block is considered as the optimization target, the optimization procedure of Fig. 2B can still be applied, and the optimal filter coefficients can be obtained from a series of weighted map functions.

[0135] Note that in some embodiments, the weighting map parameters are optimized together with the filter coefficients, and the order and number of filters are separate optimization parameters for rate-distortion optimization. In some embodiments, the filtering function of the weighting map may be dynamically signaled or may be a (possibly parametric) function applied to the decoded and upsampled picture blocks.

[0136] Although the variations of the first embodiment are described above individually, this is for ease of explanation only. Any or all of these variations may be combined.

[0137] A general overview of the first embodiment and applicable variants is given below. By applying a weighting map together with a signal enhancement filter, it is possible to improve the quality of the upsampled picture. Applying a weighted signal enhancement filter before the addition of the residual signal (and before any other processing steps after upsampling) has been shown to be an effective position in the processing chain. However, this is not necessary, and a weighted signal enhancement filter may also be applied at other processing steps.

[0138] As described above, encoder-side estimation of RD-optimized filter parameters is performed to find the best filter settings. The encoder 20 needs to estimate the best weighting map (or set of weighting maps) to use for the current picture (or picture block).

[0139] In some aspects, the weighting map calculation parameters and possibly the filter coefficients are estimated. Each filter may be specific to a particular block, which can also be taken into account in the optimization.

[0140] The filter parameters may be explicitly signaled, may be derived from the video data, or the encoder 20 may indicate the reuse of previously signaled coefficients. Additionally, previously decoded video information may be reused to find further optimized filtering parameters.

[0141] The signal enhancement filtering process is a two-step procedure. The first step is to estimate (or determine / calculate) the weighting map. In some aspects, the calculation of the weighting map can be performed by any function applied to the upsampled picture blocks.

[0142] In some aspects, the encoder 20 may provide parametric information for the weighting map calculation to select the calculation parameters. This is advantageous because the most appropriate weighting map depends on the image content. The result is a weighting map that provides spatial information about the filter weights.

[0143] The weighting map is used in the next step, where the picture block is filtered using a signal enhancement filter, with the local strength of the filtering operation being given by the weighting map. The exact implementation of the strength modification by the weighting map depends on the implementation method and may, for example, be a linear weighting of the offset calculated by the filter, or may modify the filtering procedure itself.

[0144] Applying the weighted signal enhancement filter in the manner described above serves to reduce distortions caused by low-resolution video encoding. These are the loss of high-frequency information and distortions caused by video encoding. A default upsampling filter can be used for the initial resolution change, and then the weighted signal enhancement filter described above is applied independently. In other words, this weighted signal enhancement filter does not modify the existing resampling (or upsampling) process, but rather adds / improves an enhancement step.

[0145] For example, the steps involved in generating an enhanced upsampled picture block at the decoder side are outlined in Figures 1A and 2A. At the decoder side, an upsampled image (e.g., already upsampled by a default upsampling process) is obtained. In addition, coding information is obtained that can specify aspects such as the operating mode.

[0146] In some aspects, the encoding information includes local on / off flags for different filters, weighting map functions, and parameters for those functions. Additionally, an encoding of the filter coefficients is transmitted. In some aspects, this encoding utilizes filter coefficients previously transmitted from a filter buffer to reduce encoding costs.

[0147] In some aspects, after the filter parameters are decoded, a weighting map is calculated by applying a weighting map function to the picture block. The weighting map function may be any function that maps an input picture block to an output picture block, and need not necessarily be linear. The filter receives the weighting map and the upsampled image as input. The result of the filter operation is an enhanced picture block.

[0148] Depending on the configuration, in some aspects, multiple filters are applied to a single picture. These filters may be applied to partitions (or "blocks") of the picture according to rate-distortion criteria to account for different local image distortion characteristics. Furthermore, multiple filters can be applied to the same image region (or "block" or "partition") with different weighting maps or parameters to reduce different types of artifacts in that image region.

[0149] The inputs of the (rate-distortion) optimization operation in encoder 20 are the upsampled picture block (or video) and the original / ground truth picture block (or video). In some implementations, an optimizer generates weighting maps from the upsampled picture block (or video) using a set of candidate weighting map functions. Then, given the set of weighting maps and previously decoded filters (i.e., previously used filters), signal enhancement filter parameters are optimized.

[0150] Based on the results of the RD optimization, further weighting maps with different computational parameters may be generated, and the RD optimization may be iteratively restarted with a different set of weighting maps until a sufficiently good RD point is found or a stopping criterion is met.

[0151] In some aspects, a set of multiple weighting maps is determined, and the decoder 10 selects a respective set of signal enhancement filters to apply and a picture partition (to divide the picture into blocks).

[0152] Additionally, in some aspects, signal enhancement filter parameters are optimized, selectively considering reusing parameters from a previous configuration of the filter buffer. There are several options for reusing filter parameters. One option is to signal that the previous filter parameters are fully reused. A second option is to partially reuse information from the previous filter. This could be, for example, weight map parameters or a subset of filter coefficients.

[0153] An exemplary implementation of the first embodiment includes the use of a linear filter. In this case, the encoder 20 determines the filter coefficients by least-squares optimization. It is assumed that weighting is applied by multiplying a weighting map to the filtered image. It is further assumed that the output is calculated by adding the weighted and filtered picture to the input picture. Note that even in this case where a linear filter is applied to the upsampled picture, the multiplication with the weighting map allows the overall system to solve nonlinear problems.

[0154] 4A to 4C relate to a second embodiment, in which a weighted signal enhancement filter is applied in an adaptive resolution change (ARC) scenario.

[0155] In particular, video coding systems may allow adaptive resolution change of coded video sequences, where the video sequence is temporarily coded at a resolution lower than the output resolution. To improve coding efficiency, inter-prediction between pictures of different resolutions is still performed. Therefore, upsampling of reference pictures may be required so that they match the resolution of the blocks being predicted.

[0156] In a second embodiment, a weighted signal enhancement filter is applied to the upsampled reference samples.

[0157] Figure 4A shows a block diagram illustrating exemplary operations of the encoder 20 and the decoder 10 according to an exemplary implementation of the second embodiment, Figure 4B shows a flowchart of the operation of the decoder 10 according to the second embodiment, and Figure 4C shows a flowchart of the operation of the encoder 20 according to the second embodiment.

[0158] As shown in Figure 4A, a weighted signal enhancement filter (SEF) 3001 can be integrated into a hybrid video coding system. Figure 4A shows an encoder 20 (overall image) and a decoder 10 (gray rectangular area). Except for the inclusion of the weighted signal enhancement filter 3001, the blocks in Figure 4A represent a simplified diagram of a hybrid coding scheme that is very similar to HEVC / VVC (also known as H.265 / H.266).

[0159] Typically, switching from low resolution to high resolution results in a sudden increase in coding cost or a temporary degradation in video quality. This is due to the issue that low-resolution pictures lack some high-frequency information. Here, a weighted signal enhancement filter (SEF) 3001 can be applied to the reference picture from the decoded picture buffer after upsampling. This improves the quality of the reference picture for inter-prediction and reduces the coding cost for coding the residual. Furthermore, the same filter can be applied before the picture is presented to the viewer.

[0160] An embodiment of a weighted signal enhancement filter, such as signal enhancement filter 3001, is described in further detail with reference to FIGS. 4B and 4C.

[0161] FIG. 4B shows a flow chart of the operation of the decoder 10 according to an exemplary implementation of the second embodiment.

[0162] In step 301, the decoder 10 decodes the bitstream to obtain video data and coding information, where the coding information includes weighting map indication information. Step 301 roughly corresponds to step 101 in Figure 1A, and for the sake of brevity, a detailed description will be omitted here. Furthermore, in this embodiment, inter-prediction parameters used in inter-prediction are coded into the bitstream and used in inter-prediction by the decoder 10.

[0163] In step 302, the decoder 10 obtains a reference sample based on the video data. In this embodiment, the reference sample is data coded into the bitstream for a picture block of a first picture, and is used as a reference for inter-prediction of a second picture block of a second (temporally distant) picture.

[0164] In step 303, the decoder 10 upsamples the reference samples. Step 303 corresponds roughly to step 103 of Figure 1A and will not be described in detail here for the sake of brevity.

[0165] The decoder 10 uses the weighting map indication information to determine a weighting map in step 304. Step 304 corresponds roughly to step 104 of Figure 1A and will not be described in detail here for the sake of brevity.

[0166] In step 305, the decoder 10 applies a signal enhancement filter together with a weighting map to the upsampled reference sample, thereby obtaining an enhanced reference sample, where the signal enhancement filter is applied with different weights to different regions of the reference sample. Step 305 roughly corresponds to step 105 of Figure 1A and will not be described in detail here for the sake of brevity.

[0167] In step 306, the decoder 10 obtains a prediction block by performing a prediction operation using the enhanced reference sample. In particular, the decoder 10 performs inter prediction using the enhanced reference sample as a reference sample to obtain a prediction block.

[0168] After step 306, the predicted block can be used for any desired purpose. In one example, the decoder 10 then displays the predicted block to a viewer. In another example, the decoder 10 stores the predicted block for later use. In another example, the decoder 10 transmits the predicted block to an external device for display.

[0169] A complementary method can be performed by the encoder 20 to encode the bitstream provided to the decoder 10. Figure 4C shows a flow chart of the operation of the encoder 20 according to the second embodiment.

[0170] In step 401, the encoder 20 obtains the original video data. Step 401 corresponds roughly to step 201 of Figure 1B, and will not be described in detail here for the sake of brevity.

[0171] In step 402, the encoder 20 generates a downsampled version of the original video. Video Data Step 402 corresponds roughly to step 202 of Figure 1B, and for the sake of brevity, a detailed description will not be given here.

[0172] In step 403, the encoder 20 Tabi A reference sample is obtained based on the video data, where the reference sample is a first picture block of a first picture, and is used as a reference for inter prediction of a second picture block of a second (temporally distant) picture.

[0173] Encoder 20 upsamples the reference samples in step 404. Step 404 corresponds roughly to step 204 of Figure 1B and will not be described in detail here for the sake of brevity.

[0174] In step 405, the encoder 20 applies a signal enhancement filter together with a weighting map to the upsampled reference samples, where the signal enhancement filter is applied with different weights to different regions of the picture block to obtain enhanced reference samples to recover the loss due to downsampling and upsampling of the original video data. Step 405 roughly corresponds to step 205 of Figure 1B and will not be described in detail here for the sake of brevity.

[0175] In step 406, the encoder 20 obtains a prediction block by performing a prediction operation using the enhanced reference samples. Similar to step 405, step 406 includes a rate-distortion optimization operation in which inter-prediction parameters are determined and then encoded into the bitstream. However, embodiments are not limited thereto, and any form of inter-prediction may be performed instead.

[0176] In step 407, the encoder 20 TabiIn this embodiment, the decoder 10 encodes the video data and the coding information into a bitstream, and the coding information includes weighting map indication information that indicates a weighting map. This corresponds roughly to step 206 in FIG. 1B, and a detailed description thereof will be omitted. Furthermore, in this embodiment, inter-prediction parameters used in inter-prediction are coded into the bitstream and used in inter-prediction by the decoder 10.

[0177] Typically, switching from low resolution to high resolution results in a sudden increase in coding cost or a temporary decrease in video quality. This is due to the problem that low-resolution pictures lack some high-frequency information. However, in this second embodiment, a weighted signal enhancement filter (SEF) 3001 is applied to the reference picture after upsampling the reference picture. This improves the quality of the reference picture for inter-prediction and reduces the coding cost for coding the residual. Furthermore, the same filter (or other weighted signal enhancement filter) can be applied to the picture before it is presented to the viewer, further enhancing its quality.

[0178] It should be noted that all variations described in relation to the first embodiment are equally applicable as variations to this second embodiment. Examples include the use of a weighting map function to determine the weighting map, the use of a filter function whose parameters are coded into the bitstream, and the use of multiple functions / weighting maps.

[0179] Figures 5A-5B and 6A-6B relate to a third embodiment, in which a weighted signal enhancement filter is applied in a multi-resolution coding scenario.

[0180] In particular, in multi-resolution coding, a low-resolution version of the video is coded first in the lowest layer. The generated pictures of this layer are upsampled 4003 to the resolution of the next layer. The next layer may apply enhancements to the upsampled pictures, for example by adding a residual signal. The number of resolution layers may vary depending on the application.

[0181] In the third embodiment, the weighted signal enhancement filter is applied to the upsampled picture, in other words, the weighted signal enhancement filter is applied after the upsampling step 4003.

[0182] Figure 5A shows a block diagram illustrating an exemplary operation of a decoder according to an exemplary embodiment of the third embodiment. Figure 5B shows a block diagram illustrating an exemplary operation of an encoder according to an exemplary embodiment of the third embodiment. Figure 6A shows a flowchart of the operation of a decoder according to the third embodiment. Figure 6B shows a flowchart of the operation of an encoder according to the third embodiment.

[0183] As shown in Figure 5A, a decoder first receives a bitstream 4001. A low-resolution version of a picture block is obtained in layer 0 4002, e.g., through base layer coding followed by motion compensation and intra prediction. This low-resolution version is then upsampled 4003 and converted to the resolution of the next layer. The next layer (i.e., layer 1) 4005 applies further enhancement to the upsampled picture, e.g., by adding a residual signal, resulting in a final picture block 4006. Although only two layers are shown in Figure 5A, embodiments of the present application are not limited in this respect. The number of resolution layers may vary depending on the application.

[0184] As an example, the processing for layer 0 may be very similar to single-layer coding. For example, the same scheme as shown in FIG. 4A may be used. In some examples, the coding for layer 1 is similar. However, in this case, the low-resolution (i.e., layer 0) video stream may be used to predict the high-resolution video stream (layer 1). This allows additional prediction modes to be used. In this case, it may be determined, for example, whether (inter) predicting blocks from a previous high-resolution picture is more optimal than using an upsampled low-resolution picture.

[0185] Figure 5B shows a complementary block diagram of the operation of the encoder. In a complementary manner to the diagram of Figure 5A, it can be seen that this uses input video (or picture block) 4007a as input, followed by downsampling 4008a, layer 0 processing 4002a, upsampling 4003a, application of a weighted signal enhancement filter 4004a, layer 1 processing 4005a, and multiplexing 4009a to produce bitstream 4001a. In particular, the encoder first receives input video 4007a. Then, a downsampling operation 4008a is performed on the input video 4007a, the result of which is used in layer 0 processing 4002a (e.g., motion compensation and intra prediction followed by base coding).

[0186] In some embodiments, a low-pass (or "anti-aliasing") filter is applied before the downsampling operation, which can reduce the effects of aliasing.

[0187] The result of this layer 0 processing 4002a (e.g., the result after motion compensation and intra prediction) is then upsampled 4003a, after which a signal enhancement filter 4004a is applied. The generated signal enhancement data is then provided as input for layer 1, along with the original input video 4007, on which layer 1 processing 4005a (e.g., motion compensation and intra prediction followed by base coding) is performed.

[0188] Next, the results of base layer encoding of Layer 0 and Layer 1 are multiplexed 4009a to form a bitstream 4001a, which is output to the decoder.

[0189] Applying the weighted signal enhancement filter 4004 (and 4004a) between the layer 0 processing 4002 (and 4002a) and the layer 1 processing 4005 (and 4005a) improves the quality of the upsampled signal provided as input to the layer 1 processing 4005 (and 4005a), thereby improving the overall performance of the coding.

[0190] The specific operations of the decoder 10 and the encoder 20 in the third embodiment will now be described with reference to Figures 6A and 6B. Figure 6A shows a flowchart of the operation of the decoder 10, and Figure 6B shows a flowchart of the operation of the encoder 20.

[0191] In step 501 of FIG. 6A, the decoder 10 decodes the bitstream to obtain video data and encoding information, where the encoding information includes weighting map indication information.

[0192] In step 502, the decoder 10 performs first layer processing based on the video data to obtain a picture block. In this embodiment, step 502 includes obtaining a prediction block by performing an intra prediction operation. However, embodiments are not limited thereto, and in other embodiments, step 502 may instead include any other first layer processing (e.g., an inter prediction operation) that generates a picture block.

[0193] In step 503, the decoder 10 upsamples the picture block. Step 503 corresponds roughly to step 103 of Figure 1A and will not be described in detail here for the sake of brevity.

[0194] The decoder 10 uses the weighting map indication information to determine a weighting map in step 504. Step 504 corresponds roughly to step 104 of Figure 1A and will not be described in detail here for the sake of brevity.

[0195] In step 505, the decoder 10 applies a signal enhancement filter together with a weighting map to the upsampled picture block, such that the signal enhancement filter is applied with different weights to different regions of the picture block to obtain an enhanced picture block. Step 505 roughly corresponds to step 105 of Figure 1A and will not be described in detail here for the sake of brevity.

[0196] In step 506, the decoder 10 performs second layer processing on the enhanced picture block based on the decoded bitstream to obtain a resulting picture block. In this embodiment, the second layer processing includes adding the coded residual (at high resolution in the second layer) to the bitstream. However, the embodiment is not limited thereto, and in other embodiments, step 506 may instead include any other second layer processing (e.g., a further enhancement operation on the enhanced picture block).

[0197] After step 506, the resulting picture blocks can be used for any desired purpose. In one example, the decoder 10 then displays the resulting picture blocks to a viewer. In another example, the decoder 10 stores the resulting picture blocks for later use. In another example, the decoder 10 transmits the resulting picture blocks to an external device for display.

[0198] A complementary method can be performed by the encoder 20 to encode the bitstream provided to the decoder 10. Figure 6B shows a flow chart of the operation of the encoder 20 according to the third embodiment.

[0199] In step 601, the encoder 20 obtains the original video data. Step 601 corresponds roughly to step 201 of Figure 1B, and will not be described in detail here for the sake of brevity.

[0200] In step 602, the encoder 20 generates a downsampled version of the original video data. Video Data Step 602 corresponds roughly to step 202 of Figure 1B, and for the sake of brevity, a detailed description will not be given here.

[0201] In step 603, the encoder 20 Tabi 6A , in this embodiment, the first layer processing includes performing an intra prediction operation to obtain a predicted block. However, the embodiment is not limited thereto, and in other embodiments, the first layer processing may include any other first layer processing (e.g., an inter prediction operation) that instead generates a picture block.

[0202] In this embodiment, step 603 includes a rate-distortion optimization operation in which intra-prediction parameters are determined and then encoded into the bitstream, although embodiments are not limited in this respect and any form of intra-prediction (or other first layer processing) may be performed instead.

[0203] In step 604, the encoder 20 upsamples the picture block obtained in the first layer processing. Step 604 corresponds roughly to step 204 of Figure 1B, and will not be described in detail here for the sake of brevity.

[0204] In step 605, the encoder 20 applies a signal enhancement filter together with a weighting map to the upsampled picture block, where the signal enhancement filter is applied with different weights to different regions of the picture block to obtain an enhanced picture block. Step 605 roughly corresponds to step 205 of Figure 1B and will not be described in detail here for the sake of brevity.

[0205] In step 606, the encoder 20 performs second layer processing on the enhanced picture block. In this embodiment, the second layer processing includes determining a residual (at a higher resolution in the second layer) to be applied to the enhanced picture block. Specifically, the encoder 20 compares the values ​​of the enhanced picture block with the corresponding values ​​of the original video data to determine a difference. Based on this, the encoder 20 determines a residual to apply to the enhanced picture block to arrive at the corresponding block of the original video data.

[0206] However, embodiments are not limited thereto, and in other embodiments, step 606 may instead include any other second layer processing (e.g., a different method of calculating the residual and / or further enhancement operations on the enhanced picture block).

[0207] In step 607, the encoder 20 Tabi In this embodiment, the video data and the coding information are coded into a bitstream, and the coding information includes weighting map indication information that indicates a weighting map. This corresponds roughly to step 206 in FIG. 1B, and a detailed description thereof will be omitted. Furthermore, in this embodiment, the intra prediction parameters used in the intra prediction and the residual determined in step 606 are coded into a bitstream and used by the decoder 10.

[0208] Applying a weighted signal enhancement filter to the upsampled picture block before second-layer processing is performed improves the quality of this picture block used as input for higher-layer processing. As a result, this reduces the coding cost in the higher layer. For example, if the higher layer includes adding a residual to the picture block, the coding cost required to code the residual is lower because the picture block is of high quality even before the residual is added.

[0209] While the third embodiment has been described with reference to only two layers (e.g., Layer 0 and Layer 1 shown in FIGS. 5A and 5B), it should be understood that this is for ease of explanation only and the embodiments are not limited in this respect. For example, some embodiments may use more than two processing layers.

[0210] It should further be noted that all variations described in relation to the first and second embodiments are equally applicable as variations of this third embodiment. Examples include the use of a weighting map function to determine the weighting map, the use of a filter function whose parameters are coded into the bitstream, and the use of multiple functions / weighting maps.

[0211] Figure 7 shows a schematic diagram of a decoder 10 according to one embodiment. In particular, Figure 7 shows a schematic diagram of a decoder 10 configured to perform any of the decoder methods described herein, a detailed description of which is omitted here for the sake of brevity.

[0212] 7, the decoder 10 includes a processor 11 and a computer-readable medium 12. The processor 11 and the computer-readable medium 12 may be connected via a bus system. The computer-readable medium is configured to store programs, instructions, or codes. The processor 11 is configured to execute the programs, instructions, or codes in the computer-readable medium 12 to complete the operations in the decoder method embodiments herein.

[0213] Thus, in an embodiment, the computer-readable medium 12 is configured to store a computer program executable by the processor 11, and the processor 11 is configured to execute the computer program to perform any of the steps of the decoder methods described herein.

[0214] Figure 8 shows a schematic diagram of an encoder 20 according to one embodiment. In particular, Figure 8 shows a schematic diagram of an encoder 20 configured to perform any of the encoder methods described herein, a detailed description of which is omitted here for the sake of brevity.

[0215] 8, the encoder 20 includes a processor 21 and a computer-readable medium 22. The processor 21 and the computer-readable medium 22 may be connected via a bus system. The computer-readable medium is configured to store programs, instructions, or codes. The processor 21 is configured to execute the programs, instructions, or codes in the computer-readable medium 22 to complete the operations in the decoder method embodiments herein.

[0216] Thus, in an embodiment, the computer-readable medium 22 is configured to store a computer program executable by the processor 21, and the processor 21 is configured to execute the computer program to perform any of the steps of the decoder methods described herein.

[0217] As described in detail above, embodiments provide an in-loop filtering process for the refinement of upsampled video, in which a local weighting map is used in the filtering process.

[0218] In some embodiments, multiple filters are applied to the same picture with different weighting maps.

[0219] In some embodiments, multiple filters are applied to different regions of the picture.

[0220] In some embodiments, several functions for the calculation of the weighting maps are pre-defined and only the calculation parameters and the weighting map identifiers need to be signaled.

[0221] In some embodiments, the signal enhancement filter is applied after the interpolation filter in the resampling of the reference picture.

[0222] In some embodiments, a signal enhancement filter is applied to the upsampled low resolution picture before it is presented to the viewer.

[0223] In some embodiments, the signal enhancement filter is applied after the interpolation filter in multi-resolution coding.

[0224] The use of weighted signal enhancement filters is not limited to these described applications; they merely provide an overview of highly suitable application fields. In general, weighted signal enhancement filters can be applied in any signal processing setting that requires signal enhancement and has properties that can be effectively utilized by a weighted filtering setting. This is not limited to the field of video coding / processing, but may also be applied, for example, to image coding / processing or audio coding / processing.

[0225] Embodiments of the present invention may further provide a computer-readable medium having computer-executable instructions for causing one or more processors of a computing device to perform any of the methods of the embodiments of the present invention.

[0226] Examples of computer-readable media include both volatile and non-volatile media, removable and non-removable media, including, but not limited to, solid-state memory, removable disks, hard disk drives, magnetic media, and optical disks. Generally, computer-readable media includes any type of medium suitable for storing, encoding, or transmitting a sequence of instructions executable by one or more computers to perform any one or more of the processes and functions described herein.

[0227] It should be understood that the functionality of each of the described components can be combined in many ways other than those described above. For example, in some embodiments, the functionality of multiple described devices can be combined into a single device. In other embodiments, at least one function of the described devices can be split across multiple separate (or distributed) devices.

[0228] Conditional language such as "may" is generally used to indicate that a feature / step is used in a particular embodiment, but that alternative embodiments may include alternative features or may omit such features entirely.

[0229] Furthermore, it should be understood that the method steps are not limited to the particular sequence described, but may be combined in any other suitable order, which may result in some method steps being performed in parallel in some embodiments, and furthermore, certain method steps may be omitted entirely in some embodiments.

[0230] While specific embodiments have been described, it should be understood that these are used to illustrate the general teachings of the present invention and that various modifications can be made without departing from the scope of the present invention, which is to be interpreted according to the appended claims and their equivalents.

[0231] Many further variations and modifications will occur to those skilled in the art upon reference to the foregoing exemplary embodiments, which are given by way of example only and are not intended to limit the scope of the invention, which is determined by the appended claims.

Claims

1. 1. A video data processing method performed by a decoder, comprising: decoding the bitstream to obtain video data and encoding information, the encoding information including weighting map indication information; obtaining a picture block based on the video data; upsampling the picture block; determining a weighting map using the weighting map indication; applying a signal enhancement filter together with the weighting map to the upsampled picture block, whereby the signal enhancement filter is applied with different weights to different regions of the picture block to obtain an enhanced picture block; method.

2. the signal enhancement filter comprises a linear filter optimized by a least squares optimization procedure; The method of claim 1.

3. the weighting map includes a plurality of weight values ​​each corresponding to a value of the upsampled picture block; 3. The method according to claim 1 or 2.

4. The encoded information further includes signal enhancement filter indication information, and the method further comprises: further comprising decoding the bitstream to determine the signal enhancement filter. The method of claim 1.

5. filter parameters of the signal enhancement filter are either explicitly signaled in the bitstream or derived by the decoder from video data in the bitstream; The method of claim 4.

6. the signal enhancement filter instruction information instructs the signal enhancement filter to reuse one or more filter parameters stored in a filter buffer of the decoder; 6. The method according to claim 4 or 5.

7. determining the weighting map using the weighting map indication, determining a weight map function using the weight map indication; and calculating the weighting map by applying the weighting map function to the upsampled picture block. The method according to any one of claims 1 to 6.

8. the weighting map indication information includes a weighting map identifier that identifies one of a plurality of preset weighting map functions; The method of claim 7.

9. the weighting map indication information includes parameters of the weighting map function; 9. The method according to claim 7 or 8.

10. the picture block is a predicted block, obtaining the picture block based on the video data includes obtaining the prediction block by performing a prediction operation using the video data. The method according to any one of claims 1 to 9.

11. The prediction operation is inter prediction or intra prediction. The method of claim 10.

12. the residual is encoded into the bitstream at the resolution of the upsampled picture block; The method comprises: decoding the bitstream to determine the residual; applying the residual to the enhanced prediction block.

12. The method according to claim 10 or 11.

13. the picture block is a reference sample; The method further comprises: obtaining a prediction block by performing a prediction operation using the enhanced reference sample; The method according to any one of claims 1 to 9.

14. the prediction operation includes inter prediction; the reference sample corresponds to a first picture of the video data encoded in the bitstream; the predictive block corresponds to a second picture of the video data encoded in the bitstream, the second picture being temporally separated from the first picture; the first picture is encoded in the bitstream at a lower resolution than the second picture; The method of claim 13.

15. the encoding information indicates application of a plurality of filters and a plurality of respective weighting maps to the picture block; The method according to any one of claims 1 to 14.

16. the coding information indicates the use of different weighting maps and / or signal enhancement filters for different picture blocks of a picture; The method according to any one of claims 1 to 15.

17. 1. A computer-readable medium, comprising: and computer-executable instructions stored on the computer-readable medium, the computer-executable instructions, when executed by a computing device, causing the computing device to perform the method of any of claims 1 to 16. Computer-readable medium.

18. A decoder comprising: one or more processors; a computer-readable medium comprising computer-executable instructions stored on the computer-readable medium, the computer-executable instructions, when executed by one or more processors, causing the one or more processors to perform a method according to any of claims 1 to 16; decoder.

19. 1. A video data processing method performed by an encoder, comprising: obtaining original video data; obtaining a downsampled version of the original video data; Obtaining a picture block based on the downsampled original video data; upsampling the picture block; applying a signal enhancement filter together with a weighting map to the upsampled picture block, whereby the signal enhancement filter is applied with different weights to different regions of the picture block, thereby obtaining an enhanced picture block to recover the loss caused by downsampling and upsampling of the original video data; encoding the downsampled original video data and encoding information into a bitstream, the encoding information including weight map indication information indicating the weight map; method.

20. the signal enhancement filter comprises a linear filter optimized by a least squares optimization procedure; 20. The method of claim 19.

21. the weighting map includes a plurality of weight values ​​each corresponding to a value of the upsampled picture block; 21. The method of claim 19 or 20.

22. The step of obtaining the enhanced picture block includes: performing a rate-distortion optimization operation to determine said weighting map; 20. The method of claim 19.

23. The step of performing a rate-distortion optimization operation comprises: iteratively obtaining enhanced picture blocks by applying the signal enhancement filter with different weighting maps to the upsampled picture blocks until a particular weighting map obtains an enhanced picture block that is within a threshold similarity with a corresponding original picture block obtained from the original video data.

23. The method of claim 22.

24. The method comprises: performing the rate-distortion optimization operation to determine the signal enhancement filter.

23. The method of claim 22.

25. The step of performing a rate-distortion optimization operation comprises: iteratively obtaining enhanced picture blocks by applying different signal enhancement filters with different weighting maps to the upsampled picture blocks until a particular combination of at least one signal enhancement filter and weighting map obtains an enhanced picture block that is within a threshold similarity with a corresponding original picture block obtained from the original video data; 25. The method of claim 24.

26. the encoded information further includes signal enhancement filter indication information; The method according to any one of claims 19 to 25.

27. filter parameters of the signal enhancement filter are either explicitly signaled in the bitstream or derived by a decoder from the video data in the bitstream; 27. The method of claim 26.

28. the signal enhancement filter instruction information instructs the signal enhancement filter to reuse one or more filter parameters stored in a filter buffer of a decoder; 28. The method of claim 26 or 27.

29. The weighting map may be: determining a weight map function using the weight map indication; calculating the weighting map by applying the weighting map function to the upsampled picture block; The method according to any one of claims 19 to 28.

30. The weighting map indication information is a weighting map identifier that identifies one of a plurality of pre-defined weighting map functions; 30. The method of claim 29.

31. the weighting map indication information includes parameters of the weighting map function; 31. The method of claim 29 or 30.

32. the picture block is a predicted block; obtaining the picture block based on the downsampled video data includes obtaining the prediction block by performing a prediction operation using the original video data; The method according to any one of claims 19 to 31.

33. The prediction operation is inter prediction or intra prediction.

33. The method of claim 32.

34. The method further comprises encoding a residual into the bitstream at a resolution of the upsampled picture block; The method further comprises applying the residual to the enhanced prediction block.

34. The method of claim 32 or 33.

35. the picture block is a reference sample; The method further comprises: obtaining a prediction block by performing a prediction operation using the enhanced reference sample; The method according to any one of claims 19 to 31.

36. the prediction operation includes inter prediction; the reference sample corresponds to a first picture of the video data encoded in the bitstream; the predictive block corresponds to a second picture of the video data encoded in the bitstream, the second picture being temporally separated from the first picture; the first picture is encoded in the bitstream at a lower resolution than the second picture; 36. The method of claim 35.

37. the encoding information indicates application of a plurality of filters and a plurality of respective weighting maps to the picture block; The method according to any one of claims 19 to 36.

38. the coding information indicates the use of different weighting maps and / or signal enhancement filters for different picture blocks of a picture; 40. The method according to any one of claims 19 to 39.

39. 1. A computer-readable medium, comprising: and computer-executable instructions stored on the computer-readable medium, the computer-executable instructions, when executed by a computing device, causing the computing device to perform the method of any of claims 19 to 38. Computer-readable medium.

40. 1. An encoder comprising: one or more processors; a computer-readable medium comprising computer-executable instructions stored on the computer-readable medium, the computer-executable instructions, when executed by one or more processors, causing the one or more processors to perform the method of any of claims 19 to 38; Encoder.

Images