Filter design for signal enhancement filtering for reference picture resampling.

Abstract

A video data processing method, performed by a decoder, is provided, comprising: decoding a bitstream to obtain video data and coding information, the coding information including weight map indication information for defining a weight map and filter coefficients optimized to the weight map; obtaining a picture block based on the video data; up-sampling the picture block; determining the weight map using the weight map indication information; and applying a signal enhancement filter using the filter coefficients to the up-sampled picture block together with the weight map to obtain an enhanced picture block.

Description

[Technical Field]

[0001] The present application relates to the field of computer vision, in particular to the topic of video processing and video coding, and more particularly to a method of filter design, a decoder, an encoder, and a computer-readable medium for signal-enhanced filtering for reference picture resampling. [Background technology]

[0002] Current video coding standards, such as H.265 / HEVC (High Efficiency Video Coding) and H.266 / VVC (Versatile Video Coding), support spatial scalability of coded video streams. While this support is included in the scalable second version of HEVC (SHVC), VVC natively supports spatial scalability. Adaptively changing the resolution of coded video during the coding process is known as reference picture resampling (RPR) or adaptive resolution change (ARC) in VVC. Furthermore, multi-resolution coding and layered coding enable the resolution of coded video to be scalable. Therefore, the spatial resolution at which video is coded can be adaptively changed and no longer needs to be equal to the output or input resolution of the video. The benefit of this additional flexibility is that coding lower-resolution video can potentially result in lower bitrates and reduced computational complexity, but at the cost of losing high-frequency information in the downsampling step.

[0003] Encoding video at a resolution lower than the original requires downsampling and upsampling steps in the signal processing chain. During the downsampling step, anti-aliasing filters are applied to prevent artifacts due to high frequency components in the image. During the upsampling process, interpolation filters are applied to reconstruct intensity values ​​at fractional sample positions.

[0004] In RPR, the resolution of the coded video stream may change adaptively. As a result, the encoder can code parts of the video stream at a lower resolution. RPR is applied in inter prediction whenever a picture uses a reference picture with a different resolution than the current picture. This step requires applying a resampling operation so that the reference picture blocks are mapped to the same spatial resolution as the current picture.

[0005] In multi-layer coding, video is coded at different resolution layers. In the first step, the video is coded at the lowest resolution layer. To generate the video stream for the next layer, the video is upsampled, potentially with residual coding, and further processing steps applied. This process may be applied multiple times based on the number of layers.

[0006] Finding the optimal high-resolution representation from a low-resolution picture is a key part of these coding schemes. One method is to apply a set of polyphase Finite Impulse Response (FIR) interpolation filters. While these filters provide an approximation of the high-resolution image content, they cannot recover information lost in the downsampling process and are limited to linear filtering operations. As a result, upsampled images are often blurry.

[0007] Image sharpening operations can improve image quality. However, linear high-pass filters often introduce artifacts such as overshoot and ringing. Furthermore, the distortion caused by downsampling and upsampling depends on the image content and the video coding quality (which is affected by the Quantization Parameter (QP) value). Summary of the Invention [Means for solving the problem]

[0008] SUMMARY OF THE INVENTION Embodiments of the present application provide methods, decoders, encoders, and computer-readable media for video encoding with signal enhancement filtering that overcome problems associated with conventional arrangements.

[0009] According to a first aspect, there is provided a video data processing method, to be performed by a decoder, comprising: decoding a bitstream to obtain video data and coding information, the coding information including weight map indication information for defining a weight map and filter coefficients optimized to the weight map, obtaining a picture block based on the video data, up-sampling the picture block, determining the weight map using the weight map indication information, and applying a signal enhancement filter using the filter coefficients to the up-sampled picture block together with the weight map to obtain an enhanced picture block.

[0010] In some embodiments, the weighting map comprises a scalar weighting map.

[0011] In some embodiments, the weighting map comprises a Sobel intensity map.

[0012] In some embodiments, the weighting map comprises a plurality of weight values ​​each corresponding to a value of the upsampled picture block.

[0013] In some embodiments, the signal enhancement filter indication information indicates to reuse one or more filter coefficients stored in a filter buffer of the decoder for the signal enhancement filter.

[0014] In some embodiments, determining the weight map using the weight map indication comprises determining a weight map function using the weight map indication, and calculating the weight map by applying the weight map function to the upsampled picture block.

[0015] In some embodiments, the weight map indication includes a weight map identifier that identifies one of a plurality of pre-defined weight map functions.

[0016] In some embodiments, the weighting map indication comprises parameters of the weighting map function.

[0017] In some embodiments, the picture block is a predictive block, and deriving the picture block based on the video data comprises performing a prediction operation using the video data to obtain the predictive block.

[0018] In some embodiments, the prediction operation is inter-prediction or intra-prediction.

[0019] In some embodiments, the picture block is a reference sample, and the method further comprises obtaining a prediction block by performing a prediction operation using the enhanced reference sample.

[0020] In some embodiments, the prediction operation includes inter prediction, the reference sample corresponds to a first picture of the video data encoded in the bitstream, and the predictive block corresponds to a second picture of the video data encoded in the bitstream, the second picture being temporally separated from the first picture, and the first picture being encoded in the bitstream at a lower resolution than the second picture.

[0021] In some embodiments, the encoding information indicates application of multiple filters and multiple respective weighting maps to the picture block.

[0022] In some embodiments, the encoding information indicates the use of different weighting maps and / or signal enhancement filters for different picture blocks of a picture.

[0023] According to a second aspect, a computer-readable medium is provided, comprising computer-executable instructions stored on the computer-readable medium, the computer-executable instructions, when executed by a computing device, causing the computing device to perform any of the methods described in the first aspect.

[0024] No. 3 According to an aspect, there is provided a computer-implemented video data processing method executed by a decoder, the method comprising: decoding a bitstream to obtain video data and coding information, the coding information including weight map indication information; obtaining a picture block based on the video data; up-sampling the picture block; determining the weight map using the weight map indication information; and applying a signal enhancement filter to the up-sampled picture block together with the weight map, whereby the signal enhancement filter is applied with different weights to different regions of the picture block to obtain an enhanced picture block.

[0025] In some embodiments, the signal enhancement filter comprises a Wiener filter.

[0026] In some embodiments, the weighting map comprises a plurality of weight values ​​each corresponding to a value of the upsampled picture block.

[0027] In some embodiments, the encoded information further comprises signal enhancement filter indication information, and the method further comprises decoding the bitstream to determine the signal enhancement filter.

[0028] In some embodiments, the filter parameters of the signal enhancement filter are explicitly signaled in the bitstream or are derived by the decoder from the video data of the bitstream.

[0029] In some embodiments, the signal enhancement filter instruction information instructs the signal enhancement filter to reuse one or more filter parameters stored in a filter buffer of the decoder.

[0030] In some embodiments, determining the weight map using the weight map indication comprises determining a weight map function using the weight map indication, and calculating the weight map by applying the weight map function to the upsampled picture block.

[0031] In some embodiments, the weight map indication includes a weight map identifier that identifies one of a plurality of pre-defined weight map functions.

[0032] In some embodiments, the weighting map indication comprises parameters of the weighting map function.

[0033] In some embodiments, the picture block is a predictive block, and obtaining the picture block based on the video data comprises obtaining the predictive block by performing a prediction operation using the video data.

[0034] In some embodiments, the prediction operation is inter-prediction or intra-prediction.

[0035] In some embodiments, a residual is coded into the bitstream at the resolution of the upsampled picture block, and the method further comprises decoding the bitstream to determine the residual, and applying the residual to the enhanced prediction block.

[0036] In some embodiments, the picture block is a reference sample, and the method further comprises obtaining a prediction block by performing a prediction operation using the enhanced reference sample.

[0037] In some embodiments, the prediction operation includes inter prediction, the reference sample corresponds to a first picture of the video data encoded in the bitstream, and the predictive block corresponds to a second picture of the video data encoded in the bitstream, the second picture being temporally separated from the first picture, and the first picture being encoded in the bitstream at a lower resolution than the second picture.

[0038] In some embodiments, the encoding information indicates application of multiple filters and multiple respective weighting maps to the picture block.

[0039] In some embodiments, the encoding information indicates the use of different weighting maps and / or signal enhancement filters for different picture blocks of a picture.

[0040] No. 4 According to an aspect, a computer-readable medium is provided, comprising computer-executable instructions stored on the computer-readable medium, the computer-executable instructions, when executed by a computing device, performing a first 3 causing the computing device to perform any of the methods described in the aspects.

[0041] No. 5 According to an aspect, a decoder is provided, the decoder including one or more processors and a computer-readable medium having computer-executable instructions stored on the computer-readable medium, the computer-executable instructions, when executed by the one or more processors, 3 and a computer-readable medium for causing said one or more processors to perform any of the methods described in the aspects.

[0042] No. 6 According to an aspect, there is provided a video data processing method to be performed by an encoder, the method comprising the steps of obtaining original video data; obtaining a downsampled version of the original video data; obtaining a picture block based on the downsampled original video data; and upsampling the picture block; The method includes the steps of: obtaining a weighting map from the original video data; defining a linear equation representing a signal enhancement filter that calculates an enhanced picture block based on the weighting map, filter coefficients, and the upsampled picture block; applying least squares optimization to the linear equation to obtain optimal filter coefficients for the weighting map; applying the signal enhancement filter using the optimal filter coefficients to the upsampled picture block together with the weighting map to obtain an enhanced picture block; and encoding the downsampled original video data and encoding information into a bitstream, wherein the encoding information includes weighting map indication information that indicates the weighting map and the calculated filter coefficients.

[0043] In some embodiments, the filter coefficients are calculated by calculating partial derivatives that are set to zero.

[0044] In some embodiments, the linear equations are transformed into a matrix-vector product form, and the matrix is ​​a symmetric matrix.

[0045] In some embodiments, the upsampled picture block is an upsampled low resolution picture block generated after upsampling or multi-resolution encoding of a reference picture.

[0046] In some embodiments, the weighting map comprises a scalar weighting map.

[0047] In some embodiments, the weighting map comprises a Sobel intensity map.

[0048] In some embodiments, the weighting map comprises a plurality of weight values ​​each corresponding to a value of the upsampled picture block.

[0049] In some embodiments, the weighting map indication and the calculated filter coefficients are quantized and entropy coded.

[0050] In some embodiments, the weighting map indication and the calculated filter coefficients are either explicitly signaled in the bitstream or derived by a decoder from video data in the bitstream.

[0051] In some embodiments, the weighting map is obtained by determining a weighting map function using weighting map indication information and calculating the weighting map by applying the weighting map function to the upsampled picture block.

[0052] In some embodiments, the weight map indication includes a weight map identifier that identifies one of a plurality of pre-defined weight map functions.

[0053] In some embodiments, the weighting map indication comprises parameters of the weighting map function.

[0054] In some embodiments, the picture block is a predicted block and the downsampled original Obtaining the picture block based on video data includes obtaining the prediction block by performing a prediction operation using the original video data.

[0055] In some embodiments, the prediction operation is inter-prediction or intra-prediction.

[0056] In some embodiments, the picture block is a reference sample, and the method further comprises obtaining a prediction block by performing a prediction operation using the enhanced reference sample.

[0057] In some embodiments, the prediction operation includes inter-prediction, the reference sample corresponds to a first picture of the video data encoded in the bitstream, and the predictive block corresponds to a second picture of the video data encoded in the bitstream, the second picture being temporally separated from the first picture, and the first picture being encoded in the bitstream at a lower resolution than the second picture.

[0058] In some embodiments, the encoding information indicates application of multiple filters and multiple respective weighting maps to the picture block.

[0059] In some embodiments, the encoding information indicates the use of different weighting maps and / or signal enhancement filters for different picture blocks of a picture.

[0060] No. 7 According to an aspect, a computer-readable medium is provided having computer-executable instructions stored thereon. and bitstream the computer-executable instructions are processor When executed by 6 causing said computing device to perform any of the methods described in the aspects , generating the bitstream .

[0061] No. 8 According to an aspect, an encoder is provided, the encoder including one or more processors and a computer-readable medium having computer-executable instructions stored on the computer-readable medium, the computer-executable instructions, when executed by the one or more processors, 6 and a computer-readable medium for causing said one or more processors to perform any of the methods described in the aspects.

[0062] Coding performance can be improved by applying a signal enhancement filter with localized weighting (weight map) and optimized filter coefficients. Localized weighting allows for smooth increase or decrease of filter strength in localized regions. For example, weighting can be used to increase filter strength in edge regions while decreasing filter strength in regions where signal enhancement filters typically exhibit ringing. This weight map allows for the determination of a signal enhancement filter with optimized filter coefficients in the weight map, potentially amplifying high-frequency components without introducing significant ringing. For example, in image upsampling scenarios, the amplification of high-frequency components can be used to sharpen blurred edges. In some cases, adaptive filters can be used to address coding artifacts and different characteristics of video content. The optimized filter coefficients are determined using least-squares optimization, which is computationally efficient.

[0063] These and other aspects of the present application will become more readily apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0064] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1A] 4 shows a flowchart of the operation of the decoder according to the first embodiment. [Figure 1B] 4 shows a flowchart of the operation of the encoder according to the first embodiment. [Figure 2A] FIG. 10 is a block diagram showing an exemplary operation of a decoder according to a modification of the first embodiment. [Figure 2B] FIG. 10 is a block diagram showing an exemplary operation of an encoder according to a modification of the first embodiment. [Figure 3] 1 shows a schematic diagram of a decoder according to various embodiments; [Figure 4] 1 shows a schematic diagram of an encoder according to various embodiments; DETAILED DESCRIPTION OF THE INVENTION

[0065] DETAILED DESCRIPTION OF THE INVENTION The technical solutions in the embodiments of the present application are described clearly and completely below with reference to the accompanying drawings.

[0066] These technical solutions are applicable to H.265 / HEVC or H.266 / VVC video coding systems, and in particular to the performance of RPR, ARC, multi-resolution coding, and multi-layer coding. However, it should be understood that these technical solutions are also applicable to any other video coding system that includes upsampling. Furthermore, although these principles are primarily described with reference to video processing, they are also applicable to other data formats, such as image processing and audio processing.

[0067] In the embodiments, "video" refers to one or more pictures. In other words, a video may include one picture or multiple pictures. A picture may also be called an "image."

[0068] An "encoder" is a device that can encode data into a bitstream, and a "decoder" is a device that can decode a bitstream to obtain the encoded data. A "bitstream" consists of a sequence of bits.

[0069] "Intra-prediction" and "inter-prediction" are two prediction operations that a decoder can use within the HEVC and VVC framework to process a received bitstream to obtain an original signal. In embodiments, the "original signal" or "original video" refers to the original video encoded by the encoder. Da It is used to refer to data before it is encoded. In embodiments, a reference sample may refer to spatially and / or temporally separated picture data used to predict a picture (or a region of a picture).

[0070] In particular, intra-prediction involves spatially predicting data within a single picture image without reference to other (temporally distant) pictures. In other words, data from a first region of a picture is used to predict data from other regions of the same picture image, but without relying on other samples from other pictures. In this specification, the data from the first region of a picture image is considered a "reference sample."

[0071] Inter-prediction involves predicting data between multiple pictures. In other words, data from a first region of a first picture image is used to predict data from a second region of a second picture image. The first and second regions may or may not be spatially separated. In this specification, the data from the first region of the first picture image is considered a "reference sample."

[0072] A "weighting map" is a map or mask containing scalar weights that indicate the intensity or strength of a filter application at a location (e.g., sample) in the picture. Weighting the adaptive loop filter with an appropriate per-sample mask allows for more precise filter decisions, which may result in a higher quality filtered picture. While this is beneficial, linear filters typically result in a trade-off between sharpening edge steepness and overshoot and ringing artifacts. Increasing the filter's effect at the steepest parts of the edges can mitigate these problems.

[0073] In embodiments, a "residual" may refer to a value obtained based on an original value of a region of a picture and a predicted value of that region (eg, the difference between the original value and the predicted value).

[0074] In the embodiments, a "block" may refer to a portion of a picture. For example, a picture may be divided into two or more blocks. However, this is by way of example only. If the picture is not divided, a "block" may refer to the entire picture.

[0075] A "signal enhancement filter" may refer to a filter that functions to enhance a signal, particularly an upsampled signal. Generally, in the described embodiments, the signal enhancement filter is a filter configured to reduce edge blurring (i.e., a filter that sharpens picture blocks). However, embodiments are not limited in this regard, and in other embodiments, the signal enhancement filter may be configured to provide alternative or additional signal enhancements, such as removal of blocking and / or ringing artifacts. In paragraph 1, a video data processing method to be performed by a decoder is provided, the method comprising: decoding the bitstream to obtain video data and encoding information, the encoding information including weighting map indication information for defining a weighting map and filter coefficients optimized for the weighting map; obtaining a picture block based on the video data; upsampling the picture block; determining the weighting map using the weighting map indication; applying a signal enhancement filter using the filter coefficients together with a weighting map to the upsampled picture block to obtain an enhanced picture block. In the second term, based on the first term, the weighting map comprises a scalar weighting map. In the third term, based on the first or second term, the weighting map comprises a Sobel intensity map. In the fourth term, based on any one of the first to third terms, the weighting map includes a plurality of weight values ​​each corresponding to a value of the upsampled picture block. In clause 5, based on any of clauses 1 to 4, the signal enhancement filter instruction information instructs to reuse one or more filter coefficients stored in a filter buffer of the decoder for the signal enhancement filter. In paragraph 6, the step of determining the weighting map using the weighting map indication information based on any one of paragraphs 1 to 5 includes: determining a weight map function using the weight map indication; and calculating the weighting map by applying the weighting map function to the upsampled picture block. In paragraph 7, based on paragraph 6, the weighting map indication information includes a weighting map identifier that identifies one of a plurality of preset weighting map functions. In the eighth clause, based on the sixth or seventh clause, the weighting map indication information includes a parameter of the weighting map function. In clause 9, based on any one of clauses 1 to 8, the picture block is a predicted block; Obtaining a picture block based on the video data includes performing a prediction operation using the video data to obtain the prediction block. In clause 10, based on clause 9, the prediction operation is inter-prediction or intra-prediction. In paragraph 11, based on any one of paragraphs 1 to 9, the picture block is a reference sample; The method further includes obtaining a prediction block by performing a prediction operation using the enhanced reference sample. In paragraph 12, based on paragraph 11, the prediction operation includes inter-prediction; the reference sample corresponds to a first picture of the video data encoded in the bitstream; the predictive block corresponds to a second picture of the video data encoded in the bitstream, the second picture being temporally separated from the first picture; The first picture is encoded in the bitstream at a lower resolution than the second picture. In clause 13, based on any of clauses 1 to 12, the encoding information indicates application of a plurality of filters and a plurality of respective weighting maps to the picture block. In clause 14, based on any of clauses 1 to 13, the coding information indicates the use of different weighting maps and / or signal enhancement filters for different picture blocks of a picture. In clause 15, a computer-readable medium is provided, including computer-executable instructions stored on the computer-readable medium, which, when executed by a computing device, cause the computing device to perform a method according to any of clauses 1 to 14. In clause 16, a decoder is provided, one or more processors; A computer-readable medium including computer-executable instructions stored on the computer-readable medium, which, when executed by the one or more processors, cause the one or more processors to perform the method of any of clauses 1 to 14. In paragraph 17, a video data processing method is provided that is performed by an encoder, the method comprising: obtaining original video data; obtaining a downsampled version of the original video data; obtaining a picture block based on the downsampled original video data; upsampling the picture block; obtaining a weighting map from the original video data; defining a linear equation representing a signal enhancement filter that calculates an enhancement picture block based on the weighting map, filter coefficients, and the upsampled picture block; applying least squares optimization to the linear equation to obtain optimal filter coefficients for the weighting map; applying the signal enhancement filter using the optimal filter coefficients together with the weighting map to the upsampled picture block to obtain an enhanced picture block; encoding the downsampled original video data and encoding information into a bitstream, the encoding information including weighting map indication information indicating the weighting map and the calculated filter coefficients. In term 18, based on term 17, the filter coefficients are calculated by calculating partial derivatives set to zero. In the 19th term, based on the 17th or 18th term, the linear equation is transformed into a matrix-vector product form, and the matrix is ​​a symmetric matrix. In clause 20, based on clause 17, the upsampled picture block is an upsampled low-resolution picture block generated after upsampling or multi-resolution coding of a reference picture. In a twenty-first clause, based on any one of the seventeenth to twentieth clauses, the weighting map includes a scalar weighting map. In paragraph 22, based on any of paragraphs 17 to 21, the weighting map includes a Sobel intensity map. In clause 23, based on any of clauses 17 to 22, the weighting map includes a plurality of weight values ​​each corresponding to a value of the upsampled picture block. In clause 24, the weighting map indication information and the calculated filter coefficients are quantized and entropy coded based on any of clauses 17 to 23. In clause 25, based on any of clauses 17 to 24, the weighting map indication information and the calculated filter coefficients are either explicitly signaled in the bitstream or derived by a decoder from video data in the bitstream. In paragraph 26, based on any one of paragraphs 17 to 25, the weighting map is determining a weight map function using the weight map indication; and calculating the weighting map by applying the weighting map function to the upsampled picture block. In paragraph 27, based on any one of paragraphs 17 to 26, the weighting map instruction information includes a weighting map identifier that identifies one of a plurality of preset weighting map functions. In clause 28, based on clause 26 or 27, the weighting map indication information includes a parameter of the weighting map function. In clause 29, based on any one of clauses 17 to 28, the picture block is a predicted block; Obtaining the picture block based on the downsampled video data includes obtaining the prediction block by performing a prediction operation using the original video data. In clause 30, based on any of clauses 29, the prediction operation is inter-prediction or intra-prediction. In clause 31, based on any one of clauses 17 to 30, the picture block is a reference sample; The method further includes obtaining a prediction block by performing a prediction operation using the enhanced reference sample. In paragraph 32, based on any of paragraphs 31, the prediction operation includes inter-prediction; the reference sample corresponds to a first picture of the video data encoded in the bitstream; the predictive block corresponds to a second picture of the video data encoded in the bitstream, the second picture being temporally separated from the first picture; The first picture is encoded in the bitstream at a lower resolution than the second picture. In clause 33, based on any of clauses 17 to 32, the encoding information indicates application of a plurality of filters and a plurality of respective weighting maps to the picture block. In clause 34, based on any of clauses 17 to 33, the coding information indicates the use of different weighting maps and / or signal enhancement filters for different picture blocks of a picture. In clause 35, a computer-readable medium is provided, including computer-executable instructions stored on the computer-readable medium, which, when executed by a computing device, cause the computing device to perform a method according to any of clauses 17 to 34. In paragraph 36, an encoder is provided, the encoder comprising: one or more processors; A computer-readable medium including computer-executable instructions stored on the computer-readable medium, which, when executed by the one or more processors, cause the one or more processors to perform the method of any of clauses 17 to 34.

[0076] Fig. 1A shows a flowchart of the operation of a decoder 10 according to the first embodiment. Fig. 1B shows a flowchart of the operation of an encoder 20 according to the first embodiment.

[0077] The flowchart of FIG. 1A starts at step 101, in which the decoder 10 decodes the bitstream to obtain video data and encoding information, where the encoding information includes weighting map indication information and filter coefficients optimized for the weighting map.

[0078] In step 102, the decoder 10 obtains a picture block based on the video data. In this embodiment, the video data includes a downsampled version of the original video data. In other words, the video data includes a low-resolution version of the original video data. Therefore, step 102 includes obtaining data corresponding to the picture block in the video data.

[0079] However, embodiments are not limited thereto, and the video data may include any data that the decoder 10 can use to obtain a picture block (e.g., by performing a prediction operation, such as intra-prediction or inter-prediction, using the video data).

[0080] In step 103, the decoder 10 upsamples the picture block. In this embodiment, step 103 involves applying a set of polyphase FIR interpolation filters to reconstruct intensity values ​​at fractional sample positions, thereby increasing the resolution of the picture block. However, embodiments are not limited to this, and other upsampling methods can be applied instead. In particular, there are many different methods that can be used to perform the interpolation. Essentially, upsampling presents a problem in that fractional sample positions need to be interpolated. This includes, but is not limited to, bilinear interpolation, bicubic interpolation, nearest neighbor interpolation, Lanczos interpolation, etc.

[0081] In this embodiment, step 103 includes upsampling the picture block to the resolution of the original video data, although the embodiment is not limited thereto and in other embodiments, the picture block may be upsampled to a resolution lower than the resolution of the original video data.

[0082] In step 104, the decoder 10 determines a weighting map using the weighting map indication information. In this embodiment, the weighting map indication information explicitly signals values ​​of a weighting map having a resolution corresponding to the resolution of the upsampled picture block. In one example, if the size of the upsampled picture block is 5x5, the weighting map indication information includes 25 values, one for each position of the 5x5 block. In this example, in step 104, the decoder 10 determines a weighting map having these 25 values.

[0083] However, embodiments of the present application are not limited in this respect. For example, in some embodiments, the resolution of the weighting map may be smaller than the resolution of the upsampled picture block. In such cases, a single value in the weighting map may correspond to multiple values ​​of the picture block. Furthermore, in other embodiments, the weighting map may be determined in other manners, as described below.

[0084] In step 105, the decoder 10 applies a signal enhancement filter (SEF) with optimized filter coefficients together with the weighting map to the upsampled picture block to obtain an enhanced picture block.

[0085] Thus, step 105 includes using a weighting map so that the signal enhancement filter is applied with different strengths to different regions of the upsampled picture block.

[0086] In this embodiment, the signal enhancement filter is a sharpening filter configured to sharpen blurry edges, although embodiments are not limited in this respect. For example, in other embodiments, the encoded information further indicates the particular signal enhancement filter to use and / or particular parameters of the signal enhancement filter.

[0087] After step 105, the enhanced picture blocks may be used for any desired purpose. In one example, the decoder 10 then displays the enhanced picture blocks to a viewer. In another example, the decoder 10 stores the picture blocks for later use. In another example, the decoder 10 transmits the picture blocks to an external device for display.

[0088] A complementary method may be performed by the encoder 20 to encode the bitstream provided to the decoder 10. Figure 1B shows a flow chart of the operation of the encoder 20 according to the first embodiment.

[0089] In step 201, the encoder 20 acquires original video data. For example, the encoder 20 may receive the original video data from an external server via a communication network (e.g., the Internet). However, the embodiment is not limited to a method for acquiring the original video data.

[0090] In step 202, encoder 20 obtains a downsampled version of the original video data. In this embodiment, step 202 includes encoder 20 downsampling the original video data to obtain lower resolution video data. However, embodiments are not limited in this respect. For example, in some embodiments, encoder 20 may instead receive both the original video data and the downsampled video data from an external source, e.g., an external server, via a communications network (e.g., the Internet).

[0091] In step 203, the encoder 20 obtains a picture block based on the downsampled original video data. In this embodiment, step 203 is performed in the same manner as step 102 in FIG. 1A. In other words, in this embodiment, the downsampled video data includes a lower-resolution version of the original video data. Therefore, step 203 includes obtaining data corresponding to the picture block in the video data.

[0092] However, embodiments are not limited thereto. For example, in other embodiments, step 203 may instead include performing a prediction operation using the downsampled video data, such as intra-prediction or inter-prediction, to obtain the picture block.

[0093] In step 204, encoder 20 upsamples the picture block. Step 204 is performed in a manner corresponding to step 103 of Figure 1A, and will not be described in detail here for the sake of brevity.

[0094] In step 205, the encoder 20 obtains a weighting map from the original video data.

[0095] Step 206 defines a linear equation representing a signal enhancement filter that calculates an enhancement picture block based on the weighting map, the filter coefficients, and the upsampled picture block.

[0096] In step 207, a least squares optimization is applied to the linear equation to obtain optimal filter coefficients for the weighting map.

[0097] In step 208, the signal enhancement filter using the calculated filter coefficients together with the weighting map is applied to the upsampled picture block to obtain an enhanced picture block.

[0098] In step 209, the downsampled original video data and the coding information are coded into a bitstream, the coding information including weighting map indication information indicating a weighting map, and the coding information further including the calculated filter coefficients.

[0099] As discussed above in connection with FIG. 1A, this embodiment uses a single pre-set signal enhancement filter defined by filter coefficients calculated / optimized using least-squares optimization (i.e., a filter that minimizes the squared error between the filtered signal and the ground truth signal). This finds an optimal solution for the filter coefficients given the weighting map. Of course, in some embodiments, additional side constraints are placed on the determination of the signal enhancement filter, such as filter shape or filter coefficients that must be equal.

[0100] However, although the present embodiments are described with reference to a signal enhancement filter based on the concept of a Wiener filter, the embodiments are not limited in this respect and other types of filters may instead be used as sharpening filters, such as filters based on Sobel filters or unsharp masking filters. Other non-linear options include bilateral filters, diffusion filters, and adaptive loop filters (ALFs).

[0101] This method improves the quality of the upsampled picture by reducing distortions, particularly those due to low-resolution encoding: the loss of high-frequency information and distortions caused by video encoding. This effect is achieved by optimizing the filter coefficients for a given weighting map using least-squares optimization.

[0102] In particular, local weighting provided by a weighting map can be applied to smoothly increase or decrease the strength of the filter in local regions. In some instances, the weighting map can provide weighting that increases the filter strength in edge regions but decreases it in areas where ringing typically occurs. In such a setting, the optimized filter can amplify high-frequency components without introducing significant ringing. This is particularly useful in image upsampling scenarios, where amplification of high-frequency components must be utilized to sharpen blurry edges.

[0103] In some cases, a signal enhancement filter can be applied to reduce edge blurring caused by the upsampling operation, and an appropriate weighting map can be extracted, e.g., to identify areas where ringing may occur and assign a lower weight to those areas.

[0104] Ringing is typically produced by the quantization of high-frequency components during the encoding process. Therefore, it can be assumed that ringing occurs most frequently around strong edges and corners, since these typically lead to frequency responses that contain high-frequency components. In one example, an edge detector can be used to find the strongest edges in a picture. All samples that are a certain distance from an edge and within the same block can be considered candidates for ringing. However, it should be understood that this is only one example of how ringing can be identified. Other methods can be used in addition or instead.

[0105] Likewise, the filter can be applied to other types of errors besides edge blurring, making this approach very flexible.

[0106] In some examples, the optimization described with reference to step 205 of FIG. 1B involves iterating between signal enhancement filter and weighting map function parameters. For example, setting starting weighting map parameters, then optimizing filter parameters based on the current weighting map. Next, optimizing weighting map parameters based on the found filter parameters, and so on. Of course, this is a basic form of optimization procedure. In some cases, additional bounding constraints may be set, e.g., to determine the best filter and weighting map in terms of picture quality, but also to achieve the lowest possible coding rate. This can be achieved by introducing these conditions into both of these individual optimizations and selecting the starting point for the next iteration while also considering rate costs. More generally, additional simplifications that limit computational costs may be introduced.

[0107] It should be understood that there are many ways to perform the rate-distortion optimization described with reference to step 205 of FIG. 1B in variations of this embodiment. One example involves optimizing the weighting maps of each filter according to the following case description: whereby the residual is recalculated based on the results of the previous filter. A second example is the joint optimization of the weighting maps and the signal enhancement filter. In this case, the optimization procedure depends heavily on the weighting map function. However, the most common optimization involves optimizing the signal enhancement filter over all possible weighting maps and selecting the best option in terms of RD cost. To solve this efficiently, simplifications can be applied to find a sufficiently good solution. In the case of a linearly optimizable parametric weighting map, it is possible to perform an iterative method to optimize the filter parameters based on the current weighting map. Then, the weighting map parameters are optimized based on the current filter parameters, and so on. In other words, the signal enhancement filter is a filter with one or more parameters that depend on the weighting map. This can be beneficial because jointly optimizing both would be computationally complex. Assuming one of the components is fixed at each optimization step simplifies the optimization of the remaining parameters.

[0108] In this embodiment, the weighting map provides linear weighting for the signal enhancement filter. However, embodiments are not limited in this respect, and in other embodiments, the values ​​of the weighting map can instead modify the filtering procedure itself. For example, the signal enhancement filter may be parametric. In this case, the frequency response of the edge enhancement filter may depend on the local weighting map parameters. For example, the sigma value in an unsharp masking filter (a type of sharpening filter) may depend on the weighting parameters. This means that the way the filter operates, or more specifically, the function of the filter, is parametric and does not necessarily depend linearly on the weighting map. Another example is a filter that thins (sharpens) edges by distorting an image. The strength of the distortion may depend on the current weighting map values.

[0109] In this embodiment, the weighted signal enhancement filter is applied after upsampling and before any other operations. In particular, for example, the signal enhancement filter is applied before the addition of the residual signal. This position in the processing chain has been shown to be effective. However, embodiments are not limited to this particular order, and in other embodiments, the weighted signal enhancement filter may additionally or alternatively be applied at other processing steps.

[0110] In this embodiment, the signal enhancement filter is an optimized filter for enhancing the upsampled picture, although embodiments are not limited in this respect. For example, in other embodiments, the encoding information may further indicate the particular signal enhancement filter to use and / or particular parameters of the signal enhancement filter.

[0111] Next, a first modification of the first embodiment will be described, in which the weighting map indication information and the weighting map are determined in a manner different from that described above.

[0112] As mentioned above, in the first embodiment, the weighting map indication information explicitly signals the value of a weighting map having a resolution corresponding to the resolution of the upsampled picture block. However, the embodiment is not limited to this. In a first variant of this embodiment, the weighting map function is pre-defined (e.g., stored in a memory of the decoder 10).

[0113] In this first variant, the weighting map indication indicates the parameters (or coefficients) of a preset weighting map to use. In this first variant, step 104 of Figure 1A includes the decoder 10 applying a weighting map function (with parameters encoded in the weighting map indication in the bitstream) to the upsampled picture block to determine the weighting map.

[0114] An example of a weighting map function is the Sobel Magnitude Map given by equation (1).

number

[0115] Another example of a weighting map function is the inverse of this function as shown in equation (2).

number

[0116] However, it should be understood that these are merely examples and that other weighting map functions may be used in addition or instead.

[0117] This method of applying a single weighting map function to a picture block (containing multiple sample values) applies the weighting map function multiple times to the multiple sample values ​​of the picture block, resulting in a weighting map that is calculated in the decoder 10 and that depends on the values ​​of the picture block.

[0118] As can be seen, in this variant, the calculation of the weighting maps is performed in the decoder and there is no need to code the weighting maps into the bitstream, thereby reducing coding costs. Furthermore, since the most appropriate weighting map depends on the picture content, calculating the weighting maps by applying a function to the picture blocks themselves ensures that the decoder can calculate the most appropriate weighting map.

[0119] Next, a second modification of the first embodiment will be described, in which the encoded information further includes parameters used for the signal enhancement filter.

[0120] In a second variant of the first embodiment (which can optionally be combined with the first variant described above), the encoding information further includes filter parameters of a signal enhancement filter. In other words, this second variant still involves the use of a pre-set signal enhancement filter as described above in relation to the first embodiment, but the encoder 20 can indicate which parameters (or coefficients) to use when applying the signal enhancement filter. In other words, the signal enhancement filter is adaptive.

[0121] In this variation, the filter parameters may be explicitly signaled and derived from the video data, or encoder 20 may be instructed to reuse previously signaled coefficients.

[0122] Regarding the point that the filter parameters are derived from the video data, if high-resolution and low-resolution video information is available, it is possible to estimate a filter that is approximately appropriate for the given data. This is the case for a picture immediately after its resolution has been changed to low. However, in this case, it is useful to restrict the filter to a region where it can be assumed that the motion between the high-resolution and low-resolution pictures can be compensated for and that the shape and orientation of objects will not change significantly.

[0123] Additionally, previously decoded video / picture information may be reused to find more optimized filtering parameters. This refers to the idea of ​​using information from multiple pictures to get a more detailed estimate of the location of edges or artifacts. For example, if an edge in a previous picture is discovered and ringing artifacts appear in the next picture that were not previously present, this information can be incorporated into the filter to optimize it so that these artifacts are not emphasized, or in the best case, eliminated. The use of temporal information can aid this estimation and lead to a more accurate estimate.

[0124] One option is to signal that the coefficients of the previous filter are fully reused. A second option is to partially reuse information from the previous filter. This could be, for example, weight map parameters or a subset of the filter coefficients.

[0125] Coding costs can be reduced because only the filter coefficients (and / or instructions to reuse previous coefficients) and weighting maps need to be signaled, rather than the details of the signal enhancement filter function.

[0126] In a third variant, the decoder 10 stores a filter buffer that stores previously used signal enhancement filter parameters. Based on this, the encoder 20 has the option of simply including in the encoding information an indication to use one or more previously used filter parameters, without having to include the specific parameters themselves, thereby reducing encoding costs.

[0127] Similarly, in a fourth variant, a weighting map buffer is stored in the decoder 10 to store previously used weighting map function parameters. Based on this, the encoder 20 has the option of simply including in the encoding information an indication to use one or more previously used weighting map function parameters, without having to include the particular parameters themselves.

[0128] Of course, although these variations are described individually above, this is for ease of explanation only, and any or all of these variations may be combined.

[0129] To illustrate this, an exemplary embodiment of a combination of the first, second, and third variants of the first embodiment will be described with reference to Figures 2A and 2B, where Figure 2A shows a block diagram illustrating an exemplary operation of decoder 10 and Figure 2B shows a block diagram illustrating an exemplary operation of encoder 20.

[0130] As shown in Figure 2A, the decoder 10 obtains coded information 1001 and an upsampled picture block 1002. The upsampled picture block 1002 is obtained in the manner described with reference to steps 102-103 in Figure 1A.

[0131] As described above in relation to the first modification, the coded information 1001 is coded by a preset weighting map function f w-map 1003. When such parameters are applied, the weighting map function f w-map1003 is applied to the upsampled picture block 1002 to obtain a weighting map with a resolution corresponding to the resolution of the upsampled picture block 1002 .

[0132] As described above in relation to the second and third variants, the coding information is transmitted to the coefficients stored in the filter buffer 1005 and / or to the pre-defined signal enhancement filter f filter 1 004. The signal enhancement filter f filter 1 004 is then applied to the upsampled picture block 1002 along with the weighting map, resulting in a signal enhancement filter f filter 1 004 is applied with different weights to different regions of the upsampled picture block 1002, thereby obtaining an enhanced picture block 1006.

[0133] In this example, a complementary method is performed by the encoder 20, as shown in Figure 2B. In a complementary manner to the block diagram of Figure 2A, it can be seen that the original video (or picture block) 1007a and the upsampled picture block 1002a are used as inputs. This is followed by the weighting map function f w-map 1003a, a filter buffer 1005a, and a filter coefficient optimizer 1008a are performed to generate coded information 1001a. In particular, the encoder 20 takes as input the original video data 1007a (or simply the original picture block) and the upsampled picture block 1002a, and performs filter coefficient optimization in the optimizer 1008a. This filter coefficient calculation can be used in an adaptive loop filter. To derive the optimal filter coefficients, a least-squares optimization problem, already known from the Wiener filter, is formulated.

[0134] We start with the linear (signal enhancement filter) equation:

number

[0135] JPEG2024174072000005.jpg19167

[0136] To calculate the optimal filter coefficients, a least squares optimization can be performed.

number

[0137] JPEG2024174072000007.jpg45168

number

[0138] In some embodiments, the method is used after resampling of reference pictures or for filtering upsampled images in multi-resolution coding. In both applications, the filter can be applied after the upsampling step.

[0139] In one example, the first weighting map is a Sobel magnitude map, defined as follows:

number

[0140] In this example, the second weighting map is a complementary map.

[0141] Note that s is a normalization factor, defined so that the sharpest edge yields an edge value of 1. As a result, m edge The values ​​of are in the range [0,1], but may not cover the entire range in some circumstances.

number

[0142] Since there are two weighting maps, two signal enhancement filters are optimized and the calculated offsets obtained by applying these filters to the upsampled image are added to obtain the filtered picture.

[0143] These optimized filter coefficients improve the encoder's ability to enhance picture quality. In some embodiments, the signal enhancement filter is a linear filter represented by a linear equation such as equation (3) above. Generally, such linear filters cannot be used to solve nonlinear problems. For example, edge sharpening and super-resolution problems are generally nonlinear problems, which limits the ability of such signal enhancement filters to solve these problems. However, the additional use of the weighting map described above overcomes this problem. In particular, even though the weighting map function used to calculate the weighting map is also linear, this combined use of two linear functions (i.e., the weighting map function and the signal enhancement filter function) makes it possible to solve these nonlinear problems.

[0144] It can be seen that these variations include the use of weighting map calculation parameters and filter parameters of an adaptive signal enhancement filter, whereby the filter coefficients are optimized to enhance the current picture block.

[0145] In addition to what is described above with respect to the first embodiment variant, the embodiments are not limited to the use of a single preset weighting map function and / or a single preset function. In some embodiments, multiple preset weighting map functions and / or signal enhancement filters are available (e.g., stored in memory of decoder 10). In such variants, in addition to the information described above with respect to the first and / or second variants, the encoding information further includes one or more identifiers that identify which weighting map function and / or signal enhancement filter to apply.

[0146] Thus, in some embodiments, multiple functions for the calculation of weighting maps are pre-defined, and only the calculation parameters and weighting map identifiers need to be signaled in the bitstream.Furthermore, in some embodiments, multiple signal enhancement filters are pre-defined, and only the calculation parameters and signal enhancement filter identifiers need to be signaled in the bitstream.

[0147] Although the first embodiment has been described with reference to a single signal enhancement filter and a single weighting map, embodiments are not so limited, and in some variations of the first embodiment, multiple signal enhancement filters and weighting maps are used instead.

[0148] In one such variation, multiple signal enhancement filters with multiple respective weighting maps are applied to a single picture block (i.e., multiple signal enhancement filters with multiple respective weighting maps are applied to the same region of a picture), and in this variation, signal enhancement filter identifiers are coded into the bitstream along with weighting map indication information for each identified signal enhancement filter.

[0149] A general overview of the first embodiment and applicable variants is given below. By applying a weighting map together with a signal enhancement filter, it is possible to improve the quality of the upsampled picture. Applying a weighted signal enhancement filter before the addition of the residual signal (and before any other processing steps after upsampling) has been shown to be an effective position in the processing chain. However, this is not necessary, and a weighted signal enhancement filter may also be applied at other processing steps.

[0150] As mentioned above, to find the best filter settings, an optimization based on a given weighting map is performed on the encoder side. The encoder 20 needs to find the filter coefficients to use for the current picture (or picture block).

[0151] In some aspects, the filter coefficients are optimized: each filter may be limited to a particular block, which can also be taken into account in the optimization.

[0152] The filter coefficients may be explicitly signaled, may be derived from the video data, or the encoder 20 may indicate the reuse of previously signaled filter coefficients. Additionally, previously decoded video information may be reused to find further optimized filtering coefficients.

[0153] The signal enhancement filtering process is a two-step procedure. The first step is to determine the weighting map. In some aspects, the calculation of the weighting map can be performed by any function applied to the upsampled picture blocks.

[0154] In some aspects, the encoder 20 may provide parametric information for the weighting map calculation to select the calculation parameters. This is advantageous because the most appropriate weighting map depends on the image content. The result is a weighting map that provides spatial information about the filter weights.

[0155] The weighting map is used in the next step, where the picture block is filtered using a signal enhancement filter, with the local strength of the filtering operation being given by the weighting map. The exact implementation of the strength modification by the weighting map depends on the implementation method and may, for example, be a linear weighting of the offset calculated by the filter, or may modify the filtering procedure itself.

[0156] Applying the weighted signal enhancement filter in the manner described above serves to reduce distortions caused by low-resolution video encoding. These are the loss of high-frequency information and distortions caused by video encoding. A default upsampling filter can be used for the initial resolution change, and then the weighted signal enhancement filter described above is applied independently. In other words, this weighted signal enhancement filter does not modify the existing resampling (or upsampling) process, but rather adds / improves an enhancement step.

[0157] For example, the steps involved in generating an enhanced upsampled picture block at the decoder side are outlined in Figures 1A and 2A. At the decoder side, an upsampled image (e.g., already upsampled by a default upsampling process) is obtained. In addition, coding information is obtained that can specify aspects such as the operating mode.

[0158] In some aspects, the encoding information includes local on / off flags for different filters, weighting map functions, and parameters for those functions. Additionally, an encoding of the filter coefficients is transmitted. In some aspects, this encoding utilizes filter coefficients previously transmitted from a filter buffer to reduce encoding costs.

[0159] In some aspects, after the filter parameters are decoded, a weighting map is calculated by applying a weighting map function to the picture block. The weighting map function may be any function that maps an input picture block to an output picture block, and need not necessarily be linear. The filter receives the weighting map and the upsampled image as input. The result of the filter operation is an enhanced picture block.

[0160] Depending on the configuration, in some aspects, multiple filters are applied to a single picture. These filters may be applied to partitions (or "blocks") of the picture according to rate-distortion criteria to account for different local image distortion characteristics. Furthermore, multiple filters can be applied to the same image region (or "block" or "partition") with different weighting maps or parameters to reduce different types of artifacts in that image region.

[0161] The inputs for the optimization operation in the encoder 20 are the upsampled picture blocks (or video) and , former The weighting map is a ground truth picture block (or video). In some implementations, the optimizer generates optimized filter coefficients from the upsampled picture block (or video) using least squares optimization for a given weighting map.

[0162] In some aspects, a set of multiple weighting maps is determined, and the decoder 10 selects a respective set of signal enhancement filters to apply and a picture partition (to divide the picture into blocks).

[0163] Additionally, in some aspects, signal enhancement filter parameters are optimized, selectively considering reusing parameters from a previous configuration of the filter buffer. There are several options for reusing filter parameters. One option is to signal that the previous filter parameters are fully reused. A second option is to partially reuse information from the previous filter. This could be, for example, weight map parameters or a subset of filter coefficients.

[0164] An exemplary implementation of the first embodiment includes the use of a linear filter. In this case, the encoder 20 determines the filter coefficients by least-squares optimization. It is assumed that weighting is applied by multiplying a weighting map to the filtered image. It is further assumed that the output is calculated by adding the weighted and filtered picture to the input picture. Note that even in this case where a linear filter is applied to the upsampled picture, the multiplication with the weighting map allows the overall system to solve nonlinear problems.

[0165] Figure 3 shows a schematic diagram of a decoder 10 according to one embodiment. In particular, Figure 3 shows a schematic diagram of a decoder 10 configured to perform any of the decoder methods described herein, a detailed description of which is omitted here for the sake of brevity.

[0166] 3, the decoder 10 includes a processor 11 and a computer-readable medium 12. The processor 11 and the computer-readable medium 12 may be connected via a bus system. The computer-readable medium is configured to store programs, instructions, or codes. The processor 11 is configured to execute the programs, instructions, or codes in the computer-readable medium 12 to complete the operations in the decoder method embodiments herein.

[0167] Thus, in an embodiment, the computer-readable medium 12 is configured to store a computer program executable by the processor 11, and the processor 11 is configured to execute the computer program to perform any of the steps of the decoder methods described herein.

[0168] Figure 4 shows a schematic diagram of an encoder 20 according to one embodiment. In particular, Figure 4 shows a schematic diagram of an encoder 20 configured to perform any of the encoder methods described herein, a detailed description of which is omitted here for the sake of brevity.

[0169] As shown in Fig. 4, the encoder 20 includes a processor 21 and a computer-readable medium 22. The processor 21 and the computer-readable medium 22 may be connected via a bus system. The computer-readable medium is configured to store programs, instructions, or codes. The processor 21 may be a Encoder It is configured to execute the programs, instructions, or code in the computer-readable medium 22 to complete the operations in the method embodiments.

[0170] Thus, in an embodiment, the computer readable medium 22 is configured to store a computer program executable by the processor 21, which may be any of the programs described herein. Encoder It is configured to execute a computer program to perform any of the steps of the method.

[0171] As described in detail above, embodiments provide an in-loop filtering process for the refinement of upsampled video, in which a local weighting map is used in the filtering process.

[0172] In some embodiments, multiple filters are applied to the same picture with different weighting maps.

[0173] In some embodiments, multiple filters are applied to different regions of the picture.

[0174] In some embodiments, several functions for the calculation of the weighting maps are pre-defined and only the calculation parameters and the weighting map identifiers need to be signaled.

[0175] In some embodiments, the signal enhancement filter is applied after the interpolation filter in the resampling of the reference picture.

[0176] In some embodiments, a signal enhancement filter is applied to the upsampled low resolution picture before it is presented to the viewer.

[0177] In some embodiments, the signal enhancement filter is applied after the interpolation filter in multi-resolution coding.

[0178] The use of signal enhancement filters is not limited to these described applications; they merely provide an overview of highly suitable application areas. In general, signal enhancement filters can be applied in any signal processing setting that requires signal enhancement and has properties that can be effectively utilized by a weighted filtering setting. This is not limited to the field of video coding / processing, but may also be applied, for example, to image coding / processing or audio coding / processing.

[0179] Embodiments of the present invention may further provide a computer-readable medium having computer-executable instructions for causing one or more processors of a computing device to perform any of the methods of the embodiments of the present invention.

[0180] Examples of computer-readable media include both volatile and non-volatile media, removable and non-removable media, including, but not limited to, solid-state memory, removable disks, hard disk drives, magnetic media, and optical disks. Generally, computer-readable media includes any type of medium suitable for storing, encoding, or transmitting a sequence of instructions executable by one or more computers to perform any one or more of the processes and functions described herein.

[0181] It should be understood that the functionality of each of the described components can be combined in many ways other than those described above. For example, in some embodiments, the functionality of multiple described devices can be combined into a single device. In other embodiments, at least one function of the described devices can be split across multiple separate (or distributed) devices.

[0182] Conditional language such as "may" is generally used to indicate that a feature / step is used in a particular embodiment, but that alternative embodiments may include alternative features or may omit such features entirely.

[0183] Furthermore, it should be understood that the method steps are not limited to the particular sequence described, but may be combined in any other suitable order, which may result in some method steps being performed in parallel in some embodiments, and furthermore, certain method steps may be omitted entirely in some embodiments.

[0184] While specific embodiments have been described, it should be understood that these are used to illustrate the general teachings of the present invention and that various modifications can be made without departing from the scope of the present invention, which is to be interpreted according to the appended claims and their equivalents.

[0185] Many further variations and modifications will occur to those skilled in the art upon reference to the foregoing exemplary embodiments, which are given by way of example only and are not intended to limit the scope of the invention, which is determined by the appended claims.

Claims

1. 1. A video data processing method performed by a decoder, comprising: decoding the bitstream to obtain video data and encoding information, the encoding information including weighting map indication information for defining a weighting map and filter coefficients optimized for the weighting map; obtaining a picture block based on the video data; upsampling the picture block; determining the weighting map using the weighting map indication; applying a signal enhancement filter using the filter coefficients together with a weighting map to the upsampled picture block to obtain an enhanced picture block; method.

2. the weighting map comprises a scalar weighting map; The method of claim 1.

3. the weighting map comprises a Sobel intensity map; The method of claim 1.

4. the weighting map includes a plurality of weight values ​​each corresponding to a value of the upsampled picture block; The method of claim 1.

5. the signal enhancement filter indication information instructs reusing one or more filter coefficients stored in a filter buffer of the decoder for the signal enhancement filter; The method of claim 1.

6. determining the weighting map using the weighting map indication, determining a weight map function using the weight map indication; and calculating the weighting map by applying the weighting map function to the upsampled picture block. The method of claim 1.

7. the weighting map indication information includes a weighting map identifier that identifies one of a plurality of preset weighting map functions; The method of claim 6.

8. the weighting map indication information includes parameters of the weighting map function; The method of claim 6.

9. the picture block is a predicted block, and obtaining the picture block based on the video data includes performing a prediction operation using the video data to obtain the prediction block. The method of claim 1.

10. The prediction operation is inter prediction or intra prediction.

10. The method of claim 9.

11. the picture block is a reference sample; The method further comprises: obtaining a prediction block by performing a prediction operation using the enhanced reference sample; The method of claim 1.

12. the prediction operation includes inter prediction; the reference sample corresponds to a first picture of the video data encoded in the bitstream; the predictive block corresponds to a second picture of the video data encoded in the bitstream, the second picture being temporally separated from the first picture; the first picture is encoded in the bitstream at a lower resolution than the second picture; The method of claim 11.

13. the encoding information indicates application of a plurality of filters and a plurality of respective weighting maps to the picture block; The method of claim 1.

14. the coding information indicates the use of different weighting maps and / or signal enhancement filters for different picture blocks of a picture; The method of claim 1.

15. one or more processors; a computer-readable medium comprising computer-executable instructions stored on the computer-readable medium, the computer-executable instructions, when executed by the one or more processors, causing the one or more processors to perform a method according to any one of claims 1 to 14; decoder.

16. 1. A video data processing method performed by an encoder, comprising: obtaining original video data; obtaining a downsampled version of the original video data; obtaining a picture block based on the downsampled original video data; upsampling the picture block; obtaining a weighting map from the original video data; defining a linear equation representing a signal enhancement filter that calculates an enhancement picture block based on the weighting map, filter coefficients, and the upsampled picture block; applying least squares optimization to the linear equation to obtain optimal filter coefficients for the weighting map; applying the signal enhancement filter using the optimal filter coefficients together with the weighting map to the upsampled picture block to obtain an enhanced picture block; encoding the downsampled original video data and encoding information into a bitstream, the encoding information including weight map indication information indicating the weight map and the calculated filter coefficients; method.

17. 1. A computer-readable medium, comprising: and a bitstream stored on the computer-readable medium, the computer-executable instructions, when executed by a processor of a computing device, causing the computing device to perform the method of claim 16 and generate the bitstream. Computer-readable medium.

18. one or more processors; a computer-readable medium comprising computer-executable instructions stored thereon, the computer-executable instructions, when executed by the one or more processors, causing the one or more processors to perform the method of claim 16; Encoder.

Images