Bilateral filter

Abstract

A method for filtering input samples. The method includes obtaining a first input sample, SIN_1, belonging to a first transform unit (TU) (e.g., one sample included in the TU) having a first dimension (e.g., width) and a second dimension (e.g., height). The method also includes using a first TU scale factor, CTU_1, associated with the first TU and a bilateral filter, BIF, function to calculate a first BIF offset, δBIF_1, for the first input sample. The method further includes calculating a first output sample, Sout_1, using δBIF_1 and SIN_1. The first TU scale factor is a function of: 1) a two-dimensional, 2D, lookup table, LUT, and the magnitudes of the first and second dimensions of the first TU and / or 2) an activity score (e.g., MAD, MSD) for the first TU, and / or the BIF function is a sampled piecewise linear function.

Description

BILATERAL FILTER TECHNICAL FIELD

[0001] This disclosure relates to bilateral filtering. BACKGROUND

[0002] VVC and ECM

[0003] Versatile Video Coding (VVC), also known as H.266, is a block-based video codec developed by the Joint Video Experts Team (JVET), a team of experts from ITU-T VCEG and ISO / IEC MPEG, and first published in 2020.

[0004] JVET continues study on future video codec providing coding capabilities beyond VVC. The exploratory codec is called Enhanced Compression Model (ECM). The current ECM version is ECM-9.0. ECM is used for video compression.

[0005] Video Sequence, Picture, Color Components

[0006] A video (a.k.a., “video sequence”) comprises of a series of pictures. In VVC, each picture is identified with a picture order count (POC) value. The POC value also represents the display order of the picture. A picture with a smaller POC value is displayed before another picture with a larger POC value.

[0007] Each picture consists of 2-dimensional arrays (a.k.a., blocks) of color components. Usually, each picture has three components: one luma component (Y) and two chroma components (Cb, Cr). Other color spaces can also be used, such as I, Cp, Ct. Usually, the chroma components are half the size in both the x and y dimension compared to the luma component.

[0008] CTU, CU and TU

[0009] In VVC and ECM, each component is split into square top-level blocks called Coding Tree Units (CTUs). The maximum supported size of a CTU is 256x256 in ECM-9.0. Each CTU is split into rectangular Coding Units (CUs). CU size varies from 4x4 to 256x256 in ECM-9.0. A CU consists of one or more Transform Units (TUs). Each TU is transform coded and typically varies in size from 4x4 to 128x128. If the size of the CU is higher than the maximal supported transform length, then the CU is split into two or more TUs.

[0010] Parameter Sets and Headers

[0011] ECM specifies three types of parameter sets: The picture parameter set (PPS), the sequence parameter set (SPS), and the video parameter set (VPS). The PPS contains data that is common for a whole picture, the SPS contains data that is common for a coded layer video sequence (CLVS), and the VPS contains data that is common for multiple CLVSs.

[0012] A picture is split into slices that consists of several CTUs. The decoding of one slice in the picture is independent of other slices of the same picture. Pictures, slices and CTUs have headers comprising syntax elements used by the decoder. Correspondingly, coding parameters in the picture header are common for all slices in the picture, coding parameters in the slice header are common for all CTUs in the slice.

[0013] Intra / Inter Prediction

[0014] Each CU is either intra predicted or inter predicted. If it is intra predicted, it is predicted using previously decoded blocks in the same picture. An intra picture is a picture where all CUs must be intra predicted. In inter prediction, the CU is predicted using previously decoded pictures. A picture that allows inter predicted blocks is called an inter picture.

[0015] Transform, Quantization, Entropy Coding

[0016] The difference between the original samples and predicted samples of a CU is called the residual. The residual is transformed into the frequency domain using one of the transforms supported by ECM, for instance the Discrete Cosine Transform (DCT) or the Discrete Sine Transform (DST). The resulting transform coefficients are quantized and entropy coded and the resulting bits are put into the bitstream. The level of quantization depends on the QP parameter used for the CU. In some settings, all CUs belonging to the same picture will have the same QP parameter value.

[0017] In-Loop Filters

[0018] FIG.1 illustrates a filtering flow. Because transform coefficients are quantized, video encoding produces losses. To reduce some quantization artifacts and improve video quality, in-loop filters are introduced. In-loop filtering is done on the decoder side as well as on the encoder side to use filtered pictures for inter-prediction. There are four in-loop filters in ECM-9.0: the deblocking filter (DBF), the sample adaptive offset (SAO), the bilateral filter(BIF), and the adaptive loop filter (ALF). and ALF are supplemented by cross- component (CC) counterparts.

[0019] Bilateral filter (BIF)

[0020] A BIF is designed to reduce the ringing artifacts caused by quantization of high frequency coefficients. The Gaussian form of BIF is introduced in reference [1] where its ability to improve image quality is demonstrated. The use for video coding and insertion into in-loop filtering in parallel with SAO is developed in reference [2]. State-of-the-art BIF included into ECM-9.0 is presented in a standardization contribution reference [3] by Ericsson and Huawei.

[0021] SAO, CC-SAO, and BIF use the samples output from the DBF as input samples, where each input sample belongs to a TU (e.g., each input sample is a member of a TU, which has two-dimensions (width, length) and is associated with a QP). Correspondingly, for each output sample (Sout) three offsets are calculated and clipped as shown below (where Sin is the input sample):^^ை^் ൌ ^^ ^^ ^^ ^^^^^ூே ^ ^^ௌ^ை ^ ^^^^ௌ^ை ^ ^^^ூி^.

[0022] FIG.2 illustrates a 5x5 star filtering shape that is used to calculate the BIF’s offset (i.e., ^^^ூி).

[0023] The BIF’s offset equals a sum of 12 offsets: ^^^^ ^^ ^^ൌ ൫ ^^^^ ^^⋅ ^ ∑ ^^ ^^ ^^ ^^൫ ^^^^, ^^െ ^^^^, ^^൯ ⋅ ^^^^ ^^ ^^^ ^^^^, ^^^൧ ^ ^^ ^^൯ ≫ ^^, where CTUismode andsize of the current TU (i.e., the TU to which SIN belongs), and FBIF( ^^ ^^, ^^) =LUT^,^,^^൫min ^ ൫ ^^ ^^, ^^ ^ 4൯ ≫ 3, 15 ^ ൯ for the innermost{i,j}={-1,0},{1,0},{0,-1}, and FBIF( ^^^^, ^^) = LUT^,^,^^൫min ^൫^^^^, ^^^ 4൯≫ 3, 15 ^൯>> 1 for theremaining positions, where ^^^^, ^^= ห ^^^,^െ ^^^,^ห. The lookup-table (LUT) depends on the neighboring sample locations and the current QP value. The three leastsignificant bits of the sample differences are cut off (with rounding), and the parameters 16 and 5 used in the latest stage are referred to as the BIF strength and are defined in a PPS.

[0024] All LUTs are defined by 26 rows (for QP from 17 to 42) of 16 integer values (signed 8-bit integers), i.e., base LUTs (LUTୠୟ^^,^^). For ^ ^^, ^^^ ൌ ^െ1,0^, ^1,0^, ^0,െ1^ or ^0,1^LUT^,^,^^^^^^ൌ LUTୠୟ^^,୫୧୬^^^,ସଶ^^ ^^^, LUT^,^,^^^^^^ൌ LUTୠୟ^^,୫୧୬^^^,ସଶ^^^^^≫ 1.Let ^^^^ൌ min ^width^^, height^^^. ^^^^is defined by Table 1. Table 1 - TU scale factor in ECM-9.0 Prediction mode ^^^^^ 4 4 ^ ^^^^^ 16 ^^^^^ 16 Inter-prediction2 2 1 Intra-prediction and others 3 2 1

[0025] BIF is turned off if QP ^ 17 or max^width^^, height^^^^ 128. For inter- prediction, it is also turned off if ^^^^^ 32 or CBF flag is 0, i.e., if transform coefficients are not presented.

[0026] In the slice header, it is indicated if BIF is switched on for all CTUs in the slice, switched off for all CTUs in the slice, or if the on / off flag of BIF is signaled individually for each CTU in the CTU header.

[0027] In reference [3] a bit-rate saving of 0.45% / 0.39% / 0.44% with 3% / 0% / 0% decoding time increase is reported for luma for All Intra (AI) / Random Access (RA) / Low delay B (LDB) over ECM version on 2021 / 03 / 30. For chroma, the same solution is used, the only difference is that another values for the base lookup table LUTୠୟ^^,^^are used. SUMMARY

[0028] Certain challenges presently exist. For example, there is a need to increase the performance of the conventional BIF.

[0029] Accordingly, in one aspect, there is provided a method for filtering input samples. The method includes obtaining a first input sample, SIN_1, belonging to a first transform unit (TU) (e.g., one sample included in the TU) having a first dimension (e.g., width) and a second dimension (e.g., height). The method also includes using a first TU scale factor, CTU_1, associated with the first TU and a bilateral filter, BIF, function to calculate a first BIF offset, δBIF_1,for the first input sample. The method further includes calculating a first output sample, Sout_1, using δBIF_1and SIN_1. The first TU scale factor is a function of: 1) a two-dimensional, 2D, lookup table, LUT, and the magnitudes of the first and second dimensions of the first TU and / or 2) an activity score (e.g., MAD, MSD) for the first TU, and / or the BIF function is a sampled piecewise linear function.

[0030] In another aspect there is provided an apparatus that is configured to perform the methods disclosed herein. The apparatus may include memory and processing circuitrycoupled to the memory.

[0031] In another aspect there is provided a computer program comprising instructions which when executed by processing circuitry of an apparatus causes the apparatus to perform any of the methods disclosed herein. In one embodiment, there is provided a carrier containing the computer program wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[0032] An advantage of the embodiments disclosed herein is that they improve compression efficiency, measured as Bjontegaard delta rate (BDR), compared to the conventional BIF. Furthermore, the embodiments achieve the efficiency improvement with a negligible encoder / decoder complexity increase both in terms of encoder and decoder run time and in terms of number of arithmetic operations performed. The embodiments achieve bit-rate savings of 0.14% / 0.11% / 0.04% for AI / RA / LDB with 1% of decoder run time increase. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0034] FIG.1 illustrates a conventional filtering flow.

[0035] FIG.2 illustrates a shape used to calculate a BIF’s offset.

[0036] FIG.3 illustrates a system according to some embodiments.

[0037] FIG.4 illustrates examples of BIF function forms.

[0038] FIG.5 is a flowchart illustrating a process according to some embodiments.

[0039] FIG.6 is a block diagram of an apparatus according to some embodiments. DETAILED DESCRIPTION

[0040] FIG.3 illustrates a system 300 according to an embodiment. System 300 includes an encoder 302 and a decoder 304, wherein encoder 302 is in communication with decoder 304 via a network 110 (e.g., the Internet or other network). Encoder 302 encodes a source video sequence 301 (e.g., encodes blocks of units of pictures of video sequence 301) into a bitstream comprising an encoded video sequence (e.g., encoded blocks) and transmits the bitstream to decoder 304 via network 110. In some embodiments, encoder 302 is not in communication with decoder 304, and, in such an embodiment, rather than transmittingbitstream to decoder 304, the bitstream is in a data storage unit for later use. Decoder 304 decodes the pictures included in the encoded video sequence to produce video data for display and / or for further image processing (e.g. a machine vision task). Accordingly, decoder 304 may be part of a device 303 having an image processor 305 and / or a display 306. The image processor 305 may perform machine vision tasks on the decoded pictures. The device 303 may be a mobile device, a set-top device, a head-mounted display, or any other device.

[0041] As noted above, there is a need to improve the efficiency of the conventional BIF. Accordingly, this disclosure provides, among other things, an embodiment in which the BIF offset is calculated using additional characteristics of the TU, such as, for example, shape, size, and activity score (e.g., mean absolute difference (MAD), mean squared difference, etc.). The disclosure also provides a generalization of the BIF’s function as a linear continuation of LUTs. Accordingly, in some embodiments, the TU scale factor (CTU) depends on the TU shape size and / or the activity score (e.g., MAD) of the TU, and / or the BIF function is a sampled piecewise linear function (e.g., a step function (a.k.a., piecewise constant function)).

[0042] Embodiments

[0043] In one embodiment, the TU scale factor (CTU) is calculated as a sum of two scale factors: ^^^^ൌ ^^௪^^,^^ ^^^^ୗ^,

[0044] where ^^௪^^,^is based on the TU’s shape sizes and ^^^^ୗ^is based on an activity score (e.g., based on the mean absolute difference (MAD) of the TU). In one embodiment, LUT௪,^is a 2D 8x8 lookup table with non-negative 8-bit integer values, example of such table is presented in Table 2, and let LUT^ୗbe 1D 16-entry lookup table with non-negative 8-bit integer values as exemplified in Table 3. Then these scale factors are defined as follows: ^^௪^^,^ൌ LUT௪,^^logଶwidth^^, logଶheight^^^, ^^^^ୗ^ൌ LUT^ୗ^min^AS^^≫ 4, 15^^. Table 2 - Example of TU's shape size LUTw,h24 24 24 22 22 19 11 724 24 24 22 18 11 7 24 24 24 20 20 18 11 7 22 22 20 15 14 14 11 7 22 22 20 14 11 11 11 7 19 18 18 14 11 11 10 7 11 11 11 11 11 10 10 7 7 7 7 7 7 7 7 7 Table 3 - Example of TU's AS LUT 0 0 0 1 2 5 6 8 8 12 12 12 12 16 16 16

[0045] The MAD of a real-valued random variable ^^ equals: MAD^ ^^^ ൌ ^^| ^^ െ ^^^ ^^^|, where ^^^ ^^^ is the expected value of the random variable. In practice, one may calculate (or estimate) ^^^ ^^^ as the average of the components of the 2-dimensional vector ^^, henceMAD^ ^^^ ൌ ^^௪ ∑^^ୀ^ ∑௪^ୀ^ ^ ^^^,^ െ ^^௪∑^^ୀ^ ∑௪^ୀ^ ^^^,^ ^ , where ℎ and ^^ are the number of rowsThe AS for TU iscalculated from the current channel (Y, Cr or Cb) samples in the TU. ^^^^is constant for all samples belonging to the same TU (e.g., all sample inside the TU).

[0046] In one embodiment, the function ^^^୍^൫ ^^^,^൯ is generalized as follows. As before, ^^^୍^൫ ^^^,^൯ is defined by 2D 26x16 base lookup table. For simplicity, we denote ^^ ൌ ห ^^^,^ห and. Let ^^ be the number of reduced bits (3 for luma and 2 forconstruct piecewise linear continuation of LUT, one takes the closest left and right LUT entries, i.e., ^^^ൌ LUT^ ^^ ≫ ^^^ and ^^ଶൌ LUT൫^ ^^ ^ ^1 ≪ ^^^ െ 1^ ≫ ^^൯, the reminder ^^ ൌ ^^&൫^1 ≪ ^^^ െ 1൯and calculates the linear continuation as: ^^^୍^^ ^^^ ൌ^^^^⋅൫^1 ≪ ^^^ െ ^^൯^ ^^ଶ⋅ ^^ ^൫1 ≪ ^ ^^ െ 1^൯^≫ ^^. Instead of the latest summand൫1 ≪ ^ ^^ െ 1^൯ used for rounding one can use other constants.

[0047] To avoid multiplications (complex arithmetic operations) and decrease overall complexity of the piecewise linear form of the BIF’s function, a stepwise approximation can be used where the step size equals ൫1 ≪ ^ ^^ െ 1^൯. Denote adjusted difference by ^^ᇱ ൌ ^^^,^ ^^^1, adjusted left and right entries by ^^ᇱ^ ൌ LUT^ ^^ᇱ ≫ ^^^ and ^^ᇱ ᇱଶ ൌ LUT൫^ ^^ ^൫1 ≪ ^ ^^ െ 1^൯^ ≫ ^^൯and final BIF’s value is calculated as: ^^^୍^^ ^^^ ൌ^ ^^^ᇱ^ ^^ଶᇱ^ ^^2^ ≫ 1, where ^^1 and ^^2 are defined constants. For example, in some embodiments, ^^1 ൌ 1 ≪ ^ ^^ െ 2^ and ^^2 ൌ 1.

[0048] FIG.4 shows Examples of three types of ^^^୍^^ ^^^ with ^^ ൌ 3 and base LUT defined by Table 4. Table 4 - Base LUT example 0 38 48 52 60 62 60 58 58 48 44 38 32 26 18 -14

[0049] FIG.5 is a flowchart illustrating process 500, according to some embodiments, for filtering input samples. The process 500 may begin with step s502.

[0050] Step s502 comprises obtaining a first input sample, SIN_1, belonging to a first transform unit (TU) (e.g., a sample included in the TU) having a first dimension (e.g., width) and a second dimension (e.g., height).

[0051] Step s504 comprises using a first TU scale factor, CTU_1, associated with the first TU and a bilateral filter, BIF, function to calculate a first BIF offset, δBIF_1,for the first input sample. The first TU scale factor is a function of: 1) a two-dimensional, 2D, lookup table, LUT, and the magnitudes of the first and second dimensions of the first TU and / or 2) an activity score (e.g., MAD, MSD) for the first TU. In some embodiments, the BIF function is a sampled piecewise linear function.

[0052] Step s506 comprises calculating a first output sample, Sout_1, using δBIF_1and SIN_1.

[0053] FIG.6 is a block diagram of an apparatus 600 for implementing encoder 302 and / or decoder 304, according to some embodiments. When apparatus 600 implements encoder 302, apparatus 600 may be referred to as an encoder apparatus, and when apparatus 600 implements decoder 304, apparatus 600 may be referred to as a decoder apparatus. As shown in FIG.6, apparatus 600 may comprise: processing circuitry (PC) 602, which may include one or more processors (P) 655 (e.g., one or more general purpose microprocessors and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (i.e., encoder apparatus 600 may be a distributed computing apparatus); at least one networkinterface 648 (e.g., a physical interface or air comprising a transmitter (Tx) 645 and a receiver (Rx) 647 for enabling apparatus 600 to transmit data to and receive data from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network) to which network interface 648 is connected (physically or wirelessly) (e.g., network interface 648 may be coupled to an antenna arrangement comprising one or more antennas for enabling encoder apparatus 600 to wirelessly transmit / receive data); and a storage unit (a.k.a., “data storage system”) 608, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 602 includes a programmable processor, a computer readable storage medium (CRSM) 642 may be provided. CRSM 642 may store a computer program (CP) 643 comprising computer readable instructions (CRI) 644. CRSM 642 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 644 of computer program 643 is configured such that when executed by PC 602, the CRI causes encoder apparatus 600 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, encoder apparatus 600 may be configured to perform steps described herein without the need for code. That is, for example, PC 602 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0054] Summary of Various Embodiments

[0055] A1. A method (see e.g., process 500) for filtering input samples, the method comprising: obtaining a first input sample, SIN_1, belonging to a first transform unit (TU) (e.g., one sample included in the TU) having a first dimension (e.g., width) and a second dimension (e.g., height) (step s502); using a first TU scale factor, CTU_1, associated with the first TU and a bilateral filter, BIF, function to calculate a first BIF offset, δBIF_1,for the first input sample (step s504); and calculating a first output sample, Sout_1, using δBIF_1and SIN_1(step s506), characterized in that: the first TU scale factor is a function of: 1) a two-dimensional, 2D, lookup table, LUT, and the magnitudes of the first and second dimensions of the first TU and / or 2) an activity score (e.g., MAD, MSD) for the first TU, and / or the BIF function is a sampled piecewise linear function.

[0056] A2. The method of embodiment A1, wherein the TU scale factor is a function of the 2D LUT and the magnitudes of the first and second dimensions of the first TU.

[0057] A3. The method of embodiment A2, further comprising: obtaining a first look-up coordinateusing the magnitude of the first dimension of obtaining a second look-up coordinate using the magnitude of the second dimension of the TU; and using the look-up coordinates to select a scale factor from the 2D LUT, wherein CTUis a function of the scale factor selected from the 2D LUT.

[0058] A4. The method of embodiment A3, wherein the first look-up coordinate, x, is a function of the magnitude of the first dimension; the second look-up coordinate, y, is a function of the magnitude of the second dimension.

[0059] A5. The method of embodiment A4, wherein x = log2(W), were W is the magnitude of the first dimension of the first TU, and y = log2(H), were H is the magnitude of the second dimension of the first TU.

[0060] A6. The method of any one of embodiments A1-A5, wherein the first TU scale factor is a function of the activity score for the first TU.

[0061] A7. The method of embodiment A6, wherein the first TU scale factor is a function of a scale factor selected from a one-dimensional, 1D, LUT using a look-up index, j, that is a function of the activity score, AS.

[0062] A8. The method of embodiment A7, wherein j is equal to min(AS >> s, L), where L is the length of the 1D LUT, and s is a defined value (e.g., s = 4).

[0063] A9. The method of embodiment A1, wherein the first TU scale factor is equal to the sum of SF1 and SF2, SF1 is a function of the 2D LUT and the magnitudes of the first and second dimensions of the first TU, and SF2 is a function of the activity score for the first TU.

[0064] A10. The method of embodiment A9, wherein the BIF function is a sampled piecewise linear function.

[0065] B1. A computer program (643) comprising instructions (644) which when executed by processing circuitry (602) cause the processing circuitry (602) to perform the method of any one of the above embodiments.

[0066] B2. A carrier containing the computer program of embodiment B1, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (642).

[0067] C1. An apparatus (600) for encoding a picture, the apparatus configured to perform the method of embodiment A1.

[0068] C2. The apparatus of embodiment C1, wherein the apparatus is further configured to perform the method of any one of embodiments A2-A10.

[0069] D1. An apparatus (600), the apparatus comprising: a memory (642); and processing circuitry (602) coupled to the memory (642), wherein the apparatus (600) is configured to perform the method of any one of embodiments A1-A10.

[0070] Additional Disclosure:

[0071] As noted above, the BIF offset equals a sum of 12 offsets: ^^^ூிൌ ൫ ^^^^⋅ ^ ∑sign൫ ^^^,^െ ^^^,^൯ ⋅ ^^^୍^,^,^,^^൫ห ^^^,^െ ^^^,^ห൯൧ ^ 16൯ ≫ 5, ^ ^^ ^^.1^(for 26 QPs from 17 to 42 and 16 intervals of sample difference). For the innermost positions, i.e., ^^^, ^^^ ൌ ^െ1,0^, ^1,0^, ^0,െ1^ or ^0,1^, the LUT is used as-is, i.e., ^^^୍^,^,^,^^^ ^^^ ൌ3, 15 ^^. For the remaining positions, the LUT outputit right shifted: ^^^୍^,^,^,^^^ ^^^ ൌ LUTୠୟ^^,ୡ୪୧୮^^^,^^,ସଶ^^min ^ ^ ^^ ^ 4^ ≫ 3, 15 ^^ ≫ 1.

[0072] Let^^^^is defined by the table below. TU scale factor in ECM-9.0 Prediction mode ^^^^^ 4 4 ^ ^^^^^ 16 ^^^^^ 16 Inter-prediction 2 2 1 Intra-prediction 3 2 1

[0073] A Proposal:

[0074] The base LUT size is preserved, but the content of the base LUT is changed and the calculation of ^^^୍^,^,^,^^is also changed. The proposed method uses three scale factors ( ^^^,^, ^^^,^, ^^ଶ,^) to pre-compute three LUTs for three different neighbor distances ^1,√2, and 2^, i.e.:LUT^,^,^^^ ^^^ ൌ ൫ ^^^,^ ⋅ LUTୠୟ^^,^^^ ^^^ ^ 4൯ ≫ 3,and the averagingcut off least significant bits, i.e.: ^^^୍^,^,^,^^^^^^ൌ^^^^ ^ ^^ଶ ^ 1^≫ 1,where ^^^and ^^ଶare successive entries of For chroma, the number of cut off bits is decreased from 3 to 2.

[0075] In the proposed method, the number of cut off bits in equation 1 (Eq.1) is increased from 5 to 8 and ^^^^ൌ ^^௪^^,^^ ^^^^^^ୈ, where ^^௪^^,^is based on the TU’s shape sizes and ^^^^^^ୈis based on the mean^^ ^^(MAD) of the TU. Both ^^௪,^and ^^^^ୈare calculated using LUTs.

[0076] More precisely, let LUT௪,^be a 2D 8 ൈ 8 lookup table with non-negative 8-bit integer values, and let LUT^^ୈbe a 1D 16-entry lookup table with non-negative 8-bit integer values. Then these scale factors are defined as follows: ^^௪^^,^ൌ LUT௪,^^logଶwidth^^, logଶheight^^^, ^^^^^^ୈൌ LUT^^ୈ^min^MAD^^≫ 4, 15^^.

[0077] The MAD of a (ℎ ൈ ^^)-size TU with the channel samples denoted by ^^^,^is defined asfollows:MAD ൌ ^ ^ ௪ ^ ^^௪ ∑^ୀ^ ∑^ୀ^ ^ ^^^,^ െ ^௪ ∑^ୀ^ ∑௪^ୀ^ ^^^,^ ^ .

[0078] In total, fourare introduced (for luma / chroma component, for intra / inter prediction). Note that ^^^^is a constant for all samples of the same channel inside one TU.

[0079] The value obtained in the modified version of formula (1) before right bit-shifting belongs to the interval [-15509, 15765], and so all arithmetic can be done in 15-bit singed integers. The number of arithmetic operations per sample and the size of memory required to store the LUTs in the proposed BIF are compared with [1-2] in the table below: The number of arithmetic operations in BIF per sample BIF Bit width Summations Multiplications LUT LUTs version (maximal) (per sample) (per sample) lookups (per memory sample) (bytes) ECM-9.0 12 18 0 6 832 Proposed 15 25 1 12 2816

[0080] Results

[0081] The proposal is implemented on top of ECM-9.0 and evaluated under the CTC. The calculation of the MAD is implemented using only summation and bit-shifting operators in integers (i.e., the calculation does not involve any multiplications or divisions). The following tables show the results for PSNR-based BD-rate and encoding / decoding run time. All Intra Main 10Over ECM-9.0Y U V EncT DecT Class A1 -0.15% -0.07% -0.21% 101.4% 101.0% Class A2 -0.15% -0.04% -0.21% 100.8% 100.3% Class B -0.09% -0.20% -0.17% 101.8% 101.2% Class C -0.16% -0.18% -0.21% 101.3% 101.2% Class E -0.15% -0.23% 0.09% 101.3% 99.9% Overall -0.14% -0.15% -0.15% 101.4% 100.5% Class D -0.07% -0.18% -0.20% 101.5% 100.6% Class F -0.14% -0.21% -0.25% 100.8% 100.3% Random Access Main 10Over ECM-9.0Y U V EncT DecT Class A1 -0.10% -0.23% -0.18% 100.2% Class A2 -0.15% -0.24% -0.28%99.4%Class B -0.12% -0.16% -0.09% 100.5% 100.0% Class C -0.08% -0.39% -0.24% 100.5% 99.8% Class EOverall -0.11% -0.25% -0.19% 99.8% Class D -0.08% -0.53% -0.13% 100.6% 99.9% Class F -0.10% -0.45% -0.36% 100.5% 99.9% Low delay B Main 10Over ECM-9.0 Y U V EncT DecT Class A1Class A2Class B -0.04% 0.03% 0.15%Class C -0.02% -0.82% -0.57% 100.5% 100.0% Class E -0.07% -1.75% 0.16% 99.1% 100.4% Overall -0.04% -0.70% -0.09% Class D -0.13% 0.20% -0.52% 99.1% 98.7% Class F -0.20% -0.30% -0.39%99.4%

[0082] Conclusion

[0083] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0084] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel. Further, “a” means “one or more” or “at least one,” unless the context clearly indicates otherwise.

[0085] References

[0086] [1] Tomasi, C., et. al., “Bilateral filtering for gray and color images," Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271), Bombay, India, 1998, pp.839-846.

[0087] [2] Ström, J., et. al, "Bilateral Loop Filter in Combination with SAO," 2019 Picture Coding Symposium (PCS), Ningbo, China, 2019, pp.1-5.

[0088] [3] Ström, J., et. al, "EE2: Bilateral filter in VTM, EE2 and VVenC", JVET-V0094, 22nd JVET Meeting, by teleconference, 20–28 Apr. 2021.

[0089] [4] Yin, W., et. al., “EE2-4.1: Bilateral Inloop Filter on Chroma,” JVET-X0067, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 24th Meeting, by teleconference, 6–15 October 2021.

Claims

1. An apparatus (600) for encoding a picture, the apparatus configured to: obtain a first input sample, SIN_1, belonging to a first transform unit, TU, having a first dimension and a second dimension; use a first TU scale factor, CTU_1, associated with the first TU and a bilateral filter, BIF, function to calculate a first BIF offset, δBIF_1,for the first input sample; and calculate a first output sample, Sout_1, using δBIF_1 and SIN_1, wherein: the first TU scale factor is a function of: 1) a two-dimensional, 2D, lookup table, LUT, and the magnitudes of the first and second dimensions of the first TU and / or 2) an activity score for the first TU, and / or the BIF function is a sampled piecewise linear function.

2. The apparatus of claim 1, wherein the TU scale factor is a function of the 2D LUT and the magnitudes of the first and second dimensions of the first TU.

3. The apparatus of claim 2, wherein the apparatus is further configured to: obtain a first look-up coordinate using the magnitude of the first dimension of the TU; obtain a second look-up coordinate using the magnitude of the second dimension of the TU; and use the look-up coordinates to select a scale factor from the 2D LUT, wherein CTUis a function of the scale factor selected from the 2D LUT.

4. The apparatus of claim 3, wherein the first look-up coordinate, x, is a function of the magnitude of the first dimension; the second look-up coordinate, y, is a function of the magnitude of the second dimension.

5. The apparatus of claim 4, wherein x = log2(W), were W is the magnitude of the first dimension of the first TU, and y = log2(H), were H is the magnitude of the second dimension of the first TU.

6. The apparatus of any one of claims wherein the first TU scale factor is a function of the activity score for the first TU.

7. The apparatus of claim 6, wherein the first TU scale factor is a function of a scale factor selected from a one-dimensional, 1D, LUT using a look-up index, j, that is a function of the activity score, AS.

8. The apparatus of claim 7, wherein j is equal to min(AS >> s, L), where L is the length of the 1D LUT, and s is a defined value.

9. The apparatus of claim 1, wherein the first TU scale factor is equal to the sum of SF1 and SF2, SF1 is a function of the 2D LUT and the magnitudes of the first and second dimensions of the first TU, and SF2 is a function of the activity score for the first TU.

10. The apparatus of claim 9, wherein the BIF function is a sampled piecewise linear function.

11. A method (500) for filtering input samples, the method comprising: obtaining (s502) a first input sample, SIN_1, belonging to a first transform unit, TU, having a first dimension and a second dimension; using (s504) a first TU scale factor, CTU_1, associated with the first TU and a bilateral filter, BIF, function to calculate a first BIF offset, δBIF_1,for the first input sample; and calculating (s506) a first output sample, Sout_1, using δBIF_1 and SIN_1, wherein: the first TU scale factor is a function of: 1) a two-dimensional, 2D, lookup table, LUT, and the magnitudes of the first and second dimensions of the first TU and / or 2) an activity score for the first TU, and / or the BIF function is a sampled piecewise linear function.

12. The method of claim 11, wherein the TU scale factor is a function of the 2D LUT and the magnitudes of the first and second dimensions of the first TU.

13. The method of claim 12, wherein the method further comprises: obtaining a first look-up coordinate using the magnitude of the first dimension of the TU; obtaining a second look-up coordinate using the magnitude of the second dimension of the TU; and using the look-up coordinates to select a scale factor from the 2D LUT, wherein CTU is a function of the scale factor selected from the 2D LUT.

14. The method of claim 13, wherein the first look-up coordinate, x, is a function of the magnitude of the first dimension; the second look-up coordinate, y, is a function of the magnitude of the second dimension.

15. The method of claim 14, wherein x = log2(W), were W is the magnitude of the first dimension of the first TU, and y = log2(H), were H is the magnitude of the second dimension of the first TU.

16. The method of any one of claims 11-15, wherein the first TU scale factor is a function of the activity score for the first TU.

17. The method of claim 16, wherein the first TU scale factor is a function of a scale factor selected from a one-dimensional, 1D, LUT using a look-up index, j, that is a function of the activity score, AS.

18. The method of claim 17, wherein j is equal to min(AS >> s, L), where L is the length of the 1D LUT, and s is a defined value.

19. The method of claim 11, wherein the first TU scale factor is equal to the sum of SF1 and SF2, SF1 is a function of the 2D LUT and the magnitudes of the first and second dimensions of the first TU, andSF2 is a function of the activity the first TU.

20. The method of claim 19, wherein the BIF function is a sampled piecewise linear function.

21. A computer program (643) comprising instructions (644) which when executed by processing circuitry (602) cause the processing circuitry (602) to perform the method of any one of claims 11 to 20.

22. A carrier containing the computer program of claim 21, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (642).

Images