Adaptive loop filter and cascade filtering

Abstract

The invention provides a method for performing cascade filtering in an adaptive loop filter (ALF) of a video codec, and the method is used for performing the cascade filtering in the ALF of the video codec. A video codec receives a current sample of a current pixel position of a current picture of a video. A video codec applies a first filter comprising a plurality of filter taps. The current sample and neighboring samples taken from pixel locations near the current sample are used as tap inputs for a plurality of filter taps of the first filter. The video codec applies a second filter comprising at least one filter tap. A first output of the first filter is used to generate a tap input of at least one filter tap of the second filter. The video codec updates the current sample based on a second output of the second filter. The video codec reconstructs the current picture based on the updated current sample.

Description

[0001] Related references

[0002] This application is a part of a non-provisional application claiming priority to U.S. Provisional Patent Application Nos. 63 / 370,936 and 63 / 370,938, both filed on August 10, 2022. The contents of the above-listed applications are hereby incorporated by reference. [Technical field]

[0003] The present invention generally relates to video coding and decoding. In particular, the present invention relates to a method for coding and decoding video pictures using an adaptive loop filter. [Background technology]

[0004] Unless otherwise indicated in this section, the approaches described in this section are not prior art to the claims listed below and are not admitted to be prior art by inclusion in this section.

[0005] High-Efficiency Video Coding (HEVC) is an international video codec standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). HEVC is based on a hybrid block motion compensated DCT-like transform codec architecture. The basic unit of compression, called a coding unit (CU), is a square block of 2Nx2N pixels. Each CU can be recursively divided into four smaller CUs until a predefined minimum size is reached. Each CU contains one or more prediction units (PUs).

[0006] Versatile Video Coding (VVC) is the latest international video codec standard developed by the Joint Video Expert Team (JVET) of the International Telecommunication Union-Telecommunication Standardization Sector Working Group 16, Working Party 3 (ITU-T SG16 WP3) and the International Organization for Standardization / International Electrotechnical Commission Technical Committee 29, Working Group 11 (ISO / IEC JTC1 / SC29 / WG11). The input video signal is predicted from the reconstructed signal derived from the coded picture area. The prediction residual signal is processed by block transform. The transform coefficients are quantized and entropy encoded in the bitstream together with other side information. The reconstructed signal is generated from the prediction signal and the reconstructed residual signal of the dequantized transform coefficients after inverse transform. The reconstructed signal is further processed by loop filtering to remove codec artifacts. The decoded picture is stored in the frame buffer and used to predict future pictures in the input video signal.

[0007] In VVC, coded pictures are divided into non-overlapping square block areas, represented by related coding tree units (CTUs). The leaf nodes of the codec tree correspond to codec units (CUs). A coded picture can be represented by a series of slices, each slice consisting of an integer number of CTUs. The individual CTUs in a slice are processed in raster scan order. Bidirectionally predicted (B) slices can use intra prediction or inter prediction, using up to two motion vectors and reference indices to predict the sample values ​​of each block. Predicted (P) slices use intra prediction or inter prediction, using up to one motion vector and reference index to predict the sample values ​​of each block. Intra (I) slices are decoded using only intra prediction.

[0008] CTU can be split into one or more non-overlapping codec units (CUs) using a quadtree (QT) and nested multi-type tree (MTT) structure to accommodate various local motion and texture features. CU can be further split into smaller CUs using one of five split types: quadtree split, vertical binary tree split, horizontal binary tree split, vertical center-side ternary tree split, and horizontal center-side ternary tree split.

[0009] Each CU contains one or more prediction units (PUs). The prediction unit, together with the associated CU syntax, serves as the basic unit for signaling predictor information. A specified prediction process is used to predict the values ​​of the associated pixel samples within the PU. Each CU may contain one or more transform units (TUs) to represent the prediction residual blocks. A transform unit (TU) consists of a transform block (TB) of luma samples and two corresponding transform blocks of chroma samples, each TB corresponding to one residual block sample from one color component. An integer transform is applied to the transform blocks. The horizontal values ​​of the quantized coefficients are entropy encoded in the bitstream along with other side information. The terms Coding Tree Block (CTB), Coding Block (CB), Prediction Block (PB), and Transform Block (TB) are defined as 2-D arrays of samples of monochrome components associated with CTU, CU, PU, ​​and TU. Therefore, a CTU consists of a luma CTB, two chroma CTBs, and associated syntax elements. A similar relationship is also found between CU, PU, ​​and TU.

[0010] For each inter-predicted CU, the motion parameters include motion vector, reference picture index and reference picture list usage index, as well as additional information for inter-prediction sample generation. Motion parameters can be signaled in an explicit or implicit way. When a CU is coded in skip mode, the CU is associated with one PU and has no significant residual coefficients, no coded motion vector differences or reference picture indices. A merge mode is specified, in which the motion parameters of the current CU are obtained from neighboring CUs, including spatial and temporal candidates, as well as additional schemes introduced in VVC. Merge mode can be applied to any inter-predicted CU. In contrast to merge mode is the explicit transmission of motion parameters, in which each CU explicitly signals the motion vector for each reference picture list, the corresponding reference picture index and reference picture list usage flag, as well as other required information. [Summary of the invention]

[0011] The following summary is provided for informational purposes only and is not intended to be limiting in any way. That is, the following summary is intended to introduce the concepts, highlights, advantages, and benefits of the novel and non-obvious technologies described herein. Optionally, not all implementations are further described in the detailed description below. Therefore, the following summary is not intended to determine the essential features of the subject matter, nor is it used to determine the scope of the subject matter.

[0012] Certain embodiments of the present invention provide a method for performing cascade filtering in an adaptive loop filter (ALF) of a video codec. The video codec receives a current sample at a current pixel position of a current picture of a video. The video codec applies a first filter including a plurality of filter taps. The current sample and neighboring samples taken from pixel positions near the current sample are used as tap inputs of the plurality of filter taps of the first filter. The plurality of filter taps of the first filter may use the difference between the current sample and the neighboring samples as tap inputs.

[0013] The video codec applies a second filter including at least one filter tap. The first output of the first filter is used to generate a tap input of at least one filter tap of the second filter. The video codec updates the current sample based on the second output of the second filter. The video codec reconstructs the current picture based on the updated current sample.

[0014] In some embodiments, the video codec receives or sends a specific flag to enable or disable cascade filtering. When the specific flag enables cascade filtering, the output of the second filter is used to update the current sample. When the specific flag disables cascade filtering, the output of the first filter is used to update the current sample. In some embodiments, the second filter is applied only when the difference between the current sample and the first output of the first filter is not zero. In some embodiments, the second filter is applied only when the difference between the current sample and the first output of the first filter is zero. In some embodiments, the second filter is applied only when the difference between the current sample and the first output of the first filter is greater than a threshold. In some embodiments, the second filter is applied only when the difference between the current sample and the first output of the first filter is less than a threshold.

[0015] In some embodiments, the tap input of at least one filter tap of the second filter is the clipped difference between the first output and the current sample. In some embodiments, the second filter also includes a plurality of filter taps, and the plurality of filter taps use the difference between the current sample and the adjacent samples as the tap input. The plurality of filter taps of the second filter may have the same or different filter coefficients as the first filter.

Brief Description of the Drawings

[0016] The accompanying drawings are included to provide a further understanding of the present invention and are incorporated into and constitute a part of this invention. The accompanying drawings illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the present invention. It should be appreciated that the drawings are not necessarily to scale, as some components may be shown out of proportion to the size in actual implementation in order to clearly illustrate the concepts of the present invention.

[0017] Figure 1A and Figure 1B Two diamond filter shapes are shown for use in adaptive loop filters (ALFs).

[0018] Figure 2 A system-level diagram of the loop filter is shown.

[0019] Figure 3 Filtering in cross-component ALF (CC-ALF) is demonstrated.

[0020] Figure 4 Conceptual illustration of the current sample and its neighboring samples being used as filter tap inputs for filtering the current sample.

[0021] Figure 5 A cascaded filter is conceptually shown, where the filtering results of a first filter bank are used as filtering taps of a second filter bank.

[0022] Figure 6 Conceptual illustration of a cascaded filter where the first filter result is modified by an additional filter tap whose input is the difference between the current sample and the first filter result.

[0023] Figure 7 A two-stage ALF is conceptually shown, where the filter coefficients of the first stage are different from the filter coefficients of the second stage.

[0024] Figure 8 Shows an example video encoder that implements ALF.

[0025] Fig. 9 Portions of a video encoder implementing an ALF with multiple stages or cascaded filtering are shown.

[0026] Fig.10 Conceptually illustrates the process of performing cascade filtering in ALF.

[0027] Fig.11 Shows a sample video decoder that implements ALF.

[0028] Fig.12 Portions of a video decoder implementing an ALF with multiple stages or cascaded filtering are shown.

[0029] Fig.13 Conceptually illustrates the process of performing cascade filtering in ALF.

[0030] Fig.14 Some electronic systems embodying the present invention are conceptually shown. [Specific implementation method]

[0031] In the following detailed description, numerous specific details are presented by way of example in order to provide a thorough understanding of the relevant teachings. Any variations, derivatives and / or extensions based on the teachings described herein are within the scope of the present invention. In some cases, well-known methods, procedures, components and / or circuits associated with one or more example implementations disclosed herein may be described at a relatively high level without involving details to avoid unnecessarily obscuring aspects of the teachings of the present invention.

[0032] I. Adaptive Loop Filter

[0033] A. Filter Shape

[0034] Adaptive loop filter (ALF) is a loop filtering technique used in video coding standards such as VVC. It is a block-based filter that minimizes the mean square error between the original sample and the reconstructed sample. For the luminance component, one of 25 filters is selected for each 4×4 block based on the direction and activity of the local gradient. Figure 1A-1B Two diamond filter shapes for an adaptive loop filter (ALF) are shown. Each position in the diamond corresponds to a filter tap with a filter coefficient. Figure 1A A 7x7 diamond is shown with filter coefficients C0-C12 for the luma component. Figure 1B A 5x5 diamond is shown with filter coefficients C0-C6 for the chrominance components.

[0035] B. Block Classification

[0036] For the luminance component, each 4×4 block is classified into one of 25 categories. The classification index C is based on its directionality D and activity The quantitative value of is obtained as follows:

[0037]

[0038] In order to calculate D and First, use the one-dimensional Laplacian operator to calculate the gradient in the horizontal, vertical and two diagonal directions:

[0039]

[0040] Where the indices i and j refer to the coordinates of the top left sample within the 4×4 block, and R(i,j) represents the reconstructed sample at coordinate (i,j). To reduce the complexity of block classification, a downsampled one-dimensional Laplacian calculation is applied. The same downsampled position can be used for gradient calculations in all directions. (The downsampled position can be used for vertical gradients, horizontal gradients, or diagonal gradients.) The maximum and minimum values ​​of the horizontal and vertical gradients are set to:

[0041]

[0042] The maximum and minimum values ​​of the two diagonal gradients are set to:

[0043]

[0044] To derive the value of the directionality D, these values ​​are compared with each other and with two thresholds t1 and t2:

[0045] Step 1. If and are all true, D is set to 0.

[0046] Step 2. If Then continue from step 3; otherwise continue from step 4.

[0047] Step 3. If D is set to 2; otherwise D is set to 1.

[0048] Step 4. If D is set to 4; otherwise D is set to 3.

[0049] The activity value A is calculated as follows:

[0050]

[0051] A is further quantized to the range of 0 to 4 (including 0 and 4), and the quantized value is expressed as For the chrominance components of the picture, no classification method is applied.

[0052] C. Geometric transformation of filter coefficients and clipping values

[0053] Before filtering each 4×4 luma block, a geometric transformation such as rotation or diagonal and vertical flipping is applied to the filter coefficients f(k,l) and the corresponding filter shear values ​​c(k,l) according to the gradient values ​​calculated for the block. This is equivalent to applying these transformations to the samples in the filter support region. The idea is to make different blocks where the ALF is applied more similar by aligning their directionality. Three geometric transformations are introduced, including diagonal, vertical flipping, and rotation:

[0054] Diagonal: f D(k,l)=f(l,k),c D (k,l)=c(l,k),

[0055] Flip vertically: f V (k,l)=f(k,Kl-1),c V (k,l)=c(k,Kl-1)

[0056] Rotation: f R (k,l)=f(Kl-1,k),c R (k,l)=c(Kl-1,k)

[0057] Where K is the size of the filter and 0≤k,l≤K-1 are the coefficient coordinates such that position (0,0) is at the top left corner and position (K-1,K-1) is at the bottom right corner. A transformation is applied to the filter coefficients f(k,l) and the shear value c(k,l) according to the gradient value calculated for the block. Table 1 below summarizes the relationship between the transformation and the four gradients in the four directions, showing the mapping of the gradient calculated for a block to the transformation.

[0058] Table 1:

[0059] Gradient values Transformation <![CDATA[g d2 <g d1 and g h <g v ]]> No transformation <![CDATA[g d2 <g d1 and g v <g h ]]> Diagonal <![CDATA[g d1 <g d2 and g h <g v ]]> Vertical flip <![CDATA[g d1 <g d2 and g v <g h ]]> Rotation

[0060] D. Filtering process

[0061] At the decoder side, when ALF is enabled for a CTB, each sample R'(i,j) within the CU is filtered to obtain the sample value R'(i,j) as shown below:

[0062]

[0063] Where f(k,l) represents the decoded filter coefficients, K(x,y) is the clipping function, and c(k,l) represents the decoded clipping parameters. The variables k and l vary between -L / 2 and L / 2, where L represents the filter length. The clipping function K(x,y)=min(y,max(-y,x)) corresponds to the function Clip3(-y,y,x). The clipping operation introduces nonlinearity, making the adaptive loop filter (ALF) more effective by reducing the impact of neighboring sample values ​​that differ too much from the current sample value.

[0064] E. Cross Component Adaptive Loop Filter (CC-ALF)

[0065] CC-ALF may use luma sample values ​​to refine each chroma component by applying an adaptive, linear filter to the luma channel and then using the output of this filtering operation. Figure 2 A system level diagram of a loop filter 200 is shown, wherein reconstructed or decoded samples 210 are filtered or processed by a deblock filter (DBF), a sample adaptive offset (SAO), and an adaptive filter (ALF). The reconstructed or decoded samples 210 may be generated from a prediction signal and a residual signal of a current block.

[0066] This figure shows the position relationship of CC-ALF with other loop filters. Specifically, the luma component of the SAO output is processed by a luma ALF process (ALF Y) and a pair of cross-component ALF processes (CC-ALF Cb and CC-ALFCr). These two cross-component ALF processes generate cross-component offsets for the Cb and Cb components, which are added to the output of the chroma ALF process (ALF chroma) to generate the ALF output of the chroma components. The luma and chroma components of the ALF output are then stored in the reconstructed or decoded picture buffer 290 for predictive coding and decoding of subsequent pixel blocks.

[0067] Figure 3 Filtering in a cross-component ALF (CC-ALF) is shown, performed by applying a linear, diamond filter 310 to the luma channel. One filter is used for each chroma channel, and the operation is represented as

[0068]

[0069] where (x, y) is the position of the chrominance component i being refined (x Y ,y Y ) is the brightness position based on (x,y), S i is the filter support area in the luminance component, c i (x0,y0) represents the filter coefficients. Figure 3 As shown, the luma filter support region coincides with the current chroma sample, taking into account the spatial scaling factor between the luma and chroma planes.

[0070] The CC-ALF filter coefficients may be computed by minimizing the mean squared error of each chroma channel with respect to the original chroma content. To do this, the algorithm may use a process similar to that used for coefficient derivation by the chroma ALF. Specifically, a correlation matrix is ​​derived and the coefficients are computed using a Cholesky decomposition solver in an attempt to minimize a mean squared error metric. When designing the filters, up to 8 CC-ALF filters may be designed and transmitted per picture. The results of these filters are then indicated for both chroma channels on a CTU basis.

[0071] CC-ALF filtering may use a 3x4 diamond shape with 8 filter taps, with 7 filter coefficients transmitted in the APS (possibly referenced in the slice header). Each transmitted coefficient has a 6-bit dynamic range and is limited to power-of-2 values. The 8th filter coefficient is derived in the decoder such that the sum of the filter coefficients equals 0. CC-ALF filter selection may be controlled at the CTU level for each chroma component. Boundary padding for horizontal virtual boundaries may be the same as the memory access pattern for luma ALF.

[0072] As an additional feature, the reference encoder can be configured to enable some basic subjective adjustments via a configuration file. When enabled, VTM will attenuate the application of CC-ALF in areas that were encoded with a high quantization parameter (QP), are either close to mid-grey, or contain a lot of luminance high frequencies. Algorithmically, this is achieved by disabling the application of CC-ALF in CTUs that meet any of the following conditions:

[0073] (i) The slice QP value minus 1 is less than or equal to the base QP value

[0074] (ii) The number of chroma samples with a local contrast greater than (1<<(bitDepth–2))–1 exceeds the CTU height, where local contrast is the difference between the maximum and minimum luma sample values ​​within the filter support area.

[0075] (iii) More than a quarter of the chroma samples are in the range (1<<(bitDepth–1))–16 and (1<<(bitDepth–1))+16

[0076] This is to ensure that CC-ALF does not amplify artifacts introduced earlier in the decoding path.

[0077] F. Filter parameter signal

[0078] ALF filter parameters can be signaled in an Adaptation Parameter Set (APS). In one APS, up to 25 groups of luminance filter coefficients and shear value indices and up to eight groups of chrominance filter coefficients and shear value indices can be signaled. To reduce bit overhead, filter coefficients of different classifications of the luminance component can be merged. In the slice header, the index of the APS for the current slice is signaled.

[0079] The clip value index, decoded from the Adaptive Loop Filter ParameterSet (APS), allows to determine the clip value using the clip value tables for the luma and chroma components. These clip values ​​depend on the internal bit depth. More precisely, the clip value is obtained by the following formula:

[0080] ALFClip={round(2 B-α*n )forn∈[0..N-1]}

[0081] Where B is equal to the internal bit depth, (is a predefined constant value equal to 2.35, and N is equal to 4, which is the number of clip values ​​allowed by VVC. ALFClip is then rounded to the nearest power of 2 format value.

[0082] In the slice header, up to 7 APS indices can be signaled to specify the luma filter set used for the current slice. The filtering process can be further controlled at the Coding Tree Block (CTB) level. A flag is always signaled to indicate whether ALF is applied to the luma CTB. The luma CTB can select a filter set from 16 fixed filter sets and the filter set from the APS. A filter set index is signaled for the luma CTB to indicate which filter set is applied. These 16 fixed filter sets are predefined and hard-coded in the encoder and decoder.

[0083] For chroma components, an APS index may be signaled in the slice header to indicate the chroma filter set used for the current slice. At the CTB level, if there is more than one chroma filter set in the APS, a filter index is signaled for each chroma CTB. The filter coefficients are quantized to a standard of 128. To limit the multiplication complexity, bitstream consistency is applied so that coefficient values ​​in non-central positions should be within -2 7 To 2 7 The center position coefficient is not signaled in the bitstream and is considered equal to 128.

[0084] G. ALF Simplification Using Fixed Filter

[0085] In some embodiments, ALF gradient subsampling and ALF virtual boundary processing are removed. The block size of the classification is reduced from 4x4 to 2x2. The filter size of luma and chroma, ALF coefficients are signaled, increased to 9x9.

[0086] To filter a luminance sample, three different classifiers (C0, C1, and C2) and three different sets of filters (F0, F1, and F2) can be used. Sets F0 and F1 contain fixed filters whose coefficients are trained for classifiers C0 and C1. The filter coefficients in set F2 are signaled. i Which filter in filters a given sample, by using the classifier C i The category C assigned to this sample i First, two 13x13 diamond-shaped fixed filters F0 and F1 are applied to obtain two intermediate samples R0(x,y) and R1(x,y). After that, F2 is applied to R0(x,y) and R1(x,y) and their neighboring samples to obtain the filtered samples as shown below:

[0087]

[0088] where f i,j is the clipped difference between the neighboring sample and the current sample R(x,y), g i YesR i-20 The clipped difference between (x,y) and the current sample. Filter coefficients c i , i=0,…21, is transmitted by the signal.

[0089] Figure 4 A conceptual illustration of a current sample and its neighboring samples used as filter tap inputs for filtering the current sample. The figure shows a portion of a current picture 400. The current sample is shown in the center, labeled 'R', and the neighboring samples used to generate the filter tap inputs are labeled R 0+ ,R 0- ,R 1+ ,R 1- …etc. Perform adaptive loop filtering (ALF) based on the filtered tap input to produce a filtered output for the current sample For example, use the above equation (1).

[0090] Based on the directionality D i and activity Assign a class C to each 2x2 block i :

[0091]

[0092] Among them, MD,i Represents the total number of directions D i The values ​​of horizontal, vertical and two diagonal gradients can be calculated for each sample using a one-dimensional Laplacian operator. The sum of sample gradients within a 4x4 window covering a 2x2 block of the target is used for classifier C0, while the sum of sample gradients within a 12x12 window is used for classifiers C1 and C2. The sum of horizontal, vertical and two diagonal gradients are expressed as and Determine the directionality D by comparison i :

[0093]

[0094] A set of thresholds are used. Directivity D2 is obtained using thresholds 2 and 4.5. For D0 and D1, the horizontal / vertical edge strength is first calculated and diagonal edge strength The threshold used is Th = [1.25, 1.5, 2, 3, 4.5, 8]. If Edge Strength is 0; otherwise, is the largest integer such that if Edge Strength is 0; otherwise, is the largest integer such that Table 2(a) and Table 2(b) below show the E i D and E i HV To Di mapping. That is, when horizontal / vertical edges dominate, use the following Table 2(a) to get D i Otherwise, the diagonal edges dominate, and using Table 2(b) we get D i .

[0095] Table 2(a):

[0096]

[0097]

[0098] Table 2(b):

[0099]

[0100] To gain activity The sum of the vertical and horizontal gradients A i Mapped to the range 0 to n, where n is is equal to 4, for and Equal to 15. In one ALF_APS, up to 4 luminance filter sets can be signaled, each with up to 25 filters.

[0101] II. Joint Codec of Chroma Residuals (JCCR)

[0102] Joint codec of chroma residual (JCCR) is a video codec tool that enables joint coding of chroma residual. The use (activation) of JCCR mode can be indicated by the TU level flag tu_joint_cbcr_residual_flag, and the selected mode is implicitly indicated by the chroma CBFs. The flag tu_joint_cbcr_residual_flag is present if one or both chroma CBFs of the TU are equal to 1. In the picture parameter set (PPS) and slice header, chroma quantization parameter (QP) offset values ​​are signaled for the JCCR mode to distinguish the chroma QP offset values ​​signaled by the regular chroma residual codec mode. These chroma QP offset values ​​are used to derive the chroma QP values ​​for certain blocks encoded using the JCCR mode. The JCCR mode has 3 sub-modes, as shown in Table 3 below, showing the reconstruction of the chroma residual.

[0103] Table 3: Sub-schemas of JCCR

[0104]

[0105] CSign is a symbolic value (+1 or -1) specified in the slice header and resJointC[][] is the transmitted residual. When JCCR submode 2 is activated in a TU, the chroma QP offset is added to the applied luma-derived chroma QP during quantization and decoding of that TU. For other JCCR submodes (submodes 1 and 3), chroma QPs are derived in the same way as for regular Cb or Cr blocks. The process of reconstructing the chroma residual (resCb and resCr) from the transmitted transform block is shown in Table 3. When JCCR mode is activated, a single joint chroma residual block (resJointC[x][y]) is signaled, and the residual block of Cb (resCb) and Cr (resCr) is derived taking into account information such as tu_cbf_cb, tu_cbf_cr and CSign (symbolic value specified in the slice header).

[0106] On the encoder side, according to the JCCR sub-mode, the joint chroma components are generated by the encoder as follows:

[0107] If mode is equal to 2 (single residual, reconstruction Cb = C, Cr = CSign*C), then the joint residual is determined as follows

[0108] resJointC[x][y]=(resCb[x][y]+CSign*resCr[x][y]) / 2

[0109] Otherwise, if mode is equal to 1 (single residual, reconstruction Cb = C, Cr = (CSign*C) / 2), then the joint residual is determined as follows

[0110] resJointC[x][y]=(4*resCb[x][y]+2*CSign*resCr[x][y]) / 5

[0111] Otherwise (mode equals 3, i.e. single residual, reconstruction Cr = C, Cb = (CSign*C) / 2), the joint residual is determined as follows

[0112] resJointC[x][y]=(4*resCr[x][y]+2*CSign*resCb[x][y]) / 5

[0113] In some embodiments, the three joint chroma coding sub-modes described in Table 3 above are only supported in I slices. In P and B slices, only mode 2 is supported. Therefore, in P and B slices, the syntax element tu_joint_cbcr_residual_flag only exists if both chroma cbfs are 1.

[0114] JCCR mode can be used in conjunction with chroma transform skip (TS) mode. To speed up encoder decisions, JCCR transform selection may depend on whether the independent codecs of the Cb and Cr components select DCT-2 or TS as the best transform, and whether there are non-zero coefficients in the independent chroma codecs. Specifically, if one chroma component selects DCT-2 (or TS) and the other component is all zero, or if both chroma components select DCT-2 (or TS), only DCT-2 (or TS) will be considered in JCCR encoding. Otherwise, if one component selects DCT-2 and the other selects TS, both DCT-2 and TS will be considered in JCCR encoding.

[0115] III. Adaptive Loop Filter (ALF) with Joint Chroma Filter

[0116] The video codec can select the best filter for Cb and Cr CTBs separately. Certain embodiments of the present invention provide a joint chroma filtering design in which the filtering process of one component is related to the filtering process of another component through a predetermined model. Using the predetermined model eliminates the transmission of CTB-level filter selection signaling for one component.

[0117] In some embodiments, an APS level flag is signaled to indicate whether joint chroma filtering is used. If joint chroma filtering is used, the correction value generated by the Cb filter is also applied to Cr through a predetermined linear model.

[0118]

[0119] Where m is a predefined number used to model the relationship between Cb and Cr reconstruction. In some embodiments, an additional APS level flag is sent to indicate the value of m. In some embodiments, this joint chroma filtering flag is sent at the slice level. In some embodiments, an APS level flag is sent to indicate whether joint chroma filtering is used. If joint chroma filtering is used, the filter selected for Cb will also be applied to Cr through the following predetermined linear model:

[0120]

[0121] Where m is a predefined number used to model the relationship between Cb and Cr reconstruction. The filter coefficient c i,cb (i=0,1,…,19) can be common to both components (Cr and Cb). In some embodiments, an additional APS level flag is sent to indicate the value of m. In some embodiments, this joint chroma filtering flag is sent at the slice level. In some embodiments, the value of m is determined by the Cb or Cr classification result (e.g., one class maps to one m value).

[0122] In certain embodiments, a predetermined linear model may also be applied to Cb as shown below.

[0123]

[0124] or

[0125]

[0126] In some embodiments, the concept of joint chroma filtering can be applied to chroma ALF and CCALF. In the case of CCALF, (f i,0,cb +f i,1,cb ) and (f i,0,cr +f i,1,cr ) is replaced by f i,y, which represents the corresponding neighboring differences of the brightness components.

[0127] In some embodiments, the chroma components are also classified, where the class indices for Cb and Cr are non-overlapping. For example, the video codec may classify Cb into classes 0 to 4 and Cr into classes 5 to 9. The classification is based on chroma sample values ​​only, luma sample values ​​only, or luma and chroma sample values. The video codec may apply chroma classification to chroma ALF as well as CCALF.

[0128] IV. ALF with Cascade Filtering

[0129] The disclosure of certain embodiments provides a method for cascade filtering. An APS level flag may be sent to indicate whether cascade filtering is used. In certain embodiments, cascade filtering is achieved by using fixed filtering results as additional filter taps for the brightness component APS filter. In certain embodiments, the filtering results of the first group of APS filters are used as filter taps for the second group of APS filters. In the first group of filters, the filtering equation can be written as:

[0130]

[0131] where f i,j is the clipped difference between the neighboring sample and the current sample R, g i is the shear difference between the current sample before and after filtering by a fixed filter bank, c i (i=0,1,…,21) are the filter coefficients. For the filters in the second set of filters, the filter equation can be written as:

[0132]

[0133] Figure 5 A cascade filter 500 is conceptually illustrated, in which the filtering results of the first APS filter bank are used as the filter taps of the second APS filter bank. The cascade filter 500 is an adaptive loop filter (ALF) and includes filter tap modules 511 and 512. The filter tap module 511 ("Filter Taps [0:21]") corresponds to the calculation Filter taps module 512 ("Filter Taps

[22] ") corresponds to calculating

[0134] If there are more than two filter banks in the APS, the filtering of the Nth filter bank is based on the filtering results of the (N-1)th filter bank, similar to the equation shown above.

[0135] In equation (8), the filtering result of the first APS filter bank is is used as the filter tap for the second APS filter bank. More precisely, the difference between the current sample R and the first filtering result is are clipped and used as additional filter taps (c 22 ) input. However, according to equation (7),

[0136]

[0137] The coefficients of the first and second APS filter banks (c0 to c 21 ) can be the same or different. If the coefficients of the first and second APS filter banks (c0 to c 21 ) are the same, then equation (8) can be rewritten as

[0138]

[0139] because so Equation (10) can be rewritten as:

[0140]

[0141] Therefore, the cascaded filter can be described as modifying the first APS filter by an additional filter tap The tap input is the difference between the current sample and the first filtering result. Figure 6 A cascade filter 600 is conceptually illustrated, in which a first filter result is modified by an additional filter tap with the difference between the current sample and the first filter result as the tap input. The cascade filter 600 is an adaptive loop filter, including a filter tap module 610 and a filter tap module 620. The first filter tap module 610 ("Filter Taps [0:21]") corresponds to calculating Δ1 or The second filter tap module 620 ("Filter Tap

[22] ") corresponds to calculating c 22 Clip(Δ1) or

[0142] In some embodiments, for the filters in the second filter bank, the filtering equation may be written as:

[0143]

[0144] where f′ i,j is the clipping difference between the APS filtered neighboring sample value and the current sample value before ALF, and K is the number of APS filtered neighboring samples included.

[0145] In some embodiments, cascaded filtering is used conditionally based on the results of the first filter (the earliest filter in the cascade). In some embodiments, cascaded filtering is applied only to samples whose sample values ​​are changed by the first filter (i.e. In some embodiments, cascade filtering is applied only to samples whose values ​​are not changed by the first filter (i.e. In some embodiments, cascade filtering is only applied to samples whose sample value changes are greater or less than a threshold (i.e. or ).

[0146] In some embodiments, a two-stage adaptive loop filter (ALF) is applied to the reconstructed picture. In the first stage, a normal ALF process is performed according to the following formula:

[0147]

[0148] In the second stage, another set of filters (whose coefficients are determined by c′) is performed. i denoted by ALF) and is only applied to samples where the ALF has changed in the first stage. In other words:

[0149]

[0150] where f′ i,j is the shear difference between the neighboring sample after the first stage ALF filtering and the current sample. In another embodiment, a threshold Th is set to decide whether to apply the second stage ALF to each sample, specifically:

[0151]

[0152] Figure 7 A two-stage ALF 700 is conceptually illustrated, where the filter coefficients of the first stage are different from the filter coefficients of the second stage. The two-stage ALF includes a first-stage filter tap module 710 and a second-stage filter tap module 720. The tap inputs of the second-stage filter tap module 720 may be the current sample and the neighboring samples after being filtered by the first filter 710. The filter coefficients of the filter tap 710 are different from the filter coefficients of the filter tap 720. The two-stage ALF 700 implements conditional filtering according to equation (15). Specifically, when (or the output of the first filtering stage 710) is 0 or less than a threshold, the output of the second filtering stage 720 is zero or inactive, so that

[0153] One difference between cascaded filtering and two-stage ALF is that the result of the first filter Whether to be clipped to the valid range determined by the bit depth. For example, if the input bit depth is 10 bits, in the two-stage ALF method, After the first stage ALF, it is clipped to [0,1023] and input to the second stage ALF for filtering. However, in the cascade filtering method, The overrange is allowed and fed as input to the second filter.

[0154] The above-mentioned proposed method can be implemented in an encoder and / or a decoder. For example, the proposed method can be implemented in a loop filter module of an encoder and / or a loop filter module of a decoder.

[0155] V. Example Video Encoder

[0156] Figure 8 An example video encoder 800 implementing an adaptive loop filter (ALF) is illustrated. As shown, the video encoder 800 receives an input video signal from a video source 805 and encodes the signal into a bitstream 895. The video encoder 800 has a plurality of components or modules for encoding a signal from the video source 805, including at least some components selected from a transform module 810, a quantization module 811, an inverse quantization module 814, an inverse transform module 815, an intra-frame estimation module 820, an intra-frame prediction module 825, a motion compensation module 830, a motion estimation module 835, a loop filter 845, a reconstructed picture buffer 850, an MV buffer 865, an MV prediction module 875, and an entropy encoder 890. The motion compensation module 830 and the motion estimation module 835 are part of the inter-frame prediction module 840.

[0157] In some embodiments, modules 810-890 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device or electronic device. In some embodiments, modules 810-890 are modules of hardware circuits implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 810-890 are depicted as separate modules, some modules may be combined into a single module.

[0158] The video source 805 provides an original video signal of pixel data of an uncompressed video frame. A subtractor 808 calculates the difference between the original video pixel data of the video source 805 and the predicted pixel data 813 of the motion compensation module 830 or the intra-frame prediction module 825 as a prediction residual 809. The transform module 810 converts the difference (or residual pixel data or residual signal 808) into transform coefficients (e.g., by performing a discrete cosine transform, or DCT). The quantization module 811 quantizes the transform coefficients into quantized data (or quantized coefficients) 812, which is encoded into a bitstream 895 by the entropy encoder 890.

[0159] The inverse quantization module 814 inversely quantizes the quantized data (or quantized coefficients) 812 to obtain transform coefficients, and the inverse transform module 815 inversely transforms the transform coefficients to generate a reconstructed residual 819. The reconstructed residual 819 is added to the predicted pixel data 813 to generate reconstructed pixel data 817. In some embodiments, the reconstructed pixel data 817 is temporarily stored in a line buffer (not shown) for intra-frame prediction and spatial MV prediction. The reconstructed pixels are filtered by the loop filter 845 and stored in the reconstructed picture buffer 850. In some embodiments, the reconstructed picture buffer 850 is a storage external to the video encoder 800. In some embodiments, the reconstructed picture buffer 850 is a storage internal to the video encoder 800.

[0160] The intra prediction module 820 performs intra prediction based on the reconstructed pixel data 817 to generate intra prediction data. The intra prediction data is provided to the entropy encoder 890 to be encoded into a bitstream 895. The intra prediction data is also used by the intra prediction module 825 to generate the predicted pixel data 813.

[0161] The motion estimation module 835 performs inter-frame prediction by generating MVs to reference pixel data of previously decoded frames stored in the reconstructed picture buffer 850. These MVs are provided to the motion compensation module 830 to generate predicted pixel data.

[0162] Instead of encoding complete actual MVs in the bitstream, the video encoder 800 uses MV prediction to generate predicted MVs, and the difference between the MVs used for motion compensation and the predicted MVs is encoded as residual motion data and stored in the bitstream 895.

[0163] The MV prediction module 875 generates prediction MVs, i.e., motion compensation MVs for performing motion compensation, based on the reference MVs generated for encoding the previous video frame. The MV prediction module 875 retrieves the reference MVs of the previous video frame from the MV buffer 865. The video encoder 800 stores the MVs generated for the current video frame in the MV buffer 865 as reference MVs for generating prediction MVs.

[0164] The MV prediction module 875 creates predicted MVs using reference MVs. The predicted MVs can be calculated by spatial MV prediction or temporal MV prediction. The difference (residual motion data) between the predicted MVs and the motion compensation MVs (MC MVs) of the current frame is encoded into the bitstream 895 by the entropy encoder 890.

[0165] The entropy encoder 890 uses entropy coding techniques, such as context-adaptive binary arithmetic coding and decoding (CABAC) or Huffman coding, to encode various parameters and data into a bitstream 895. The entropy encoder 890 encodes various header elements, flags, and quantized transform coefficients 812 and residual motion data as syntax elements into the bitstream 895. The bitstream 895 is in turn stored in a storage device or transmitted to a decoder via a communication medium (such as a network).

[0166] The loop filter 845 performs filtering or smoothing operations on the reconstructed pixel data 817 to reduce encoding and decoding artifacts, especially at the boundaries of pixel blocks. In some embodiments, the filtering or smoothing operations performed by the loop filter 845 include a deblocking filter (DBF), a sample adaptive offset (SAO), and / or an adaptive loop filter (ALF).

[0167] Fig. 9 Portions of a video encoder 800 that implements multiple stages or cascaded filtering of an ALF are described. Specifically, the figure describes components of a loop filter 845 of the video encoder 800. As shown, the loop filter 845 receives reconstructed pixel data 817 for a current block (e.g., a current CTB) and produces a filtered output to be stored in a reconstructed picture buffer 850. The incoming pixel data is processed in the loop filter 845 by a deblocking filter module (DBF) 902 and a sample adaptive offset (SAO) module 904. The processed samples produced by the DBF 902 and the SAO 904 are provided to an adaptive loop filter (ALF) module 906. By referring to the above Figure 2 The loop filter 200 is described, which includes DBF, SAO and ALF.

[0168] The ALF module 906 generates a correction value to be added to the current sample, which is provided by the SAO module 904. The correction value is generated by applying a filter 920 to the current sample and its neighboring samples. The filter 920 may be generated by referring to the above Figure 5-6 described in the cascade filter, or by referring to the above Figure 7 The filter coefficients of filter 920 may be signaled in the bitstream in an APS by entropy encoder 890. Inputs to the filter taps of filter 920 are provided by filter tap generator 910, which provides the difference between the current sample and its neighboring samples as input to at least some of the filter taps.

[0169] The filter tap generator 910 may provide the neighboring samples required by the filter 920 (i.e., filter footprint) from multiple different sources. The multiple sources of samples may include the output of the SAO module 904, the output of the DBF module 902, and the reconstructed pixel data 817 (i.e., the input sample data before DBF). The multiple sources of samples selected by the filter tap generator 910 may also include residual samples of the current block (reconstructed residual 819) and predicted samples of intra-frame or inter-frame prediction of the current block (predicted pixel data 813). In some embodiments, the multiple sources of the filter tap input may also include samples of neighboring blocks of the current block (provided by the reconstructed picture buffer 850). The multiple sources of samples may also include a line buffer 915, which temporarily stores some reconstructed pixel data 817.

[0170] The samples passed into the ALF module 906 are thus combined with their corresponding correction values ​​to generate the output of the ALF module 906, which is also the output of the loop filter 845. The output of the loop filter 845 is stored in the reconstructed picture buffer 850 for use in encoding subsequent blocks.

[0171] Fig.10 The process 1000 of performing cascade filtering in an ALF is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the encoder 800 perform the process 1000 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing the encoder 800 performs the process 1000.

[0172] The encoder receives a current sample at a current pixel position of a current picture of a video at block 1010. The encoder applies a first filter having a plurality of filter taps at block 1020. The current sample and neighboring samples taken from pixel positions near the current sample are used as tap inputs of the plurality of filter taps of the first filter. In some embodiments, cascade filtering is always applied, and the process continues to block 1030. In some embodiments, cascade filtering is conditional, and the encoder determines (at 1025) whether to enable or disable cascade filtering. In some embodiments, the encoder sends a specific flag to enable or disable cascade filtering. In some embodiments, the encoder determines whether to enable or disable cascade filtering based on whether there is a difference or a sufficient difference between the current sample and the first output of the first filter. If cascade filtering is to be enabled, the process continues to 1030. If cascade filtering is to be disabled (e.g., etc.), the process continues to 1060.

[0173] The encoder applies a second filter (e.g., C 22). The first output of the first filter (e.g., ) is used to generate the tap input of at least one filter tap. In some embodiments, the tap input of at least one filter tap of the second filter is the clipped difference between the first output and the current sample (e.g., ). In some embodiments, the second filter includes a plurality of filter taps that use the difference between the current sample and the adjacent samples as tap inputs. The plurality of filter taps of the second filter may have different (or the same) filter coefficients as the first filter. The encoder generates a filter based on the second output of the second filter (e.g., ) to update the current sample.

[0174] The encoder reconstructs the current picture based on the updated current sample at block 1050. For example, the updated current sample may be stored in a reconstructed picture buffer as part of the current picture and may be used as a reference for subsequent samples or subsequent blocks of the current picture or subsequent pictures in the encoded video.

[0175] In some embodiments, when the second filter is inactive or is bypassed when cascade filtering or multi-stage filtering is disabled, the encoder updates the current sample based on the first output of the first filter at block 1060 and the process continues to 1050 .

[0176] VI. Example Video Decoder

[0177] In some embodiments, an encoder may signal (or generate) one or more syntax elements in a bitstream so that a decoder may parse the one or more syntax elements from the bitstream.

[0178] Fig.11 An example video decoder 1100 implementing an adaptive loop filter (ALF) is shown. As shown, the video decoder 1100 is an image decoding or video decoding circuit that receives a bitstream 1195 and decodes the contents of the bitstream into pixel data of a video frame for display. The video decoder 1100 has several components or modules that decode the bitstream 1195, including some components selected from an inverse quantization module 1111, an inverse transform module 1110, an intra prediction module 1125, a motion compensation module 1130, a loop filter 1145, a decoded picture buffer 1150, an MV buffer 1165, an MV prediction module 1175, and a parser 1190. The motion compensation module 1130 is part of the inter prediction module 1140.

[0179] In some embodiments, modules 1110-1190 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 1110-1190 are modules of hardware circuits implemented by electronic devices of one or more ICs. Although modules 1110-1190 are depicted as separate modules, some modules may be combined into a single module.

[0180] The parser 1190 (or entropy decoder) receives the bitstream 1195 and performs initial parsing according to the syntax defined by the video codec or image codec standard. The parsed syntax elements include various header elements, flags, and quantization data (or quantization coefficients) 1112. The parser 1190 uses entropy coding and decoding techniques such as context-adaptive binary arithmetic coding (CABAC) or Huffman encoding to parse various syntax elements.

[0181] The inverse quantization module 1111 inversely quantizes the quantized data (or quantized coefficients) 1112 to obtain transform coefficients, and the inverse transform module 1110 inversely transforms the transform coefficients 1116 to generate a reconstructed residual signal 1119. The reconstructed residual signal 1119 is added to the predicted pixel data 1113 from the intra-prediction module 1125 or the motion compensation module 1130 to generate decoded pixel data 1117. The decoded pixel data is filtered by the loop filter 1145 and stored in a decoded picture buffer 1150. In some embodiments, the decoded picture buffer 1150 is a storage external to the video decoder 1100. In some embodiments, the decoded picture buffer 1150 is a storage internal to the video decoder 1100.

[0182] The intra prediction module 1125 receives intra prediction data from the bitstream 1195 and, based thereon, generates predicted pixel data 1113 from decoded pixel data 1117 stored in the decoded picture buffer 1150. In some embodiments, the decoded pixel data 1117 is also stored in a line buffer (not shown) for intra prediction and spatial motion vector (MV) prediction.

[0183] In some embodiments, the contents of the decoded picture buffer 1150 are used for display. The display device 1155 either directly retrieves the contents of the decoded picture buffer 1150 for display, or retrieves the contents of the decoded picture buffer into a display buffer. In some embodiments, the display device receives pixel values ​​from the decoded picture buffer 1150 via pixel transfer.

[0184] The motion compensation module 1130 generates predicted pixel data 1113 from the decoded pixel data 1117 stored in the decoded picture buffer 1150 based on motion compensation motion vectors (MC MVs). These motion compensation motion vectors are decoded by adding the residual motion data received from the bitstream 1195 to the predicted motion vectors received from the motion vector prediction module (MV prediction module) 1175.

[0185] The motion vector prediction module 1175 generates a predicted motion vector based on a reference motion vector generated for decoding a previous video frame, for example, a motion compensated motion vector for performing motion compensation. The motion vector prediction module 1175 retrieves the reference motion vector of the previous video frame from the motion vector buffer (MV buffer) 1165. The video decoder 1100 stores the motion compensated motion vector generated for decoding the current video frame in the motion vector buffer 1165 as a reference motion vector for generating a predicted motion vector.

[0186] The loop filter 1145 performs filtering or smoothing operations on the decoded pixel data 1117 to reduce encoding and decoding artifacts, particularly at the boundaries of pixel blocks. In some embodiments, the filtering or smoothing operations performed by the loop filter 1145 include a deblock filter (DBF), a sample adaptive offset (SAO), and / or an adaptive loop filter (ALF).

[0187] Fig.12A portion of a video decoder 1100 that implements an adaptive loop filter (ALF) having multiple stages or cascaded filtering is shown. Specifically, the figure shows components of a loop filter 1145 of the video decoder 1100. As shown, the loop filter 1145 receives decoded pixel data 1117 for a current block (e.g., a current CTB) and produces a filtered output to be stored in a decoded picture buffer 1150. The incoming pixel data is processed in the loop filter 1145 by a deblocking filter module (DBF) 1202 and a sample adaptive offset (SAO) module 1204. The processed samples produced by the DBF 1202 and the SAO 1204 are provided to an adaptive loop filter (ALF) module 1206. By reference to the above Figure 2 The example of loop filter 200 with DBF, SAO and ALF is described.

[0188] The ALF module 1206 generates a correction value (Corr. Value) to be added to the current sample, which is provided by the SAO module 1204. The correction value is generated by applying a filter 1220 to the current sample and its neighboring samples. The filter 1220 can be as described above. Figure 5-6 cascaded filters as described above, or Figure 7 The filter coefficients of filter 1220 may be parsed from the bitstream in the APS by entropy decoder 1190. The inputs to the filter taps of filter 1220 are provided by filter tap generator 1210, which provides the difference between the current sample and its neighboring samples as inputs to at least some of the filter taps.

[0189] The filter tap generator 1210 may provide the neighboring samples (i.e., filter footprint) required by the filter 1220 from a plurality of different sources. The plurality of sources of samples may include the output of the SAO module 1204, the output of the DBF module 1202, and the decoded pixel data 1117, i.e., the input sample data before the DBF. The plurality of sample sources selected by the filter tap generator 1210 may also include the residual samples of the current block and the prediction samples (predicted pixel data 1113) of the inter-frame or intra-frame prediction of the current block. In some embodiments, the plurality of sources of the filter tap input may also include the samples of the neighboring blocks of the current block provided by the decoded picture buffer 1150. The plurality of sources of samples may also include a line buffer 1215, which temporarily stores some decoded pixel data 1117.

[0190] Therefore, the samples passed into the ALF module 1206 are combined with their corresponding correction values ​​to generate the output of the ALF module 1206, which is also the output of the loop filter 1145. The output of the loop filter 1145 is stored in the decoded picture buffer 1150 for use in decoding subsequent blocks.

[0191] Fig.13 The process 1300 of performing cascade filtering in an adaptive loop filter (ALF) is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the decoder 1100 perform the process 1300 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing the decoder 1100 performs the process 1300.

[0192] The decoder receives (at block 1310) a current sample at a current pixel position of a current picture of a video. The decoder applies (at block 1320) a first filter having a plurality of filter taps. The current sample and neighboring samples taken from pixel positions near the current sample are used as tap inputs to the plurality of filter taps of the first filter.

[0193] In some embodiments, cascade filtering is always applied, and the process continues to block 1330. In some embodiments, cascade filtering is conditional, and the decoder determines (at 1325) whether to enable or disable cascade filtering. In some embodiments, the decoder receives a specific flag to enable or disable cascade filtering. In some embodiments, the decoder determines whether to enable or disable cascade filtering based on whether there is a difference or a sufficient difference between the current sample and the first output of the first filter. If cascade filtering is to be enabled, the process continues to 1330. If cascade filtering is to be disabled (e.g., etc.), the process continues to 1360.

[0194] The decoder application (at block 1330) has at least one filter tap (e.g., C 22 ). The first output of the first filter (e.g., ) is used to generate a tap input of at least one filter tap of the second filter. In some embodiments, the tap input of at least one filter tap of the second filter is the clipped difference between the first output and the current sample (e.g., ). In some embodiments, the second filter includes a plurality of filter taps that use the difference between the current sample and the adjacent samples as tap inputs. The plurality of filter taps of the second filter may have filter coefficients that are different (or the same) as those of the first filter. The decoder is based on the second output of the second filter (e.g., ) to update the current sample.

[0195] The decoder reconstructs the current picture based on the updated current sample (at block 1350). For example, the updated current sample may be stored in a reconstructed picture buffer as part of the current picture, which may be used as a reference for decoding subsequent samples or subsequent blocks of the current picture or video. The decoder may provide the reconstructed current picture for display.

[0196] In some embodiments, when the second filter is inactive or bypassed when cascade filtering or multi-stage filtering is disabled, the decoder updates the current sample based on the first output of the first filter (at block 1360 ). The process then continues to 1350 .

[0197] VII. Example Electronic System

[0198] Many of the above features and applications are implemented as a software process specified by a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more computing or processing units (e.g., one or more processors, cores of a processor, or other processing units), they cause the processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, random access memory (RAM) chips, hard disks, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), etc. Computer-readable media do not include carrier waves and electronic signals transmitted through wireless or wired connections.

[0199] In this specification, the term "software" is intended to include firmware residing in read-only memory or application programs stored in magnetic storage that can be read into memory and processed by a processor. In addition, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while remaining independent software inventions. In some embodiments, multiple software inventions can also be implemented as independent programs. Finally, any combination of independent programs that together implement the software inventions described herein are within the scope of the present invention. In some embodiments, when the software program is installed on one or more electronic systems for operation, one or more specific machine implementations are defined that execute and perform the operations of the software program.

[0200] Fig.14The electronic system 1400 for implementing certain embodiments of the present invention is conceptually illustrated. The electronic system 1400 may be a computer (e.g., a desktop computer, a personal computer, a tablet computer, etc.), a phone, a PDA, or any other type of electronic device. Such an electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 1400 includes a bus 1405, a processing unit 1410, a graphics processing unit (GPU) 1415, a system memory 1420, a network 1425, a read-only memory (ROM) 1430, a permanent storage device 1435, an input device 1440, and an output device 1445.

[0201] Bus 1405 collectively represents all system, peripheral, and chipset buses that communicatively connect numerous devices within electronic system 1400. For example, bus 1405 communicatively connects processing unit 1410 with GPU 1415, read-only memory 1430, system memory 1420, and permanent storage device 1435.

[0202] From these various storage units, the processing unit (1410) retrieves execution instructions and data to perform the processes of the present invention. The processing unit can be a single processor or a multi-core processor in different embodiments. Some instructions are passed to and executed by the GPU (1415). The GPU (1415) can offload various calculations or supplement the image processing provided by the processing unit (1410).

[0203] The read-only memory (ROM, 1430) stores static data and instructions used by the processing unit (1410) and other modules of the electronic system. On the other hand, the permanent storage device (1435) is a read-write storage device. This device is a non-volatile storage unit that stores instructions and data even when the electronic system (1400) is turned off. Some embodiments of the present invention use a mass storage device (such as a magnetic disk or optical disk and its corresponding drive) as the permanent storage device (1435).

[0204] Other embodiments use removable storage devices (such as floppy disks, flash memory devices, etc. and their corresponding drivers) as permanent storage devices. Like the permanent storage device (1435), the system memory (1420) is a read-write storage device. However, unlike the storage device (1435), the system memory (1420) is a volatile read-write memory, such as a random access memory. The system memory (1420) stores some instructions and data used by the processor at runtime. In some embodiments, processes according to the present invention are stored in the system memory (1420), the permanent storage device (1435) and / or the read-only memory (1430). For example, various storage units include instructions for processing multimedia clips according to some embodiments. From these different storage units, the processing unit (1410) retrieves execution instructions and data to perform the processes of some embodiments.

[0205] The bus (1405) also connects input and output devices (1440 and 1445). Input devices (1440) enable a user to communicate information and select commands to the electronic system. Input devices (1440) include alphanumeric keyboards and pointing devices (also known as "cursor control devices"), cameras (e.g., webcams), microphones or similar devices for receiving voice commands, etc. Output devices (1445) display images generated by the electronic system or output data in other ways. Output devices (1445) include printers and display devices, such as cathode ray tubes (CRTs) or liquid crystal displays (LCDs), and speakers or similar audio output devices. Some embodiments include devices such as touch screens as both input devices and output devices.

[0206] Finally, if Fig.14 As shown, bus 1405 also connects electronic system 1400 to a network (1425) via a network adapter (not shown). In this way, the computer can become part of a computer network (such as a local area network ("LAN"), a wide area network ("WAN") or an intranet, or a network of networks, such as the Internet. Any or all components of electronic system 1400 may be used with the present invention.

[0207] Some embodiments include electronic components, such as microprocessors, storage, and memory, which store computer program instructions in a machine-readable or computer-readable medium (also referred to as a computer-readable storage medium, a machine-readable medium, or a machine-readable storage medium). Some examples of such computer-readable media include RAM, ROM, a read-only compact disk (CD-ROM), a recordable compact disk (CD-R), a rewritable compact disk (CD-RW), a read-only digital versatile disk (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini-SD card, micro-SD card, etc.), magnetic and / or solid-state hard disks, read-only and recordable A computer readable medium may store a computer program that contains a set of instructions for performing various operations and is executed by at least one processing unit. Examples of computer programs or computer code include machine code, such as produced by a codec, and files containing high-level code that are executed by a computer, electronic component, or microprocessor using an interpreter.

[0208] Although the above discussion refers primarily to microprocessors or multi-core processors that execute software, many of the features and applications described above are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuits themselves. In addition, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices.

[0209] As used in this specification and any claims of this application, the terms "computer", "server", "processor", and "memory" refer to electronic or other technical devices. These terms do not include people or groups of people. For the purposes of this specification, the terms display or displaying means displaying on an electronic device. As used in this specification and any claims of this application, the terms "computer-readable medium", "computer-readable medium", and "machine-readable medium" are entirely limited to tangible, physical objects that store information in a form that can be read by a computer. These terms do not include any wireless signals, wired download signals, and any other transient signals.

[0210] Although the present invention has been described with reference to numerous specific details, those skilled in the art will recognize that the present invention may be embodied in other specific forms without departing from the spirit of the present invention. Fig.10 and Fig.13) conceptually illustrate the processes. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. In addition, the process can be implemented using several sub-processes or as part of a larger macro process. Therefore, it will be understood by those skilled in the art that the present invention should not be limited by the above illustrative details, but should be defined by the appended claims.

[0211] The subject matter described herein sometimes shows that different components are included or connected to other different components. It should be understood that the architecture described in this way is only an example, and many other architectures can actually be implemented to achieve the same function. Conceptually, the arrangement of any components to achieve the same function is actually "associated" in order to achieve the desired function. Therefore, any two components combined to achieve a specific function here can be regarded as "associated" with each other to achieve the desired function, regardless of the architecture or intermediate components. Similarly, any two components associated in this way can also be regarded as "operably connected" or "operably coupled" to each other to achieve the desired function, and any two components that can be associated in this way can also be regarded as "operably coupled" to each other to achieve the desired function. Specific examples of operable coupling include, but are not limited to, components that can be physically matched and / or physically interacted and / or components that can be wirelessly interacted and / or wirelessly interacted and / or components that can interact logically and / or logically interact.

[0212] Furthermore, with respect to virtually any plural and / or singular terms used herein, one having technical skill can translate from the plural to the singular and / or from the singular to the plural as appropriate to the context and / or application. Various singular / plural permutations may be expressly listed herein for clarity.

[0213] Furthermore, those skilled in the art will appreciate that the terms used herein, particularly in the appended claims, such as the claim bodies, are generally considered to be "open" terms, e.g., "comprising" should be interpreted as "including but not limited to," "having" should be interpreted as "having at least," "including" should be interpreted as "including but not limited to," etc. Those skilled in the art will also understand that if a specific number of an introduced claim recitation is intended, such intent will be explicitly stated in the claim, and in the absence of such a statement, no such intent is present. For example, as an aid to understanding, the following appended claims may contain the use of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be interpreted to mean that any particular claim that introduces a claim recitation by the indefinite article "a" or "an" only includes implementation of one such recitation, even if the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an," e.g., "a" and / or "an" should be interpreted as "at least one" or "one or more"; the same rule applies to the use of definite articles used to introduce claim recitations. Furthermore, even if a specific number of claim recitations introduced is explicitly recited, those skilled in the art will recognize that such a recitation should be interpreted as at least the recited number, e.g., merely reciting "two recitations," without other modifiers, means at least two recitations, or two or more recitations. Furthermore, where conventions similar to "at least one of A, B, and C, etc." are used, such constructions are generally constructed in accordance with the meaning understood by those skilled in the art, e.g., "a system having at least A, B, and C" would include, but are not limited to, systems including only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together. Where conventions similar to "at least one of A, B, or C, etc." are used, such constructions are generally constructed in accordance with the meaning understood by those skilled in the art, e.g., "a system having at least A, B, or C" would include, but are not limited to, systems including only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together. Those skilled in the art will also understand that virtually any disjunctive word and / or phrase presenting two or more alternative terms in the description, claims, or drawings should be understood to contemplate the possibility of including one, either, or both of the terms. For example, the phrase "A or B" will be understood to include the possibility of "A" or "B" or "A and B."

[0214] As can be seen from the above, various implementations of the present invention have been described herein for illustrative purposes, and various modifications may be made without departing from the scope and spirit of the present invention. Therefore, the various implementations disclosed herein are not meant to be restrictive, and the true scope and spirit are indicated by the following claims.

Claims

1. A video encoding and decoding method comprises: Receive a current sample at a current pixel position of a current picture of a video; applying a first filter including a plurality of filter taps, wherein the current sample and neighboring samples taken from pixel locations near the current sample are used as tap inputs to the plurality of filter taps of the first filter; applying a second filter comprising at least one filter tap, wherein a first output of the first filter is used to generate a tap input of at least one filter tap of the second filter; updating the current sample based on a second output of the second filter; as well as The current picture is reconstructed based on the updated current samples.

2. The video encoding and decoding method according to claim 1, characterized in that: The method further comprises: Receive or send a specific flag to enable or disable cascade filtering; When the specific flag enables cascade filtering, the output of the second filter is used to update the current sample; as well as When the special flag disables cascade filtering, the output of the first filter is used to update the current sample.

3. The video encoding and decoding method according to claim 1, characterized in that: The second filter is applied only when the difference between the current sample and the first output of the first filter is non-zero.

4. The video encoding and decoding method according to claim 1, characterized in that: The second filter is applied only when the difference between the current sample and the first output of the first filter is zero.

5. The video encoding and decoding method according to claim 1, characterized in that: The second filter is applied only when the difference between the current sample and the first output of the first filter is greater than a threshold.

6. The video encoding and decoding method according to claim 1, characterized in that: The second filter is applied only when the difference between the current sample and the first output of the first filter is less than a threshold.

7. The video encoding and decoding method according to claim 1, characterized in that: A tap input to at least one filter tap of the second filter is a clipped difference between the first output and the current sample.

8. The video encoding and decoding method according to claim 1, wherein: The second filter further includes a plurality of filter taps using differences between the current sample and adjacent samples as tap inputs.

9. The video encoding and decoding method according to claim 8, characterized in that: The plurality of filter taps of the second filter have different filter coefficients than those of the first filter.

10. The video encoding and decoding method according to claim 1, characterized in that: The current sample and the neighboring samples used as the tap input of the second filter are filtered by the first filter.

11. The video encoding and decoding method according to claim 1, wherein: The plurality of filter taps of the first filter use the difference between a current sample and adjacent samples as tap inputs.

12. The video encoding and decoding method according to claim 1, characterized in that: The first filter and the second filter are part of an adaptive loop filter (ALF) in a video coding system, wherein filtered samples of a current block are provided for coding subsequent blocks of the current picture.

13. An electronic device comprising: The video codec circuit is configured to perform the following operations: Receive a current sample at a current pixel position of a current picture of a video; applying a first filter comprising a plurality of filter taps, wherein the current sample and neighboring samples taken from pixel locations near the current sample are used as tap inputs to the plurality of filter taps of the first filter; applying a second filter comprising at least one filter tap, wherein a first output of the first filter is used to generate a tapped input of at least one filter tap of the second filter; updating the current sample based on a second output of the second filter; as well as The current picture is reconstructed based on the updated current samples.

Images