Adaptive loop filter with virtual boundaries and diverse origins

Abstract

The present invention provides a method for implementing an adaptive loop filter (ALF) in a video system, and a method for implementing an adaptive loop filter (ALF) in a video system. A video codec receives data of a pixel block of a current block of a current image to be encoded or decoded as a video. A video codec receives a current sample of a current block. The video codec applies a filter to the current sample to generate a correction value. A plurality of adjacent samples from two or more different sources are used as inputs to the filter. When a first adjacent sample is within the virtual boundary, the first adjacent sample is used as an input to the filter. When a first adjacent sample exceeds the virtual boundary, the first adjacent sample is excluded as an input to the filter. The video codec adds the correction value to the current sample as a filtered sample of the current block.

Description

[0001] Cross-references

[0002] This application is part of a non-provisional application claiming priority to U.S. Provisional Patent Application No. 63 / 368,509 filed on July 15, 2022. The above application is hereby referenced and incorporated herein in its entirety. Technical Field

[0003] The present invention generally relates to video encoding and decoding, and more particularly to a method for encoding and decoding video images using an adaptive loop filter (ALF). Background Art

[0004] Unless otherwise indicated herein, the methods described in this section are not prior art in the scope of the following patent claims and are not admitted to be prior art by inclusion of this section.

[0005] High-Efficiency Video Coding (HEVC) is an international video codec standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). HEVC is based on a hybrid block-based motion compensated DCT-like transform codec architecture. The basic unit of compression is called a coding unit (CU), which is a 2N×2N square pixel block. Each CU can be recursively divided into four smaller CUs until a predefined minimum size is reached. Each CU contains one or more prediction units (PUs).

[0006] Versatile Video Coding (VVC) is the latest international video codec standard developed by ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11 Joint Video Expert Team (JVET). The input video signal is predicted from the reconstructed signal, which is derived from the coded and decoded image region. The prediction residual signal is processed by block transform. The transform coefficients are quantized and entropy coded along with other side information in the bitstream. The reconstructed signal is generated from the prediction signal and the reconstructed residual signal after inverse transforming the dequantized transform coefficients. The reconstructed signal is also processed by loop filtering to remove coding artifacts. The decoded image is stored in a buffer and used to predict future images in the input video signal.

[0007] In VVC, the coded image is divided into non-overlapping square block areas represented by related coding tree units (CTUs). The leaf nodes of the coding tree correspond to coding units (CUs). The coded image can be represented by a set of slices, each slice comprising an integer number of CTUs. Each CTU in a slice is processed in raster scan order. Bi-predicted (B) slices can be decoded using intra-picture prediction or inter-picture prediction with up to two motion vectors and reference indices to predict the sample values ​​of each block. Predicted (P) slices are decoded using intra-picture prediction or inter-picture prediction with up to one motion vector and reference index to predict the sample values ​​of each block. Intra-picture (I) slices are decoded using only intra-picture prediction.

[0008] Using a quadtree (QT) with a nested multi-type-tree (MTT) structure, the CTU can be divided into one or more non-overlapping codec units (CUs) to accommodate various local motion and texture features. The CU is further divided into smaller CUs using one of five partition types: quadtree partition, vertical binary tree partition, horizontal binary tree partition, vertical center-side ternary tree partition, and horizontal center-side ternary tree partition.

[0009] Each CU contains one or more prediction units (PUs). The prediction unit, together with the associated CU syntax, serves as a basic unit for indicating prediction sub-information. The values ​​of the relevant pixel samples within the PU are predicted using a specified prediction process. Each CU may contain one or more transform units (TUs) representing prediction residual blocks. The transform unit (TU) includes a transform block (TB) of luma samples and two corresponding transform blocks of chroma samples, each TB corresponding to a residual block sample of a color component. An integer transform is applied to the transform block. The layer values ​​of the quantized coefficients are entropy encoded and decoded in the bitstream along with other side information. The terms Coding Tree Block (CTB), Coding Block (CB), Prediction Block (PB), and Transform Block (TB) are defined to specify a 2D sample array of a color component associated with a CTU, CU, PU, ​​and TU, respectively. Therefore, a CTU includes a luma CTB, two chroma CTBs, and related syntax elements. Similar relationships apply to CU, PU, ​​and TU.

[0010] For each inter-predicted CU, motion parameters including motion vector, reference picture index and reference picture list usage index and additional information are used to generate inter-predicted samples. Motion parameters can be indicated in an explicit or implicit way. When the CU is encoded and decoded using skip mode, the CU is associated with one PU and has no valid residual coefficients, no coded motion vector delta or reference picture index. A merge mode is specified, in which the motion parameters of the current CU are obtained from neighboring CUs, including spatial candidates and temporal candidates, as well as the additional schedule introduced in VVC. Merge mode can be applied to any inter-predicted CU. An alternative to merge mode is the explicit transmission of motion parameters, in which the motion vector, the corresponding reference picture index and reference picture list usage flag for each reference picture list and other required information are explicitly indicated for each CU. Summary of the invention

[0011] The following summary of the invention is provided for illustration only and is not intended to be limiting in any way. That is, the following summary of the invention is provided to introduce the concepts, highlights, benefits and advantages of the novel and non-obvious technologies described herein. Selected embodiments will be further described in the following specific examples. Therefore, the following summary of the invention is not intended to identify the essential characteristics of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

[0012] Some embodiments of the present application provide a method for implementing ALF. A video encoder receives data of a pixel block of a current block of a current image to be encoded or decoded as a video. A video codec receives a current sample of the current block. The video codec applies a filter to the current sample to generate a correction value. A plurality of adjacent samples from two or more different sources are used as inputs to the filter. When the first adjacent sample is within a virtual boundary, the first adjacent sample is used as input to the filter. When the first adjacent sample exceeds the virtual boundary, the first adjacent sample is excluded as input to the filter. The video codec adds the correction value to the current sample as a filtered sample of the current block.

[0013] In some embodiments, the virtual boundary is a horizontal boundary of several rows above or below a horizontal boundary of a coding tree unit (CTU). A plurality of adjacent samples from more than two different sources include a first sample filtered by a deblocking filter (DBF) and a second sample not filtered by the deblocking filter. A plurality of adjacent samples from more than two different sources include (i) samples before applying a sample adaptive offset (SAO), (ii) filtered samples generated by a fixed filter, (iii) reconstructed residual samples after inverse transformation, (iv) predicted samples generated by inter-picture prediction or intra-picture prediction, and (v) at least two of the samples processed by a deblocking filter and a sample adaptive offset.

[0014] In some embodiments, the excluded first neighboring samples are replaced by padding samples as input to the filter, and the second neighboring samples located at filter positions symmetrical to the first neighboring samples are also replaced by padding samples as input to the filter.

[0015] In some embodiments, when the first neighboring sample exceeds the virtual boundary, the first difference between the first neighboring sample and the current sample is set to zero. The first neighboring sample may be a sample that is not filtered by DBF (or before DBF). In some embodiments, for a second neighboring sample located at a filter position symmetrical to the first neighboring sample, when the first neighboring sample exceeds the virtual boundary, the second difference between the second neighboring sample and the current sample is also set to zero (even if the second neighboring sample does not exceed the virtual boundary).

[0016] In some embodiments, when the first neighboring sample is beyond the virtual boundary and the first neighboring sample is from the first source, the second neighboring sample from the first source within the virtual boundary is also excluded as the input of the filter, or none of the neighboring samples from the first source is used as the input of the filter. In some embodiments, if the second neighboring sample is the current sample (or at the center of the filter), the second neighboring sample is used as the input of the filter. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings are included to provide a further understanding of the present invention and are incorporated into and constitute a part of the present invention. The drawings illustrate embodiments of the present invention and together with the description are used to explain the principles of the present invention. It should be noted that the drawings are not necessarily drawn to scale, as some components may be shown out of proportion to the size in the actual embodiment in order to clearly illustrate the concepts of the present invention.

[0018] Figure 1A-B shows two diamond filter shapes for the adaptive loop filter (ALF).

[0019] Figure 2 A system level schematic diagram of a loop filter is shown, wherein reconstructed or decoded samples are filtered or processed by a deblock filter (DBF), a sample adaptive offset (SAO), and an adaptive filter (ALF).

[0020] Figure 3 Filtering in cross-component-ALF (CC-ALF) is shown.

[0021] Figure 4A -B shows modified block classification at virtual boundaries near horizontal CTU boundaries.

[0022] Figure 5A -B schematically illustrates a padding operation at a virtual boundary for generating filter taps for ALF filtering.

[0023] Fig. 6A -B shows the filter shape applied to the samples before DBF.

[0024] Fig. 7A -C schematically illustrates symmetric and asymmetric processing across virtual boundaries for ALF filtering when the difference between the current sample and adjacent samples is used to generate the filter tap inputs.

[0025] Figure 8 An exemplary video encoder implementing an in-loop filter is shown.

[0026] Fig. 9 Portions of a video encoder implementing an ALF with virtual boundaries based on samples from a plurality of sources are shown.

[0027] Fig.10 The process of performing ALF filtering based on a virtual boundary using samples from a plurality of sources is schematically illustrated.

[0028] Fig.11 An exemplary video decoder implementing an in-loop filter is shown.

[0029] Fig.12 Part of a video decoder implementing an ALF with virtual boundaries based on samples from a plurality of sources is shown.

[0030] Fig.13 The process of performing ALF filtering based on a virtual boundary using samples from a plurality of sources is schematically illustrated.

[0031] Fig.14 An electronic system for implementing some embodiments of the present application is schematically shown. DETAILED DESCRIPTION

[0032] In the detailed description below, many specific details are set forth by way of example to provide a comprehensive understanding of the relevant teachings. Any variants, derivatives and / or extensions based on the teachings of the present application are within the scope of protection of the present application. In some cases, in order to avoid unnecessary confusion of various aspects of the teachings of the present application, well-known methods, processes, components and / or circuits related to one or more example embodiments disclosed herein may be described at a relatively high level without details.

[0033] I.ALF (Adaptive Loop Filter)

[0034] A. Filter Shape

[0035] ALF is an in-loop filtering technique used in video codec standards such as VVC. It is a block-based filter that minimizes the mean square error between the original and reconstructed samples. For the luminance component, one of 25 filters is selected for each 4×4 block based on the direction and activity of the local gradient. Figure 1A -B shows two diamond filter shapes for ALF. Each position in the diamond corresponds to a filter tap with a filter coefficient. Figure 1A A 7x7 diamond is shown with taps having filter coefficients C0-C12 applied to the luma component. Figure 1B A 5x5 diamond is shown with filter coefficients C0-C6 applied to the chrominance components.

[0036] B. Block Classification

[0037] For the luminance component, each 4×4 block is classified into one of 25 classes. The class index C is based on its directionality D and activity The quantitative value of is derived as follows:

[0038]

[0039] To calculate D and First, use the one-dimensional Laplace to calculate the gradient in the horizontal, vertical, and two diagonal directions:

[0040]

[0041] Where the indices i and j refer to the coordinates of the top left sample within the 4×4 block, and R(i,j) refers to the reconstructed sample at coordinate (i,j). To reduce the complexity of block classification, a one-dimensional Laplacian calculation of subsamples is used. The same subsample position can be used for gradient calculations in all directions (the subsample position can be used for vertical gradients, horizontal gradients, or diagonal gradients). The maximum and minimum values ​​of D for the horizontal and vertical gradients are set to:

[0042]

[0043] The maximum and minimum values ​​of the gradients in the two diagonal directions are set to:

[0044]

[0045] To derive the value of the directionality D, these values ​​are compared with each other and with two thresholds t1 and t2:

[0046] Step 1. If and is true, then D is set to 0.

[0047] Step 2. If Then continue from step 3; otherwise continue from step 4.

[0048] Step 3. If If yes, set D to 2; otherwise, set D to 1.

[0049] Step 4. If Then set D to 4; otherwise, set D to 3.

[0050] The activity value A is calculated as:

[0051]

[0052] A is further quantized to the range of 0 to 4, inclusive, and the quantized values ​​are expressed as For the chrominance components in the image, no classification method is applied.

[0053] C. Geometric Transformation of Filter Coefficients and Clipping Values

[0054] Before filtering each 4×4 luma block, geometric transformations such as rotation, or diagonal and vertical flipping are applied to the filter coefficients f(k,l) and the corresponding filter clipping values ​​c(k,l) according to the gradient values ​​calculated for the block. This is equivalent to applying these transformations to the samples in the filter support region. The scheme is to make different blocks to which the ALF is applied more similar by aligning the directionality of different blocks. There are three geometric transformations, including diagonal flipping, vertical flipping, and rotation, which are described as:

[0055] Diagonal: f D (k,l)=f(l,k),c D (k,l)=c(l,k),

[0056] Flip vertically: f V (k,l)=f(k,Kl-1),c V (k,l)=c(k,Kl-1)

[0057] Rotation: f R (k,l)=f(Kl-1,k),c R (k,l)=c(Kl-1,k)

[0058] Where K is the size of the filter, 0≤k,l≤K-1 are the coefficient coordinates, so that the position (0,0) is in the upper left corner and the position (K-1,K-1) is in the lower right corner. The transform is applied to the filter coefficients f(k,l) and the clipping value c(k,l) according to the gradient value calculated for the block. The relationship between the transform and the four gradients in four directions is summarized in Table 1 below, which shows the mapping of the gradient calculated for a block and the transform.

[0059] Table 1:

[0060] Gradient Value Transform <![CDATA[g d2 <g d1 and g h <g v ]]> No transformation <![CDATA[g d2 <g d1 and g v <g h ]]> diagonal <![CDATA[g d1 <g d2 and g h <g v ]]> Flip Vertically <![CDATA[g d1 <g d2 and g v <g h ]]> Rotation

[0061] D. Filtering process

[0062] On the decoder side, when ALF is enabled for CTB, each sample R'(i,j) in the CU is filtered to obtain the sample value R'(i,j) as shown below:

[0063]

[0064] Where f(k,l) represents the decoded filter coefficients, K(x,y) is the clipping function, and c(k,l) represents the decoded clipping parameters. The variables k and l vary between -L / 2 and L / 2, where l represents the filter length. The clipping function K(x,y) = min(y,max(-y,x)), which corresponds to the function Clip3(-y,y,x). The clipping operation introduces nonlinearity to make the ALF more efficient by reducing the impact of neighboring sample values ​​that differ too much from the current sample value.

[0065] E.CC-ALF(Cross Component Adaptive Loop Filter)

[0066] CC-ALF can use luma sample values ​​to refine each chroma component by applying an adaptive linear filter to the luma channel and then using the output of this filtering operation for chroma refinement. Figure 2 A system level diagram of a loop filter is shown, wherein reconstructed or decoded samples 210 are filtered or processed by DBF, SAO and ALF. The reconstructed or decoded samples 210 may be generated from a prediction signal and a residual signal of a current block.

[0067] The figure shows the displacement of CC-ALF relative to other loop filters. Specifically, the luma component of the SAO output is processed by a luma ALF process (ALF Y) and a pair of cross-component ALF processes (CC-ALF Cb and CC-ALF Cr). The two cross-component ALF processes generate a Cb component and a cross-component offset of the Cb component to add it to the output of the chroma ALF process (ALF Chroma) to generate the ALF output of the chroma component. The luma component and chroma component of the ALF output are then stored in the reconstructed or decoded image buffer 290 for predictive coding and decoding of subsequent blocks.

[0068] Figure 3 Filtering in CC-ALF is shown, which is accomplished by applying a linear diamond filter 310 to the luma channel. One filter is used per chroma channel, and the operation is represented as:

[0069]

[0070] where (x, y) is the position of the chrominance component i being refined, (x Y ,y Y ) is the brightness position based on (x,y), S i is the filter support area in the luminance component, c i (x0,y0) represents the filter coefficients. Figure 3As shown, after taking into account the spatial scaling factor between the luma plane and the chroma plane, the luma filter support area is the area that is collocated with the current chroma sample.

[0071] The CC-ALF filter coefficients can be calculated by minimizing the mean square error of each chroma channel with respect to the original chroma content. To implement this scheme, the algorithm can use a coefficient derivation process similar to that used for chroma ALF. Specifically, a correlation matrix is ​​derived and the coefficients are calculated using a Cholesky decomposition solver in an attempt to minimize the mean square error metric. When designing the filters, up to 8 CC-ALF filters can be designed and transmitted per image. The resulting filters are then indicated for each of the two chroma channels on a CTU basis.

[0072] CC-ALF filtering can use a 3x4 diamond with 8 filter taps and 7 filter coefficients transmitted in the Adaptation Parameter Set (APS) (which can be referenced in the slice header). Each transmitted coefficient has a 6-bit dynamic range and is limited to a power of 2 value. The 8th filter coefficient is derived at the decoder so that the sum of the filter coefficients is equal to 0. For each chroma component, CC-ALF filter selection can be controlled at the CTU level. The border filling of the horizontal virtual border can be the same memory access mode as the luma ALF.

[0073] As an additional feature, the reference encoder can be used to enable some basic subjective tuning via a configuration file. When enabled, VTM attenuates the application of CC-ALF in areas that are encoded with a high QP and are close to mid-grey or contain a lot of luminance high frequencies. Algorithmically, this is done by disabling the application of CC-ALF in CTUs where any of the following conditions are true:

[0074] (i) The slice QP value minus 1 is less than or equal to the base QP value;

[0075] (ii) the number of chroma samples with a local contrast greater than (1<<(bitDepth–2))–1 exceeds the CTU height, where local contrast is the difference between the maximum and minimum luma sample values ​​within the filter support area;

[0076] (iii) More than a quarter of the chroma samples are in the range between (1<<(bitDepth–1))–16and(1<<(bitDepth–1))+16.

[0077] This is to ensure that CC-ALF does not amplify artifacts introduced earlier in the decoding path.

[0078] F. Filter parameter marking

[0079] ALF filter parameters are indicated in the APS. In one APS, up to 25 sets of luma filter coefficients and clipping value indices and up to 8 sets of chroma filter coefficients and clipping value indices can be indicated. In order to reduce bit overhead, filter coefficients of different classifications of luma components can be merged. In the slice header, the index of the APS for the current slice is indicated.

[0080] The clipping value index decoded from the APS allows to determine the clipping value using a table of clipping values ​​for luma and chroma components. These clipping values ​​depend on the internal bit depth. More precisely, the clipping value is obtained using the following formula:

[0081] ALFClip={round(2 B-α*n )for n∈[0..N-1]}

[0082] Where B is equal to the internal bit depth, α is a predefined constant value equal to 2.35, and N is equal to 4, which is the number of crop values ​​allowed in VVC. ALFClip is then rounded to the nearest value in power of 2 format.

[0083] In the slice header, up to 7 APS indices can be indicated to specify the luma filter set for the current slice. The filtering process can be further controlled at the CTB level. A flag is always indicated to indicate whether the ALF is applied to the luma CTB. The luma CTB can select a filter set from 16 fixed filter sets and a filter set from the APS. The filter set index of the luma CTB is indicated to indicate which filter set is applied. The 16 fixed filter sets are predefined and hard-coded in the encoder and decoder.

[0084] For chroma components, the APS index can be signaled in the slice header to indicate the chroma filter set being used for the current slice. At the CTB level, if there is more than one chroma filter in the APS, the filter index is signaled for each chroma CTB. The filter coefficients are quantized with a normalization equal to 128. To limit the multiplication complexity, bitstream consistency is applied so that coefficient values ​​at non-central positions will be around -2 7 To 2 7-1 In the range, including -2 7 and 2 7-1 The center position coefficient is not signaled in the bitstream and is assumed to be equal to 128.

[0085] G. Simplified ALF filtering using fixed filters

[0086] In some embodiments, ALF gradient subsamples and ALF virtual boundary processing are removed. The block size of the classification is reduced from 4x4 to 2x2. The filter size of the ALF coefficients is increased to 9x9 for luma and chroma.

[0087] To filter the luminance samples, three different classifiers (C0, C1, and C2) and three different filter sets (F0, F1, and F2) can be used. Sets F0 and F1 contain fixed filters with coefficients trained for classifiers C0 and C1. The coefficients of the filters in F2 are labeled. Which filter in set Fi is used for a given sample is determined by the filter selected using classifier C0. i The category C assigned to this sample i First, two 13x13 diamond fixed filters F0 and F1 are applied to derive two intermediate samples R0(x,y) and R1(x,y). Then, F2 is applied to R0(x,y) and R1(x,y) and adjacent samples to derive the filtered samples as:

[0088]

[0089] Among them, f i,j is the cropping difference between the adjacent sample and the current sample R(x,y), g i YesR i-20 The clipping difference between (x,y) and the current sample. Filter coefficient c i , i=0,…21 are marked.

[0090] Based on directionality D i and activity Category C i Assigned to each 2x2 block.

[0091]

[0092] Among them, M D,i Indicates directionality D i The total number of . The value of the horizontal gradient, vertical gradient, and two diagonal gradients can be calculated for each sample using a one-dimensional Laplacian. The sum of the sample gradients within the 4×4 window covering the target 2×2 block is used for classifier C0, and the sum of the sample gradients within the 12×12 window is used for classifiers C1 and C2. The sum of the horizontal gradient, vertical gradient, and two diagonal gradients are expressed as and Directionality D i Determined by comparing the following to a set of thresholds:

[0093]

[0094] Directivity D2 is derived using thresholds 2 and 4.5. For D0 and D1, first calculate the horizontal / vertical edge strength and diagonal edge strength Threshold Th = [1.25, 1.5, 2, 3, 4.5, 8] is used. If The edge strength value is 0, otherwise, is the largest integer, so if The edge strength is 0, otherwise, is the largest integer, so Table 2(a) and Table 2(b) below show and The mapping with Di. That is, horizontal edges / vertical edges dominate, and Di is derived using Table 2(a) below. Otherwise, diagonal edges dominate, and D is derived using Table 2(b) below. i .

[0095] Table 2(a):

[0096]

[0097] Table 2(b):

[0098]

[0099]

[0100] To gain activity Vertical gradient and horizontal gradient A i The sum of is mapped to the range from 0 to n, where for n is equal to 4, for and n is equal to 15. In ALF_APS, up to 4 luma filter sets are indicated, each set may have up to 25 filters.

[0101] II. ALF with Virtual Boundaries and Multiple Sources

[0102] H. Virtual Boundary Filtering

[0103] To reduce the ALF's need for line buffers (temporary storage of recently reconstructed pixels), a modified block classification and filtering is employed for samples near horizontal CTU boundaries. Figure 4A-B shows modified block classification at a virtual boundary near a horizontal CTU boundary. A virtual boundary 410 is shown, which is defined by shifting the horizontal CTU boundary 405 by 4 sample rows for luma components (or 2 sample rows for chroma components). The virtual boundary 410 is between pixel row 'J' and pixel row 'K', while the CTU boundary is between pixel row 'N' and pixel row 'O'.

[0104] Figure 4A It is shown that, for a one-dimensional Laplacian gradient calculation, when the 4x4 block used to calculate the C0 classifier is above the virtual boundary 410, only the samples located above the virtual boundary 410 are used for the C1 and C2 classifications. Figure 4B 4 shows that, for a one-dimensional Laplace gradient calculation, when the 4x4 block used to calculate the C0 classifier is below the virtual boundary 410, only the samples below the virtual boundary 410 are used for the C1 and C2 classifications. By considering the reduced number of samples used in the one-dimensional Laplace gradient calculation, the quantization of the activity value A is scaled accordingly. In some embodiments, Figure 4A -B also shows that when samples outside the virtual boundary 410 are needed for a one-dimensional Laplace gradient calculation (eg, for the C1 and C2 classifiers), padding samples are used (by duplicating samples immediately inside the virtual boundary 410).

[0105] For the filtering process, symmetric padding operations at the virtual boundaries are used for the luma and chroma components. Figure 5A -B schematically illustrates a padding operation at a virtual boundary for generating filter taps for ALF filtering. The figure shows a diamond filter, where each position in the diamond corresponds to a filter tap with a coefficient. Figure 5A It is shown that when the sample being filtered (the current sample) is below the virtual boundary, the adjacent samples above the virtual boundary (required for filtering) are not available and are therefore filled in. The corresponding samples at symmetrical positions (bottom side of the diamond) are also filled in, even though the actual corresponding samples at those symmetrical positions may be available. Figure 5B It shows that when the sample being filtered (the current sample) is above a virtual boundary, the adjacent samples below the virtual boundary (needed for filtering) are not available and are therefore filled in. The corresponding samples at the symmetrical position (top side of the diamond) are also filled in.

[0106] Although not shown, in some embodiments, sample padding due to virtual boundaries is applied asymmetrically. For example, if samples at the top of a diamond filter are unavailable due to a virtual boundary and are replaced by padding samples, actual samples at corresponding symmetrical locations at the bottom of the diamond filter are used for filtering without padding.

[0107] In contrast to the symmetrical padding method used at the horizontal boundaries of the ALF (CTU boundaries or virtual boundaries), when cross-boundary filtering is disabled, a simple padding process (e.g., directly copying samples immediately inside the virtual boundary) can be applied to slice boundaries, tile boundaries, and sub-image boundaries. The simple padding process is also applied to picture boundaries. The padding samples are used for classification and filtering processes. To compensate for padding when filtering samples just above or below the virtual boundary, the filter strength can be reduced for these cases of chroma and luma (e.g., the filter strength is divided by 8 or right-shifted by 3).

[0108] J. ALF with multiple sources

[0109] In some embodiments, for the luma component, the samples before DBF are used as filter taps for ALF. Specifically, the filtered samples can be derived as:

[0110]

[0111] Among them, h i,j is the clipping difference between the adjacent sample before DBF and the current sample R(x,y), f i,j is the cropping difference between the adjacent sample and the current sample R(x,y), g i YesR i-20 The cropped difference between (x,y) and the current sample R(x,y). Fig. 6A -B shows the filter shape applied to the samples before DBF. Fig. 6A A 3x3 diamond filter with N=24 is shown. Figure 6B A 5x5 diamond filter with N=28 is shown. In some embodiments, in the APS, a flag is signaled to indicate whether the samples before the DBF are used for ALF. In some embodiments, this flag is always set to true by the encoder.

[0112] K. ALF with virtual boundaries and multiple sources

[0113] As previously mentioned, in some embodiments, the difference between the sample to be processed (current sample) and the adjacent sample is used as the filter tap of the ALF. In some embodiments, the adjacent sample is DBF processed (h i,j ) before. In some embodiments, a virtual boundary as described in Section H above is applied so that when a neighboring sample is not available due to a virtual boundary, the difference between the neighboring sample and the sample to be processed is set to zero. That is, as shown in FIG5 , the filter footprint is modified, but the padding process used in the upper and lower sides is replaced by setting the difference between the neighboring sample and the sample to be processed to zero. For some embodiments, this method can be used in luma ALF, chroma ALF, and / or CC-ALF.

[0114] Virtual boundary processing can be symmetric or asymmetric. In some embodiments, the process of setting the difference between adjacent samples and samples to be processed to zero is applied to corresponding symmetric positions. In some embodiments, the process of setting the difference between adjacent samples and samples to be processed to zero is only applied to samples that become unavailable through the virtual boundary. Available samples at corresponding symmetric positions will still be used for filtering.

[0115] Fig. 7A -C schematically illustrates symmetric and asymmetric processing across virtual boundaries for ALF filtering when the difference between the current sample and the adjacent sample is used to generate the filter tap input. The figure shows a diamond filter 700, where each position in the diamond corresponds to a filter tap. One of the filter taps n0 has a filter tap based on the sample to be processed (current sample) R and the adjacent sample R 0+ and R 0- The value is determined by the difference between

[0116] Fig. 7A shows a scenario where none of the samples required for the filter tap input are outside the virtual boundaries. In this case, the adjacent samples R 0+ and R 0- is available, so the difference between the adjacent sample and the sample to be processed is not set to zero, that is, the value of the filter tap n0 is calculated as n0 = (R 0+ -R)+(R 0- -R).

[0117] Figure 7B 710. 0+ Not available, and R and R 0+ The difference between is set to zero. However, the adjacent sample R at the bottom 0- is available, and R and R 0- The difference between is used as is, rather than being set to zero. The value of the filter tap n0 is calculated as n0 = 0 + (R 0- -R).

[0118] Figure 7C The symmetric filling process is shown when some samples near the top exceed the virtual boundary 710. 0+ Not available, and R and R 0+ The difference between them is set to zero. Although the adjacent sample R at the bottom 0- is available, but to maintain symmetry, R and R 0- The difference is set to zero. The value of filter tap n0 is calculated as n0=0.

[0119] In some embodiments, except for samples before ALF (e.g., DBF h i,j In addition to the samples before the DBF, the luma ALF also includes the plural sources in the filter footprint. The plural sources can be samples before DBF, samples before SAO, samples after applying the ALF fixed filter, reconstructed residuals after inverse transform, and / or samples before the reconstruction stage (using inter / intra predictors). In some embodiments, to further reduce buffer usage, the ALF virtual boundary processing is also applied to the plural sources. For example, as shown above Figure 5A As described in -B, when using samples before DBF in ALF, and if the required samples before DBF are not available (e.g., the samples before DBF are on the other side of the virtual boundary), a padding process is used to avoid accessing these samples. The padding process can be asymmetric or symmetric. In some embodiments, the padding process is replaced by setting the difference between the required (adjacent) samples before DBF and the samples to be processed to zero, as described above. Fig. 7A -C described.

[0120] In some embodiments, if one of the required samples before DBF is not available, (all) filter taps for the samples before DBF are removed. In some embodiments, if one of the required samples before DBF is not available, the filter taps for the samples before DBF are reduced to a single tap, which corresponds to the position of the sample to be processed (e.g., the center position of the diamond filter, or if the required sample is the current sample). In some embodiments, the virtual boundary processing for samples before ALF and the virtual boundary processing for multiple sources are the same. That is, the same virtual boundary processing is applied to all input sources of the luma ALF.

[0121] In some embodiments, multiple sources are also used in the chroma ALF and / or CC-ALF to further improve the codec performance. In some embodiments, the chroma samples before DBF and the chroma samples before SAO are added to the filter footprint of the chroma ALF. In some embodiments, the luma samples before DBF and the luma samples before SAO are included in the filter footprint of the CC-ALF.

[0122] In some embodiments, the plurality of sources may also be from different components (Y / Cr / Cb). For example, in some embodiments, for luma ALF, chroma samples before DBF may be included in the luma ALF filter footprint. In some embodiments, luma samples before DBF, chroma samples before DBF, luma samples before SAO, and chroma samples before SAO are included in the luma ALF filter footprint. For another example, in some embodiments, luma samples before DBF and luma samples before SAO are included in the filter footprint of chroma ALF.

[0123] In some embodiments, the filter footprint of the chroma ALF may include two chroma components. For example, in some embodiments, both Cr and Cb component samples are included as filter taps for filtering luma or chroma samples. For another example, when the ALF is applied to the Cb samples, the Cr samples before the ALF are used in the chroma ALF.

[0124] In some embodiments, the plurality of sources of ALF can also come from the intermediate ALF filtering results. For example, the luma samples after applying different fixed filters can be added to the filter footprint of the luma ALF. For another example, the luma samples after applying different fixed filters can be added to the filter footprint of the CC-ALF.

[0125] In the above method for complex sources for ALF filtering, the filter taps for the complex sources can be higher order parameters. For example, consider the sample to be processed R and the target sample N, instead of using (N – R), the squared difference (N 2 –R 2 ) is used as an additional tap. For another example, the input may be sign(N–R)*((N–R)*(N–R)), where sign(x) is used to return "+1" when x is non-negative and "-1" when x is negative. In some embodiments, when a plurality of sources are used in the ALF, nonlinear operations (e.g., clipping operations) may also be applied.

[0126] The above-mentioned proposed method can be implemented in an encoder and / or a decoder. For example, the proposed method can be implemented in an in-loop filter module of an encoder and / or an in-loop filter module of a decoder.

[0127] III. Example Video Encoder

[0128] Figure 8An exemplary video encoder 800 implementing an in-loop filter is shown. As shown, the video encoder 800 receives an input video signal from a video source 805 and encodes the signal into a bitstream 895. The video encoder 800 has several components or modules for encoding the signal from the video source 805, including at least some components selected from a transform module 810, a quantization module 811, an inverse quantization module 814, an inverse transform module 815, an intra-image estimation module 820, an intra-image prediction module 825, a motion compensation module 830, a motion estimation module 835, an in-loop filter 845, a reconstructed image buffer 850, a motion vector (MV) buffer 865, a motion vector prediction module 875, and an entropy encoder 890. The motion compensation module 830 and the motion estimation module 835 are part of the inter-image prediction module 840.

[0129] In some embodiments, modules 810-890 are modules of software instructions being executed by one or more processing units (e.g., processors) of a computing device or electronic device. In some embodiments, modules 810-890 are modules of hardware circuits implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 810-890 are shown as separate modules, some of these modules can be combined into a single module.

[0130] The video source 805 provides a raw video signal, which represents pixel data for each video frame without compression. The subtractor 808 calculates the difference between the raw video pixel data of the video source 805 and the predicted pixel data 813 from the motion compensation module 830 or the intra-frame image prediction module 825 as a prediction residual 809. The transform module 810 transforms the difference (or residual pixel data or residual signal 808) into transform coefficients (e.g., by performing a discrete cosine transform or DCT). The quantization module 811 quantizes the transform coefficients into quantized data (or quantized coefficients) 812, which are encoded into the bitstream 895 by the entropy encoder 890.

[0131] The inverse quantization module 814 dequantizes the quantized data (or quantized coefficients) 812 to obtain transform coefficients, and the inverse transform module 815 inversely transforms the transform coefficients to generate a reconstructed residual 819. The reconstructed residual 819 is added to the predicted pixel data 813 to generate reconstructed pixel data 817. In some embodiments, the reconstructed pixel data 817 is temporarily stored in a line buffer (not shown) for intra-frame image prediction and spatial MV prediction. The reconstructed pixels are filtered by the in-loop filter 845 and stored in a reconstructed image buffer 850. In some embodiments, the reconstructed image buffer 850 is a storage external to the video codec 800. In some embodiments, the reconstructed image buffer 850 is a storage internal to the video encoder 800.

[0132] The image intra-picture estimation module 820 performs intra-picture prediction based on the reconstructed pixel data 817 to generate intra-picture prediction data. The intra-picture prediction data is provided to the entropy encoder 890 to be encoded into a bitstream 895. The intra-picture prediction data is also used by the intra-picture prediction module 825 to generate predicted pixel data 813.

[0133] The motion estimation module 835 performs inter-picture prediction by generating motion vectors to reference pixel data of previously decoded frames stored in the reconstructed picture buffer 850. These motion vectors are provided to the motion compensation module 830 to generate predicted pixel data.

[0134] Instead of encoding the complete actual MV in the bitstream, the video codec 800 uses MV prediction to generate a predicted MV, and the difference between the MV used for motion compensation and the predicted MV is encoded as residual motion data and stored in the bitstream 895.

[0135] The motion vector prediction module 875 generates a predicted motion vector, i.e., a motion compensated motion vector used to perform motion compensation, based on the reference motion vector generated for encoding the previous video information frame. The motion vector prediction module 875 retrieves the reference motion vector from the previous video information frame from the motion vector buffer 865. The video encoder 800 stores these motion vectors generated for the current video information frame in the motion vector buffer 865 as reference motion vectors for generating the predicted motion vector.

[0136] The motion vector prediction module 875 uses the reference motion vector to create a predicted motion vector. The predicted motion vector can be calculated by spatial motion vector prediction or temporal motion vector prediction. The difference (residual motion data) between the predicted motion vector and the motion compensation motion vector (MC MV) of the current information frame is encoded into a bit stream 895 by the entropy encoder 890.

[0137] Using entropy coding techniques, such as context-adaptive binary arithmetic coding and decoding (CABAC) or Huffman coding, the entropy encoder 890 encodes various parameters and data into a bitstream 895. The entropy encoder 890 encodes various header elements, flags, and quantized transform coefficients 812 and residual motion data as syntax elements into the bitstream 895. In turn, the bitstream 895 is stored in a storage device or transmitted to a decoder via a communication medium such as a network.

[0138] The in-loop filter 845 performs filtering or smoothing operations on the reconstructed pixel data 817 to reduce coding artifacts, especially artifacts located at the boundaries of pixel blocks. In some embodiments, the filtering or smoothing operations performed by the in-loop filter 845 include DBF, SAO and / or ALF.

[0139] Fig. 9 A portion of a video encoder 800 that implements an ALF with virtual boundaries based on samples from a plurality of sources is shown. Specifically, the figure illustrates components of an in-loop filter 845 of the video encoder 800. As shown, the in-loop filter 845 receives reconstructed pixel data 817 of a current block (e.g., a current CTB) and generates a filtered output to be stored in a reconstructed image buffer 850. The input pixel data is processed in the in-loop filter 845 by a DBF module 902 and a SAO module 904. The processed samples generated by the DBF and SAO are provided to an ALF module 906. Figure 2 An example of an in-loop filter 200 with DBF, SAO, and ALF is described.

[0140] The ALF module 906 generates a correction value to be added to the current sample, which is the output of the SAO module 904. The correction value is generated by applying a filter 920 to samples adjacent to the current sample. The entropy encoder 890 can indicate the filter coefficients of the filter 920 in the bitstream. The input of the filter tap of the filter 920 is provided by the filter tap generator 910.

[0141] The filter tap generator 910 may provide the neighboring samples (i.e., filter footprint) required by the filter 920 from a plurality of different sources. The plurality of sample sources may include the output of the SAO module 904, the output of the DBF module 902, and the reconstructed pixel data 817 as the input sample data before the DBF. The plurality of sample sources selected by the filter tap generator 910 may also include the residual samples (reconstructed residual 819) of the current block and the prediction samples (predicted pixel data 813) of the inter-picture or intra-picture prediction of the current block. In some embodiments, the plurality of sources of the filter tap input may also include samples of neighboring blocks of the current block (provided by the reconstructed image buffer 850).

[0142] The filter tap generator 910 may also apply a virtual boundary so that samples beyond the virtual boundary will not be used as data for the filter taps of the filter 920. In some embodiments, the virtual boundary is a horizontal boundary several rows above or below the horizontal boundary of the CTU. In some embodiments, the filter tap generator 910 includes a line buffer 915 (temporary local storage) for storing samples required by the filter 920, and the use of the virtual boundary limits the size of the line buffer. The virtual boundary may be set by the entropy encoder 890. In some embodiments, the filter tap generator 910 performs padding to replace samples beyond the virtual boundary, and samples at symmetrical positions of the replaced samples may also be replaced by padding. The padding process for required samples beyond the virtual boundary may refer to the above. Figure 5A -B.

[0143] In some embodiments, the difference between the current sample and the adjacent sample is used to generate the filter tap input to filter 920. If the adjacent sample is beyond the virtual boundary, filter tap generator 910 may replace the difference with a zero value. Fig. 7A -C describes the use of the difference between the current sample and the adjacent sample to generate the filter tap input when the adjacent sample is beyond the virtual boundary.

[0144] In some embodiments, if any required samples for the filter taps from a particular source (e.g., before the DBF) are not available (e.g., beyond a virtual boundary), the filter tap generator 910 will discard all filter taps that require samples from the same particular source except for one filter tap corresponding to the center position of the (diamond) filter, i.e., the current sample.

[0145] Therefore, the input samples of the ALF module 906 are combined with their corresponding correction values ​​to generate the output of the ALF module 906, which is also the output of the in-loop filter 845. The output of the in-loop filter 845 is stored in the reconstructed image buffer 850 for encoding subsequent blocks.

[0146] Fig.10 Schematically illustrates a process 1000 for performing ALF filtering based on a virtual boundary using samples from a plurality of sources. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing encoder 800 perform process 1000 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing encoder 800 performs process 1000.

[0147] The encoder receives (block 1010) data to be encoded as a current pixel block in a current image of a video. The encoder receives (block 1020) current samples of the current block. The current samples may be samples processed by other in-loop filters such as SAO and DBF.

[0148] The encoder applies a filter to the current sample (block 1030) to generate a correction value by using a plurality of adjacent samples from two or more different sources as inputs to the filter and by excluding samples beyond a virtual boundary from being input to the filter. In some embodiments, the virtual boundary is a horizontal boundary of several rows above or below a horizontal boundary of the CTU. The plurality of adjacent samples from two or more different sources may include a first sample filtered by DBF and a second sample not filtered by DBF. The plurality of adjacent samples from two or more different sources may include at least two of (i) samples before applying SAO, (ii) filtered samples generated by a fixed filter, (iii) reconstructed residual samples after inverse transformation, (iv) predicted samples generated by inter-picture prediction or intra-picture prediction, and (v) samples processed by DBF and SAO.

[0149] In some embodiments, the excluded first neighboring samples may be replaced by padding samples as input to the filter, and the second neighboring samples located at filter positions symmetrical to the first neighboring samples are also replaced by padding samples as input to the filter.

[0150] In some embodiments, when the first neighboring sample exceeds the virtual boundary, the first difference between the first neighboring sample and the current sample is set to zero. The first neighboring sample may be a sample that is not filtered by DBF (or before DBF). In some embodiments, for a second neighboring sample at a filter position symmetrical to the first neighboring sample, when the first neighboring sample exceeds the virtual boundary, the second difference between the second neighboring sample and the current sample is also set to zero (even if the second neighboring sample does not exceed the virtual boundary).

[0151] In some embodiments, when the first neighboring sample exceeds the virtual boundary and the first neighboring sample comes from the first source, the second neighboring sample from the first source within the virtual boundary is also excluded as the input of the filter, or none of the neighboring samples from the first source is used as the input of the filter. In some embodiments, if the second neighboring sample is the current sample (or at the center of the filter), the second neighboring sample is used as the input of the filter.

[0152] The encoder adds the correction value to the current sample (block 1040) as a filtered sample of the current block. The filtered sample can be used as a reference for encoding a plurality of subsequent blocks of the current image.

[0153] IV. Example Video Decoder

[0154] In some embodiments, the encoder may indicate (or generate) one or more syntax elements in the bitstream so that the decoder may parse the one or more syntax elements from the bitstream.

[0155] Fig.11 An exemplary video encoder 1100 with an in-loop filter is shown. As shown, the video decoder 1100 is an image decoding or video decoding circuit that receives a bitstream 1195 and decodes the content of the bitstream into pixel data of a video information frame for display. The video decoder 1100 has several components or modules for decoding the bitstream 1195, including some components selected from an inverse quantization module 1111, an inverse transform module 1110, an intra-picture prediction module 1125, a motion compensation module 1130, an in-loop filter 1145, a decoded image buffer 1150, a motion vector buffer 1165, a motion vector prediction module 1175, and a parser 1190. The motion compensation module 1130 is a part of the inter-picture prediction module 1140.

[0156] In some embodiments, modules 1110-1190 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 1110-1190 are hardware circuit modules implemented by one or more ICs of an electronic device. Although modules 1110-1190 are illustrated as separate modules, some of these modules may be combined into a single module.

[0157] The parser 1190 (or entropy decoder) receives the bitstream 1195 and performs initial parsing according to the syntax defined by the video codec or image codec standard. The parsed syntax elements include various header elements, flags, and quantized data (or quantized coefficients) 1112. The parser 1190 parses out the various syntax elements by using entropy coding techniques such as context-adaptive binary arithmetic coding (CABAC) or Huffman coding.

[0158] The inverse quantization module 1111 dequantizes the quantized data (or quantized coefficients) 1112 to obtain transform coefficients, and the inverse transform module 1110 inversely transforms the transform coefficients 1116 to generate reconstructed residuals 1119. The reconstructed residuals 1119 are added to the predicted pixel data 1113 from the intra-picture prediction module 1125 or the motion compensation module 1130 to generate decoded pixel data 1117. The decoded pixel data is filtered by the in-loop filter 1145 and stored in the decoded image buffer 1150. In some embodiments, the decoded image buffer 1150 is a storage external to the video decoder 1100. In some embodiments, the decoded image buffer 1150 is a storage internal to the video decoder 1100.

[0159] The intra-picture prediction module 1125 receives the intra-picture prediction data from the bitstream 1195 and generates predicted pixel data 1113 based on the intra-picture prediction data from the decoded pixel data 1117 stored in the decoded image buffer 1150. In some embodiments, the decoded pixel data 1117 is also stored in a line buffer (not shown) for intra-picture prediction and spatial MV prediction.

[0160] In some embodiments, the contents of the decoded image buffer 1150 are used for display. The display device 1155 retrieves the contents of the decoded image buffer 1150 for direct display, or retrieves the contents of the decoded image buffer into a display buffer. In some embodiments, the display device receives pixel values ​​from the decoded image buffer 1150 via pixel transfer.

[0161] Based on the motion compensated MV (MC MV), the motion compensation module 1130 generates predicted pixel data 1113 from the decoded pixel data 1117 stored in the decoded picture buffer 1150. These motion compensated MVs are decoded by adding the residual motion data received from the bitstream 1195 to the predicted MV received from the motion vector prediction module 1175.

[0162] The motion vector prediction module 1175 generates a predicted MV, for example, a motion compensated MV for performing motion compensation, based on a reference MV generated for decoding a previous video information frame. The motion vector prediction module 1175 retrieves the reference motion vector of the previous video information frame from the motion vector buffer 1165. The video decoder 1100 also stores the motion compensated motion vector generated for decoding the current video information frame in the motion vector buffer 1165 as a reference motion vector for generating a predicted motion vector.

[0163] The in-loop filter 1145 performs filtering or smoothing operations on the decoded pixel data to reduce encoding and decoding artifacts, especially artifacts located at the boundaries of pixel blocks. In some embodiments, the filtering or smoothing operations performed by the in-loop filter 1145 include a deblocking filter (DBF), a sample adaptive offset (SAO) and / or an adaptive loop filter (ALF).

[0164] Fig.12 A portion of a video decoder 1100 that implements an ALF with virtual boundaries based on samples from a plurality of sources is shown. Specifically, the figure illustrates components of an in-loop filter 1145 of the video decoder 1100. As shown, the in-loop filter 1145 receives reconstructed pixel data 1117 of a current block (e.g., a current CTB) and generates a filtered output to be stored in a decoded image buffer 1150. The input pixel data is processed in the in-loop filter 1145 by a DBF module 1202 and a SAO module 1204. The processed samples generated by the DBF and SAO are provided to an ALF module 1206. Figure 2 An example of an in-loop filter 200 with DBF, SAO, and ALF is described.

[0165] The ALF module 1206 generates a correction value to be added to the current sample, which is the output of the SAO module 1204. The correction value is generated by applying a filter 1220 to samples adjacent to the current sample. The entropy decoder 1190 can parse the bitstream to receive the filter coefficients of the filter 1220. The input of the filter taps of the filter 1220 is provided by the filter tap generator 1210.

[0166] The filter tap generator 1210 may provide the neighboring samples required by the filter 1220 from a plurality of different sources. The plurality of sample sources may include the output of the SAO module 1204, the output of the DBF module 1202, and the reconstructed pixel data 1117 as the input sample data before the DBF. The plurality of sample sources selected by the filter tap generator 1210 may also include the residual samples (reconstructed residual 1119) of the current block and the prediction samples (predicted pixel data 1113) of the inter-picture or intra-picture prediction of the current block. In some embodiments, the plurality of sources of the filter tap input may also include samples of neighboring blocks of the current block (provided by the decoded image buffer 1150).

[0167] The filter tap generator 1210 may also apply a virtual boundary so that samples beyond the virtual boundary will not be used as material for the filter taps of the filter 1220. In some embodiments, the virtual boundary is a horizontal boundary several rows above or below the horizontal boundary of the CTU. In some embodiments, the filter tap generator 1210 includes a line buffer 1215 (temporary local storage) for storing samples required by the filter 1220, and the use of the virtual boundary limits the size of the line buffer 1215. The virtual boundary may be set by the entropy decoder 1190. In some embodiments, the filter tap generator 1210 performs padding to replace samples beyond the virtual boundary, and samples at symmetrical positions of the replaced samples may also be replaced by padding. The padding process for required samples beyond the virtual boundary may refer to the above. Figure 5A -B.

[0168] In some embodiments, the difference between the current sample and the adjacent sample is used to generate the filter tap input to filter 1220. If the adjacent sample is beyond the virtual boundary, filter tap generator 1210 may replace the difference with a zero value. Fig. 7A -C describes the use of the difference between the current sample and the adjacent sample to generate the filter tap input when the adjacent sample is beyond the virtual boundary.

[0169] In some embodiments, if any required samples for the filter taps from a particular source (e.g., before the DBF) are not available (e.g., beyond a virtual boundary), the filter tap generator 1210 will discard all filter taps that require samples from the same particular source, except for one filter tap corresponding to the center position of the (diamond) filter, i.e., the current sample.

[0170] Therefore, the input samples of the ALF module 1206 are combined with their corresponding correction values ​​to generate the output of the ALF module 1206, which is also the output of the in-loop filter 1145. The output of the in-loop filter 1145 is stored in the decoded picture buffer 1150 for decoding subsequent blocks.

[0171] Fig.13 Schematically illustrates a process 1300 for performing ALF filtering based on a virtual boundary using samples from a plurality of sources. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the decoder 1100 perform the process 1300 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing the decoder 1100 performs the process 1300.

[0172] The decoder receives (block 1310) data of a current pixel block in a current picture to be decoded as video. The decoder receives (block 1320) current samples of the current block. The current samples may be samples processed by other in-loop filters such as SAO and DBF.

[0173] The decoder applies a filter to the current sample (block 1330) to generate a correction value by using a plurality of neighboring samples from two or more different sources as inputs to the filter and by excluding samples beyond a virtual boundary from being input to the filter. In some embodiments, the virtual boundary is a horizontal boundary of a number of pixel rows above or below a horizontal boundary of the CTU. The plurality of neighboring samples from two or more different sources may include a first sample filtered by DBF and a second sample not filtered by DBF. The plurality of neighboring samples from two or more different sources may include at least two of (i) samples before applying SAO, (ii) filtered samples generated by a fixed filter, (iii) reconstructed residual samples after inverse transformation, (iv) predicted samples generated by inter-picture prediction or intra-picture prediction, and (v) samples processed by DBF and SAO.

[0174] In some embodiments, the excluded first neighboring samples may be replaced by padding samples as input to the filter, and the second neighboring samples located at filter positions symmetrical to the first neighboring samples are also replaced by padding samples as input to the filter.

[0175] In some embodiments, when the first neighboring sample exceeds the virtual boundary, the first difference between the first neighboring sample and the current sample is set to zero. The first neighboring sample may be a sample that is not filtered by DBF (or before DBF). In some embodiments, for a second neighboring sample at a filter position symmetrical to the first neighboring sample, when the first neighboring sample exceeds the virtual boundary, the second difference between the second neighboring sample and the current sample is also set to zero (even if the second neighboring sample does not exceed the virtual boundary).

[0176] In some embodiments, when the first neighboring sample exceeds the virtual boundary and the first neighboring sample comes from the first source, the second neighboring sample from the first source within the virtual boundary is also excluded as the input of the filter, or none of the neighboring samples from the first source is used as the input of the filter. In some embodiments, if the second neighboring sample is the current sample (or at the center of the filter), the second neighboring sample is used as the input of the filter.

[0177] The decoder adds the correction value to the current sample (block 1340) as a filtered sample of the current block, which can be used as a reference for a plurality of subsequent blocks of the reconstructed current image. The filtered sample can also be provided as part of the reconstructed current image for display.

[0178] IV. Example Electronic Systems

[0179] Many of the above features and applications can be implemented as software processes, which are specified as sets of instructions recorded on a computer readable storage medium (also referred to as a computer readable medium). When these instructions are executed by one or more computing units or processing units (e.g., one or more processors, processor cores, or other processing units), the instructions cause the processing units to perform the actions represented by the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, random access memory (RAM) chips, hard disks, erasable programmable read only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. Computer readable media do not include carriers and telecommunications signals connected wirelessly or wired.

[0180] In this specification, the term "software" is meant to include firmware in a read-only memory or an application stored in a magnetic storage device, which can be read into the memory for processing by the processor. At the same time, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program, while retaining different software inventions. In some embodiments, multiple software inventions can be implemented as independent programs. Finally, any combination of independent programs that implement the software inventions described herein together is within the scope of the present invention. In some embodiments, when installed to operate on one or more electronic systems, the software program defines one or more specific machine implementations that execute and implement the operations of the software program.

[0181] Fig.14An electronic system 1400 is conceptually shown as being implemented in some embodiments of the present invention. The electronic system 1400 may be a computer (e.g., a desktop computer, a personal computer, a tablet computer, etc.), a phone, a PDA, or other types of electronic devices. The electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 1400 includes a bus 1405, a processing unit 1410, a graphics-processing unit (GPU) 1415, a system memory 1420, a network 1425, a read-only memory (ROM) 1430, a permanent storage device 1435, an input device 1440, and an output device 1445.

[0182] The bus 1405 collectively represents all system buses, peripheral buses, and chipset buses that are communicatively coupled to a number of internal devices of the electronic system 1400. For example, the bus 1405 is communicatively coupled to the processing unit 1410 via the GPU 1415, the ROM 1430, the system memory 1420, and the permanent storage device 1435.

[0183] From these various memory units, the processing unit 1410 retrieves instructions to execute and data to process in order to perform the processes of the present invention. In various embodiments, the processing unit may be a single processor or a multi-core processor. Certain instructions are transmitted to and executed by the GPU 1415. The GPU 1415 may offload various calculations or supplement the image processing provided by the processing unit 1410.

[0184] ROM 1430 stores static data and instructions required by processing unit 1410 or other modules of the electronic system. On the other hand, permanent storage device 1435 is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when electronic system 1400 is turned off. Some embodiments of the present invention use a large-capacity storage device (such as a magnetic disk or optical disk and its corresponding magnetic disk drive) as permanent storage device 1435.

[0185] Other embodiments use removable storage devices (such as floppy disks, flash memory devices, etc., and their corresponding disk drives) as permanent storage devices. Like permanent storage device 1435, system memory 1420 is a read-write memory device. However, unlike storage device 1435, system memory 1420 is a volatile read-write memory, such as random access memory. System memory 1420 stores instructions and data that some processors need when running. In some embodiments, processing according to the present invention is stored in system memory 1420, permanent storage device 1435 and / or read-only memory 1430. For example, various memory units include instructions for processing multimedia clips according to some embodiments. For these various memory units, processing unit 1410 retrieves executed instructions and processed data in order to perform processing of certain embodiments.

[0186] The bus 1405 is also connected to input devices 1440 and output devices 1445. The input devices 1440 enable a user to communicate information and select commands to the electronic system. The input devices 1440 include alphanumeric keyboards and pointing devices (also known as "cursor control devices"), cameras (such as webcams), microphones or similar devices for receiving voice commands, etc. The output devices 1445 display images generated by the electronic system or data output in other ways. The output devices 1445 include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD), and speakers or similar audio output devices. Some embodiments include devices such as touch screens that function as both input devices and output devices.

[0187] Finally, if Fig.14 As shown, bus 1405 also couples electronic system 1400 to network 1425 via a network adapter (not shown). In this manner, the computer may be part of a computer network (e.g., a local area network (LAN), a wide area network (WAN), or an intranet) or a network of networks (e.g., the Internet). Any or all components of electronic system 1400 may be used in conjunction with the present invention.

[0188] Some embodiments include electronic components, such as microprocessors, storage devices, and memory, which store computer program instructions to machine-readable media or computer-readable media (optionally referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable Computer readable media may store computer programs executed by at least one processing unit and include sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as that produced by a compiler, and files containing high-level code executed by a computer, electronic component, or microprocessor using an interpreter.

[0189] While the above discussion refers primarily to microprocessors or multi-core processors that execute software, many of the above functions and applications are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuits themselves. In addition, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices.

[0190] As used in the specification and any claims of the present invention, the terms "computer", "server", "processor" and "memory" refer to electronic devices or other technical devices. These terms do not include people or groups. For the purpose of this specification, the term display or display device refers to displaying on an electronic device. As used in the specification and any claims of the present invention, the terms "computer-readable medium", "computer-readable medium" and "machine-readable medium" are entirely limited to tangible, physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals and any other transient signals.

[0191] While the present invention has been described in conjunction with many specific details, those skilled in the art will recognize that the present invention may be implemented in other specific forms without departing from the spirit of the present invention. Fig.10 and Fig.13 ) conceptually illustrates the processes. The specific operations of these processes may not be performed in the exact order shown and described. These specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. In addition, the processes are implemented by using several sub-processes, or as part of a larger macro process. Therefore, it will be understood by those skilled in the art that the present invention is not limited by the foregoing illustrative details, but is defined by the scope of the claims.

[0192] Additional Notes

[0193] The subject matter described herein sometimes represents different components, which are contained in or connected to other different components. It is understood that the described structure is only an example and can actually be implemented by many other structures to implement the same function. Conceptually, any arrangement of components that implement the same function is actually "associated" in order to implement the desired function. Therefore, regardless of the structure or intermediate components, any two components combined to implement a specific function are considered to be "interrelated" to implement the desired function. Similarly, any two associated components are considered to be "operably connected" or "operably coupled" to each other to implement a specific function. Any two components that can be associated with each other are also considered to be "operably coupled" to each other to implement a specific function. Specific examples of operable connections include, but are not limited to, physically pairable and / or physically interacting components, and / or wirelessly interactive and / or wirelessly interactive components, and / or logically interactive and / or logically interactive components.

[0194] Furthermore, with respect to the use of substantially any plural and / or singular terms, those of ordinary skill in the art can translate from the plural to the singular and / or from the singular to the plural as appropriate to the context and / or application. For clarity, the various singular / plural permutations are expressly provided herein.

[0195] In addition, it will be understood by those skilled in the art that, in general, the terms used in the present invention, especially in the claims, and the subject matter of the claims, are generally used as "open" terms, for example, "including" should be interpreted as "including but not limited to", "having" should be interpreted as "at least having", "including" should be interpreted as "including but not limited to", etc. It will be further understood by those skilled in the art that if a specific number of claim contents are intended to be introduced, it will be explicitly indicated in the claim, and will not be displayed in the absence of such contents. For example, to aid understanding, the claims may include the phrases "at least one" and "one or more" to introduce the contents of the claims. However, the use of these phrases should not be interpreted as implying the use of the indefinite article "a" to introduce the contents of the claims and limit any particular patent scope. Even when the same claim includes the introductory phrases "one or more" or "at least one", the indefinite article, such as "a", should be interpreted to mean at least one or more, and the same is true for the use of explicit descriptions used to introduce the claims. Furthermore, even when an introductory content explicitly refers to a specific number, one of ordinary skill in the art will recognize that such content should be interpreted as indicating the number of references, for example, "two references" without other modifications means at least two references, or two or more references. Furthermore, where expressions similar to "at least one of A, B, and C" are used, they are generally expressed so that one of ordinary skill in the art can understand the expression, for example, "a system includes at least one of A, B, and C" will include but is not limited to a system with A alone, a system with B alone, a system with C alone, a system with A and B, a system with A and C, a system with B and C, and / or a system with A, B, and C, etc. One of ordinary skill in the art will further understand that any separated words and / or phrases represented by two or more alternative terms, whether in the specification, claims, or drawings, should be understood to include the possibility of one of these terms, one of them, or both of these terms. For example, "A or B" should be understood as the possibility of "A", or "B", or "A and B".

[0196] As can be seen from the foregoing, various embodiments have been described herein for illustrative purposes, and various modifications may be made without departing from the scope and spirit of the invention. Therefore, the various embodiments disclosed herein are not intended to be limiting, and the scope of the claims represents the true scope and spirit.

Claims

1. A video encoding and decoding method, characterized in that: include: receiving data of a pixel block of a current block of a current image to be encoded or decoded as a video; receiving a current sample of the current block; Apply the filter to the current sample to generate a correction value where: A plurality of adjacent samples from two or more different sources are used as input to the filter. When the first neighboring sample is within the virtual boundary, the first neighboring sample is used as the input of the filter, When the first adjacent sample exceeds the virtual boundary, the first adjacent sample is excluded as an input of the filter; as well as The correction value is added to the current sample as a filtered sample of the current block.

2. The video encoding and decoding method as claimed in claim 1, characterized in that: in, The filter is an adaptive loop filter (ALF) of a video coding and decoding system, wherein the filtered samples of the current block are provided for coding and decoding a plurality of subsequent blocks of the current image.

3. The video encoding and decoding method as claimed in claim 1, characterized in that: in, The plurality of adjacent samples from the more than two different sources include first samples filtered by a deblocking filter (DBF) and second samples not filtered by the DBF.

4. The video encoding and decoding method as claimed in claim 1, wherein: in, The plurality of adjacent samples from the more than two different sources include at least two of the following: (i) Samples before applying Sample Adaptive Offset (SAO), (ii) filtered samples generated by a fixed filter, (iii) the reconstructed residual samples after inverse transformation, (iv) predicted samples generated by inter-picture prediction or intra-picture prediction, and (v) Sample processed by the deblocking filter and the sample adaptive offset.

5. The video encoding and decoding method as claimed in claim 1, wherein: in, The excluded first neighboring samples are replaced by padded samples as input to the filter.

6. The video encoding and decoding method as claimed in claim 5, characterized in that: in, The second adjacent sample located at the filter position symmetrical to the first adjacent sample is replaced by the padding sample as the input of the filter.

7. The video encoding and decoding method as claimed in claim 1, characterized in that: in, When the first neighboring sample exceeds the virtual boundary, a first difference between the first neighboring sample and the current sample is set to zero.

8. The video encoding and decoding method as claimed in claim 7, characterized in that: in, The second neighboring sample is located at a filter position symmetrical to the first neighboring sample, wherein when the first neighboring sample exceeds the virtual boundary, a second difference between the second neighboring sample and the current sample is set to zero.

9. The video encoding and decoding method as claimed in claim 1, wherein: in, When the first neighboring sample exceeds the virtual boundary and the first neighboring sample comes from a first source, the second neighboring sample from the first source within the virtual boundary is also excluded as an input of the filter.

10. The video encoding and decoding method as claimed in claim 9, characterized in that: in, If the second neighboring sample is at the position of the current sample, the second neighboring sample is used as input to the filter.

11. The video encoding and decoding method as claimed in claim 9, characterized in that: in, All samples from the first source are excluded from use as input to the filter.

12. An electronic device, characterized in that: include: A video codec circuit is provided for performing a plurality of operations, including: receiving data of a pixel block of a current block of a current image to be encoded or decoded as a video; receiving a current sample of the current block; Apply the filter to the current sample to generate a correction value where: A plurality of adjacent samples from two or more different sources are used as input to the filter, When the first neighboring sample is within the virtual boundary, the first neighboring sample is used as the input of the filter, When the first neighboring sample exceeds the virtual boundary, the first neighboring sample is excluded as an input of the filter; and The correction value is added to the current sample as the filtered sample of the current block.

13. A video decoding method, characterized in that: include: receiving data of a pixel block of a current block of a current image to be decoded as a video; receiving a current sample of the current block; Apply the filter to the current sample to generate a correction value where: A plurality of adjacent samples from two or more different sources are used as input to the filter, When the first neighboring sample is within the virtual boundary, the first neighboring sample is used as the input of the filter, When the first adjacent sample exceeds the virtual boundary, the first adjacent sample is excluded as an input of the filter; adding the correction value to the current sample as a filtered sample of the current block; and The filtered samples are provided as references for reconstructing a plurality of subsequent blocks of the current image.

14. A video encoding method, characterized in that: include: receiving data of a pixel block of a current block of a current image to be encoded as video; receiving a current sample of the current block; Apply the filter to the current sample to generate a correction value where: A plurality of adjacent samples from two or more different sources are used as input to the filter, When the first neighboring sample is within the virtual boundary, the first neighboring sample is used as the input of the filter, When the first adjacent sample exceeds the virtual boundary, the first adjacent sample is excluded as an input of the filter; adding the correction value to the current sample as a filtered sample of the current block; and The filtered samples are provided as references for encoding a plurality of subsequent blocks of the current image.

Images