Method and apparatus for video coding

Abstract

Methods and apparatuses of video coding are disclosed. According to one method, a sum of filter coefficients for each filter in one or more candidate cross-component loop filters is constrained to a fixed value. One or more syntax elements indicating a value related to a total number of candidate filters in a filter set are signaled or parsed in an adaptation parameter set (APS).

Description

[0001] Related citations

[0002] This invention claims priority to: U.S. Provisional Application No. 62 / 900,702, filed September 16, 2019; U.S. Provisional Application No. 62 / 940,321, filed November 26, 2019; U.S. Provisional Application No. 62 / 954,024, filed December 27, 2019; and U.S. Provisional Application No. 62 / 956,356, filed January 2, 2020. The entire contents of these provisional applications are incorporated herein by reference. Technical Field

[0003] This invention relates to adaptive loop filter (ALF) processing in video encoding and decoding systems. Specifically, this invention relates to constraining cross-component ALF processing in video encoders or decoders. Background Technology

[0004] Motion estimation is an efficient inter-frame coding / decoding technique to utilize temporal redundancy in video sequences. Motion-compensated inter-frame coding / decoding has been widely used in various international video coding / decoding standards. Motion estimation employed in various coding / decoding standards is typically a block-based technique, where motion information such as coding / decoding modes and motion vectors is determined for each macroblock or similar block configuration. Additionally, intra-frame coding / decoding is adaptively applied, where the image is processed without reference to any other image. Inter-frame prediction residuals or intra-frame prediction residuals are further processed, typically through transform, quantization, and entropy coding, to generate a compressed video bitstream. Coding artifacts are introduced during encoding, particularly during quantization. To mitigate these artifacts, additional processing is applied to the reconstructed video to enhance image quality in newer coding / decoding systems. This additional processing is often configured with in-loop operation so that the encoder and decoder can derive the same reference image for improved system performance.

[0005] Figure 1AAn example adaptive inter-frame / intra-frame video coding / decoding system incorporating in-loop processing including adaptive ALF is shown. For inter-frame prediction, motion estimation (ME) / motion compensation (MC) 112 is used to provide prediction data based on video data from one or more other images. Switch 114 selects either intra-frame prediction 110 or inter-frame prediction data, and the selected prediction data is provided to adder 116 to form prediction error, also known as residual. The prediction error is then processed by transformation (T) 118, followed by quantization (Q) 120. The transformed and quantized residual is then encoded by entropy encoder 122 to form a video bitstream corresponding to compressed video data. The bitstream associated with the transform coefficients is then packaged with auxiliary information such as motion, pattern, and other information associated with image regions. The auxiliary information can also be entropy encoded to reduce the required bandwidth. Thus, as Figure 1A As shown, data related to auxiliary information is provided to the entropy encoder 122. When the inter-frame prediction mode is used, the reference image must also be reconstructed at the encoder. Therefore, the transformed and quantized residuals are processed by inverse quantization (IQ) 124 and inverse transformation (IT) 126 to recover the residuals. Then, at reconstruction (REC) 128, the residuals are added back to the prediction data 136 to reconstruct the video data. The reconstructed video data can be stored in the reference picture buffer 134 and used for prediction of other frames.

[0006] like Figure 1A As shown, the input video data undergoes a series of processing steps in the coding system. Due to these processing steps, the reconstructed video data from REC 128 may suffer various forms of degradation. Therefore, various in-loop processing steps are applied to the reconstructed video data to improve video quality before it is stored in the reference image buffer 134. In the video coding system, various in-loop filters, such as the Deblocking Filter (DF) 130, the Sample Adaptive Offset (SAO) 131, and the Adaptive Loop Filter (ALF) 132, are used to enhance image quality.

[0007] For Figure 1A The corresponding decoder of the encoder in Figure 1B The video bitstream is shown in the diagram. It is decoded by entropy decoder 142 to recover the transformed and quantized residuals. On the decoder side, only motion compensation (MC) 113 is performed; ME / MC is not performed. The decoding process is similar to the reconstruction loop on the encoder side. The recovered transformed and quantized residuals, SAO / ALF information, and other system information are used to reconstruct the video data. The reconstructed video is further processed by DF 130, SAO 131, and ALF 132 to generate the final enhanced decoded video.

[0008] Cross-component ALF processing

[0009] In JVET-O0636 (Kiran Misra, et al., “Cross-Component Adaptive Loop Filter for Chroma”, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC1 / SC 29 / WG 11, 15th Meeting: Gothenburg, SE, 3-12 July 2019, Document: JVET-O0636), the Cross-Component Adaptive Loop Filter (CC-ALF) was proposed. CC-ALF uses luminance sample values ​​to refine each chroma component. Figure 2A The structure of the CC-ALF for other loop filters is shown according to JVET-O0636. Figure 2A In the process, ALF processing is performed after each SAO (210, 212, and 214). In conventional video coding systems, only ALF luma 220 and ALF chroma 230 are used. In JVET-0636, two additional ALF filters, CC ALF Cb 222 and CC ALF Cr 224, are used to derive the adjustment signals, which are then added to the ALF-processed Cb and Cr using adders 240 and 242, respectively.

[0010] According to JVET-O0636, CC-ALF operates by applying a linear diamond filter to the luminance channel of each chromaticity component, such as... Figure 2BAs shown. Filter coefficients are transmitted in the APS, scaled by a factor of 2^10, and rounded for fixed-point representation. The application of the filter is controlled on a variable block size and transmitted via the context coding flag received for each sample block. For each chroma component, the block size and CC-ALF enable flag are received at the fragment level. The syntax and semantics of CC-ALF are also disclosed in JVET-O0636.

[0011] As described above, CC-ALF operates by applying a linear diamond filter to the luminance channel of each chromaticity component. Figure 2C An example of the placement of CC-ALF 250 for luminance samples is shown. Symbol legend 260 is shown, where empty circles represent luminance samples and circles filled with shaded lines represent chrominance samples. For example, diamond filters 253 and 255 are shown for chrominance samples 252 and 254, respectively. The dynamic range of filter coefficients is limited to 6-bit signed values. Filter selection is performed at the CTU level, supporting a maximum of four filters. Symmetric line selection is used at virtual boundaries (for better alignment with ALF). The time-layer coefficient buffer is not used (for better consistency with the APS concept). The residual output is clipped to -2. BitDepthC -1 to 2 BitDepthC-1 -1 (including endpoint values).

[0012] The application of the filter is controlled on a variable block size and transmitted via the context coding flag received for each sample block. For each chroma component, the block size and CC-ALF enable flag are received at the fragment level. The syntax and semantics of CC-ALF are provided in the appendix. The following block sizes (in chroma samples) are supported in the literature: 16x16, 32x32, 64x64, and 128x128.

[0013] Virtual boundaries reduce the use of line buffers.

[0014] To eliminate the row buffer requirements of SAO and ALF, we introduce the concept of a virtual boundary (VB). For example... Figure 3 As shown, VB is the horizontal largest coding unit (LCU) boundary moved upwards by N lines. For each LCU, SAO and ALF can process the lines above the VB before the lower LCU arrives, but cannot process the lines below the VB until the lower LCU arrives, due to DF. Considering DF in HM-4.0, the proposed interval between the VB and the horizontal LCU boundary is set to three lines (i.e., Figure 3 N=3 in the value is used for brightness and one line (i.e. Figure 3 N=1 in the value is used for chroma. SAO and ALF are modified to ensure that each pending position on the VB side does not require any data access from the other side of the VB, unless the data can be used in a timely manner without the use of any additional line buffer.

[0015] At the 11th JVET meeting, the draft of the next-generation video codec standard (Versatile Video Coding, or VVC) adopted an ALF based on 4x4 block adaptability. For a 4x4 block, an 8x8 window is needed to compute the block's attributes, and the filter footprint is one 7x7 diamond filter (for luma) and one 5x5 diamond filter (for chroma). Therefore, the required line buffers without virtual boundaries are an additional 7 luma line buffers and 3 chroma line buffers on top of 4 luma DF line buffers and 2 chroma DF line buffers. The concept of virtual boundaries can still be applied in VVC to reduce the use of line buffers. In the above description, the Largest Coding Unit (LCU) was renamed the coding tree unit (CTU) in VVC. However, some modifications were proposed to suit the functionality of the current ALF design.

[0016] Nonlinear ALF operations

[0017] In VTM 3.0 (Versatile Video Coding (VVC) Test Model Version 3.0), the filtering process of the adaptive loop filter is executed as follows:

[0018] O(x, y) = ∑ (i，j) w(i,j).I(x+i,y+j), (1)

[0019] In the above formula, sample I(x+i, y+j) is the input sample, O(x, y) is the filtered output sample (i.e., the filtering result), and w(i, j) represents the filtering coefficients. Since ALF is applied to the reconstructed sample, sample I(x+i, y+j) corresponds to the reconstructed sample. The center reconstruction position at the center of the ALF processing corresponds to (i, j) = (0, 0), i.e., I(i, j). In fact, in VTM 3.0, it uses integer algorithms to implement fixed-point precision calculations.

[0020]

[0021] In the above formula, L represents the filter length, where w(i,j) are the fixed-point precision filter coefficients.

[0022] In the following expression, formula (1) can be reformulated without affecting coding efficiency:

[0023] O(x, y) = I(x, y) + ∑ (i，j)≠(0，0 )w(i,j)·(I(x+i,y+j)-I(x,y)). (3)

[0024] In the above formula, except for w(0,0), w(i,j) is the same as the filter coefficient in formula (1), w(0,0) in formula (3) is equal to 1 and is equal to (1-∑ (i，j)≠(0，0) w(i,j)).

[0025] Using the filter formula (3) above, when the neighboring sample values ​​(I(x+i, y+j)) are completely different from the current sample value (I(x, y)) being filtered, we can easily introduce nonlinearity to improve the efficiency of ALF by using a simple clipping function to reduce the influence of the neighboring sample values ​​(I(x+i, y+j)).

[0026] In JVET-M0385 (J. Taquet, et al., “Non-Linear Adaptive Loop Filter”, in Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC29 / WG 11, 13th Meeting: Marrakech, MA, 9-18 Jan. 2019, Document: JVET-M0385), a nonlinear ALF is disclosed. It introduces an adaptive pruning operation on the input sample values ​​of the adaptive loop filter in the test software VTM3.0. The purpose of this adaptive pruning operation is to introduce some nonlinearity to limit the difference between the input sample value to be filtered and the other adjacent input sample values ​​of the filter.

[0027] According to JVET-M0385, the ALF filter operation is modified as follows:

[0028] O′(x, y)=I(x, y)+∑ (i，j)≠(0，0) w(i,j).K(I(x+i,y+j)-I(x,y),k(i,j)), (4)

[0029] In the above equation, O′(x, y) corresponds to the modified ALF filter output, I(x, y) corresponds to the sample before ALF, K(d, b) = min(b, max(-b, d)) is the clipping function, and k(i, j) is the clipping parameter, which depends on the filter coefficient at (i, j). The encoder performs optimization to find the optimal k(i, j).

[0030] In the implementation according to JVET-M0385, the clipping parameter k(i,j) is specified for each ALF filter, where one clipping value is sent for each filter coefficient. This means that 12 clipping values ​​are sent in the bitstream for each luminance filter and 6 clipping values ​​are sent for each chrominance filter.

[0031] To limit transmission costs and encoder complexity, the cropped values ​​were restricted to a small subset of possible values ​​in JVET-M0385. Furthermore, only four possible values ​​were used for inter-frame segments, and three for intra-frame segments.

[0032] Because the variance of local differences in luminance is typically higher than that in chrominance, two separate sets are used for luminance and chrominance filters. Furthermore, each set contains the maximum sample value, so clipping can be disabled when not needed.

[0033] Table 1 provides a set of clipping values ​​proposed according to JVET-M0385.

[0034] Table 1: Allowable clipping values

[0035]

[0036] Using Golomb encoding corresponding to the index of the clipped value in the set, the clipped value is encoded in the fragment header.

[0037] Reduce ALF buffer requirements using virtual boundaries

[0038] In JVET-M301 ((A. Kotra, et al., “Non-CE: Loop filter line buffer reduction”, in Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 13th Meeting: Marrakech, MA, 9-18 Jan. 2019, Document: JVET-M0301)), a mechanism for reducing the line buffer requirements of an Adaptive Loop Filter (ALF) is disclosed. This document uses the concept of virtual boundaries (VB), which is a horizontal CTU boundary shifted upwards by “N” samples. Modified ALF block classification and modified ALF filtering are applied to samples near the virtual boundaries to reduce the number of line buffers required. The modified ALF block classification uses only samples above the VB to classify a given 4×4 block above the VB. Similarly, for classification of 4×4 blocks below VB, samples belonging to rows below VB are used. The modified ALF filtering can be achieved by using either a conditionally disabled original ALF filter or a truncated version of the original ALF filter, or a combination of both.

[0039] ALF filtering truncated on virtual boundaries

[0040] The truncated version of the filter is used to filter luminance samples belonging to rows close to the virtual boundary. These truncated versions of the filter can be used for N=4 and N=6 cases. Similarly, the truncated version of the filter is also used for chroma ALF filtering.

[0041] For a truncated ALF, the coefficients must be normalized; that is, the sum of the remaining coefficients must be the same as the sum of the original coefficients. Figures 4A-F illustrate an example of coefficient normalization used for truncated luminance ALF filter processing at the virtual boundary. Figure 4A The modified ALF coefficients are shown for the case where the top position (i.e., position c0) is outside the virtual boundary. The coefficient c2 is modified to c2′, where c2′ = c2 + c0. Figure 4B This shows the case where the bottom position (i.e., c0) is outside the virtual boundary. Figure 4C The modified ALF coefficients are shown for the first two rows (i.e., positions c0, c1, c2, and c3) outside the virtual boundary. Therefore, coefficients c5, c6, and c7 are modified to c5', c6', and c7', where c5' = c5 + c1, c6' = c6 + c2 + c0, and c7' = c7 + c3. Figure 4D The modified ALF coefficients are shown for the bottom two rows (i.e., positions c0, c1, c2, and c3) outside the virtual boundary. Figure 4E The modified ALF coefficients are shown for the case where the first three rows (i.e., positions c0 to c8) are outside the virtual boundary. Therefore, coefficients c10, c11, and c12 are modified to c10′, c11′, and c12′, where c10′ = c10 + c4 + c8, c11′ = c11 + c5 + c7 + c1 + c3, and c12′ = c12 + 2*c6 + 2*c2 + 2*c0. Figure 4F The modified ALF coefficients are shown for the case where the bottom three rows (i.e., positions c0 to c8) are outside the virtual boundary.

[0042] Figure 5A -D shows an example of truncated chroma ALF filter processing at virtual boundaries. Figure 5A The modified ALF coefficients are shown for the case where the top position (i.e., position c0) is outside the virtual boundary. The coefficient c2 is modified to c2′, where c2′ = c2 + c0. Figure 5B This shows the case where the bottom position (i.e., c0) is outside the virtual boundary. Figure 5C This shows the modified ALF coefficients for the first two rows (i.e., positions c0, c1, c2, and c3) outside the virtual boundary. Therefore, coefficients c5 and c6 are modified to c5′ and c6′, where c5′ = c5 + c1 + c3, and c6′ = c6 + 2*c2 + 2*c0. Figure 5D The modified ALF coefficients are shown for the bottom two rows (i.e., positions c0, c1, c2, and c3) outside the virtual boundary.

[0043] In this application, various other ALF boundary problems are addressed, and solutions for improving performance, unifying multiple systems, or simplifying ALF boundary handling are disclosed. Summary of the Invention

[0044] A method and apparatus for loop filter processing for video reconstruction are disclosed. According to one method, reconstructed samples are received. A target Cross-Component Adaptive Loop Filter (CCALF) is determined, wherein the target CCALF belongs to a filter set including one or more candidate filters, and the sum of the filter coefficients of each candidate filter is constrained to a fixed value. The target CCALF is applied to the associated reconstructed luminance samples to generate CCALF-filtered chrominance samples for the target reconstructed chrominance samples. By combining the CCALF-filtered chrominance samples with regular ALF-filtered chrominance samples, a final filtered chrominance sample is generated, wherein the final filtered chrominance sample is used as the video decoding output or for further video encoding or decoding processing.

[0045] In one embodiment, the fixed value is equal to 0. In another embodiment, when the number of unique filter coefficients for each candidate filter is equal to M, only (M-1) filter coefficients for each candidate filter are decoded on the decoder side, where M is an integer greater than 1. In one embodiment, in addition to (M-1) filter coefficients, one filter coefficient is either not decoded on the decoder side or encoded on the encoder side, and the unencoded filter coefficient corresponds to any selected coefficient among the unique filter coefficients of each candidate filter.

[0046] According to another method, one or more syntax elements are sent in the Adaptation Parameter Set (APS) of the video bitstream on the encoder side, or the one or more syntax elements are parsed in the APS of the video bitstream on the decoder side, wherein the one or more syntax elements indicate values ​​related to the total number of candidate filters in the filter set. A target CCALF is determined, wherein the target CCALF belongs to a filter set including one or more candidate filters, and the sum of the filter coefficients of each of the one or more candidate filters is constrained to a fixed value. The target CCALF is applied to the associated reconstructed luma samples to generate CCALF-filtered chroma samples for the target reconstructed chroma samples. By combining the CCALF-filtered chroma samples and the regular ALF-filtered chroma samples, a final filtered chroma sample is generated, wherein the final filtered chroma sample is used as the video decoding output or used for further video encoding or decoding processing.

[0047] In one embodiment, a syntax element is sent or parsed for each color component. In one embodiment, the one or more syntax elements include a value corresponding to the total number of candidate filters in the filter set minus 1. Attached Figure Description

[0048] Figure 1A An exemplary adaptive inter-frame / intra-frame video coding system combining DF, SAO, and ALF in-loop processing is shown.

[0049] Figure 1B An exemplary adaptive inter-frame / intra-frame video decoding system incorporating DF, SAO, and ALF in-loop processing is shown.

[0050] Figure 2A The structure of CC-ALF for other loop filters according to JVET-O0636 is shown, where ALF processing is performed after the corresponding SAO.

[0051] Figure 2B The diagram shows a diamond filter applied to the luminance channel for each chromaticity component according to JVET-O0636.

[0052] Figure 2C An example placement is shown relative to the brightness sample CC-ALF 250.

[0053] Figure 3 This shows an example of a VB formed by moving the horizontal LCU boundary up N rows.

[0054] Figure 4A -F illustrates an example of coefficient normalization for truncated brightness ALF filter processing at virtual boundaries, where Figure 4A The modified ALF coefficients are shown when the top position is outside the virtual boundary; Figure 4B This shows the case where the bottom position is outside the virtual boundary. Figure 4C The first two rows show the modified ALF coefficients for the case outside the virtual boundary. Figure 4D The bottom two rows show the modified ALF coefficients for the case outside the virtual boundary. Figure 4E The first three rows show the modified ALF coefficients for the case outside the virtual boundary. Figure 4F The bottom three rows show the modified ALF coefficients for the case outside the virtual boundary.

[0055] Figure 5A -D illustrates an example of truncated chroma ALF filter processing at virtual boundaries, where Figure 5A The modified ALF coefficients are shown when the top position is outside the virtual boundary. Figure 5B This shows the case where the bottom position is outside the virtual boundary. Figure 5C The first two rows show the modified ALF coefficients for the case outside the virtual boundary. Figure 5D The bottom two rows show the modified ALF coefficients for the case outside the virtual boundary.

[0056] Figure 6 An example of a 3x4 rhombus CCALF with coefficients c0 to c6 is shown.

[0057] Figure 7 A flowchart illustrating an exemplary ALF process for reconstructing video according to an embodiment of the present invention is shown, wherein the sum of the filter coefficients of each of one or more candidate filters is constrained to a fixed value.

[0058] Figure 8 A flowchart illustrating an exemplary ALF process for reconstructing video according to an embodiment of the present invention is shown, wherein one or more syntax elements for indicating values ​​related to the total number of candidate filters in the filter set are sent or parsed in the APS. Detailed Implementation

[0059] The following description represents the preferred mode of carrying out the invention. This description is made for the purpose of illustrating the general principles of the invention and should not be considered limiting. The scope of the invention is best determined by referring to the appended claims.

[0060] Method 1: Corresponding luminance sample values ​​in CC-ALF

[0061] In CC-ALF, an additional filter is added for the chroma component, and the filtered output is a weighted sum of the original chroma filter and the additional filter. If the video format is YUV444 or RGB444, the corresponding luminance sample of the chroma sample to be processed can be well defined by using the same spatial location across different color components. However, if the video format is YUV420 or YUV444, defining the corresponding luminance sample in CC-ALF becomes problematic. In one embodiment, the corresponding luminance sample is downsampled based on the YUV format or the chroma sampling position. For example, if the video format is YUV420, and the chroma sampling position is (0, 0.5) based on the luminance position, the corresponding luminance sample could be one of four luminance samples. In another example, the corresponding luminance sample could be the average of the remaining two luminance samples. In yet another example, the corresponding luminance sample could be the average of the left (right) top and bottom luminance samples. In yet another case, the corresponding luminance sample could be the average of four luminance samples. The filter footprint can be defined based on either the chroma position or the luminance position. If the filter footprint is defined based on the chroma sample position, then the input to the CC-ALF filtering process is the corresponding luminance sample mentioned above. If the filter footprint is defined based on the luminance sample position, then the center position of the filter shape in CC-ALF is defined by the corresponding luminance sample of the chroma sample to be processed. In one embodiment, the definition of the corresponding luminance sample depends on the video format or the sampling position of the chroma components.

[0062] In another embodiment, a nonlinear clipping operation can be applied to CC-ALF. In yet another embodiment, the difference between neighboring samples and the sample to be processed is used in the filtering process, rather than directly using neighboring samples. The sample to be processed can be a sample value in the chromaticity component or a sample value of the corresponding luminance sample in CC-ALF.

[0063] In the current design, the output of the chroma ALF is cropped to a valid range (e.g., [0, 1023] for 10-bit video compression, [0, 255] for 8-bit video compression). The output of the CCALF is added to this cropped value, and the cropping operation is performed again to ensure the final output is within the valid range. Therefore, there are two cropping operations: one applied to the chroma ALF output and another applied to the CCALF output. In one embodiment, when the CC-ALF is applied, the cropping operation used to limit the output value to a valid range in the chroma ALF processing can be discarded; therefore, the cropping operation is only applied after the CC-ALF output is added to the chroma ALF output.

[0064] In another embodiment, the accuracy of the filter coefficients in the additional filters in the CC-ALF is consistent with the original ALF design.

[0065] In one embodiment, the filter shape of the CC-ALF is aligned with the filter shape of the luminance ALF filter. In one embodiment, the CC-ALF filter may include 25 taps with 23 coefficients, where the top and bottom coefficients are equal, and the left and rightmost coefficients are equal. In one embodiment, the CC-ALF filter may include 25 taps with 22 coefficients, where the top, bottom, leftmost, and rightmost coefficients are all equal. In one embodiment, the symmetry of the CC-ALF filter is aligned with the symmetry of the ALF filter. In one embodiment, when the filter shapes of the CC-ALF and luminance ALF are the same, the boundary processing of the CC-ALF, including ALF virtual boundary processing, ALF boundary processing for slice boundaries / brick boundaries / tile boundaries / 360-face virtual boundaries / sub-image boundaries / image boundaries, is the same as the ALF boundary processing of the luminance ALF.

[0066] In one embodiment, the filter shape of the CC-ALF can be aligned with the filter shape of the chroma ALF filter. In one embodiment, the CC-ALF filter may include 13 taps with 11 coefficients, where the top and bottom coefficients are equal, and the left and right coefficients are equal. In one embodiment, the CC-ALF filter may include 13 taps with 10 coefficients, where the top, bottom, left, and right coefficients are all equal. In one embodiment, the CC-ALF filter is symmetrical, similar to the ALF filter, and therefore contains 13 taps and only 7 coefficients. In one embodiment, when the filter shapes of the CC-ALF and chroma ALF are the same, the boundary processing of the CC-ALF, including ALF virtual boundary processing, and the ALF boundary processing at fragment boundaries / brick boundaries / square boundaries / 360-face virtual boundaries / sub-image boundaries / image boundaries, is the same as the ALF boundary process for the chroma ALF.

[0067] In one embodiment, the CC-ALF filter is shared between the chroma U and chroma V color components. In one embodiment, only one `alf_crosscomponent_cbcr_filter_signal_flag` exists in the APS. In one example, if this flag exists, the CC-ALF coefficients exist in the APS and can be decoded.

[0068] In one embodiment, the CC-ALF coefficients of chrominance U are defined from the luminance component and can be reused for chrominance V. In another embodiment, the CC-ALF coefficients of chrominance V are defined from the luminance component and can be reused for chrominance U. In one embodiment, the CC-ALF coefficients of chrominance V can be obtained from the CC-ALF coefficients of chrominance U through additional encoding of symbol and / or bit-shift information. In another embodiment, the CC-ALF coefficients of chrominance U can be obtained from the CC-ALF coefficients of chrominance V through additional encoding of symbol and / or bit-shift information.

[0069] In one embodiment, CC-ALF is applied to only one of the chroma components. In one example, a flag may be explicitly sent at the CTB level, CU level, or block level to indicate whether one or both chroma components are using CC-ALF. In one embodiment, if not sent, the CC-ALF coefficient is inferred to be equal to zero.

[0070] In one embodiment, `slice_cross_component_alf_cb_reuse_temporal_layer_filter` is never sent in the slice header, meaning that when CCALF is enabled, the CC-ALF filter selection must always be sent for each slice. In another embodiment, only the CCALF filter coefficients stored in the APS can be referenced by different slices or images. That is, there is no data dependency of CCALF filter coefficients between different slices or different images.

[0071] In another embodiment, multiple CC-ALF filters can be used in a single segment / brick / square / image. When multiple CC-ALF filters are used, the selection of the multiple CC-ALF filters can be explicitly sent at the CTB level, CU level, or block level. In another embodiment, the selection of multiple CC-ALF filters is implicitly derived based on the selection of multiple chroma filters. When multiple CC-ALF filters are used, they can be emitted independently in a single APS or coupled with multiple chroma filters. For example, two sets of multiple filters, one used for CC-ALF and the other for chroma filters, are emitted separately in a single APS. Or, in another case, only one set of multiple filters is emitted in a single APS, and each candidate in that set includes at least one of a chroma filter and a CC-ALF filter.

[0072] In another embodiment, the chroma ALF filter and the CC-ALF filter cannot be used simultaneously. That is, for each chroma sample to be processed, at most one of the chroma ALF filter and the CC-ALF filter can be applied. In one embodiment, the filter shapes of the two filters can be further unified. In another embodiment, after an on / off flag is sent and that on / off flag indicates that one of the chroma ALF filter and the CC-ALF filter is applied, a flag is sent for each filter to indicate which filter type is a chroma ALF filter or a CC-ALF filter. In one embodiment, a flag indicating that the chroma ALF filter is enabled is sent first, and if the flag equals 1, the chroma ALF filter is applied. If the flag is false, another flag indicating whether the CC-ALF filter is enabled is sent. If the flag is true, the CC-ALF filter is used. Otherwise, no filter is applied. In another embodiment, a flag indicating that the CC-ALF filter is enabled is sent first, and if the flag equals 1, the CC-ALF filter is applied. If the flag is false, another flag indicating whether the chroma ALF filter is enabled is sent. If the flag is true, the chroma ALF filter is used. Otherwise, no filter is applied. In the above embodiments, the selection of the chroma ALF filter or CC-ALF filter can be at the fragment level, image level, sequence level, CTB level, or block level.

[0073] In another embodiment, the proposed method is applied to the luma ALF. That is, the ALF output of the luma component is also affected by the corresponding chroma sampling. In one example, two filters are added, with one filter's input being the Cb component and the other filter's input being the Cr component. The ALF output is a weighted sum of the outputs of these two additional filters and the original filter. In another case, only one filter is added, but the input of this filter is one of the combinations including Cb, Cr, and the average values ​​of Cb and Cr.

[0074] Method 2: The sum of the constrained CCALF-CCALF coefficients equals 0

[0075] According to this method, the sum of CCALF coefficients is constrained. Due to this constraint, data redundancy is introduced between CCALF coefficients, and specific information does not need to be transmitted. Therefore, this method can improve coding efficiency.

[0076] In CC-ALF, a linear filter filters the luminance sample values ​​and generates residual corrections for the chrominance channels from the co-located filter output.

[0077] In embodiments of the invention, the sum of the filter coefficients of all filters is limited by a certain number N. In one embodiment, to maintain the chromaticity average of the Cb and Cr components without introducing any severe color changes in chromaticity, it is recommended that the sum of the filter taps of each CCALF filter be equal to 0. Therefore, the CCALF filter has a response of 0 at DC and will eliminate all chromaticity averages.

[0078] In one embodiment, when all filter coefficients are unique, the final unique coefficient of the filter can be determined knowing the sum (N) of all unique coefficients on the decoder side and the number (M) of unique filter coefficients. Therefore, only M-1 unique filter coefficients must be decoded. It should be noted that when the filter coefficients are unique, their values ​​are different.

[0079] In one embodiment, the number of unique filter coefficients (M) in the CCALF filter is less than the total number of coefficients in the filter. Figure 6 An example of a 3×4 diamond-shaped CCALF is shown. It has seven (M=7) unique coefficients c0 to c6. The total number of filter coefficients is eight. In one embodiment, knowing the sum of all filter coefficients N, only six (M-1=6) coefficients c0 to c5 must be decoded, and coefficient c6 can be defined at the decoder as follows:

[0080] c6 = N - (2*c0 + c1 + c2 + c3 + c4 + c5).

[0081] In one embodiment, any of the unique coefficients can be skipped, not just the last unique coefficient. In one embodiment, coefficient c2 is skipped and defined on the decoder side. In another embodiment, coefficient c5 is skipped and defined on the decoder side.

[0082] In one embodiment, the requirement for bitstream consistency is that the last coefficient of the filter coefficients CcAlfApsCoeffCb[adaptive_parameter_set_id][k][7] of the k-th cross component Cb should be in the range [-32, 31].

[0083] In one embodiment, this requirement applies to any coefficient in the coefficients of the k-th cross-component Cb filter, CcAlfApsCoeffCb[adaptive_parameter_set_id][k][i], where i is 0 to the number of coefficients in the k-th cross-component Cb filter minus 1.

[0084] In one embodiment, bitstream consistency is required: the last coefficient of the k-th cross component Cr filter coefficient CcAlfApsCoeffCr[adaptive_parameter_set_id][k][7] should be in [-32, 31].

[0085] In one embodiment, the requirement is applied to any coefficient Cr filter coefficient CcAlfApsCoeffCr[adaptation_parameterset_id][k][i] of the k-th cross component, where i is the number of coefficients in the k-th cross component Cr filter minus 1.

[0086] In one embodiment, this requirement applies to all coefficients of the Cb / Cr filter in the k-th cross component, namely CcAlfApsCoeffCb / Cr[adaptation_parameter_set_id][k][i], where i is the number of coefficients from 0 to the number of coefficients in the Cb / Cr filter in the k-th cross component.

[0087] Method 3: Constrained CCALF - Disabling CCALF in lossless and near-lossless encoding and decoding scenarios

[0088] In HEVC, if the CU is encoded using lossless or near-lossless modes (including palette mode for 4:4:4 content), all loop filters (i.e., SAO, ALF, DF (deblocking filter)) are disabled.

[0089] In the current version of VVC, when the CU is encoded in lossless mode, all loop filters are disabled by the encoder settings. Additionally, if the CU is encoded using palette mode, SAO, ALF, and DF are disabled on both the encoder and decoder sides.

[0090] In embodiments of the invention, when the CU encodes using lossless or near-lossless modes (including palette modes for 4:4:4 content), all loop filters on the encoder and decoder sides must be disabled.

[0091] In one embodiment, for CUs encoded in lossless or near-lossless modes (including palette modes for 4:4:4 content), CCALF should be disabled on both the encoder and decoder.

[0092] In one embodiment, when dual trees are used to divide the coding tree nodes (treeType equals DUAL_TREE_LUMA or DUAL_TREE_CHROMA), all loop filters can be disabled for luma and chroma respectively (i.e., different solutions can be used for luma and chroma).

[0093] In one embodiment, when a dual tree is used to divide the coding tree nodes, treeType is equal to DUAL_TREE_CHROMA, and at least one CU (chroma CU or the luminance CU corresponding to the chroma CU) is encoded using lossless mode, then CCALF is automatically disabled for the chroma CU.

[0094] In one embodiment, when the dual tree is not used to divide the coding tree nodes (i.e., treeType equals SINGLE_TREE), and at least one of the CUs (chroma CU or the luminance CU corresponding to the chroma CU) is encoded in lossless mode, CCALF is automatically disabled for the chroma CU.

[0095] In one embodiment, when the dual tree is not used to divide the coding tree nodes, treeType equals DUAL_TREE_CHROMA, and the chroma CU is encoded using lossless mode, then CCALF is automatically disabled for that chroma CU.

[0096] In one embodiment, when the dual tree is not used to partition the encoding tree nodes (treeType equals SINGLE_TREE), and the chroma CU is encoded using lossless mode, CCALF is automatically disabled for that chroma CU.

[0097] In one embodiment, when a dual tree is used to divide the coding tree nodes (treeType equals DUAL_TREE_CHROMA), and both the chroma CU and the corresponding luma CU are encoded using lossless mode, CCALF is automatically disabled for the chroma CU.

[0098] In one embodiment, when the dual tree is not used to partition the encoding tree nodes (treeType equals SINGLE_TREE), and both the chroma CU and the corresponding luma CU are encoded using lossless mode, CCALF is automatically disabled for the chroma CU.

[0099] In one embodiment, if the residual samples of chromaticity components Cb and Cr are both encoded as a single transform block (e.g., tu_joint_cbcr_residual_flag[xC][yC] == 1) and at least one CU (Cb or Cr) is encoded using lossless mode, then CCALF is automatically disabled for both chromaticity CUs.

[0100] In one embodiment, if the residual samples of chrominance components Cb and Cr are both encoded as a single transform block (e.g., tu_joint_cbcr_residual_flag[xC][yC] == 1) and at least one CU(Cb) or Cr is encoded using lossless mode, then the corresponding chrominance CU, CCALF is automatically disabled.

[0101] Method 4: Number of filters in the CCALF file

[0102] To determine the number of CCALF filters, an additional syntax element needs to be sent. In the conventional approach, this additional syntax element is sent in the slice header (SH). According to another conventional method disclosed in JVET-P1008 (K. Misra, et al., “CE5-related: On the design of CC-ALF”, Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 16th Meeting: Geneva, CH, 1-11 October 2019, JVET-P1008-v2), it discloses a syntax design for the number of filters in the CCALF, which is based on VVC draft 6 (J. Chen, et al., “Cross-Component Adaptive Loop Filter for Chroma”, Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 15th Meeting: Gothenburg, SE, 3-12 July). 2019, Document: JVET-O2001-vE). According to JVET-P1008, the syntax elements alf_ctb_cross_component_cb_idc[] and alf_ctb_cross_component_cr_idc[][] are transmitted at the CTU level. The syntax element alf_ctb_cross_component_cb_idc[][] indicates whether the cross-component Cb filter is applied to the Cb color component samples. The syntax element alf_ctb_cross_component_cr_idc[][] indicates whether the cross-component Cr filter is applied to the Cr color component samples. As shown in the table below, both syntax elements are binarized using truncated Rice (TR) codes:

[0103] Table 2

[0104]

[0105]

[0106] In Table 2, cMax and cRiceParam are the two inputs provided to the binarization processing of alf_ctb_cross_component_cb_idc[][] and alf_ctb_cross_component_cr_idc[][] using truncated Rice. Details of the binarization processing are described in JVET-O2001-vE. The syntax elements slice_cross_component_cb_filters_signalled_minus1 and slice_cross_component_cr_filters_signalled_minus1 are sent in the slice header (SH).

[0107] In JVET-P1008, several syntax elements were introduced, briefly summarized below. `slice_cross_component_alf_cb_enabled_flag` specifies whether the cross-component Cb filter is applied to the Cb color component. If `slice_cross_component_alf_cb_enabled_flag` does not exist, it is inferred to be equal to 0.

[0108] When `slice_cross_component_alf_cb_enabled_flag` equals 1, the bitstream consistency requirement is that the ALF APS pointed to by `slice_cross_component_alf_cb_aps_id` must be the same for all segments of the current image. Increasing `slice_cross_component_cb_filters_signalled_minus1` by 1 specifies the number of cross-component Cb filters. The value of `slice_cross_component_cb_filters_signalled_minus1` should be in the range of 0 to 3.

[0109] When slice_cross_component_alf_cb_enabled_flag equals 1, the requirement for bitstream consistency is that slice_cross_component_cb_filters_signalled_minus1 should be less than or equal to the value of alf_cross_component_cb_filters_signalled_minus1 in the ALF APS pointed to by slice_cross_component_alf_cb_aps_id of the current slice.

[0110] The `slice_cross_component_alf_cr_enabled_flag` specifies whether the cross-component Cr filter is applied to the Cr color component. If `slice_cross_component_alf_cr_enabled_flag` does not exist, it is inferred to be equal to 0.

[0111] `slice_cross_component_alf_er_aps_id` is the same as `slice_cross_component_alf_cb_aps_id`, but it specifies the Cr color component for the segment it points to. When `slice_cross_component_alf_cr_enabled_flag` is equal to 1, the bitstream consistency requirement is that the ALFAPS pointed to by `slice_cross_component_alf_cr_aps_id` must be the same for all segments of the current image.

[0112] Incrementing 1 by 1 to slice_cross_component_cr_filters_signalled_minus1 specifies the number of cross-component Cr filters. The value of slice_cross_component_cr_filters_signalled_minus1 should be in the range of 0 to 3.

[0113] When slice_cross_component_alf_cr_enabled_flag equals 1, the requirement for bitstream consistency is that slice_cross_component_cr_filters_signalled_minus1 is less than or equal to the value of alf_cross_component_cr_filters_signalled_minus1 in the referenced ALF APS pointed to by slice_cross_component_alf_cr_aps_id of the current slice.

[0114] In embodiments of the present invention, pic_cross_component_cb_filters_signalled_minus1, pic_cross_component_cr_filters_signalled_minus1, slice_cross_component_cb_filters_signalled_minus1, and slice_cross_component_cr_filters_signalled_minus1 do not exist in the CC-ALF syntax, and alf_ctb_cross_component_cb_idc[][] and alf_ctb_cross_component_cr_idc[][] are encoded as shown in Table 3 below:

[0115] Table 3

[0116]

[0117]

[0118] According to the proposed syntax design, the syntax slice_cross_component_cb_filters_signalled_minus1 and slice_cross_component_cr_filters_signalled_minus1 in Table 2 are replaced by alf_cross_component_cb_filters_signalled_minus1 and alf_cross_component_cr_filters_signalled_minus1 in Table 3, respectively. In other words, instead of using syntax elements sent in the segment header, the number of filters is sent in the ALF APS and directly referenced to define the number of filters used for CC-ALF encoding. Therefore, unlike regular syntax and signaling, the CC-ALF design is consistent with the ALF design. Furthermore, since different segments in an image can reference different ALF APSs, if the number of CC-ALF filters exists in the ALF APS, 4 bits can be saved per segment (2 bits per color component).

[0119] Any of the methods proposed above can be implemented in the encoder and / or decoder. For example, any of the proposed methods can be implemented in the in-loop filtering module of the encoder and / or decoder. Alternatively, any of the proposed methods can be implemented as a circuit coupled to the in-loop filtering module of the encoder and / or decoder.

[0120] Figure 7 A flowchart illustrating an exemplary ALF processing of reconstructed video according to an embodiment of the present invention is shown, wherein the sum of the filter coefficients of each of one or more candidate filters is constrained to a fixed value. The steps shown in the flowchart can be implemented as program code executable on one or more processors (e.g., one or more CPUs) on the encoder or decoder side. The steps shown in the flowchart can also be implemented in hardware, such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to the method, in step 710, a reconstructed sample is received. In step 720, a target Cross-Component Adaptive Loop Filter (CCALF) is determined, wherein the target CCALF belongs to a set including one or more candidate filters, and the sum of the filter coefficients of each of the one or more candidate filters is constrained to a fixed value. In step 730, the target CCALF is applied to the relevant reconstructed luma sample to generate a CCALF-filtered chroma sample of the target reconstructed chroma sample. In step 740, a final filtered chroma sample is generated by combining the CCALF-filtered chroma sample with a regular ALF-filtered chroma sample, wherein the final filtered chroma sample is used as the video decoding output or for further video encoding or decoding processing.

[0121] Figure 8 A flowchart illustrating an exemplary ALF processing of reconstructed video according to an embodiment of the present invention is shown, wherein one or more syntax elements indicating values ​​related to the total number of candidate filters in a filter set are transmitted or parsed in an APS. According to the method, in step 810, reconstructed samples are received. In step 820, one or more syntax elements are transmitted in the APS of the video bitstream on the encoder side, or said one or more syntax elements are parsed in the APS of the video bitstream on the decoder side, wherein said one or more syntax elements indicate values ​​related to the total number of candidate filters in a filter set. In step 830, a target CCALF is determined, wherein the target CCALF belongs to a filter set including one or more candidate filters, and the sum of the filter coefficients of each of said one or more candidate filters is constrained to a fixed value. In step 840, the target CCALF is applied to the associated reconstructed luminance samples to generate CCALF-filtered chrominance samples for the target reconstructed chrominance samples. In step 850, a final filtered chroma sample is generated by combining CCALF filtered chroma samples and regular ALF filtered chroma samples, wherein the final filtered chroma sample is used as the video decoding output or is used for further video encoding or decoding processing.

[0122] The flowchart shown is intended to illustrate an example of video encoding and decoding according to the present invention. Those skilled in the art can modify each step, rearrange the steps, split the steps, or combine the steps to practice the invention without departing from its spirit. In this disclosure, specific syntax and semantics have been used to illustrate examples for implementing embodiments of the invention. Those skilled in the art can practice the invention by substituting equivalent syntax and semantics without departing from its spirit.

[0123] The foregoing description is provided to enable those skilled in the art to practice the invention in the context of a particular application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments. Therefore, the invention is not intended to be limited to the specific embodiments shown and described, but is consistent with the widest scope of the principles and novel features disclosed herein. In the above detailed description, various specific details are shown to provide a thorough understanding of the invention. However, those skilled in the art will understand that the invention can be practiced.

[0124] The embodiments of the present invention described above can be implemented in various hardware, software code, or a combination of both. For example, embodiments of the present invention may be one or more circuits integrated into a video compression chip or program code integrated into video compression software to perform the processes described herein. Embodiments of the present invention may also be program code executed on a digital signal processor (DSP) to perform the processes described herein. The present invention may also relate to numerous functions performed by a computer processor, digital signal processor, microprocessor, or field-programmable gate array (FPGA). These processors may be configured to perform specific tasks according to the present invention by executing machine-readable software code or firmware code that defines the specific methods embodied in the present invention. The software code or firmware code may be developed in different programming languages ​​and in different formats or styles. The software code may also be compiled for different target platforms. However, different code formats, software code styles and languages, and other means of configuring the code to perform the tasks according to the present invention will not depart from the spirit and scope of the present invention.

[0125] The invention may be practiced in other specific forms without departing from the spirit or essential characteristics thereof. The described examples should be considered illustrative rather than restrictive in all respects. Therefore, the scope of the invention is indicated by the appended claims rather than the foregoing description. All modifications falling within the equivalent meaning and scope of the claims should be included within their scope.

Claims

1. A video encoding / decoding method, comprising: Receive multiple reconstructed chromaticity samples and multiple associated reconstructed luminance samples from a color image; A target cross-component adaptive loop filter is determined, wherein the target cross-component adaptive loop filter belongs to a filter set including one or more candidate filters, and the sum of multiple filter coefficients of each of the one or more candidate filters is constrained to a fixed value, wherein the fixed value is 0; The target cross-component adaptive loop filter is applied to the plurality of correlated reconstructed luminance samples to generate cross-component adaptive loop filter chrominance samples of the target reconstructed chrominance samples; as well as By combining the cross-component adaptive loop filter chroma samples with a conventional adaptive loop filter, a final filtered chroma sample is generated, which is used as the video decoding output or for further video encoding or decoding processing.

2. The video encoding and decoding method as described in claim 1, characterized in that, The fixed value is equal to 0.

3. The video encoding and decoding method as described in claim 1, characterized in that, When the number of multiple filter coefficients for each candidate filter is equal to M, only (M-1) filter coefficients in each candidate filter are decoded on the decoder side, where M is an integer greater than 1.

4. The video encoding and decoding method as described in claim 3, characterized in that, A filter coefficient, instead of (M-1) filter coefficients, is not decoded on the decoder side or encoded on the encoder side, and wherein the one filter coefficient corresponds to any selected coefficient among the plurality of filter coefficients in each candidate filter.

5. A video encoding / decoding apparatus, the apparatus comprising one or more electronic circuits or processors configured to: Receive multiple reconstructed chromaticity samples and multiple associated reconstructed luminance samples from a color image; A target cross-component adaptive loop filter is determined, wherein the target cross-component adaptive loop filter belongs to a filter set including one or more candidate filters, and the sum of multiple filter coefficients of each of the one or more candidate filters is constrained to a fixed value, wherein the fixed value is 0; The target cross-component adaptive loop filter is applied to the plurality of correlated reconstructed luminance samples to generate cross-component adaptive loop filter chrominance samples of the target reconstructed chrominance samples; as well as By combining the cross-component adaptive loop filter chroma samples with a conventional adaptive loop filter, a final filtered chroma sample is generated, which is used as the video decoding output or for further video encoding or decoding processes.

Images