Method and apparatus for video encoding and decoding using CC-ALF based on non-linear cross component relationship

Abstract

The embodiment discloses a video coding and decoding method and device using CC-ALF based on a non-linear cross component relation. In this embodiment, a video decoding device obtains a reconstructed frame from a sample adaptive offset (SAO) filter, and inputs the reconstructed frame into an adaptive loop filter (ALF) to generate an ALF output. Here, the ALF outputs include a luma ALF output and a chroma ALF output. A video decoding device generates a correction value of a chroma component by inputting a luma ALF output into a non-linear cross component ALF (CC-ALF), and generates an improved chroma ALF output by adding the correction value of the chroma component and the chroma ALF output.

Description

Technical Field

[0001] The invention relates to a video coding and decoding method and device using CC-ALF based on nonlinear cross component relationship. Background Art

[0002] The statements in this section merely provide background information related to the present invention and may not necessarily constitute prior art.

[0003] Since video data has a large amount of data compared to audio data or still image data, video data requires a large amount of hardware resources (including memory) to store or transmit the video data without being compressed.

[0004] Accordingly, the encoder is generally used to compress and store or transmit video data. The decoder receives the compressed video data, decompresses the received compressed video data, and plays the decompressed video data. Video compression technologies include H.264 / Advanced Video Codec (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Codec (VVC), which improves the coding efficiency of HEVC by about 30% or more.

[0005] However, as image size, resolution and frame rate gradually increase, the amount of data to be encoded will also increase. Accordingly, new compression technologies are needed that provide higher encoding and decoding efficiency and improved image enhancement effects than existing compression technologies.

[0006] The cross-component adaptive loop filter (CC-ALF), which is a component of the in-loop filter of VVC, corrects the chroma component in parallel with the adaptive loop filter (ALF) based on the correlation between the current chroma sample and its corresponding luma sample. When generating the correction value of the chroma sample, the conventional CC-ALF applies a linear filtering operation to the input luma sample. Therefore, in order to improve the efficiency of video coding and decoding and enhance the video quality, it is necessary to consider a method for generating the correction value of the chroma sample more efficiently. Summary of the invention

[0007] Technical issues

[0008] The present invention seeks to provide a video coding method and apparatus that improves chrominance components by utilizing linear modeling and nonlinear modeling when applying in-loop filtering to a reconstructed video signal.

[0009] Technical Solution

[0010] At least one aspect of the present invention provides a method for filtering a reconstructed frame by a video decoding device. The method includes obtaining a reconstructed frame, which is the output of a sample adaptive offset (SAO) filter. The method also includes generating an adaptive loop filter output (ALF output) by inputting the reconstructed frame into an adaptive loop filter (ALF). Here, the ALF output includes a luma ALF output and a chroma ALF output. The method also includes generating a correction value of a chroma component by inputting the luma ALF output into a nonlinear cross component ALF (nonlinear CC-ALF). The method also includes generating an enhanced chroma ALF output by summing the correction value of the chroma component and the chroma ALF output.

[0011] Another aspect of the present invention provides a method for filtering a reconstructed frame by a video encoding device. The method includes obtaining a reconstructed frame, which is the output of a sample adaptive offset (SAO) filter. The method also includes generating an adaptive loop filter output (ALF output) by inputting the reconstructed frame into an adaptive loop filter (ALF). Here, the ALF output includes a luma ALF output and a chroma ALF output. The method also includes generating a correction value of a chroma component by inputting the luma ALF output into a nonlinear cross component ALF (nonlinear CC-ALF). The method also includes generating a first enhanced chroma ALF output by summing the correction value of the chroma component and the chroma ALF output.

[0012] Another aspect of the present invention provides a computer-readable recording medium storing a bitstream generated by a video encoding method. The video encoding method includes obtaining a reconstructed frame, which is the output of a sample adaptive offset (SAO) filter. The video encoding method also includes generating an adaptive loop filter output (ALF output) by inputting the reconstructed frame into an adaptive loop filter (ALF). Here, the ALF output includes a luminance ALF output and a chrominance ALF output. The video encoding method also includes generating a correction value of a chrominance component by inputting the luminance ALF output into a nonlinear cross component ALF (nonlinear CC-ALF). The video encoding method also includes generating an enhanced chrominance ALF output by summing the correction value of the chrominance component and the chrominance ALF output.

[0013] Beneficial Effects

[0014] As described above, the present invention provides a video coding method and apparatus that improves chrominance components by utilizing linear modeling and nonlinear modeling when applying in-loop filtering to a reconstructed video signal. Therefore, the video coding method and apparatus improve video coding efficiency and enhance video quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of a video encoding device that can implement the technology of the present invention.

[0016] Figure 2 A method of partitioning blocks using a quadtree plus binary tree ternary tree (QTBTTT) structure is shown.

[0017] Figure 3a and Figure 3b Multiple intra prediction modes are shown including a wide-angle intra prediction mode.

[0018] Figure 4 Shows the neighboring blocks of the current block.

[0019] Figure 5 is a block diagram of a video decoding device that can implement the technology of the present invention.

[0020] Figure 6 is a schematic diagram showing the configuration of an adaptive loop filter (ALF).

[0021] Figure 7 is a schematic diagram showing the application of a cross-component adaptive loop filter (CC-ALF).

[0022] Figure 8 is a schematic diagram showing the configuration of CC-ALF.

[0023] Fig. 9 is a flowchart of a method for filtering a reconstructed frame by a video encoding apparatus according to at least one embodiment of the present invention.

[0024] Fig.10 is a flowchart of a method for filtering a reconstructed frame by a video decoding device according to at least one embodiment of the present invention. DETAILED DESCRIPTION

[0025] Hereinafter, some embodiments of the present invention will be described in detail with reference to the attached illustrative drawings. In the following description, the same reference numerals represent the same elements, although the elements are shown in different drawings. In addition, in the following description of some embodiments, when it is considered that the detailed description of related known components and functions obscures the subject matter of the present invention, the detailed description of related known components and functions may be omitted for the sake of clarity and brevity.

[0026] Figure 1 is a block diagram of a video encoding device that can implement the technology of the present invention. Figure 1 A video encoding device and components of the device are described in detail.

[0027] The encoding device may include: an image divider 110, a predictor 120, a subtractor 130, a transformer 140, a quantizer 145, a rearrangement unit 150, an entropy encoder 155, an inverse quantizer 160, an inverse transformer 165, an adder 170, a loop filtering unit 180 and a memory 190.

[0028] Each component of the encoding device can be implemented as hardware or software, or as a combination of hardware and software. In addition, the function of each component can be implemented as software, and the microprocessor can also be implemented to execute the function of the software corresponding to each component.

[0029] A video consists of one or more sequences including multiple images. Each image is divided into multiple regions, and encoding is performed on each region. For example, an image is divided into one or more tiles or / and slices. Here, one or more tiles can be defined as a tile group. Each tile or / and slice is divided into one or more coding tree units (coding tree unit, CTU). In addition, each CTU is divided into one or more coding units (coding unit, CU) through a tree structure. The information applied to each coding unit (CU) is encoded as the syntax of the CU, and the information commonly applied to the CU included in a CTU is encoded as the syntax of the CTU. In addition, the information commonly applied to all blocks in a slice is encoded as the syntax of the slice header, and the information applied to all blocks constituting one or more images is encoded as a picture parameter set (Picture Parameter Set, PPS) or a picture header. In addition, the information commonly referenced by multiple images is encoded as a sequence parameter set (Sequence Parameter Set, SPS). In addition, the information commonly referenced by one or more SPSs is encoded as a video parameter set (Video Parameter Set, VPS). In addition, information commonly applied to one tile or tile group may also be encoded as syntax of the tile or tile group header.The syntax included in the SPS, PPS, slice header, tile or tile group header may be referred to as a high-level syntax.

[0030] The image divider 110 determines the size of a coding tree unit (CTU). Information on the size of the CTU (CTU size) is encoded as a syntax of an SPS or a PPS and transmitted to a video decoding device.

[0031] The image divider 110 divides each image constituting a video into a plurality of coding tree units (CTUs) having a predetermined size, and then recursively divides the CTUs by using a tree structure. A leaf node in the tree structure becomes a coding unit (CU), which is a basic unit of coding.

[0032] The tree structure can be a quadtree (quadtree, QT), in which a higher node (or parent node) is split into four lower nodes (or child nodes) of the same size. The tree structure can also be a binary tree (binarytree, BT), in which a higher node is split into two lower nodes. The tree structure can also be a ternary tree (ternarytree, TT), in which a higher node is split into three lower nodes at a ratio of 1:2:1. The tree structure can also be a structure in which two or more structures in the QT structure, BT structure, and TT structure are mixed. For example, a quadtree plus binary tree (quadtree plus binarytree, QTBT) structure can be utilized, or a quadtree plus binary tree ternary tree (quadtree plus binarytreeternarytree, QTBTTT) structure can be utilized. Here, a binary tree ternary tree (BTTT) is added to the tree structure to be called a multiple-type tree (multiple-type tree, MTT).

[0033] Figure 2 is a schematic diagram for describing a method of partitioning a block by using a QTBTTT structure.

[0034] like Figure 2 As shown, the CTU can be first split into a QT structure. The quadtree splitting can be recursive until the size of the split block reaches the minimum block size (MinQTSize) of the leaf node allowed in QT. The first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoder 155 and notified to the video decoding device with a signal. When the leaf node of QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in BT, the leaf node can be further split into at least one of a BT structure or a TT structure. There may be multiple splitting directions in the BT structure and / or the TT structure. For example, there may be two directions, namely, a direction for horizontally splitting blocks of the corresponding node and a direction for vertically splitting blocks of the corresponding node. As Figure 2 As shown, when MTT splitting starts, the entropy encoder 155 encodes a second flag (mtt_split_flag) indicating whether the node is split, and a flag indicating the splitting direction (vertical or horizontal) and / or a flag indicating the splitting type (binary or trifurcated) when the node is split, and notifies the video decoding device of the same by signal.

[0035] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of the lower layer, the CU split flag (split_cu_flag) indicating whether the node is split may also be encoded. When the value of the CU split flag (split_cu_flag) indicates that each node is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU, which is the basic unit of encoding. When the value of the CU split flag (split_cu_flag) indicates that each node is split, the video encoding device first starts encoding the first flag in the above scheme.

[0036] When QTBT is used as another example of a tree structure, there may be two types, namely, a type in which the block of the corresponding node is horizontally split into two blocks of the same size (i.e., symmetrical horizontal splitting) and a type in which the block of the corresponding node is vertically split into two blocks of the same size (i.e., symmetrical vertical splitting). The entropy encoder 155 encodes a split flag (split_flag) indicating whether each node of the BT structure is split into blocks of the lower layer and split type information indicating the split type, and transmits them to the video decoding device. On the other hand, there may be another type in which the block of the corresponding node is split into two blocks that are asymmetric to each other. The asymmetric form may include a form in which the block of the corresponding node is split into two rectangular blocks with a size ratio of 1:3, or may also include a form in which the block of the corresponding node is split in a diagonal direction.

[0037] The CU may have various sizes according to the QTBT or QTBTTT partitioned from the CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block". When QTBTTT partitioning is adopted, the shape of the current block may also be a rectangular shape in addition to a square shape.

[0038] The predictor 120 predicts the current block to generate a predicted block. The predictor 120 includes an intra predictor 122 and an inter predictor 124.

[0039] Typically, each of the current blocks in an image may be predictively encoded. Typically, the prediction of the current block may be performed by utilizing an intra-prediction technique (which utilizes data from an image including the current block) or an inter-prediction technique (which utilizes data from an image encoded prior to the image including the current block). Inter-prediction includes both unidirectional prediction and bidirectional prediction.

[0040] The intra-frame predictor 122 predicts pixels in the current block by using pixels (reference pixels) located adjacent to the current block in the current image including the current block. There are multiple intra-frame prediction modes according to the prediction direction. For example, Figure 3aAs shown, the plurality of intra prediction modes may include two non-directional modes including a planar mode and a DC mode, and may include 65 directional modes. Neighboring pixels and algorithm equations to be used are defined differently according to each prediction mode.

[0041] In order to perform efficient directional prediction for a current block with a rectangular shape, we can additionally utilize Figure 3b The directional modes are shown in the dashed arrows (#67 to #80, intra-frame prediction modes #-1 to #-14). The directional modes can be called "wide angle intra-frame prediction modes". Figure 3b In the figure, the arrows indicate the corresponding reference samples used for prediction, rather than representing the prediction direction. The prediction direction is opposite to the direction indicated by the arrow. When the current block has a rectangular shape, the wide-angle intra prediction mode is a mode in which prediction is performed in a direction opposite to the specific direction mode without additional bit transmission. In this case, in the wide-angle intra prediction mode, some wide-angle intra prediction modes available for the current block can be determined by the ratio of the width to the height of the current block having a rectangular shape. For example, when the current block has a rectangular shape with a height less than the width, a wide-angle intra prediction mode (intra prediction modes #67 to #80) with an angle less than 45 degrees is available. When the current block has a rectangular shape with a width greater than the height, a wide-angle intra prediction mode with an angle greater than -135 degrees is available.

[0042] The intra-frame predictor 122 may determine an intra-frame prediction to be used for encoding the current block. In some examples, the intra-frame predictor 122 may encode the current block by utilizing multiple intra-frame prediction modes, and may also select an appropriate intra-frame prediction mode to be utilized from a test mode. For example, the intra-frame predictor 122 may calculate a rate-distortion value by utilizing a rate-distortion analysis of multiple tested intra-frame prediction modes, and may also select an intra-frame prediction mode having the best rate-distortion characteristic in the test mode.

[0043] The intra-frame predictor 122 selects an intra-frame prediction mode among a plurality of intra-frame prediction modes and predicts the current block by using adjacent pixels (reference pixels) and an algorithm equation determined according to the selected intra-frame prediction mode. Information about the selected intra-frame prediction mode is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0044] The inter-frame predictor 124 generates a prediction block for the current block by using a motion compensation process. The inter-frame predictor 124 searches for a block that is most similar to the current block in a reference image that has been encoded and decoded earlier than the current image, and generates a prediction block for the current block by using the searched block. In addition, a motion vector (MV) is generated, which corresponds to the displacement between the current block in the current image and the prediction block in the reference image. Typically, motion estimation is performed on the luminance (luma) component, and the motion vector calculated based on the luminance component is used for both the luminance component and the chrominance component. The motion information including information of the reference image and information about the motion vector used to predict the current block is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0045] The inter-frame predictor 124 may also perform interpolation of a reference image or a reference block to increase the accuracy of the prediction. In other words, a subsample is interpolated between two consecutive integer samples by applying a filter coefficient to a plurality of consecutive integer samples including two integer samples. When performing a process of searching for a block most similar to the current block on the interpolated reference image, a decimal unit precision may be represented for the motion vector instead of an integer sample unit precision. For each target region to be encoded, for example, a unit such as a slice, a tile, a CTU, a CU, etc., the precision or resolution of the motion vector may be set differently. When such an adaptive motion vector resolution (AMVR) is applied, information about the motion vector resolution to be applied to each target region should be signaled for each target region. For example, when the target region is a CU, information about the motion vector resolution applied to each CU is signaled. The information about the motion vector resolution may be information representing the precision of the motion vector difference to be described below.

[0046] On the other hand, the inter-frame predictor 124 may perform inter-frame prediction by utilizing bidirectional prediction. In the case of bidirectional prediction, two reference images and two motion vectors representing the block positions most similar to the current block in each reference image are utilized. The inter-frame predictor 124 selects a first reference image and a second reference image from a reference image list 0 (RefPicList0) and a reference image list 1 (RefPicList1), respectively. The inter-frame predictor 124 also searches for a block most similar to the current block in the corresponding reference image to generate a first reference block and a second reference block. In addition, a prediction block of the current block is generated by averaging or weighted averaging the first reference block and the second reference block. In addition, motion information including information about two reference images used to predict the current block and including information about two motion vectors is transmitted to the entropy encoder 155. Here, the reference image list 0 may be composed of images that are before the current image in the display order in the pre-reconstructed image, and the reference image list 1 may be composed of images that are after the current image in the display order in the pre-reconstructed image. However, although not particularly limited thereto, the pre-reconstructed image that is after the current image in the display order may be additionally included in the reference image list 0. Conversely, a pre-reconstructed image preceding the current image may also be additionally included in the reference image list 1 .

[0047] In order to minimize the amount of bits consumed for encoding motion information, various methods may be utilized.

[0048] For example, when the reference image and motion vector of the current block are the same as those of the neighboring block, information capable of identifying the neighboring block is encoded to transmit the motion information of the current block to the video decoding device. This method is called merge mode.

[0049] In the merge mode, the inter predictor 124 selects a predetermined number of merge candidate blocks (hereinafter, referred to as 'merge candidates') from neighboring blocks of the current block.

[0050] As the neighboring blocks for deriving the merge candidate, all or some of the left block A0, the lower left block A1, the upper block B0, the upper right block B1 and the upper left block B2 adjacent to the current block in the current image may be used, such as Figure 4 As shown. In addition, in addition to the current image where the current block is located, a block located in a reference image (which may be the same as or different from the reference image used to predict the current block) may also be used as a merge candidate. For example, a co-located block of the current block in the reference image or a block adjacent to the co-located block may be additionally used as a merge candidate. If the number of merge candidates selected by the above method is less than the preset number, a zero vector is added to the merge candidate.

[0051] The inter-frame predictor 124 configures a merge list including a predetermined number of merge candidates by using adjacent blocks. A merge candidate to be used as motion information of the current block is selected from among the merge candidates included in the merge list, and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0052] Merge skip mode is a special case of merge mode. After quantization, when all transform coefficients for entropy coding are close to zero, only the neighbor block selection information is transmitted without the residual signal. By using merge skip mode, relatively high coding efficiency can be achieved for images with slight motion, still images, screen content images, etc.

[0053] Hereinafter, merge mode and merge skip mode are collectively referred to as merge / skip mode.

[0054] Another method for encoding motion information is the advanced motion vector prediction (AMVP) mode.

[0055] In the AMVP mode, the inter-frame predictor 124 derives a motion vector prediction candidate for a motion vector of a current block by using neighboring blocks of the current block. As neighboring blocks for deriving motion vector prediction candidates, Figure 4 All or some of the left block A0, lower left block A1, upper block B0, upper right block B1 and upper left block B2 adjacent to the current block in the current image shown. In addition, in addition to the current image where the current block is located, blocks located in a reference image (which may be the same as or different from the reference image used to predict the current block) may also be used as adjacent blocks for deriving motion vector prediction candidates. For example, the same block of the current block in the reference image or a block adjacent to the same block may be used. If the number of motion vector candidates selected by the above method is less than the preset number, a zero vector is added to the motion vector candidates.

[0056] The inter-frame predictor 124 derives a motion vector prediction candidate by using the motion vectors of the neighboring blocks, and determines a motion vector prediction of the motion vector of the current block by using the motion vector prediction candidate. In addition, a motion vector difference is calculated by subtracting the motion vector prediction from the motion vector of the current block.

[0057] The motion vector prediction can be obtained by applying a predefined function (e.g., median and mean calculations, etc.) to the motion vector prediction candidate. In this case, the video decoding device also knows the predefined function. In addition, since the neighboring blocks used to derive the motion vector prediction candidate are blocks that have completed encoding and decoding, the video decoding device may also already know the motion vectors of the neighboring blocks. Therefore, the video encoding device does not need to encode the information for identifying the motion vector prediction candidate. Accordingly, in this case, information about the motion vector difference and information about the reference image used to predict the current block are encoded.

[0058] On the other hand, the motion vector prediction can also be determined by selecting any one of the motion vector prediction candidates. In this case, information for identifying the selected motion vector prediction candidate is additionally encoded together with information about the motion vector difference used to predict the current block and information about the reference image.

[0059] The subtractor 130 generates a residual block by subtracting a prediction block generated by the intra predictor 122 or the inter predictor 124 from the current block.

[0060] The transformer 140 transforms the residual signal in the residual block having the pixel value of the spatial domain into the transform coefficient of the frequency domain. The transformer 140 may transform the residual signal in the residual block by using the entire size of the residual block as the transform unit, or may also divide the residual block into a plurality of sub-blocks, and may perform the transform by using the sub-block as the transform unit. Alternatively, the residual block is divided into two sub-blocks, namely, a transform region and a non-transform region, so as to transform the residual signal by using only the transform region sub-block as the transform unit. Here, the transform region sub-block may be one of two rectangular blocks having a size ratio of 1:1 based on the horizontal axis (or the vertical axis). In this case, the entropy encoder 155 encodes a flag (cu_sbt_flag) indicating that only the sub-block is transformed, and direction (vertical / horizontal) information (cu_sbt_horizontal_flag) and / or position information (cu_sbt_pos_flag), and notifies the video decoding device with a signal. In addition, the size of the transform region subblock may have a size ratio of 1:3 based on the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_quad_flag) for dividing the corresponding partition is additionally encoded by the entropy encoder 155 and signaled to the video decoding apparatus.

[0061] On the other hand, the transformer 140 may perform transformation of the residual block separately in the horizontal direction and the vertical direction. For the transformation, various types of transformation functions or transformation matrices may be used. For example, paired transformation functions for horizontal transformation and vertical transformation may be defined as a multiple transform set (MTS). The transformer 140 may select a transformation function pair with the highest transformation efficiency in the MTS, and may transform the residual block in each of the horizontal and vertical directions. Information (mts_idx) about the transformation function pair in the MTS is encoded by the entropy encoder 155 and notified to the video decoding device with a signal.

[0062] The quantizer 145 quantizes the transform coefficients output from the transformer 140 using the quantization parameters and outputs the quantized transform coefficients to the entropy encoder 155. The quantizer 145 may also quantize the relevant residual block immediately without transforming any block or frame. The quantizer 145 may also apply different quantization coefficients (scaling values) according to the position of the transform coefficients in the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to the video decoding device.

[0063] The rearrangement unit 150 may perform rearrangement of coefficient values ​​on the quantized residual value.

[0064] The rearrangement unit 150 may change the 2D coefficient array into a 1D coefficient sequence by utilizing coefficient scanning. For example, the rearrangement unit 150 may utilize a zig-zag scan or a diagonal scan to scan the DC coefficient to the coefficient of the high frequency region to output a 1D coefficient sequence. Depending on the size of the transform unit and the intra-frame prediction mode, the zig-zag scan may also be replaced by a vertical scan that scans the 2D coefficient array in the column direction and a horizontal scan that scans the 2D block type coefficient in the row direction. In other words, depending on the size of the transform unit and the intra-frame prediction mode, the scanning method to be utilized may be determined among zig-zag scanning, diagonal scanning, vertical scanning, and horizontal scanning.

[0065] The entropy encoder 155 encodes the sequence of 1D quantized transform coefficients output from the rearrangement unit 150 by using various encoding schemes including Context-based Adaptive Binary Arithmetic Code (CABAC), Exponential Golomb, etc. to generate a bitstream.

[0066] In addition, the entropy encoder 155 encodes information related to block segmentation (e.g., CTU size, CTU segmentation flag, QT segmentation flag, MTT segmentation type, and MTT segmentation direction, etc.) so that the video decoding device can segment the block equivalent to the video encoding device. In addition, the entropy encoder 155 encodes information about the prediction type indicating whether the current block is encoded by intra-frame prediction or inter-frame prediction. The entropy encoder 155 encodes intra-frame prediction information (i.e., information about intra-frame prediction mode) or inter-frame prediction information (merge index in the case of merge mode, and information about reference image index and motion vector difference in the case of AMVP mode) according to the prediction type. In addition, the entropy encoder 155 encodes information related to quantization (i.e., information about quantization parameters and information about quantization matrices).

[0067] The inverse quantizer 160 inversely quantizes the quantized transform coefficient output from the quantizer 145 to generate a transform coefficient. The inverse transformer 165 transforms the transform coefficient output from the inverse quantizer 160 from the frequency domain to the spatial domain to reconstruct a residual block.

[0068] The adder 170 adds the reconstructed residual block and the prediction block generated by the predictor 120 to reconstruct the current block. When intra-prediction is performed on the next block, the pixels in the reconstructed current block are used as reference pixels.

[0069] The loop filtering unit 180 performs filtering on the reconstructed pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transform / quantization. The loop filtering unit 180 as an in-loop filter may include all or some of a deblocking filter 182, a sample adaptive offset (SAO) filter 184, and an adaptive loop filter (ALF) 186.

[0070] The deblocking filter 182 filters the boundaries between the reconstructed blocks to remove blocking artifacts that occur due to block unit encoding / decoding, and the SAO filter 184 and the ALF 186 perform additional filtering on the deblocking filtered video. The SAO filter 184 and the ALF 186 are filters for compensating for the difference between the reconstructed pixels and the original pixels that occur due to lossy coding. The SAO filter 184 applies an offset in units of CTU to enhance subjective image quality and encoding efficiency. On the other hand, the ALF 186 performs block unit filtering and applies different filters to compensate for distortion by dividing the boundaries of the corresponding blocks and the degree of the amount of change. Information about the filter coefficients to be used for the ALF can be encoded and notified to the video decoding device with a signal.

[0071] The reconstructed blocks filtered by the deblocking filter 182, the SAO filter 184, and the ALF 186 are stored in the memory 190. When all blocks in one image are reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks within an image to be subsequently encoded.

[0072] The video encoding device may store a bit stream of encoded video data in a nonvolatile storage medium or transmit the bit stream to a video decoding device through a communication network.

[0073] Figure 5 is a functional block diagram of a video decoding device that can implement the technology of the present invention. Figure 5 , describes a video decoding device and components of the device.

[0074] The video decoding apparatus may include an entropy decoder 510 , a rearrangement unit 515 , an inverse quantizer 520 , an inverse transformer 530 , a predictor 540 , an adder 550 , a loop filtering unit 560 , and a memory 570 .

[0075] Similar to Figure 1 Each component of the video encoding device and the video decoding device can be implemented as hardware or software, or as a combination of hardware and software. In addition, the function of each component can be implemented as software, and the microprocessor can also be implemented to execute the function of the software corresponding to each component.

[0076] The entropy decoder 510 extracts information related to block partitioning by decoding a bitstream generated by a video encoding apparatus to determine a current block to be decoded, and extracts prediction information required to reconstruct the current block and information about a residual signal.

[0077] The entropy decoder 510 determines the size of the CTU by extracting information about the CTU size from a sequence parameter set (SPS) or a picture parameter set (PPS), and divides the image into CTUs of the determined size. In addition, the CTU is determined as the highest level (i.e., the root node) of the tree structure, and the partition information of the CTU can be extracted to partition the CTU by using the tree structure.

[0078] For example, when CTU is segmented by using the QTBTTT structure, the first flag (QT_split_flag) related to the segmentation of QT is first extracted to segment each node into four nodes of the lower layer. In addition, the second flag (mtt_split_flag) related to the segmentation of MTT, the segmentation direction (vertical / horizontal) and / or the segmentation type (binary / tripartite) are extracted with respect to the node corresponding to the leaf node of QT to segment the corresponding leaf node into the MTT structure. As a result, each node below the leaf node of QT is recursively segmented into a BT or TT structure.

[0079] As another example, when a CTU is split by using a QTBTTT structure, a CU split flag (split_cu_flag) indicating whether the CU is split is extracted. When the corresponding block is split, a first flag (QT_split_flag) may also be extracted. During the splitting process, for each node, 0 or more recursive MTT splits may occur after 0 or more recursive QT splits. For example, for a CTU, MTT splits may occur immediately, or, conversely, multiple QT splits may occur only.

[0080] As another example, when CTU is segmented by using the QTBT structure, the first flag (QT_split_flag) related to the segmentation of the QT is extracted to segment each node into four nodes of the lower layer. In addition, the segmentation flag (split_flag) indicating whether the node corresponding to the leaf node of the QT is further segmented into BT and the segmentation direction information are extracted.

[0081] On the other hand, when the entropy decoder 510 determines the current block to be decoded by using the partitioning of the tree structure, the entropy decoder 510 extracts information about the prediction type indicating whether the current block is intra-predicted or inter-predicted. When the prediction type information indicates intra-prediction, the entropy decoder 510 extracts syntax elements of intra-prediction information (intra-prediction mode) for the current block. When the prediction type information indicates inter-prediction, the entropy decoder 510 extracts information representing syntax elements of inter-prediction information, that is, a motion vector and a reference image to which the motion vector refers.

[0082] In addition, the entropy decoder 510 extracts quantization-related information and extracts information on a quantized transform coefficient of the current block as information on a residual signal.

[0083] The rearrangement unit 515 may change the sequence of 1D quantized transform coefficients entropy-decoded by the entropy decoder 510 into a 2D coefficient array (ie, block) again in the reverse order of the coefficient scanning order performed by the video encoding device.

[0084] The inverse quantizer 520 inversely quantizes the quantized transform coefficients, and inversely quantizes the quantized transform coefficients by using quantization parameters. The inverse quantizer 520 may also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in 2D. The inverse quantizer 520 may perform inverse quantization by applying a matrix of quantization coefficients (scaling values) from a video encoding device to a 2D array of quantized transform coefficients.

[0085] The inverse transformer 530 reconstructs a residual signal by inversely transforming the inversely quantized transform coefficient from the frequency domain to the spatial domain to generate a residual block of the current block.

[0086] In addition, when the inverse transformer 530 inversely transforms a partial region (sub-block) of the transform block, the inverse transformer 530 extracts a flag (cu_sbt_flag) that only the sub-block of the transform block is transformed, direction (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and / or position information (cu_sbt_pos_flag) of the sub-block. The inverse transformer 530 also inversely transforms the transform coefficient of the corresponding sub-block from the frequency domain to the spatial domain to reconstruct the residual signal, and fills the region that is not inversely transformed with a value of "0" as the residual signal to generate a final residual block of the current block.

[0087] In addition, when MTS is applied, the inverse transformer 530 determines the transform index or transform matrix to be applied in each of the horizontal and vertical directions by using the MTS information (mts_idx) signaled from the video encoding device. The inverse transformer 530 also performs inverse transform on the transform coefficients in the transform block in the horizontal and vertical directions by using the determined transform function.

[0088] The predictor 540 may include an intra predictor 542 and an inter predictor 544. The intra predictor 542 is activated when the prediction type of the current block is intra prediction, and the inter predictor 544 is activated when the prediction type of the current block is inter prediction.

[0089] The intra predictor 542 determines an intra prediction mode of a current block among a plurality of intra prediction modes according to syntax elements of the intra prediction mode extracted from the entropy decoder 510. The intra predictor 542 also predicts the current block by using neighboring reference pixels of the current block according to the intra prediction mode.

[0090] The inter predictor 544 determines a motion vector of a current block and a reference image to which the motion vector refers by using a syntax element of the inter prediction mode extracted from the entropy decoder 510 .

[0091] The adder 550 reconstructs the current block by adding the residual block output from the inverse transformer 530 and the prediction block output from the inter predictor 544 or the intra predictor 542. When intra-predicting a subsequent block to be decoded, pixels within the reconstructed current block are used as reference pixels.

[0092] The loop filtering unit 560 as an in-loop filter may include a deblocking filter 562, an SAO filter 564, and an ALF 566. The deblocking filter 562 performs deblocking filtering on the boundary between the reconstructed blocks in order to remove block artifacts that occur due to block unit decoding. The SAO filter 564 and the ALF 566 perform additional filtering on the reconstructed blocks after deblocking filtering in order to compensate for the difference between the reconstructed pixels and the original pixels that occurs due to lossy encoding. The filter coefficients of the ALF are determined by using information about the filter coefficients decoded from the bitstream.

[0093] The reconstructed blocks filtered by the deblocking filter 562, the SAO filter 564, and the ALF 566 are stored in the memory 570. When all blocks in one image are reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks within an image to be subsequently encoded.

[0094] In some embodiments, the present invention relates to encoding and decoding video images as described above. More specifically, the present invention provides a video encoding and decoding method and apparatus for improving chrominance components by utilizing linear models and nonlinear models when applying in-loop filtering to a reconstructed video signal.

[0095] The following implementation scheme may be performed by the loop filtering unit 180 in the video encoding apparatus. The following implementation scheme may also be performed by the loop filtering unit 560 in the video decoding apparatus.

[0096] The video encoding device may generate signaling information associated with the present embodiment in terms of optimizing rate distortion when encoding the current block. The video encoding device may encode the signaling information using the entropy encoder 155 and send the encoded signaling information to the video decoding device. The video decoding device may decode the signaling information associated with decoding of the current block from the bitstream using the entropy decoder 510.

[0097] In the following description, the term 'target block' may be used interchangeably with a current block or a coding unit (CU), or may refer to some regions of a coding unit.

[0098] In addition, a flag value of true indicates a case where the flag is set to 1. In addition, a flag value of false indicates a case where the flag is set to 0.

[0099] The following embodiments are described with reference to a video decoding apparatus, but may be implemented in a video encoding apparatus in the same or similar manner.

[0100] I. In-loop Filter

[0101] As described above, in order to remove the artifacts remaining after compression, an in-loop filter including a deblocking filter (182 or 562), an SAO filter (184 or 564), an ALF (186 or 566), and a luminance mapping and chroma sampling (Luma mapping and chromasampling, LMCS) is used. In the encoding and decoding loops, the in-loop filter is applied to the reconstructed image in the order of LMCS, deblocking filter, SAO filter, and ALF, and the output image is stored in a decoded picture buffer (DPB) in the memory (190 or 570).

[0102] As described above, the deblocking filter (182 or 562) and the SAO filter (184 or 564) are used to remove blocking artifacts and ringing artifacts in HEVC.

[0103] Compared with HEVC, ALF (186 or 566) and LMCS are new in-loop filters added to VVC. ALF (186 or 566) uses filter coefficients determined based on the Wiener-Hopf equation to reduce the mean square error (MSE) between the original sample and the reconstructed sample. LMCS adjusts the dynamic range of pixel values ​​in the image to improve the objective quality of the reconstructed image.

[0104] II. Adaptive Loop Filter (ALF)

[0105] The ALF (186 or 566) of VVC uses an adaptive linear filter based on the Wiener-Hopf equation to approximate the reconstructed video frame to the original frame. The video encoding device uses the output samples of the SAO 184 to calculate the filter coefficients of the ALF 186 based on rate-distortion optimization, and sends the filter coefficients to the video decoding device. Figure 6As shown, the ALF (186 or 566) is configured as a 7×7 diamond and a 5×5 diamond for luminance and chrominance samples, respectively. The filter shape and size can be determined by considering the trade-off between codec efficiency and computational complexity. For example, by utilizing a symmetric FIR filter, the computational complexity of the ALF (186 or 566) can be reduced.

[0106] To derive Figure 6 The filter coefficients c shown in i , using samples at their relevant positions. The filtered sample I(x, y) at the current position (x, y) can be calculated as shown in Equation 1 according to 7-bit precision arithmetic.

[0107] [Equation 1]

[0108]

[0109] In this paper, r i is the difference between the current sample and its adjacent samples calculated according to Equation 2.

[0110] [Equation 2]

[0111] r i =min(b i ,max(-b i ,I(x+x i ,y+y i )-I(x,y)))

[0112] +min(b i ,max(-b i ,I(xx i ,yy i )-I(x,y)))

[0113] Here, b i is the shearing parameter.

[0114] ALF (186 or 566) uses up to 25 sets of filter coefficients for the luminance component and applies the filter coefficients to 4×4 sub-blocks. Based on the gradient information of the local block calculated using the Laplacian filter, the 4×4 sub-block is classified into one of 25 categories. Specifically, the classification index of a class is derived from a combination of five directional attributes and five activity attributes of the sub-block, which represent the intensity and direction of the texture component. In addition, before filtering, geometric transformations such as 90-degree rotation, diagonalization, and verticalization can be applied to the filter coefficients. By taking into account different directions using geometric transformations, a wider variety of block features can be processed by using fewer sets of filter coefficients.

[0115] In addition to the sub-block level, application decisions may also be made at the CTU level. For chroma components, up to eight filters are utilized at the CTU level. Chroma ALF can be enabled only when luma ALF is enabled at that level.

[0116] On the other hand, an adaptive parameter set (APS) is used to convey ALF filter parameters, including a set of filter coefficients. As described above, up to 25 sets of filter coefficients can be calculated for luma components, and up to 8 sets of filter coefficients can be calculated for chroma components. If the same ALF coefficients are used for different slices, the index of the reference APS can be signaled instead of redundantly resending the same ALF coefficients.

[0117] III. Cross-Component ALF (CC-ALF)

[0118] In video applications such as High-Dynamic Range (HDR) and Wide Color Gamut (WCG), the reconstruction of video color is crucial. The Cross-Component Adaptive Loop Filter (CC-ALF) modifies the chroma samples in parallel with the ALF by exploiting the correlation between the current chroma sample and the corresponding luma sample.

[0119] Figure 7 is a schematic diagram showing the application of cross-component ALF.

[0120] like Figure 7 As shown, in order to generate correction values ​​of chrominance samples according to the input luma samples, a CC-ALF 702 is provided to perform a linear filtering operation. The linear filtering operation utilizes the luma samples (R Y ()) as input to generate the correlation value (ΔR) with each chrominance sample (i∈{Cb,Cr}) i ()), as shown in Equation 3.

[0121] [Equation 3]

[0122]

[0123] Here, (x, y) is the position of each chroma sample, and (x c ,y c ) is the position of the brightness sample corresponding to (x, y). 0 ,y 0 ) means around (x c ,y c ) filters support offset, and c i (x 0 ,y 0 ) represents the filter coefficient. Si represents the target area to be filtered relative to the brightness component. Then, Figure 7 As shown, the correlation value is used as a correction value to enhance the output of the chrominance ALF566.

[0124] In one example, if Figure 8 As shown, a 3×4 diamond high pass filter may be applied to the luma samples to generate correction values. To generate correction values ​​for each chroma sample, the luma samples that have passed through the SAO filter 564 corresponding to each chroma sample position are utilized.

[0125] The video encoding apparatus may determine four sets of filter coefficients for CC-ALF 702 for each chrominance component. Unlike a typical ALF, the CC-ALF filter has coefficients that are not symmetric, but these coefficients have the following characteristics. First, the sum of the CC-ALF coefficients is zero. Second, the absolute value of the CC-ALF coefficient is 0 or a power of 2.

[0126] The video encoding device signals one of the four groups of each chrominance component on a CTU basis. The CC-ALF filter coefficients may be sent together with the ALF parameters of the APS. In order to use CC-ALF 702 at the sequence level, ALF 566 also needs to be used in the sequence. Similarly, in order to use CC-ALF 702 at the slice or picture level, ALF 566 also needs to be used in the slice or picture.

[0127] On the other hand, considering the filter size of the luma and chroma ALF 566, in order to save the line buffer, the virtual boundaries of luma and chroma are formed four and two rows above the CTU boundary, respectively. When CC-ALF 702 is applied to the 4:2:0 chroma format, there is no difficulty in aligning the luma and chroma line buffers based on the position of the virtual boundary. However, when CC-ALF 702 is applied to the 4:2:2 or 4:4:4 chroma format, the difference in the position of the virtual boundary causes misalignment between the luma and chroma line buffers at the third and fourth rows above the CTU boundary. Therefore, for the 4:2:2 and 4:4:4 chroma formats, CC-ALF 702 is not applied to samples in the third and fourth rows above the CTU boundary.

[0128] The following embodiments are described with reference to a video decoding apparatus, but may be implemented in a video encoding apparatus in the same or similar manner.

[0129] IV. Embodiments according to the present invention

[0130] Cross-component Linear Model prediction (CCLM) is a technique for predicting chrominance samples by identifying the correlation between luminance and chrominance components.

[0131] When the CCLM mode is applied to the intra-frame prediction of the current chroma block, the video decoding device determines the area in the luma image corresponding to the current chroma block, wherein such luma image is referred to as the "corresponding luma area" hereinafter. The prediction used for the linear model may be the left reference pixel and the upper reference pixel of the corresponding luma area, as well as the left reference pixel and the upper reference pixel of the target chroma block. In the following, the left reference pixel and the upper reference pixel are generally referred to as reference pixels, adjacent pixels or neighboring pixels. In addition, the reference pixels in the chroma channel are denoted as chroma reference pixels, and the reference pixels in the luma channel are denoted as luma reference pixels.

[0132] In CCLM prediction, a linear model between reference pixels in the corresponding luminance region and reference pixels in the chrominance block is derived, and then a prediction block of a prediction value of a target chrominance block is generated by applying the linear model to the reconstructed pixels in the corresponding luminance region. For example, the linear model parameters α and β can be derived from samples of adjacent rows in the current block by a linear minimum mean square error (LMMSE) method. Alternatively, the derivation for the linear model can be four pairs of pixels from pixels in adjacent pixel rows of the current chrominance block combined with pixels in the corresponding luminance region. The video decoding device can derive α and β representing the linear model based on the four pairs of pixels, as shown in Equation 4.

[0133] [Equation 4]

[0134]

[0135] Here, for the corresponding brightness pixels of the four pairs of pixels, X a and X b represents the average of the two minimum values ​​and the average of the two maximum values, respectively. In addition, for chrominance pixels, Y a and Y b The video decoding device can then use a linear model to calculate the pixel value rec' of the corresponding brightness area. L (i, j) generates the predicted value pred of the current chrominance block C (i, j), as shown in Equation 5.

[0136] [Equation 5]

[0137] pred C(i,j)=α·rec′ L (i,j)+β

[0138] As described above, the CCLM mode is divided into three modes, CCLM_LT, CCLM_L, and CCLM_T, according to the positions of the neighboring pixels used in the derivation of the linear model. The CCLM_LT mode utilizes two pixels from the neighboring pixels adjacent to the left and upper sides of the current chroma block in each direction. The CCLM_L mode utilizes four pixels from the neighboring pixels adjacent to the left side of the current chroma block. Finally, the CCLM_T mode utilizes four pixels from the neighboring pixels adjacent to the upper side of the current chroma block.

[0139] As mentioned above, CCLM prediction assumes a linear correlation between the luminance component and the chrominance component. The present invention describes the correlation between the luminance component and the chrominance component, where a nonlinear model is added in addition to the linear model.

[0140] In addition to the linear model, an example of using a nonlinear model for predicting chrominance samples by using luma samples can be expressed as shown in Equation 6.

[0141] [Equation 6]

[0142] pred C (i,j)=α 0 ·(((rec′ L (i,j)) 2 +midValue)>>bitDepth)+α 1 ·rec′ L (i,j)+α 2 midValue

[0143] Here, the parameter α of the nonlinear model 0 , α 1 and α 2 It can be derived from adjacent reconstructed samples. For example, for a video using 10 bits, the bit depth (bitDepth) is set to 10 and the midValue is set to 512.

[0144] As mentioned above, Figure 7 As shown, the CC-ALF 702 performs filtering by deriving a correction value ΔR based on the correlation between the luma component and the chroma component, and by adding the derived correction value to the output of the conventional chroma ALF 566 .

[0145] CC-ALF 702 corrects the chroma samples in parallel with the ALF by exploiting the correlation between the current chroma sample and the luma sample at that position. Figure 7As shown in FIG. 7 , in order to generate correction values ​​of chrominance samples according to the input luminance samples, CC-ALF 702 performs a linear filtering operation. The linear filtering operation utilizes the luminance samples (R Y ()) as input to generate the correlation value (ΔR) with each chrominance sample (i∈{Cb,Cr}) i ()), as shown in Equation 7.

[0146] [Equation 7]

[0147]

[0148] Here, (x, y) is the position of each chroma sample, and (x c ,y c ) is the position of the brightness sample corresponding to (x, y). 0 ,y 0 ) means around (x c ,y c ) filters support offset, and c i (x 0 ,y 0 ) represents the filter coefficient. S i Indicates the target area to be filtered in the luma component.

[0149] In order to effectively remove artifacts, the nonlinear CC-ALF according to the present embodiment utilizes nonlinear modeling (NM) of the luminance component to derive a correction value ΔR, as shown in Equation 8.

[0150] [Equation 8]

[0151]

[0152] Here, R Y represents the reconstructed sample of the luma component. (x, y) is the position of each chroma sample, and (x c ,y c ) is the position of the brightness sample corresponding to (x, y). 0 ,y 0 ) means around (x c ,y c ) filters support offset, and c i (x 0 ,y 0 ) represents the filter coefficient. S i Indicates the target area to be filtered in the luma component.

[0153] The nonlinear CC-ALF generates a nonlinear modeling value of the luminance samples using a nonlinear model for the luminance samples within the target area to be filtered. The nonlinear CC-ALF can then generate a correction value ΔR based on the product of the nonlinear modeling value and the filter coefficient of the nonlinear CC-ALF.

[0154] Nonlinear modeling can be performed by one of the following methods.

[0155] In one example, nonlinear modeling can utilize a polynomial model, as shown in Equation 9.

[0156] [Equation 9]

[0157] NM=α n ·(R Y (x C +x 0 ,y C +y 0 )) n +α n-1 ·(R Y (x C +x 0 ,y C +y 0 )) n-1 +…+α 1 ·(R Y (x C +x 0 ,y C +y 0 ))+α 0

[0158] As another example, the Michaelis-Menten hyperbolic model may be utilized, as shown in Equation 10.

[0159] [Equation 10]

[0160]

[0161] As another example, a Fourier transform based model may be utilized, as shown in Equation 11.

[0162] [Equation 11]

[0163] NM=α 1 cos(R Y (x C +x 0 ,y C +y 0 )+α 0 )

[0164] As another example, an exponential function model may be utilized, as shown in Equation 12.

[0165] [Equation 12]

[0166]

[0167] In Equation 9 to Equation 12, R Y Represents the reconstructed samples of the luminance component. (x c ,y c ) is the position of the luma sample corresponding to the position (x, y) of each chroma sample. (x, y) 0 ,y 0 ) means around (x c ,y c ) filters support offset. In addition, α j Represents the coefficients of the nonlinear model for the luminance component.

[0168] In one example, by combining the nonlinear model as described above, CC-ALF 702 can be derived as the sum of a linear model and a nonlinear model. In other words, CC-ALF 702 can be a linear model, a nonlinear model, or a combination of a linear model and a nonlinear model.

[0169] Alternatively, nonlinear CC-ALF and linear CC-ALF can be adaptively utilized. For example, in the polynomial-based nonlinear model shown in Equation 9, by replacing α with the linear coefficient 1 and α 0 All values ​​except become zero, and the nonlinear CC-ALF can be converted to linear CC-ALF.

[0170] The nonlinear CC-ALF, like the conventional CC-ALF, can have a diamond-shaped target area to be filtered of size 3×4.

[0171] Therefore, when nonlinear CC-ALF is applied to 4:2:2 or 4:4:4 chroma formats, due to the difference in the position of the virtual boundary, the luma and chroma line buffers are not aligned with each other in the third and fourth rows above the CTU boundary. Therefore, for 4:2:2 and 4:4:4 chroma formats, nonlinear CC-ALF may not be applied to samples in the third and fourth rows above the CTU boundary.

[0172] In order to indicate whether non-linear CC-ALF is applied, the video encoding device may signal a flag (hereinafter referred to as "non-linear CC-ALF-flag") to the video decoding device. The application or non-application of non-linear CC-ALF may be determined at the sequence, picture, sub-picture, slice, tile and / or coding tree unit (CTU) level.

[0173] To use the non - linear CC - ALF at a specific application level, the linear CC - ALF 702 and ALF 566 are also required to be used at that level. For example, to use the non - linear CC - ALF at the slice level, ALF 566 is also required to be used at that slice level. In terms of flags, if the flag indicating whether to apply ALF 566 (hereinafter referred to as the "ALF flag") and the flag indicating whether to apply the linear CC - ALF 702 (hereinafter referred to as the "CC - ALF flag") are both true, then the non - linear CC - ALF flag can be signaled and parsed. Therefore, if the non - linear CC - ALF flag is true, the non - linear CC - ALF can be applied. On the other hand, if the non - linear CC - ALF flag is false, the CC - ALF flag can determine whether to apply the linear CC - ALF.

[0174] The non - linear CC - ALF filter coefficients can be sent in the form of an Adaptive Parameter Set (APS). Alternatively, a fixed filter predefined by the protocol between the video encoding device and the video decoding device can be used. Alternatively, the same filter as the one used by the linear CC - ALF 702 can be used.

[0175] As another example, the video encoding device can determine which filter to use, whether it is a filter configured according to the APS or a fixed filter, and then can signal the determined index and the filter coefficients according to the APS to the video decoding device.

[0176] The video encoding device can also signal the coefficients of the non - linear model of the luminance component as well as the filter coefficients. In this case, if α n is zero, then all α k for n < k can be set to zero.

[0177] Now referring to Fig. 9 and Fig.10 , a method of filtering a reconstructed frame by using the non - linear CC - ALF is described.

[0178] Fig. 9 is a flowchart of a method for a video encoding device to filter a reconstructed frame according to at least one embodiment of the present invention.

[0179] Fig. 9 The example of

[0180] shows the case where the ALF flag is set to true.

[0181] The video encoding device obtains a reconstructed frame (S900). Here, the reconstructed frame is the output of the SAO filter 184.

[0181] The video encoding apparatus inputs the reconstructed frame into the ALF to generate an ALF output (S902). Here, the ALF output includes a luma ALF output and a chroma ALF output.

[0182] The video encoding apparatus inputs the luma ALF output into the nonlinear CC-ALF to generate correction values ​​of chroma components (S904).

[0183] The video encoding apparatus generates a nonlinear modeling value of the luminance sample by using a nonlinear model for the luminance sample within the target area to be filtered. Then, the video encoding apparatus can generate a correction value based on the product of the nonlinear modeling value and the filter coefficient of the nonlinear CC-ALF.

[0184] The video encoding apparatus sums the correction value of the chroma component and the chroma ALF output to generate a first enhanced chroma ALF output (S906).

[0185] The video encoding apparatus inputs the luma ALF output into the linear CC-ALF to generate correction values ​​of chroma components (S908).

[0186] The video encoding apparatus may generate a correction value based on a product of a luma sample within a target area to be filtered and a filter coefficient of a linear CC-ALF.

[0187] The video encoding apparatus generates a second enhanced chroma ALF output by summing the correction value of the chroma component and the chroma ALF output (S910).

[0188] Based on the chroma ALF output, the first enhanced chroma ALF output, and the second enhanced chroma ALF output, the video encoding apparatus determines a CC-ALF flag (S912). Here, the CC-ALF flag indicates whether linear CC-ALF is applied.

[0189] In terms of rate-distortion optimization, the video encoding device may determine the CC-ALF flag. For example, if the chroma ALF output is optimal, the video encoding device may set the CC-ALF flag to false. On the other hand, if the first enhanced chroma ALF output or the second enhanced chroma ALF output is optimal, the video encoding device may set the CC-ALF flag to true.

[0190] The video encoding apparatus encodes the CC-ALF flag (S914).

[0191] The video encoding apparatus checks the CC-ALF flag (S916).

[0192] If the CC-ALF flag is true (Yes in S916 ), the video encoding apparatus determines a non-linear CC-ALF flag based on the first enhancement chroma ALF output and the second enhancement chroma ALF output ( S918 ).

[0193] In terms of rate-distortion optimization, the video encoding device may determine a nonlinear CC-ALF flag. For example, if the first enhanced chroma ALF output is optimal, the video encoding device may set the nonlinear CC-ALF flag to true. On the other hand, if the second enhanced chroma ALF output is optimal, the video encoding device may set the nonlinear CC-ALF flag to false.

[0194] The video encoding apparatus encodes a non-linear CC-ALF flag (S920).

[0195] If the CC-ALF flag is false (No in S916 ), the video encoding apparatus may omit the steps of determining the non-linear CC-ALF flag and encoding the non-linear CC-ALF flag ( S918 and S920 ).

[0196] Fig.10 is a flowchart of a method for filtering a reconstructed frame by a video decoding device according to at least one embodiment of the present invention.

[0197] Fig.10 The example of depicts the case where both the ALF flag and the CC-ALF flag are true.

[0198] The video decoding apparatus obtains a reconstructed frame ( S1000 ). Here, the reconstructed frame is the output of the SAO filter 564 .

[0199] The video decoding apparatus inputs the reconstructed frame into the ALF to generate an ALF output (S1002). Here, the ALF output includes a luma ALF output and a chroma ALF output.

[0200] The video decoding apparatus decodes a non-linear CC-ALF flag from the bitstream (S1004). Here, the non-linear CC-ALF flag indicates whether the non-linear CC-ALF is enabled or disabled.

[0201] The video decoding apparatus checks the non-linear ALF flag (S1006).

[0202] If the non-linear ALF flag is true (Yes in S1006 ), the video decoding apparatus performs the following steps ( S1008 and S1010 ).

[0203] The video decoding apparatus inputs the luma ALF output into the nonlinear CC-ALF to generate correction values ​​of chroma components (S1008).

[0204] The video decoding apparatus generates a nonlinear modeling value of the luminance sample by using a nonlinear model for the luminance sample within the target area to be filtered. Then, the video decoding apparatus can generate a correction value based on the product of the nonlinear modeling value and the filter coefficient of the nonlinear CC-ALF.

[0205] The video decoding apparatus sums the correction value of the chroma component and the chroma ALF output to generate an enhanced chroma ALF output (S1010).

[0206] On the other hand, if the non-linear ALF flag is false (No in S1006), the video decoding apparatus inputs the luma ALF output into the linear CC-ALF to generate correction values ​​of the chroma components (S1020).

[0207] The video decoding apparatus may generate a correction value based on the product of the luma sample within the target area to be filtered and the filter coefficient of the linear CC-ALF 702 .

[0208] Although the steps in the various flowcharts are described as being performed sequentially, these steps merely illustrate the technical ideas of some embodiments of the present invention. Therefore, a person of ordinary skill in the art to which the present invention belongs can perform the steps by changing the order described in the various figures or by performing two or more steps in parallel. Therefore, the steps in the various flowcharts are not limited to the order shown in the order in which they occur.

[0209] It should be understood that the above description presents illustrative embodiments that can be implemented in various other ways. The functions described in some embodiments can be implemented by hardware, software, firmware and / or their combination. It should also be understood that the functional components described in the present invention are marked as "... units" to highlight the possibility of their independent implementation.

[0210] On the other hand, the various methods or functions described in some embodiments can be implemented as instructions stored in a non-volatile recording medium, which can be read and executed by one or more processors. Non-volatile recording media can include various types of recording devices that store data in a form that is readable by a computer system. For example, non-volatile recording media can include storage media, such as erasable programmable read-only memory (EPROM), flash drives, optical disk drives, magnetic hard drives, and solid-state drives (SSDs), etc.

[0211] Although embodiments of the present invention are described for illustrative purposes, it will be appreciated by those skilled in the art that various modifications, additions and substitutions may be made without departing from the spirit and scope of the present invention. Therefore, embodiments of the present invention are described for simplicity and clarity. The scope of the technical ideas of embodiments of the present invention is not limited by illustration. Accordingly, it will be appreciated by those skilled in the art that the scope of the present invention should not be limited by the embodiments clearly described above, but by the claims and their equivalents.

[0212] Reference numerals

[0213] 180: Loop filter unit

[0214] 186:ALF

[0215] 560: Loop filter unit

[0216] 566:ALF

[0217] 702: CC-ALF

[0218] CROSS-REFERENCE TO RELATED APPLICATIONS

[0219] This application claims priority to and the benefit of Korean Patent Application No. 10-2022-0103224, filed on August 18, 2022, and Korean Patent Application No. 10-2023-0093234, filed on July 18, 2023, which are hereby incorporated by reference in their entirety.

Claims

1. A method for filtering a reconstructed frame by a video decoding device, the method comprising: Obtaining a reconstructed frame, the reconstructed frame being an output of a sample adaptive offset (SAO) filter; generating an adaptive loop filter output (ALF output) by inputting the reconstructed frame into an adaptive loop filter (ALF), the ALF output comprising a luma ALF output and a chroma ALF output; Generating correction values ​​for chrominance components by inputting the luma ALF output into a nonlinear cross-component ALF (nonlinear CC-ALF); as well as An enhanced chroma ALF output is generated by summing the correction values ​​for the chroma components and the chroma ALF output.

2. The method according to claim 1, further comprising: decoding a non-linear CC-ALF flag from the bitstream, the non-linear CC-ALF flag indicating whether non-linear CC-ALF is enabled or disabled; as well as Check the non-linear CC-ALF flag, Among them, when the non-linear CC-ALF flag is true, the correction values ​​of the chroma components continue to be generated and the enhanced chroma ALF output is generated.

3. The method according to claim 2, further comprising, when the non-linear CC-ALF flag is false, Corrected values ​​for the chrominance components are generated by inputting the luma ALF output into a linear cross-component ALF (linear CC-ALF).

4. The method according to claim 1, wherein: The non-linear CC-ALF is applied at a common level with linear CC-ALF and ALF, which is the level of sequence, picture, sub-picture, slice, tile and / or coding tree unit (CTU).

5. The method according to claim 1, wherein: Generating correction values ​​for chrominance components involves: Generates corrected values ​​for the chrominance components, generating nonlinear modeled values ​​of the luminance samples by applying a nonlinear model to the luminance samples within the target region to be filtered; and The correction value is generated based on the product of the nonlinear modeling value and the filter coefficient of the nonlinear CC-ALF.

6. The method according to claim 5, wherein: The nonlinear model includes: Polynomial models, hyperbolic models, Fourier transform-based models, or exponential function models.

7. The method according to claim 5, wherein: The nonlinear CC-ALF has a diamond-shaped brightness area of ​​size 3×4 as a target area to be filtered.

8. The method according to claim 1, wherein: When applied to 4:2:2 and 4:4:4 chroma formats, non-linear CC-ALF is not applied to samples in the third and fourth rows above the coding tree unit (CTU) boundary.

9. The method according to claim 1, wherein: The nonlinear CC-ALF has filter coefficients that are transmitted from a video encoding apparatus in the form of an adaptive parameter set (APS) or are predefined values.

10. The method according to claim 5, further comprising: The coefficients of the nonlinear model are decoded from the bitstream.

11. The method according to claim 10, wherein: Generating correction values ​​for chrominance components involves: The nonlinear CC-ALF and the linear CC-ALF are adaptively utilized, but the nonlinear CC-ALF is transformed into the linear CC-ALF by setting the coefficients of the nonlinear model that are not related to the linear CC-ALF to zero.

12. A method for filtering a reconstructed frame by a video encoding device, the method comprising: Obtaining a reconstructed frame, the reconstructed frame being an output of a sample adaptive offset (SAO) filter; generating an adaptive loop filter output (ALF output) by inputting the reconstructed frame into an adaptive loop filter (ALF), the ALF output comprising a luma ALF output and a chroma ALF output; Generating correction values ​​for chrominance components by inputting the luma ALF output into a nonlinear cross-component ALF (nonlinear CC-ALF); as well as A first enhanced chroma ALF output is generated by summing the correction values ​​for the chroma components and the chroma ALF output.

13. The method according to claim 12, further comprising: Generating correction values ​​for chrominance components by inputting the luma ALF output into a linear cross-component ALF (linear CC-ALF); as well as A second enhanced chroma ALF output is generated by summing the correction values ​​for the chroma components and the chroma ALF output.

14. The method according to claim 13, further comprising: determining a CC-ALF flag indicating whether to apply a linear CC-ALF based on the chroma ALF output, the first enhanced chroma ALF output, and the second enhanced chroma ALF output; as well as Encode the CC-ALF flag.

15. The method according to claim 14, further comprising: Check the CC-ALF flag, wherein when the CC-ALF flag is true, determining the non-linear CC-ALF flag based on the first enhanced chroma ALF output and the second enhanced chroma ALF output; and Encode the non-linear CC-ALF flag.

16. A computer-readable recording medium storing a bit stream generated by a video encoding method, the video encoding method comprising: Obtaining a reconstructed frame, the reconstructed frame being an output of a sample adaptive offset (SAO) filter; generating an adaptive loop filter output (ALF output) by inputting the reconstructed frame into an adaptive loop filter (ALF), the ALF output comprising a luma ALF output and a chroma ALF output; Generating correction values ​​for chrominance components by inputting the luma ALF output into a nonlinear cross-component ALF (nonlinear CC-ALF); as well as An enhanced chroma ALF output is generated by summing the correction values ​​for the chroma components and the chroma ALF output.

Images