AI-based video decoding device and method for performing chroma component prediction, and video encoding device and method

Abstract

The invention provides an AI-based video decoding method and device. The method comprises the following steps: obtaining a joint chroma residual sample of a current block, Cb component prediction information of the current block and Cr component prediction information of the current block from a bit stream; determining a prediction sample of the Cb component of the current block at least based on the Cb component prediction information; determining a prediction sample of the Cr component of the current block at least based on the Cr component prediction information; and inputting the joint chrominance residual sample, the prediction sample of the Cb component and the prediction sample of the Cr component into a neural network, and obtaining a reconstructed sample of the Cb component of the current block and a reconstructed sample of the Cr component of the current block from the output of the neural network so as to reconstruct the current block.

Description

Technical Field

[0001] The present disclosure relates to a video decoding method and apparatus and a video encoding method and apparatus, and more particularly, to a method and apparatus for performing prediction on a Cb component and a Cr component, which are chrominance components of an image, based on artificial intelligence (AI). Background Art

[0002] A codec such as H.266 Versatile Video Codec (VVC) jointly encodes and decodes residual samples of the Cb component and the Cr component via joint encoding and decoding by using the similarity or relationship between the Cb component and the Cr component, thereby saving the bit rate required to encode and decode each chrominance component separately.

[0003] The relationship between the Cb component and the Cr component is adaptively selected based on a determined number of cases so that joint chroma residual samples for the Cb component and the Cr component are encoded and decoded.

[0004] Although rule-based chrominance component prediction algorithms have demonstrated good performance, with the increase in image resolution and the diversification of image content, artificial intelligence (AI)-based chrominance component prediction algorithms that can flexibly consider image characteristics are needed. Summary of the invention

[0005] According to an embodiment of the present disclosure, a video decoding method based on artificial intelligence (AI) includes: obtaining a joint chroma residual sample of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block from a bit stream, wherein the current block includes a Cb component and a Cr component; determining a prediction sample of the Cb component of the current block based at least on the Cb component prediction information; determining a prediction sample of the Cr component of the current block based at least on the Cr component prediction information; and reconstructing the current block by inputting the joint chroma residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component into a neural network, and obtaining a reconstructed sample of the Cb component of the current block and a reconstructed sample of the Cr component of the current block from the output of the neural network.

[0006] According to an embodiment of the present disclosure, a video decoding device based on artificial intelligence (AI) includes: a memory storing at least one instruction; and at least one processor configured to operate according to at least one instruction. The at least one processor is also configured to obtain a joint chroma residual sample of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block from a bitstream, and the current block includes a Cb component and a Cr component. The at least one processor is also configured to determine a predicted sample of the Cb component of the current block based at least on the Cb component prediction information. The at least one processor is also configured to determine a predicted sample of the Cr component of the current block based at least on the Cr component prediction information. The at least one processor is also configured to reconstruct the current block by inputting the joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component into a neural network, and obtaining a reconstructed sample of the Cb component of the current block and a reconstructed sample of the Cr component of the current block from the output of the neural network.

[0007] According to an embodiment of the present disclosure, a video encoding method based on artificial intelligence (AI) includes: generating initial joint chroma residual samples of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block; determining the prediction samples of the Cb component of the current block based at least on the Cb component prediction information, and determining the prediction samples of the Cr component of the current block based at least on the Cr component prediction information; and encoding the joint chroma residual samples of the current block by inputting the initial joint chroma residual samples, the prediction samples of the Cb component, and the prediction samples of the Cr component into a neural network, and obtaining reconstructed samples of the Cb component of the current block and reconstructed samples of the Cr component of the current block from the output of the neural network.

[0008] According to an embodiment of the present disclosure, an AI-based video encoding device may include: a memory storing at least one instruction; and at least one processor configured to operate according to at least one instruction. At least one processor may be configured to generate an initial joint chroma residual sample of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block. At least one processor is also configured to determine the predicted sample of the Cb component of the current block based at least on the Cb component prediction information. At least one processor is also configured to determine the predicted sample of the Cr component of the current block based at least on the Cr component prediction information. At least one processor is also configured to encode the joint chroma residual sample of the current block by inputting the initial joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component into a neural network, and obtaining the reconstructed sample of the Cb component of the current block and the reconstructed sample of the Cr component of the current block from the output of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which: Figure 1 A diagram for describing the relationship between a Cb component, a Cr component, and a joint chroma residual according to an embodiment of the present disclosure is shown; Figure 2 A diagram for describing a relational expression between a Cb component, a Cr component, and a joint chroma residual according to an embodiment of the present disclosure is shown; Figure 3 A diagram for describing a neural network for artificial intelligence (AI)-based chrominance component prediction according to an embodiment of the present disclosure is shown; Figure 4 A diagram for describing a neural network for AI-based chrominance component prediction according to an embodiment of the present disclosure is shown; Figure 5 A diagram for describing a method of training a neural network for AI-based chrominance component prediction according to an embodiment of the present disclosure is shown; Figure 6 A diagram for describing a method of training a neural network for AI-based chrominance component prediction according to an embodiment of the present disclosure is shown; Figure 7 A table showing correlation between a neural network model for AI-based chrominance component prediction and a prediction type according to an embodiment of the present disclosure is shown; Figure 8 A table showing correlation between a neural network model for AI-based chrominance component prediction and a prediction type according to an embodiment of the present disclosure is shown; Fig. 9 A diagram for describing a neural network and a scaling factor for AI-based chrominance component prediction according to an embodiment of the present disclosure is shown; Fig.10 shows a table showing the correlation between scale factor sets and prediction types according to an embodiment of the present disclosure; Fig.11 A flowchart of an AI-based video decoding method according to an embodiment of the present disclosure is shown; Fig.12 A block diagram showing a configuration of an AI-based video decoding device according to an embodiment of the present disclosure; Fig.13 A block diagram showing a configuration of an AI-based video decoding device according to an embodiment of the present disclosure; Fig.14 A flowchart of an AI-based video encoding method according to an embodiment of the present disclosure is shown; Fig.15 A block diagram showing a configuration of an AI-based video encoding device according to an embodiment of the present disclosure; and Fig.16 A block diagram showing the configuration of an AI-based video encoding device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0010] Since the present disclosure allows various changes and many embodiments, specific embodiments will be shown in the drawings and described in detail in the written description. However, this is not intended to limit the present disclosure to a specific mode of practice, and it will be understood that all changes, equivalents and alternatives that do not depart from the spirit and technical scope of the present disclosure are included in the present disclosure.

[0011] In the description of the embodiments of the present disclosure, when it is considered that the detailed explanation of the related art may unnecessarily obscure the essence of the present disclosure, the detailed explanation of the related art is omitted. In addition, the numbers (e.g., "first", "second", etc.) in the description of the specification are only used to distinguish one element from another element.

[0012] Throughout the disclosure, the expression "at least one of a, b, or c" means only a, only b, only c, a and b, a and c, b and c, and all or variations of a, b, and c.

[0013] In the present disclosure, it will also be understood that when an element is referred to as being “connected to” or “coupled with” another element, the element may be directly connected to or coupled with the other element, or may be indirectly connected to or coupled with the other element by interposing an intermediate element between the element and the other element.

[0014] In the present disclosure, with respect to an element represented as a "unit" or a "module", two or more elements may be combined into one element, or one element may be divided into two or more elements according to subdivided functions. In addition, each element described below may perform some or all of the functions performed by another element in addition to its own main function, and some of the main functions of each element may be completely performed by another element.

[0015] Furthermore, in the present disclosure, an “image” or a “picture” may refer to a still image, a moving image including a plurality of consecutive still images (or frames), or a video.

[0016] In addition, in the present disclosure, "neural network" is a representative example of an artificial neural network model that simulates brain nerves, and is not limited to an artificial neural network model that uses a specific algorithm. The neural network may be referred to as a deep neural network.

[0017] In addition, in the present disclosure, "weight" is a value used in the operation process of each layer constituting a neural network, and, for example, the weight can be used when an input value is applied to a predetermined operation expression. The weight is a value as a training result, and the weight can be updated by using separate training data.

[0018] In addition, in the present disclosure, a "current block" refers to a block that is a current target to be processed. The current block may be a slice, a tile, a maximum coding unit, a coding unit, a prediction unit, or a transform unit partitioned from a current image.

[0019] In addition, in the present disclosure, "sample" is data assigned to a sampling position in data of an image, a block, a filter kernel, a feature map, etc., and refers to data to be processed. For example, a sample may include a pixel in a two-dimensional image.

[0020] In the following, reference is made to Figure 1 and Figure 2 , a method for predicting chrominance components without using a neural network will be described.

[0021] Figure 1 A diagram for describing the relationship between a Cb component, a Cr component, and a joint chroma residual according to an embodiment of the present disclosure is shown.

[0022] Reference Figure 1 , a graph 100 shows a relationship between an actual value 140 of a residual sample of a Cr component, an actual value 110 of a residual sample of a Cb component, and a joint chroma residual sample 120 during a video encoding process. The joint chroma residual sample 120 may be determined as an average value between a value 130 obtained by multiplying -1 and the actual value 140 of the residual sample of the Cr component and the actual value 110 of the residual sample of the Cb component.

[0023] The graph 150 shows an example of a residual sample 160 of a Cb component and a residual sample 170 of a Cr component, which can be obtained by using the joint chroma residual sample 140 in a video decoding process. In detail, the same value as the joint chroma residual sample 140 may be determined as a reconstructed value of the residual sample 160 of the Cb component, and a value generated by multiplying the joint chroma residual sample 140 by -1 may be determined as a reconstructed value of the residual sample 170 of the Cr component.

[0024] Figure 2 A diagram for describing a relational expression among a Cb component, a Cr component, and a joint chroma residual according to an embodiment of the present disclosure is shown.

[0025] Reference Figure 2, tu_cbf_cb may be a syntax element corresponding to Cb coded block flag (cbf) information, and may indicate whether the current transform block includes a Cb component whose transform coefficient level is not zero. The parameter tu_cbf_cr may be a syntax element corresponding to Cr cbf information, and may indicate whether the current transform block includes a Cr component whose transform coefficient level is not zero. The parameter tu_joint_cbcr_residual_flag may be a syntax element corresponding to joint chroma information, and may indicate whether one chroma sample is encoded to indicate a residual sample of a Cb component in the current transform block and a residual sample of a Cr component corresponding thereto.

[0026] In the video decoding process, sps_joint_cbcr_enabled_flag may be obtained from a sequence parameter set. When sps_joint_cbcr_enabled_flag indicates 1, it indicates that a joint chroma scheme is enabled, in which one chroma sample is encoded to indicate a residual sample of a Cb component and a residual sample of a Cr component corresponding thereto in a block included in the current sequence.

[0027] Therefore, tu_joint_cbcr_residual_flag as the joint chroma information may be obtained in the following cases: first, the joint chroma scheme is enabled via sps_joint_cbcr_enabled_flag in the current sequence, the prediction type of the coding unit including the current transform block is the intra prediction type, and the Cr cbf information or the Cb cbf information indicates 1. In addition, in the case where the prediction type of the coding unit including the current transform block is the inter prediction type, only when both the Cr cbf information and the Cbcbf information indicate 1, the parameter tu_joint_cbcr_residual_flag as the joint chroma information may be obtained.

[0028] Therefore, when the prediction type of the coding unit is an intra-frame prediction type, and the current block includes a Cr component with a transform coefficient level of 1 or includes a Cb component with a transform coefficient level of 1, it can be determined via joint chrominance information whether to encode one chrominance sample to indicate a residual sample of the Cb component and a residual sample of the Cr component corresponding thereto. When the prediction type of the coding unit is an inter-frame prediction type, and the current block includes a Cr component with a transform coefficient level of 1 and includes a Cb component with a transform coefficient level of 1, it can be determined via joint chrominance information whether to encode one chrominance sample to indicate a residual sample of the Cb component and a residual sample of the Cr component corresponding thereto. Here, when the obtained joint chrominance information indicates 0, it means that each of the residual sample of the Cb component and the residual sample of the Cr component corresponding thereto can be obtained from the bitstream. On the other hand, when the joint chrominance information indicates 1, it means that one chrominance sample is obtained from the bitstream, and the residual sample of the Cb component and the residual sample of the Cr component corresponding thereto can be reconstructed from the one chrominance sample.

[0029] When Cb cbf information (tu_cbf_cb) and Cr cbf information (tu_cbf_cr) are obtained, weights for reconstructed values ​​of residual samples of the Cb component (resCb) and reconstructed values ​​of residual samples of the Cr component (resCr) may be determined according to the Cb cbf information and the Cr cbf information.

[0030] For example, when tu_cbf_cb is 1 and tu_cbf_cr is 0, the reconstructed value (resCb) of the residual sample of the Cb component may be determined as the same value as the joint chroma residual sample (resJointC), and the reconstructed value (resCr) of the residual sample of the Cr component may be determined as a value obtained by multiplying ±1 / 2 by the joint chroma residual sample (resJointC) based on the CSign value obtained from the picture parameter set. In this case, the weight of the Cb component is 1, the weight of the Cr component is ±1 / 2, and the joint mode index is 1.

[0031] For example, when tu_cbf_cb is 1 and tu_cbf_cr is 1, the reconstructed value (resCb) of the residual sample of the Cb component may be determined as the same value as the joint chroma residual sample (resJointC), and the reconstructed value (resCr) of the residual sample of the Cr component may be determined as a value obtained by multiplying the joint chroma residual sample (resJointC) by ±1 based on the CSign value. In this case, the weight of the Cb component is 1, the weight of the Cr component is ±1, and the joint mode index is 2.

[0032] For example, when tu_cbf_cb is 0 and tu_cbf_cr is 1, the reconstructed value (resCb) of the residual sample of the Cb component may be determined as a value obtained by multiplying the joint chroma residual sample (resJointC) by ±1 / 2 based on the CSign value, and the reconstructed value (resCr) of the residual sample of the Cr component may be determined as the same value as the joint chroma residual sample (resJointC). In this case, the weight of the Cb component is ±1 / 2, the weight of the Cr component is 1, and the joint mode index is 3.

[0033] When the prediction type of the coding unit is an intra prediction mode, index values ​​1, 2, and 3 may all be used for the joint mode index. For example, when the prediction type of the coding unit is an intra prediction mode, each of the weight of the Cb component and the weight of the Cr component may be determined according to the joint mode index.

[0034] When the prediction type of the coding unit is the inter prediction mode, only the joint mode index 2 may be allowed. Therefore, in the case where the prediction type of the coding unit is the inter prediction mode and the joint mode index is 2 (for example, only when tu_cbf_cb is 1 and tu_cbf_cr is 1), a joint chroma scheme may be allowed, the weight of the Cb component may be determined to be 1, and the weight of the Cr component may be determined to be ±1.

[0035] In the following, reference is made to Figures 3 to 16 , a method for predicting chrominance components based on artificial intelligence (AI) by using a neural network will now be described.

[0036] Figure 3 A diagram is shown for describing a neural network for AI-based chrominance component prediction according to an embodiment of the present disclosure.

[0037] like Figure 3 As shown, the neural network 340 may include a first convolution layer 341, a first activation layer 342, a second convolution layer 343, a second activation layer 344, a third convolution layer, a third activation layer, a fourth convolution layer, a fourth activation layer, a fifth convolution layer, a fifth activation layer, a sixth convolution layer, a sixth activation layer, a seventh convolution layer 345, a seventh activation layer 346, an eighth convolution layer 347, an eighth activation layer 348, a ninth convolution layer 349, and a summation layer 380. The summation layer 380 may be referred to as an adder.

[0038] The prediction sample 310 of the Cb component of the current block, the joint chroma residual sample 320 of the current block, and the prediction sample 330 of the Cr component of the current block are input to the first convolution layer 341. The prediction sample 310 of the Cb component of the current block, the joint chroma residual sample 320 of the current block, and the prediction sample 330 of the Cr component of the current block may be concatenated as one data item input to the first convolution layer 341.

[0039] According to the diagram, the first convolution layer 341 is marked with 3×3, which means that a convolution process is performed on the prediction sample 310 of the Cb component of the current block, the joint chrominance residual sample 320 of the current block, and the prediction sample 330 of the Cr component of the current block by using a filter kernel of size 3×3. In an embodiment of the present disclosure, there may be 16 channels 350, such as Figure 3 Therefore, according to the convolution processing result, the feature map of 16 channels 350 can be generated by 16 filter kernels, each filter kernel having a size of 3×3.

[0040] The feature map generated by the first convolutional layer 341 may indicate unique features of the prediction sample 310 of the Cb component of the current block, the joint chroma residual sample 320 of the current block, and the prediction sample 330 of the Cr component of the current block. For example, the feature map may indicate vertical direction features, horizontal direction features, or edge features of the prediction sample 310 of the Cb component of the current block, the joint chroma residual sample 320 of the current block, and the prediction sample 330 of the Cr component of the current block. In an embodiment of the present disclosure, a feature map may be generated for each of the prediction sample 310 of the Cb component of the current block, the joint chroma residual sample 320 of the current block, and the prediction sample 330 of the Cr component of the current block.

[0041] The feature map output from the first convolutional layer 341 is input to the first activation layer 342.

[0042] The first activation layer 342 may apply a nonlinear feature to the feature map. The nonlinear feature may be a feature that exhibits one or more nonlinear characteristics. The first activation layer 342 may include a sigmoid function, a Tanh function, and a rectified linear unit (ReLU) function, or any other suitable function known to those of ordinary skill in the art.

[0043] Applying the nonlinear feature to the feature map by the first activation layer 342 may indicate that some sample values ​​of the feature map are changed. Here, the change may be performed by applying the nonlinear feature thereto.

[0044] The first activation layer 342 may determine whether to transfer the sample values ​​of the feature map to the second convolution layer 343. For example, some of the sample values ​​from the feature map may be activated by the first activation layer 342 and may be transferred to the second convolution layer 343, and some sample values ​​may not be activated by the first activation layer 342 and may not be transferred to the second convolution layer 343. Unique features of the prediction sample 310 of the Cb component of the current block, the joint chroma residual sample 320 of the current block, and the prediction sample 330 of the Cr component of the current block indicated by the feature map may be emphasized by the first activation layer 342.

[0045] The feature map of 16 channels 350 output from the first activation layer 342 is output to the second convolutional layer 343.

[0046] According to the diagram, the second convolution layer 343 is marked with 3×3, which may indicate that a convolution process is performed on the input feature map by using a filter kernel of size 3×3. The output from the second convolution layer 343 is input to the second activation layer 344. The second activation layer 344 may apply nonlinear features to the input feature map.

[0047] The feature map output from the second activation layer 344 passes through repeated operations of the convolution layer and the activation layer, and is sequentially input to and output from the third convolution layer, the third activation layer, the fourth convolution layer, the fourth activation layer, the fifth convolution layer, the fifth activation layer, the sixth convolution layer, and the sixth activation layer. The output from the sixth activation layer is input to the seventh convolution layer 345.

[0048] According to the diagram, the seventh convolution layer 345 is marked with 3×3, which may indicate that a convolution process is performed on the input feature map by using a filter kernel of size 3×3. The output from the seventh convolution layer 345 is input to the seventh activation layer 346. The feature map of 16 channels 355 output from the seventh activation layer 346 is input to the eighth convolution layer 347, and the output from the eighth convolution layer 347 is input to the eighth activation layer 348.

[0049] The output from the eighth activation layer 348 is input to the ninth convolution layer 349. The ninth convolution layer 349 outputs two feature maps by using two filter kernels each having a size of 3×3.

[0050] The two feature maps output from the ninth convolutional layer 349 are summed with the predicted samples 310 of the Cb component of the current block and the predicted samples 330 of the Cr component of the current block at the summing layer 380, respectively, and as a result of the summing, reconstructed samples 360 of the Cb component of the current block and reconstructed samples 370 of the Cr component of the current block are output.

[0051] Since the two feature maps output from the ninth convolutional layer 349 are respectively summed with the predicted samples 310 of the Cb component of the current block and the predicted samples 330 of the Cr component of the current block, the reconstructed samples 360 of the Cb component of the current block and the reconstructed samples 370 of the Cr component of the current block are output, and the two feature maps output from the ninth convolutional layer 349 can be respectively referred to as the residual samples of the Cb component of the current block and the residual samples of the Cr component of the current block.

[0052] Although Figure 3 The neural network 340 is shown to include 9 convolutional layers and 8 activation layers, but this is merely an example, and according to an embodiment of the present disclosure, the number of convolutional layers and activation layers included in the neural network 340 may vary.

[0053] According to an embodiment of the present disclosure, the neural network 340 may be implemented via a recurrent neural network (RNN). For example, a convolutional neural network (CNN) structure of the neural network 340 according to an embodiment of the present disclosure may be changed to an RNN structure.

[0054] because Figure 3 The neural network 340 includes a summation layer 380, so the reconstructed sample 360 ​​of the Cb component of the current block and the reconstructed sample 370 of the Cr component of the current block can be obtained from the neural network 340. Figure 4 A neural network is described that does not include summation layer 380.

[0055] Figure 4 A diagram is shown for describing a neural network for AI-based chrominance component prediction according to an embodiment of the present disclosure.

[0056] like Figure 4 As shown, the neural network 440 may include a first convolutional layer 441, a first activation layer 442, a second convolutional layer 443, a second activation layer 444, a third convolutional layer, a third activation layer, a fourth convolutional layer, a fourth activation layer, a fifth convolutional layer, a fifth activation layer, a sixth convolutional layer, a sixth activation layer, a seventh convolutional layer 445, a seventh activation layer 446, an eighth convolutional layer 447, an eighth activation layer 448 and a ninth convolutional layer 449.

[0057] The prediction sample 410 of the Cb component of the current block, the joint chroma residual sample 420 of the current block, and the prediction sample 430 of the Cr component of the current block are input to the first convolution layer 441. The prediction sample 410 of the Cb component of the current block, the joint chroma residual sample 420 of the current block, and the prediction sample 430 of the Cr component of the current block may be concatenated as one data item input to the first convolution layer 441.

[0058] According to the diagram, the first convolution layer 441 is marked with 3×3, which may indicate that a convolution process is performed on the prediction sample 410 of the Cb component of the current block, the joint chrominance residual sample 420 of the current block, and the prediction sample 430 of the Cr component of the current block by using a filter kernel of size 3×3. In an embodiment of the present disclosure, there may be 16 channels, such as Figure 4 Therefore, according to the convolution processing result, the feature map of 16 channels 450 can be generated by 16 filter kernels, each filter kernel having a size of 3×3.

[0059] The feature map generated by the first convolutional layer 441 may indicate unique features of the prediction sample 410 of the Cb component of the current block, the joint chroma residual sample 420 of the current block, and the prediction sample 430 of the Cr component of the current block. For example, the feature map may indicate vertical direction features, horizontal direction features, or edge features of the prediction sample 410 of the Cb component of the current block, the joint chroma residual sample 420 of the current block, and the prediction sample 430 of the Cr component of the current block. In an embodiment of the present disclosure, a feature map may be generated for each of the prediction sample 410 of the Cb component of the current block, the joint chroma residual sample 420 of the current block, and the prediction sample 430 of the Cr component of the current block.

[0060] The feature map output from the first convolutional layer 441 is input to the first activation layer 442.

[0061] The first activation layer 442 may apply a nonlinear feature to the feature map. The nonlinear feature may be a feature that exhibits one or more nonlinear characteristics. The first activation layer 442 may include a sigmoid function, a Tanh function, and a ReLU function, or any other suitable function known to those of ordinary skill in the art.

[0062] Applying the nonlinear feature to the feature map by the first activation layer 442 may indicate that some sample values ​​of the feature map are changed. Here, the change may be performed by applying the nonlinear feature thereto.

[0063] The first activation layer 442 may determine whether to transfer the sample values ​​of the feature map to the second convolution layer 443. For example, some of the sample values ​​from the feature map may be activated by the first activation layer 442 and may be transferred to the second convolution layer 443, and some sample values ​​may not be activated by the first activation layer 442 and may not be transferred to the second convolution layer 443. Unique features of the prediction sample 410 of the Cb component of the current block, the joint chroma residual sample 420 of the current block, and the prediction sample 430 of the Cr component of the current block indicated by the feature map may be emphasized by the first activation layer 442.

[0064] The feature map of 16 channels 450 output from the first activation layer 442 is output to the second convolutional layer 443.

[0065] According to the diagram, the second convolution layer 443 is marked with 3×3, which may indicate that a convolution process is performed on the input feature map by using a filter kernel of size 3×3. The output from the second convolution layer 443 is input to the second activation layer 444. The second activation layer 444 may apply nonlinear features to the input feature map.

[0066] The feature map output from the second activation layer 444 passes through repeated operations of the convolution layer and the activation layer, and is sequentially input to and output from the third convolution layer, the third activation layer, the fourth convolution layer, the fourth activation layer, the fifth convolution layer, the fifth activation layer, the sixth convolution layer, and the sixth activation layer. The output from the sixth activation layer is input to the seventh convolution layer 445.

[0067] According to the diagram, the seventh convolution layer 445 is marked with 3×3, which may indicate that a convolution process is performed on the input feature map by using a filter kernel of size 3×3. The output from the seventh convolution layer 445 is input to the seventh activation layer 446. The feature map of 16 channels 455 output from the seventh activation layer 446 is input to the eighth convolution layer 447, and the output from the eighth convolution layer 447 is input to the eighth activation layer 448.

[0068] The output from the eighth activation layer 448 is input to the ninth convolution layer 449. The ninth convolution layer 449 outputs two feature maps by using two filter kernels each having a size of 3×3.

[0069] The two feature maps output from the ninth convolutional layer 449 may be respectively referred to as residual samples 460 of the Cb component of the current block and residual samples 470 of the Cr component of the current block.

[0070] Although Figure 4 The neural network 440 is shown to include 9 convolutional layers and 8 activation layers, but this is merely an example, and according to an embodiment of the present disclosure, the number of convolutional layers and activation layers included in the neural network 440 may vary.

[0071] In addition, according to an embodiment of the present disclosure, the neural network 440 may be implemented via RNN. For example, the CNN structure of the neural network 440 according to an embodiment of the present disclosure is changed to the RNN structure.

[0072] A residual sample 460 of a Cb component of a current block and a residual sample 470 of a Cr component of a current block may be obtained from the neural network 440 .

[0073] Figure 5 A diagram for describing a method of training a neural network for AI-based chrominance component prediction according to an embodiment of the present disclosure is shown.

[0074] Figure 5 A method of training a neural network 520 for AI-based chrominance component prediction is shown. Figure 5 The prediction samples 505 of the Cb component of the current block for training shown in the figure may correspond to the prediction samples of the Cb component of the current block described above, the joint chroma residual samples 510 of the current block for training may correspond to the joint chroma residual samples of the current block described above, and the prediction samples 515 of the Cr component of the current block for training may correspond to the prediction samples of the Cr component of the current block described above.

[0075] According to the method of training a neural network 520 disclosed in the present invention, according to an embodiment, the neural network 520 is trained so that the reconstructed samples 535 of the Cb component of the current block for training become equal to or similar to the original samples 545 of the Cb component of the current block for training, wherein the reconstructed samples 535 are the result of summing the residual samples 525 of the Cb component of the current block for training output from the neural network 520 and the predicted samples 505 of the Cb component of the current block for training, and the reconstructed samples 540 of the Cr component of the current block for training become equal to or similar to the original samples 550 of the Cr component of the current block for training, wherein the reconstructed samples 540 are the result of summing the residual samples 530 of the Cr component of the current block for training output from the neural network 520 and the predicted samples 515 of the Cr component of the current block for training. For example, loss information 555 corresponding to the sum of the difference between the reconstructed sample 535 of the Cb component of the current block for training and the original sample 545 of the Cb component of the current block for training and the difference between the reconstructed sample 540 of the Cr component of the current block for training and the original sample 550 of the Cr component of the current block for training can be used to train the neural network 520.

[0076] In more detail, first, a prediction sample 505 of the Cb component of the current block for training, a joint chroma residual sample 510 of the current block for training, and a prediction sample 515 of the Cr component of the current block for training are obtained. For example, the prediction sample 505 of the Cb component may be determined based on at least the Cb component prediction information obtained from the bitstream, and the prediction sample 515 of the Cr component of the current block may be determined based on at least the Cr component prediction information obtained from the bitstream.

[0077] The prediction sample 505 of the Cb component of the current block for training, the joint chrominance residual sample 510 of the current block for training, and the prediction sample 515 of the Cr component of the current block for training are input to the neural network 520, and then the residual sample 525 of the Cb component of the current block for training and the residual sample 530 of the Cr component of the current block for training are obtained output from the neural network 520. The neural network 520 can perform operations according to predetermined weights.

[0078] The residual samples 525 of the Cb component of the current block for training are summed with the predicted samples 505 of the Cb component of the current block for training, so as to obtain the reconstructed samples 535 of the Cb component of the current block for training, and the residual samples 530 of the Cr component of the current block for training are summed with the predicted samples 515 of the Cr component of the current block for training, so as to obtain the reconstructed samples 540 of the Cr component of the current block for training.

[0079] Loss information 555 corresponding to the sum of the difference between the reconstructed sample 535 of the Cb component of the current block for training and the original sample 545 of the Cb component of the current block for training and the difference between the reconstructed sample 540 of the Cr component of the current block for training and the original sample 550 of the Cr component of the current block for training is output, and a weight set for the neural network 520 is updated according to the loss information 555. The neural network 520 may update the weights to reduce or minimize the loss information 555.

[0080] Figure 5 The heart shape shown in may correspond to an example object in the current block for training, and the remaining portion may correspond to the background. For the residual sample 525 of the Cb component of the current block for training output from the neural network 520, a weight of 1 may be applied to each of the object 526 and the background 527 of the current block for training, and for the residual sample 530 of the Cr component of the current block for training, a weight of 1 may be applied to the object 531 of the current block for training, and a weight of 1 / 2 may be applied to the background 532. Based on this configuration, by using the predicted samples of the Cb component and the Cr component, the relationship between the Cb component and the Cr component may become more general, more local, and more specific via the trained neural network model. For example, instead of a fixed relationship regarding pixels corresponding to the Cb component and the Cr component, the Cb component and the Cr component may be reconstructed so that different Cb components and Cr components from each other have a relationship for an object, a background, a foreground, etc., or different Cb components and Cr components from each other have a relationship for different parts of an object.

[0081] In an embodiment of the present disclosure, the loss information 555 may include at least one of an L1 norm value, an L2 norm value, a structural similarity (SSIM) value, a peak signal-to-noise ratio-human visual system (PSNR-HVS) value, a multi-scale SSIM (MS-SSIM) value, a variance inflation factor (VIF) value, or a video multi-method assessment fusion (VMAF) value, wherein the at least one value is related to the sum of the difference between the reconstructed sample 535 of the Cb component of the current block used for training and the original sample 545 of the Cb component of the current block used for training and the difference between the reconstructed sample 540 of the Cr component of the current block used for training and the original sample 550 of the Cr component of the current block used for training.

[0082] According to an embodiment of the present disclosure, the training of the neural network 520 may be performed by a training device. The training device may be an AI-based video encoding device 1600 or an AI-based video decoding device 1300. According to an embodiment of the present disclosure, the training device may be an external server. In this case, the neural network 520 and the weights trained by the external server may be sent to the AI-based video encoding device 1600 or the AI-based video decoding device 1300 through a network. In addition, the trained neural network may be downloaded by one or more devices.

[0083] According to an embodiment of the present disclosure, in addition to the prediction samples 505 of the Cb component of the current block for training, the joint chroma residual samples 510 of the current block for training, and the prediction samples 515 of the Cr component of the current block for training, another input may be added to the neural network 520. For example, the added another input may be the quantization step size of the current block (in the case where the current block is an intra block or an inter block), the quantization error of the current block (in the case where the current block is an intra block or an inter block), the quantization step size or quantization step size error of the reference frame of the current block (in the case where the current block is an inter block), a block obtained by downsampling a reconstructed luminance block corresponding to the current chroma block of the current block based on a chroma format, or any other suitable method (for example, when the chroma format is 4:2:0, the block is reduced by a factor of 2).

[0084] The quantization error of the current block may include a quantization error value calculated from the quantization parameter as a sample value. The value may indicate a quantization error for each sample included in the current block. The size of the quantization error map including the quantization error value to be additionally input to the neural network may correspond to the size of the current block.

[0085] The quantization error value may indicate the amount of error that may be caused due to quantization and inverse quantization applied to residual samples during encoding and decoding processes for the original block.

[0086] A large quantization error value may indicate that the difference between the transform coefficient before quantization and the transform coefficient after inverse quantization may be large. The greater the difference between the transform coefficient before quantization and the transform coefficient after inverse quantization increases, the more the similarity between the original block and the current block obtained by decoding the encoded data decreases.

[0087] Because errors due to quantization and dequantization correspond to artifacts, AI-based video encoding and video decoding must be performed by considering the quantization error value.

[0088] In an embodiment of the present disclosure, a quantization error value may be calculated by using Equation 1 below.

[0089] [Equation 1] Quantization error value = quantization step size ^ 2 / 12 Referring to Equation 1, the quantization error value may be proportional to a value obtained by squaring the quantization step size.

[0090] In an embodiment of the present disclosure, the quantization step size is a value used to quantize the transform coefficient, and the transform coefficient can be quantized by dividing the transform coefficient by the quantization step size. On the other hand, the quantized transform coefficient can be dequantized by multiplying the quantized transform coefficient by the quantization step size.

[0091] The quantization step size can be approximated by using Equation 2 below.

[0092] [Equation 2] Quantization step size = 2^(quantization parameter / n) / quantization scale [quantization parameter%n] In Equation 2, quantization scale [quantization parameter % n] indicates a scale value indicated by a quantization parameter among predetermined n scale values. In the High Efficiency Video Coding (HEVC) codec, 6 scale values ​​(26214, 23302, 20560, 18396, 16384, and 14564) are defined, and thus, according to the HEVC codec, n is 6.

[0093] Referring to Equation 1 and Equation 2, the more the quantization parameter increases, the more the quantization step size may increase, and the more the quantization error value may increase.

[0094] The neural network may additionally consider a quantization error or a quantization step size, thereby more effectively removing the quantization error in the current block when reconstructing the current block.

[0095] In an embodiment of the present disclosure, a neural network may use a block obtained by downsampling a reconstructed luminance block corresponding to a current chrominance block of a current block based on a chrominance format, and thus the current chrominance block may be reconstructed more efficiently by using information of the reconstructed luminance block.

[0096] According to an embodiment of the present disclosure, the number of models of the neural network 520 may be plural. For example, a model may be selected from a plurality of models according to whether the type of the slice including the current block is an intra-frame slice or an inter-frame slice.

[0097] According to an embodiment of the present disclosure, the number of models of the neural network 520 may be multiple. For example, a model may be selected from multiple models according to whether the encoding block type of the current block is an intra-frame block or an inter-frame block.

[0098] According to an embodiment of the present disclosure, the model of the neural network 520 may be plural in number. For example, one model may be selected from a plurality of models according to the QP range of the strip including the current block. In detail, when the QP range is equal to or greater than 0 and less than a, model 1 may be selected from a plurality of models, when the QP range is equal to or greater than a and less than b, model 2 may be selected from a plurality of models, and when the QP range is equal to or greater than b and less than c, model 3 may be selected from a plurality of models.

[0099] According to an embodiment of the present disclosure, the model of the neural network 520 may be plural in number. For example, one model may be selected from a plurality of models based on the Cb component and the Cr component of the current block. In detail, one of the residual of the Cb component and the residual of the Cr component is used unchanged, a weight of 1 may be applied, and the model of the neural network may be applied to the other of the residual of the Cb component and the residual of the Cr component.

[0100] Figure 6 A diagram for describing a method of training a neural network for AI-based chrominance component prediction according to an embodiment of the present disclosure is shown.

[0101] and Figure 5 Compared with the neural network 520, Figure 6 The difference between the neural network 620 is that the output of the neural network is the reconstructed samples of the Cb component and the reconstructed samples of the Cr component, rather than the residual samples of the Cb component and the residual samples of the Cr component.

[0102] Figure 6 A method of training a neural network 620 for AI-based chrominance component prediction is shown. Figure 6 The predicted samples 605 of the Cb component of the current block for training shown in the figure may correspond to the predicted samples of the Cb component of the above-mentioned current block, the joint chrominance residual samples 610 of the current block for training may correspond to the joint chrominance residual samples of the above-mentioned current block, and the predicted samples 615 of the Cr component of the current block for training may correspond to the predicted samples of the Cr component of the above-mentioned current block.

[0103] According to the method of training the neural network 620 of the present disclosure, according to an embodiment of the present disclosure, the neural network 620 is trained so that the reconstructed sample 635 of the Cb component of the current block for training output from the neural network 620 becomes equal to or similar to the original sample 645 of the Cb component of the current block for training, and the reconstructed sample 640 of the Cr component of the current block for training becomes equal to or similar to the original sample 650 of the Cr component of the current block for training. For example, the loss information 655 corresponding to the sum of the difference between the reconstructed sample 635 of the Cb component of the current block for training and the original sample 645 of the Cb component of the current block for training and the difference between the reconstructed sample 640 of the Cr component of the current block for training and the original sample 650 of the Cr component of the current block for training may be used to train the neural network 620.

[0104] In more detail, first, a prediction sample 605 of the Cb component of the current block for training, a joint chroma residual sample 610 of the current block for training, and a prediction sample 615 of the Cr component of the current block for training are obtained. For example, the prediction sample 605 of the Cb component may be determined based on at least the Cb component prediction information obtained from the bitstream, and the prediction sample 615 of the Cr component of the current block may be determined based on at least the Cr component prediction information obtained from the bitstream.

[0105] The predicted sample 605 of the Cb component of the current block for training, the joint chrominance residual sample 610 of the current block for training, and the predicted sample 615 of the Cr component of the current block for training are input to the neural network 620, and then the reconstructed sample 635 of the Cb component of the current block for training and the reconstructed sample 640 of the Cr component of the current block for training are obtained output from the neural network 620. The neural network 620 can perform operations according to predetermined weights.

[0106] Loss information 655 corresponding to the sum of the difference between the reconstructed sample 635 of the Cb component of the current block for training and the original sample 645 of the Cb component of the current block for training and the difference between the reconstructed sample 640 of the Cr component of the current block for training and the original sample 650 of the Cr component of the current block for training is output, and a weight set for the neural network 620 is updated according to the loss information 655. The neural network 620 may update the weights to reduce or minimize the loss information 655.

[0107] The loss information 655 may include at least one value among an L1 norm value, an L2 norm value, an SSIM value, a PSNR-HVS value, an MS-SSIM value, a VIF value or a VMAF value, which at least one value is related to the sum of the difference between the reconstructed sample 635 of the Cb component of the current block used for training and the original sample 645 of the Cb component of the current block used for training and the difference between the reconstructed sample 640 of the Cr component of the current block used for training and the original sample 650 of the Cr component of the current block used for training.

[0108] According to an embodiment of the present disclosure, the training of the neural network 620 may be performed by a training device. The training device may be an AI-based video encoding device 1500 or an AI-based video decoding device 1200. According to an embodiment of the present disclosure, the training device may be an external server. In this case, the neural network 620 and the weights trained by the external server may be sent to the AI-based video encoding device 1500 or the AI-based video decoding device 1200. In an embodiment of the present disclosure, the trained neural network may be downloaded to one or more devices.

[0109] According to an embodiment of the present disclosure, in addition to the prediction samples 605 of the Cb component of the current block for training, the joint chroma residual samples 610 of the current block for training, and the prediction samples 615 of the Cr component of the current block for training, another input may be added to the neural network 620. For example, the added another input may be a quantization step size of the current block, a quantization error of the current block, a quantization step size or a quantization step size error of a reference frame of the current block (in the case where the current block is an inter-frame block), a block obtained by downsampling a reconstructed luminance block corresponding to the current chroma block of the current block based on a chroma format, or any other suitable method (for example, when the chroma format is 4:2:0, the block is reduced by a factor of 2).

[0110] According to an embodiment of the present disclosure, the number of models of the neural network 620 may be plural. For example, a model may be selected from a plurality of models according to whether the type of the slice including the current block is an intra-frame slice or an inter-frame slice.

[0111] According to an embodiment of the present disclosure, the number of models of the neural network 620 may be multiple. For example, a model may be selected from multiple models according to whether the encoding block type of the current block is an intra-frame block or an inter-frame block.

[0112] According to an embodiment of the present disclosure, the model of the neural network 620 may be plural in number. For example, one model may be selected from a plurality of models according to the QP range of the slice including the current block. In detail, when the QP range is equal to or greater than 0 and less than a, model 1 may be selected from a plurality of models, when the QP range is equal to or greater than a and less than b, model 2 may be selected from a plurality of models, and when the QP range is equal to or greater than b and less than c, model 3 may be selected from a plurality of models.

[0113] According to an embodiment of the present disclosure, the model of the neural network 620 may be plural in number. For example, one model may be selected from a plurality of models based on the Cb component and the Cr component of the current block. In detail, one of the residual of the Cb component and the residual of the Cr component is used unchanged, a weight of 1 may be applied, and the model of the neural network may be applied to the other of the residual of the Cb component and the residual of the Cr component.

[0114] Figure 7 A table showing the correlation between a neural network model for AI-based chrominance component prediction and a prediction type according to an embodiment of the present disclosure is shown. The correlation may be pre-stored in the AI-based video encoding device 1500 or the video encoding device 1600 or the AI-based video decoding device 1200 or the video decoding device 1300.

[0115] Reference Figure 7 , the neural network model may vary according to the prediction type. When the prediction type is an intra-frame prediction type, one of the multiple models, Model 1, Model 2, and Model 3 may be used, and when the prediction type is an inter-frame prediction type, one of the multiple models, Model 1 and Model 2 may be used.

[0116] In addition, refer to the above Figure 2 , the model to be used may vary according to the Cb cbf information (tu_cbf_cb) and the Cr cbf information (tu_cbf_cr).

[0117] In more detail, when Cb cbf information (tu_cbf_cb) and Cr cbf information (tu_cbf_cr) are obtained, models for reconstructed values ​​(resCb) of residual samples of Cb components and reconstructed values ​​(resCr) of residual samples of Cr components may be determined according to the Cb cbf information and the Cr cbf information.

[0118] For example, when tu_cbf_cb is 1 and tu_cbf_cr is 0, models for reconstructed values ​​of residual samples of the Cb component (resCb) and reconstructed values ​​of residual samples of the Cr component (resCr) may be determined from the neural network model as model 1. In this case, the joint mode index is 1.

[0119] For example, when tu_cbf_cb is 1 and tu_cbf_cr is 1, models for reconstructed values ​​of residual samples of the Cb component (resCb) and reconstructed values ​​of residual samples of the Cr component (resCr) may be determined from the neural network model as Model 2. In this case, the joint mode index is 2.

[0120] For example, when tu_cbf_cb is 0 and tu_cbf_cr is 1, models for reconstructed values ​​of residual samples of the Cb component (resCb) and reconstructed values ​​of residual samples of the Cr component (resCr) may be determined from the neural network model as model 3. In this case, the joint mode index is 3.

[0121] In an embodiment of the present disclosure, when the prediction type of the coding unit is an intra-frame prediction mode, index value 1, index value 2, and index value 3 can all be used for joint mode indexing. For example, when the prediction type of the coding unit is an intra-frame prediction mode, the model for the Cb component and the Cr component can be determined according to the joint mode index.

[0122] In an embodiment of the present disclosure, when the prediction type of the coding unit is the inter prediction mode, only the joint mode index 2 may be allowed. Therefore, in the case where the prediction type of the coding unit is the inter prediction mode and the joint mode index is 2, that is, only when tu_cbf_cb is 1 and tu_cbf_cr is 1, the joint chroma scheme may be allowed and the model for the Cb component and the Cr component may be determined as model 2.

[0123] Figure 8 A table showing the correlation between a neural network model for AI-based chrominance component prediction and a prediction type according to an embodiment of the present disclosure is shown. The correlation may be pre-stored in the AI-based video encoding device 1500 or the video encoding device 1600 or the AI-based video decoding device 1200 or the video decoding device 1300.

[0124] Reference Figure 8, when the prediction type is an intra-frame prediction type, one of the following schemes can be used: a scheme for obtaining the Cb component by multiplying the joint chroma residual with a weight of 1 and obtaining the Cr component by applying the joint chroma residual to model 1 of the neural network ([1, model 1]), a scheme for obtaining both the Cb component and the Cr component by applying the joint chroma residual to model 2 of the neural network ([model 2]), and a scheme for obtaining the Cb component by applying the joint chroma residual to model 3 of the neural network and obtaining the Cr component by multiplying the joint chroma residual with weight 1 ([model 3, 1]), and when the prediction type is an inter-frame prediction type, a scheme for obtaining the Cb component by multiplying the joint chroma residual with weight 1 and obtaining the Cr component by applying the joint chroma residual to model 2 of the neural network ([1, model 2]) can be used.

[0125] In addition, refer to the above Figure 2 , the model to be used may vary according to the Cb cbf information (tu_cbf_cb) and the Cr cbf information (tu_cbf_cr).

[0126] In more detail, when Cb cbf information (tu_cbf_cb) and Cr cbf information (tu_cbf_cr) are obtained, models or weights for reconstructed values ​​of residual samples of Cb components (resCb) and Cr components (resCr) may be determined according to the Cb cbf information and the Cr cbf information.

[0127] For example, when tu_cbf_cb is 1 and tu_cbf_cr is 0, the weight of the reconstructed value (resCb) of the residual sample of the Cb component may be determined as 1, and the model of the reconstructed value (resCr) of the residual sample of the Cr component may be determined as model 1. In this case, the joint mode index is 1.

[0128] For example, when tu_cbf_cb is 1 and tu_cbf_cr is 1, models for reconstructed values ​​of residual samples of the Cb component (resCb) and reconstructed values ​​of residual samples of the Cr component (resCr) may be determined from the neural network model as Model 2. In this case, the joint mode index is 2.

[0129] For example, when tu_cbf_cb is 0 and tu_cbf_cr is 1, the model of the reconstructed value (resCb) of the residual sample of the Cb component may be determined as model 3 from the neural network model, and the weight of the reconstructed value (resCr) of the residual sample of the Cr component may be determined as 1. In this case, the joint mode index is 3.

[0130] In an embodiment of the present disclosure, when the prediction type of the coding unit is an intra prediction mode, index value 1, index value 2, and index value 3 can all be used for the joint mode index. For example, when the prediction type of the coding unit is an intra prediction mode, the weights or models for the Cb component and the Cr component can be determined according to the joint mode index.

[0131] In an embodiment of the present disclosure, when the prediction type of the coding unit is the inter prediction mode, only the joint mode index 2 may be allowed. Therefore, in the case where the prediction type of the coding unit is the inter prediction mode and the joint mode index is 2 (for example, only when tu_cbf_cb is 1 and tu_cbf_cr is 1), the joint chroma scheme may be allowed and the model for the Cb component and the Cr component may be determined as model 2.

[0132] Fig. 9 A diagram is shown for describing a neural network and scaling factors for AI-based chroma component prediction according to an embodiment of the present disclosure.

[0133] Reference Fig. 9 , the prediction sample 905 of the Cb component of the current block, the joint chroma residual sample 910 of the current block, and the prediction sample 915 of the Cr component of the current block are input to the neural network 920. Before the prediction sample 905 of the Cb component of the current block and the prediction sample 915 of the Cr component of the current block are added to the residual sample 925 of the Cb component and the residual sample 930 of the Cr component output via the neural network 920, the residual sample 925 of the Cb component and the residual sample 930 of the Cr component may be multiplied by an additional scale factor 945 and an additional scale factor 950 (e.g., also referred to as a first scale factor 945 and a second scale factor 950), respectively.

[0134] In detail, the residual sample 925 of the Cb component output via the neural network 920 is multiplied by the first scale factor 945, thereby refining the residual sample 925. Subsequently, the prediction sample 905 of the Cb component of the current block is added to the refined residual sample of the Cb component, so that the reconstructed sample 935 of the Cb component of the current block can be obtained. The residual sample 930 of the Cr component output via the neural network 920 is multiplied by the second scale factor 950, thereby refining the residual sample 930. Subsequently, the prediction sample 915 of the Cr component of the current block is added to the refined residual sample of the Cr component, so that the reconstructed sample 940 of the Cr component of the current block can be obtained.

[0135] Even when a trained neural network model is used, when the residual sample value of the Cb component or the residual sample value of the Cr component is different from the original residual value, the residual sample needs to be refined so that a scaling factor can be additionally used. Fig.10 An example describing a scale factor.

[0136] In addition, in the above Figure 6 In the neural network, even when reconstructed samples of the Cb component of the current block and reconstructed samples of the Cr component of the current block are obtained by receiving inputs of predicted samples of the Cb component of the current block, joint chrominance residual samples of the current block, and predicted samples of the Cr component of the current block, a scaling factor may be additionally used.

[0137] According to an embodiment of the present disclosure, the residual sample of the Cb component of the current block can be obtained by subtracting the predicted sample of the Cb component of the current block from the reconstructed sample of the Cb component of the current block which is one of the outputs of the neural network, and the residual sample of the Cr component of the current block can be obtained by subtracting the predicted sample of the Cr component of the current block from the reconstructed sample of the Cr component of the current block which is another output of the neural network. The refined residual sample of the Cb component can be obtained by multiplying the residual sample of the Cb component of the current block by a first scale factor, and the refined residual sample of the Cr component can be obtained by multiplying the residual sample of the Cr component of the current block by the first scale factor. The refined reconstructed sample of the Cb component can be obtained by adding the predicted sample of the Cb component of the current block to the refined residual sample of the Cb component, and the refined reconstructed sample of the Cr component can be obtained by adding the predicted sample of the Cr component of the current block to the refined residual sample of the Cr component.

[0138] Since refinement via scaling may be required for a neural network trained to immediately output reconstructed samples, in the case of such a neural network, residual samples may be obtained by subtracting the predicted samples from the reconstructed samples, and then refined sample values ​​may be obtained by applying the scaling factor to the residual samples.

[0139] Fig.10 A table showing the correlation between a scale factor set and a prediction type according to an embodiment of the present disclosure is shown. The correlation may be pre-stored in the AI-based video encoding apparatus 1600 or the AI-based video decoding apparatus 1300 .

[0140] Reference Fig.10 , when the prediction type is an intra-frame prediction type, the scale factor set may include 1, 1 / 2, and 1 / 4, and when the prediction type is an inter-frame prediction type, the scale factor set may include 1 and 1 / 2.

[0141] For example, one of a plurality of scale factors included in a scale factor set that varies according to a prediction type may be selected and used. An index of a corresponding scale factor value may be signaled for each block or for each slice.

[0142] According to an embodiment of the present disclosure, the same scale factor may be used for the Cb component and the Cr component.

[0143] According to an embodiment of the present disclosure, when a scale factor is signaled for each block or for each slice, different scale factors may be used for a Cb component and a Cr component.

[0144] The prediction type may be a prediction type of a slice or a prediction type of a coding block. That is, a scale factor set may be selected based on a slice type or a scale factor set may be selected based on a block type.

[0145] Fig.11 A flowchart of an AI-based video decoding method according to an embodiment of the present disclosure is shown.

[0146] In operation S1110, the AI-based video decoding apparatus 1200 or the AI-based video decoding apparatus 1300 may obtain a joint chroma residual sample of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block from a bitstream.

[0147] According to an embodiment of the present disclosure, the joint chroma residual sample of the current block may be an average value between an actual value of the residual sample of the Cb component of the current block and an actual value of the residual sample of the Cr component of the current block.

[0148] According to an embodiment of the present disclosure, the joint chroma residual samples of the current block may be equal to the actual values ​​of the residual samples of the Cb component of the current block.

[0149] According to an embodiment of the present disclosure, the joint chroma residual samples of the current block may be equal to the actual values ​​of the residual samples of the Cr component of the current block.

[0150] According to an embodiment of the present disclosure, a joint chroma residual sample of a current block may be determined by an encoder by performing a sum of transform differences (SATD) or a rate-distortion optimization (RDO) calculation and may be transmitted in a bitstream.

[0151] In S1130, the AI-based video decoding apparatus 1200 or the AI-based video decoding apparatus 1300 may determine the prediction sample of the Cb component of the current block and the prediction sample of the Cr component of the current block based on the Cb component prediction information and the Cr component prediction information. For example, the prediction sample of the Cb component may be determined based on at least the Cb component prediction information obtained from the bitstream, and the prediction sample of the Cr component of the current block may be determined based on at least the Cr component prediction information obtained from the bitstream.

[0152] In S1150, the AI-based video decoding device 1200 or the AI-based video decoding device 1300 may reconstruct the current block by applying joint chroma residual samples, predicted samples of the Cb component, and predicted samples of the Cr component to a neural network to obtain reconstructed samples of the Cb component of the current block and reconstructed samples of the Cr component of the current block.

[0153] According to an embodiment of the present disclosure, the AI-based video decoding device 1300 may obtain at least one of the residual samples of the Cb component of the current block or the residual samples of the Cr component of the current block by applying the joint chrominance residual samples, the predicted samples of the Cb component, and the predicted samples of the Cr component to a neural network. The AI-based video decoding device 1300 may reconstruct the current block by using at least one of the residual samples of the Cb component or the residual samples of the Cr component, the predicted samples of the Cb component, and the predicted samples of the Cr component to obtain the reconstructed samples of the Cb component and the reconstructed samples of the Cr component.

[0154] According to an embodiment of the present disclosure, the AI-based video decoding device 1300 can refine the residual samples of the Cb component by applying a first scaling factor to the residual samples of the Cb component obtained via a neural network, and can refine the residual samples of the Cr component by applying a second scaling factor to the residual samples of the Cr component.

[0155] According to an embodiment of the present disclosure, the first scale factor and the second scale factor may be determined according to a scale factor value indicated by an index obtained from a bitstream and included in a scale factor set.

[0156] According to an embodiment of the present disclosure, a scale factor set may be determined from among a plurality of scale factor sets according to at least one of a type of a slice including a current block or a type of the current block.

[0157] According to an embodiment of the present disclosure, the plurality of scale factor sets may include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0158] According to an embodiment of the present disclosure, a neural network may be trained to determine the correlation between Cb and Cr for each sample of a current block for training by receiving input values ​​of a joint chroma residual sample for training, a prediction sample of a Cb component for training, and a prediction sample of a Cr component for training.

[0159] According to an embodiment of the present disclosure, for the correlation between Cb and Cr, weights may be determined for the residual samples of the Cb component and the residual samples of the Cr component, respectively.

[0160] According to an embodiment of the present disclosure, the input value of the neural network may also include at least one of a quantization step size of the current block, a quantization error of the current block, or a block obtained by downsampling a reconstructed luminance block corresponding to a current chroma block of the current block based on a chroma format.

[0161] According to an embodiment of the present disclosure, a neural network may be trained based on first loss information and second loss information, wherein the first loss information corresponds to the difference between the original sample of the Cb component of the original block for training and the reconstructed sample of the Cb component of the reconstructed block for training obtained via the neural network, and the second loss information corresponds to the difference between the original sample of the Cr component of the original block for training and the reconstructed sample of the Cr component of the reconstructed block for training obtained via the neural network.

[0162] According to an embodiment of the present disclosure, a model of a neural network may be determined based on Cb cbf information indicating whether a transform coefficient level of a Cb component of a current block includes a non-zero Cb component and Cr cbf information indicating whether a transform coefficient level of a Cr component of a current block includes a non-zero Cr component, the Cb cbf information and the Cr cbf information being obtained from a bitstream.

[0163] According to an embodiment of the present disclosure, a model of a neural network may be determined according to at least one of a type of a current block, a type of a slice including the current block, a QP range of a slice, or a Cb component or a Cr component of the current block.

[0164] Fig.12 A block diagram showing the configuration of an AI-based video decoding device according to an embodiment of the present disclosure is shown.

[0165] Reference Fig.12 , the AI-based video decoding device 1200 may include a decoder 1210, a predictor 1220, an AI-based image processor 1230, and a reconstructor 1240.

[0166] The decoder 1210, the predictor 1220, the AI-based image processor 1230, and the reconstructor 1240 may be implemented via one or more processors. The decoder 1210, the predictor 1220, the AI-based image processor 1230, and the reconstructor 1240 may operate according to instructions stored in a memory, where the instructions cause the one or more processors to perform one or more operations.

[0167] Although Fig.12The decoder 1210, the predictor 1220, the AI-based image processor 1230, and the reconstructor 1240 are shown separately, but the decoder 1210, the predictor 1220, the AI-based image processor 1230, and the reconstructor 1240 may be implemented via one processor. For example, the decoder 1210, the predictor 1220, the AI-based image processor 1230, and the reconstructor 1240 may be implemented via a dedicated processor, or may be implemented via a combination of a general-purpose processor (such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU)) and software. In addition, the dedicated processor may include a memory containing instructions for implementing an embodiment of the present disclosure, or the dedicated processor may include a memory processor configured to use an external memory.

[0168] The decoder 1210, the predictor 1220, the AI-based image processor 1230, and the reconstructor 1240 may be configured via multiple processors. In this case, they may be implemented via a combination of dedicated processors, or may be implemented via a combination of multiple general-purpose processors (such as AP, CPU, or GPU) and software. In addition, the processor may include an AI-dedicated processor. As another example, the AI-dedicated processor may be configured as a chip separate from the processor.

[0169] The decoder 1210 may obtain a joint chroma residual sample of a current block, Cb component prediction information of a current block, and Cr component prediction information of a current block from a bitstream.

[0170] According to an embodiment of the present disclosure, the joint chroma residual sample of the current block may be an average value between an actual value of the residual sample of the Cb component of the current block and an actual value of the residual sample of the Cr component of the current block.

[0171] According to an embodiment of the present disclosure, the joint chroma residual samples of the current block may be equal to the actual values ​​of the residual samples of the Cb component of the current block.

[0172] According to an embodiment of the present disclosure, the joint chroma residual samples of the current block may be equal to the actual values ​​of the residual samples of the Cr component of the current block.

[0173] According to an embodiment of the present disclosure, the joint chroma residual sample of the current block may be determined by an encoder by performing SATD or RDO calculation and may be transmitted in a bitstream.

[0174] In an embodiment of the present disclosure, the decoder 1210 transmits Cb component prediction information of the current block and Cr component prediction information of the current block to the predictor 1220 .

[0175] The predictor 1220 may determine a prediction sample of a Cb component of a current block and a prediction sample of a Cr component of a current block based on the Cb component prediction information and the Cr component prediction information.

[0176] According to an embodiment of the present disclosure, the AI-based image processor 1230 may include a reference Figure 5 Neural network 520 described or referenced Figure 6 Neural network 620 is described.

[0177] According to an embodiment of the present disclosure, a neural network included in the AI-based image processor 1230 may be trained to determine a correlation between Cb and Cr for each sample of a current block for training by receiving input values ​​of a joint chroma residual sample for training, a prediction sample of a Cb component for training, and a prediction sample of a Cr component for training.

[0178] According to an embodiment of the present disclosure, for the correlation between Cb and Cr, weights may be determined for the residual samples of the Cb component and the residual samples of the Cr component, respectively.

[0179] According to an embodiment of the present disclosure, a neural network may be trained based on first loss information and second loss information, wherein the first loss information corresponds to the difference between the original sample of the Cb component of the original block for training and the reconstructed sample of the Cb component of the reconstructed block for training obtained via the neural network, and the second loss information corresponds to the difference between the original sample of the Cr component of the original block for training and the reconstructed sample of the Cr component of the reconstructed block for training obtained via the neural network.

[0180] The AI-based image processor 1230 may obtain a reconstructed sample of the Cb component and a reconstructed sample of the Cr component by receiving inputs of the joint chroma residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component.

[0181] According to an embodiment of the present disclosure, the input value of the neural network may also include at least one of a quantization step size of the current block, a quantization error of the current block, or a block obtained by downsampling a reconstructed luminance block corresponding to a current chroma block of the current block based on a chroma format.

[0182] According to an embodiment of the present disclosure, the AI-based image processor 1230 can obtain reconstructed samples of the Cb component and reconstructed samples of the Cr component by additionally receiving an input of at least one of a quantization step size of the current block, a quantization error of the current block, or a block obtained by downsampling a reconstructed luminance block corresponding to a current chroma block of the current block based on a chroma format.

[0183] According to an embodiment of the present disclosure, the decoder 1210 may further obtain Cb cbf information indicating whether a transform coefficient level of a current block regarding a Cb component includes a non-zero Cb component and Cr cbf information indicating whether a transform coefficient level of a current block regarding a Cr component includes a non-zero Cr component from a bitstream. A model of a neural network included in the AI-based image processor 1230 may be determined based on the Cb cbf information and the Cr cbf information.

[0184] According to an embodiment of the present disclosure, a model of a neural network included in the AI-based image processor 1230 may be determined according to at least one of a type of a current block, a type of a slice including the current block, a QP range of a slice, or a Cb component or a Cr component of the current block.

[0185] The reconstructor 1240 may reconstruct the current block by using the reconstructed sample of the Cb component and the reconstructed sample of the Cr component obtained by the AI-based image processor 1230 .

[0186] The output of the neural network included in the AI-based image processor 1230 may not be the reconstructed sample of the Cb component and the reconstructed sample of the Cr component, but may be one of the residual sample of the Cb component and the residual sample of the Cr component. In this case, an adder may be further included to add the residual sample of the Cb component and the residual sample of the Cr component to the predicted sample of the Cb component and the predicted sample of the Cr component. Fig.13 This is described in an AI-based video decoding device.

[0187] Fig.13 A block diagram showing the configuration of an AI-based video decoding device according to an embodiment of the present disclosure is shown.

[0188] Reference Fig.13 , the AI-based video decoding device 1300 may include a decoder 1310, a predictor 1320, an AI-based image processor 1330, an adder 1335 and a reconstructor 1340.

[0189] The decoder 1310, the predictor 1320, the AI-based image processor 1330, the adder 1335, and the reconstructor 1340 may be implemented via one or more processors. The decoder 1310, the predictor 1320, the AI-based image processor 1330, the adder 1335, and the reconstructor 1340 may operate according to instructions stored in a memory.

[0190] Although Fig.13The decoder 1310, the predictor 1320, the AI-based image processor 1330, the adder 1335, and the reconstructor 1340 are shown separately, but the decoder 1310, the predictor 1320, the AI-based image processor 1330, the adder 1335, and the reconstructor 1340 may be implemented via one processor. In this case, the decoder 1310, the predictor 1320, the AI-based image processor 1330, the adder 1335, and the reconstructor 1340 may be implemented via a dedicated processor, or may be implemented via a combination of a general-purpose processor (such as an AP, a CPU, or a GPU) and software. In addition, the dedicated processor may include a memory including instructions for implementing an embodiment of the present disclosure, or the dedicated processor may include a memory processor configured to use an external memory.

[0191] The decoder 1310, the predictor 1320, the AI-based image processor 1330, the adder 1335, and the reconstructor 1340 may be configured via a plurality of processors. In this case, they may be implemented via a combination of dedicated processors, or may be implemented via a combination of a plurality of general-purpose processors (such as an AP, a CPU, or a GPU) and software. In addition, the processor may include an AI-dedicated processor. As another example, the AI-dedicated processor may be configured as a chip separate from the processor.

[0192] The decoder 1310 may obtain a joint chroma residual sample of a current block, Cb component prediction information of a current block, and Cr component prediction information of a current block from a bitstream.

[0193] According to an embodiment of the present disclosure, the joint chroma residual sample of the current block may be an average value between an actual value of the residual sample of the Cb component of the current block and an actual value of the residual sample of the Cr component of the current block.

[0194] According to an embodiment of the present disclosure, the joint chroma residual samples of the current block may be equal to the actual values ​​of the residual samples of the Cb component of the current block.

[0195] According to an embodiment of the present disclosure, the joint chroma residual samples of the current block may be equal to the actual values ​​of the residual samples of the Cr component of the current block.

[0196] According to an embodiment of the present disclosure, the joint chroma residual sample of the current block may be determined by an encoder by performing SATD or RDO calculation and may be transmitted in a bitstream.

[0197] The decoder 1310 transmits the Cb component prediction information of the current block and the Cr component prediction information of the current block to the predictor 1320 .

[0198] The predictor 1320 may determine a prediction sample of a Cb component of a current block and a prediction sample of a Cr component of a current block based on the Cb component prediction information and the Cr component prediction information.

[0199] The AI-based image processor 1330 may include a reference Figure 5 Neural network 520 is described.

[0200] According to an embodiment of the present disclosure, a neural network included in the AI-based image processor 1330 may be trained to determine a correlation between Cb and Cr for each sample of a current block for training by receiving input values ​​of a joint chroma residual sample for training, a prediction sample of a Cb component for training, and a prediction sample of a Cr component for training.

[0201] According to an embodiment of the present disclosure, for the correlation between Cb and Cr, weights may be determined for the residual samples of the Cb component and the residual samples of the Cr component, respectively.

[0202] According to an embodiment of the present disclosure, a neural network may be trained based on first loss information and second loss information, wherein the first loss information corresponds to the difference between the original sample of the Cb component of the original block for training and the reconstructed sample of the Cb component of the reconstructed block for training obtained via the neural network, and the second loss information corresponds to the difference between the original sample of the Cr component of the original block for training and the reconstructed sample of the Cr component of the reconstructed block for training obtained via the neural network.

[0203] The AI-based image processor 1330 may obtain at least one of a residual sample of a Cb component of a current block or a residual sample of a Cr component of a current block by receiving input of a joint chroma residual sample, a prediction sample of a Cb component, and a prediction sample of a Cr component.

[0204] According to an embodiment of the present disclosure, the input value of the neural network may also include at least one of a quantization step size of the current block, a quantization error of the current block, or a block obtained by downsampling a reconstructed luminance block corresponding to a current chroma block of the current block based on a chroma format.

[0205] According to an embodiment of the present disclosure, the AI-based image processor 1330 can obtain at least one of the residual samples of the Cb component or the residual samples of the Cr component by additionally receiving an input of at least one of the quantization step size of the current block, the quantization error of the current block, or a block obtained by downsampling a reconstructed luminance block corresponding to the current chroma block of the current block based on a chroma format.

[0206] The adder 1335 may obtain the reconstructed samples of the Cb component and the reconstructed samples of the Cr component by using at least one of the residual samples of the Cb component or the residual samples of the Cr component, the predicted samples of the Cb component, and the predicted samples of the Cr component. For example, the adder 1335 may obtain the reconstructed samples of the Cb component by adding the residual samples of the Cb component to the predicted samples of the Cb component, and may obtain the reconstructed samples of the Cr component by adding the residual samples of the Cr component to the predicted samples of the Cr component.

[0207] According to an embodiment of the present disclosure, the decoder 1310 may further obtain an index indicating a scale factor from a bitstream. The AI-based image processor 1330 may further obtain an index indicating a scale factor included in a scale factor set from the decoder 1310. The AI-based image processor 1330 may apply the obtained scale factor to the residual sample of the Cb component of the current block and the residual sample of the Cr component of the current block, respectively, thereby obtaining a refined residual sample of the Cb component of the current block and a refined residual sample of the Cr component of the current block. In detail, the AI-based image processor 1330 may refine the residual sample of the Cb component by applying a first scale factor to the residual sample of the Cb component, and may refine the residual sample of the Cr component by applying a second scale factor to the residual sample of the Cr component. The adder 1335 may obtain a reconstructed sample of the Cb component by adding the refined residual sample of the Cb component to the predicted sample of the Cb component, and may obtain a reconstructed sample of the Cr component by adding the refined residual sample of the Cr component to the predicted sample of the Cr component.

[0208] According to an embodiment of the present disclosure, a scale factor set may be determined from among a plurality of scale factor sets according to at least one of a type of a slice including a current block or a type of the current block.

[0209] According to an embodiment of the present disclosure, the plurality of scale factor sets may include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0210] According to an embodiment of the present disclosure, the decoder 1310 may further obtain Cb cbf information indicating whether a transform coefficient level of a Cb component of a current block includes a non-zero Cb component and Cr cbf information indicating whether a transform coefficient level of a Cr component of a current block includes a non-zero Cr component from a bitstream. A model of a neural network included in the AI-based image processor 1330 may be determined based on the Cb cbf information and the Cr cbf information.

[0211] According to an embodiment of the present disclosure, a model of a neural network included in the AI-based image processor 1330 may be determined according to at least one of a type of a current block, a type of a slice including the current block, a QP range of a slice, or a Cb component or a Cr component of the current block.

[0212] The reconstructor 1340 may reconstruct the current block by using the reconstructed sample of the Cb component and the reconstructed sample of the Cr component obtained by the adder 1335 .

[0213] Fig.14 A flowchart of an AI-based video encoding method according to an embodiment of the present disclosure is shown.

[0214] In operation S1410, the AI-based video encoding apparatus 1500 or the AI-based video encoding apparatus 1600 may generate an initial joint chroma residual sample of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block.

[0215] According to an embodiment of the present disclosure, the initial joint chroma residual sample of the current block may be an average value between an actual value of the residual sample of the Cb component of the current block and an actual value of the residual sample of the Cr component of the current block.

[0216] According to an embodiment of the present disclosure, the initial joint chroma residual samples of the current block may be equal to the actual values ​​of the residual samples of the Cb component of the current block.

[0217] According to an embodiment of the present disclosure, the initial joint chroma residual samples of the current block may be equal to the actual values ​​of the residual samples of the Cr component of the current block.

[0218] In operation S1430, the AI-based video encoding apparatus 1500 or the AI-based video encoding apparatus 1600 may determine a prediction sample of a Cb component of a current block and a prediction sample of a Cr component of a current block based on the Cb component prediction information and the Cr component prediction information. For example, the prediction sample of the Cb component may be determined based on at least the Cb component prediction information obtained from the bitstream, and the prediction sample of the Cr component of the current block may be determined based on at least the Cr component prediction information obtained from the bitstream.

[0219] In S1450, the AI-based video encoding device 1500 or the AI-based video encoding device 1600 may encode the joint chroma residual samples of the current block by applying the initial joint chroma residual samples, the predicted samples of the Cb component, and the predicted samples of the Cr component to a neural network to obtain a reconstructed sample of the Cb component of the current block and a reconstructed sample of the Cr component of the current block.

[0220] According to an embodiment of the present disclosure, the AI-based video encoding device 1500 or the AI-based video encoding device 1600 may obtain at least one of the residual samples of the Cb component of the current block or the residual samples of the Cr component of the current block by applying the initial joint chroma residual samples, the predicted samples of the Cb component, and the predicted samples of the Cr component to a neural network. The AI-based video encoding device 1600 may encode the joint chroma residual samples of the current block by using at least one of the residual samples of the Cb component or the residual samples of the Cr component, the predicted samples of the Cb component, and the predicted samples of the Cr component to obtain the reconstructed samples of the Cb component and the reconstructed samples of the Cr component.

[0221] According to an embodiment of the present disclosure, a joint chroma residual sample of a current block may be determined by performing SATD or RDO calculation and may be encoded.

[0222] According to an embodiment of the present disclosure, Cb component prediction information and Cr component prediction information of the current block may be additionally encoded.

[0223] According to an embodiment of the present disclosure, the AI-based video encoding device 1600 can refine the residual samples of the Cb component by applying a first scale factor to the residual samples of the Cb component obtained via a neural network, and can refine the residual samples of the Cr component by applying a second scale factor to the residual samples of the Cr component.

[0224] According to an embodiment of the present disclosure, the first scale factor and the second scale factor may be encoded as an index indicating a scale factor value included in a scale factor set. That is, the index indicating the scale factor value may be determined by performing SATD or RDO calculation and may be signaled by being included in a bitstream.

[0225] According to an embodiment of the present disclosure, a scale factor set may be determined from among a plurality of scale factor sets according to at least one of a type of a slice including a current block or a type of the current block.

[0226] According to an embodiment of the present disclosure, the plurality of scale factor sets may include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0227] According to an embodiment of the present disclosure, a neural network may be trained to determine the correlation between Cb and Cr for each sample of a current block for training by receiving input values ​​of a joint chroma residual sample for training, a prediction sample of a Cb component for training, and a prediction sample of a Cr component for training.

[0228] According to an embodiment of the present disclosure, for the correlation between Cb and Cr, weights may be determined for the residual samples of the Cb component and the residual samples of the Cr component, respectively.

[0229] According to an embodiment of the present disclosure, the input value of the neural network may also include at least one of a quantization step size of the current block, a quantization error of the current block, or a block obtained by downsampling a reconstructed luminance block corresponding to a current chroma block of the current block based on a chroma format.

[0230] According to an embodiment of the present disclosure, a neural network may be trained based on first loss information and second loss information, wherein the first loss information corresponds to the difference between the original sample of the Cb component of the original block for training and the reconstructed sample of the Cb component of the reconstructed block for training obtained via the neural network, and the second loss information corresponds to the difference between the original sample of the Cr component of the original block for training and the reconstructed sample of the Cr component of the reconstructed block for training obtained via the neural network.

[0231] According to an embodiment of the present disclosure, a model of a neural network may be determined based on Cb cbf information indicating whether a transform coefficient level of a Cb component of a current block includes a non-zero Cb component and Cr cbf information indicating whether a transform coefficient level of a Cr component of a current block includes a non-zero Cr component, the Cb cbf information and the Cr cbf information being obtained from a bitstream.

[0232] According to an embodiment of the present disclosure, a model of a neural network may be determined according to at least one of a type of a current block, a type of a slice including the current block, a QP range of a slice, or a Cb component or a Cr component of the current block.

[0233] Fig.15 A block diagram showing the configuration of an AI-based video encoding device according to an embodiment of the present disclosure is shown.

[0234] Reference Fig.15 , the AI-based video encoding device 1500 may include a generator 1510, a predictor 1520, an AI-based image processor 1530, and an encoder 1540.

[0235] The generator 1510, the predictor 1520, the AI-based image processor 1530, and the encoder 1540 may be implemented via one or more processors. The generator 1510, the predictor 1520, the AI-based image processor 1530, and the encoder 1540 may operate according to instructions stored in a memory.

[0236] Although Fig.15The generator 1510, the predictor 1520, the AI-based image processor 1530, and the encoder 1540 are shown separately, but the generator 1510, the predictor 1520, the AI-based image processor 1530, and the encoder 1540 may be implemented via one processor. In this case, the generator 1510, the predictor 1520, the AI-based image processor 1530, and the encoder 1540 may be implemented via a dedicated processor, or may be implemented via a combination of a general-purpose processor (AP, CPU, or GPU) and software. In addition, the dedicated processor may include a memory containing instructions for implementing an embodiment of the present disclosure, or the dedicated processor may include a memory processor configured to use an external memory.

[0237] The generator 1510, the predictor 1520, the AI-based image processor 1530, and the encoder 1540 may be configured via a plurality of processors. In this case, they may be implemented via a combination of dedicated processors, or may be implemented via a combination of a plurality of general-purpose processors (such as an AP, a CPU, or a GPU) and software. In addition, the processor may include an AI-dedicated processor. As another example, the AI-dedicated processor may be configured as a chip separate from the processor.

[0238] The generator 1510 may generate an initial joint chroma residual sample of the current block, Cb component prediction information of the current block, and Cr component prediction information of the current block.

[0239] According to an embodiment of the present disclosure, the initial joint chroma residual sample of the current block may be an average value between an actual value of the residual sample of the Cb component of the current block and an actual value of the residual sample of the Cr component of the current block.

[0240] According to an embodiment of the present disclosure, the initial joint chroma residual samples of the current block may be equal to the actual values ​​of the residual samples of the Cb component of the current block.

[0241] According to an embodiment of the present disclosure, the initial joint chroma residual samples of the current block may be equal to the actual values ​​of the residual samples of the Cr component of the current block.

[0242] The generator 1510 transmits the Cb component prediction information of the current block and the Cr component prediction information of the current block to the predictor 1520 .

[0243] The predictor 1520 may determine a prediction sample of a Cb component of a current block and a prediction sample of a Cr component of a current block based on the Cb component prediction information and the Cr component prediction information.

[0244] According to an embodiment of the present disclosure, the AI-based image processor 1530 may include a reference Figure 5 Neural network 520 described or referenced Figure 6 Neural network 620 is described.

[0245] According to an embodiment of the present disclosure, a neural network included in the AI-based image processor 1530 may be trained to determine a correlation between Cb and Cr for each sample of a current block for training by receiving input values ​​of a joint chroma residual sample for training, a prediction sample of a Cb component for training, and a prediction sample of a Cr component for training.

[0246] According to an embodiment of the present disclosure, for the correlation between Cb and Cr, weights may be determined for the residual samples of the Cb component and the residual samples of the Cr component, respectively.

[0247] According to an embodiment of the present disclosure, a neural network may be trained based on first loss information and second loss information, wherein the first loss information corresponds to the difference between the original sample of the Cb component of the original block for training and the reconstructed sample of the Cb component of the reconstructed block for training obtained via the neural network, and the second loss information corresponds to the difference between the original sample of the Cr component of the original block for training and the reconstructed sample of the Cr component of the reconstructed block for training obtained via the neural network.

[0248] The AI-based image processor 1530 may obtain a reconstructed sample of the Cb component and a reconstructed sample of the Cr component by receiving inputs of an initial joint chroma residual sample, a prediction sample of the Cb component, and a prediction sample of the Cr component.

[0249] According to an embodiment of the present disclosure, the input value of the neural network may also include at least one of a quantization step size of the current block, a quantization error of the current block, or a block obtained by downsampling a reconstructed luminance block corresponding to a current chroma block of the current block based on a chroma format.

[0250] According to an embodiment of the present disclosure, the AI-based image processor 1530 can obtain reconstructed samples of the Cb component and reconstructed samples of the Cr component by additionally receiving input of at least one of a quantization step size of the current block, a quantization error of the current block, or a block obtained by downsampling a reconstructed luminance block corresponding to a current chroma block of the current block based on a chroma format.

[0251] According to an embodiment of the present disclosure, the generator 1510 may further generate Cb cbf information indicating whether the transform coefficient level of the current block regarding the Cb component includes a non-zero Cb component and Cr cbf information indicating whether the transform coefficient level of the current block regarding the Cr component includes a non-zero Cr component. A model of a neural network included in the AI-based image processor 1530 may be determined based on the Cb cbf information and the Cr cbf information.

[0252] According to an embodiment of the present disclosure, a model of a neural network included in the AI-based image processor 1530 may be determined according to at least one of a type of a current block, a type of a slice including the current block, a QP range of a slice, or a Cb component or a Cr component of the current block.

[0253] The encoder 1540 may encode the joint chroma residual samples of the current block by using the reconstructed samples of the Cb component and the reconstructed samples of the Cr component obtained by the AI-based image processor 1530 .

[0254] According to an embodiment of the present disclosure, a joint chroma residual sample of a current block may be determined by performing SATD or RDO calculation and may be encoded.

[0255] According to an embodiment of the present disclosure, the encoder 1540 may additionally encode the Cb component prediction information and the Cr component prediction information of the current block obtained by the generator 1510 .

[0256] The output of the neural network included in the AI-based image processor 1530 may not be the reconstructed sample of the Cb component and the reconstructed sample of the Cr component, but may be one of the residual sample of the Cb component and the residual sample of the Cr component. In this case, an adder may be further included to add the residual sample of the Cb component and the residual sample of the Cr component to the predicted sample of the Cb component and the predicted sample of the Cr component. Fig.16 This is described in an AI-based video encoding device.

[0257] Fig.16 A block diagram showing the configuration of an AI-based video encoding device according to an embodiment of the present disclosure is shown.

[0258] Reference Fig.16 , the AI-based video encoding device 1600 may include a generator 1610, a predictor 1620, an AI-based image processor 1630, an adder 1635, and an encoder 1640.

[0259] The generator 1610, the predictor 1620, the AI-based image processor 1630, the adder 1635, and the encoder 1640 may be implemented via one or more processors. The generator 1610, the predictor 1620, the AI-based image processor 1630, the adder 1635, and the encoder 1640 may operate according to instructions stored in a memory.

[0260] Although Fig.16The generator 1610, the predictor 1620, the AI-based image processor 1630, the adder 1635, and the encoder 1640 are shown separately, but the generator 1610, the predictor 1620, the AI-based image processor 1630, the adder 1635, and the encoder 1640 may be implemented via one processor. In this case, the generator 1610, the predictor 1620, the AI-based image processor 1630, the adder 1635, and the encoder 1640 may be implemented via a dedicated processor, or may be implemented via a combination of a general-purpose processor (such as an AP, a CPU, or a GPU) and software. In addition, the dedicated processor may include a memory containing instructions for implementing an embodiment of the present disclosure, or the dedicated processor may include a memory processor configured to use an external memory.

[0261] The generator 1610, the predictor 1620, the AI-based image processor 1630, the adder 1635, and the encoder 1640 may be configured via a plurality of processors. In this case, they may be implemented via a combination of dedicated processors, or may be implemented via a combination of a plurality of general-purpose processors (such as an AP, a CPU, or a GPU) and software. In addition, the processor may include an AI-dedicated processor. As another example, the AI-dedicated processor may be configured as a chip separate from the processor.

[0262] The generator 1610 may generate an initial joint chroma residual sample of the current block, Cb component prediction information of the current block, and Cr component prediction information of the current block.

[0263] According to an embodiment of the present disclosure, the initial joint chroma residual sample of the current block may be an average value between an actual value of the residual sample of the Cb component of the current block and an actual value of the residual sample of the Cr component of the current block.

[0264] According to an embodiment of the present disclosure, the initial joint chroma residual samples of the current block may be equal to the actual values ​​of the residual samples of the Cb component of the current block.

[0265] According to an embodiment of the present disclosure, the initial joint chroma residual samples of the current block may be equal to the actual values ​​of the residual samples of the Cr component of the current block.

[0266] The generator 1610 transmits the Cb component prediction information of the current block and the Cr component prediction information of the current block to the predictor 1620 .

[0267] The predictor 1620 may determine a prediction sample of a Cb component of a current block and a prediction sample of a Cr component of a current block based on the Cb component prediction information and the Cr component prediction information.

[0268] According to an embodiment of the present disclosure, the AI-based image processor 1630 may include a reference Figure 5Neural network 520 is described.

[0269] According to an embodiment of the present disclosure, a neural network included in the AI-based image processor 1630 may be trained to determine a correlation between Cb and Cr for each sample of a current block for training by receiving input values ​​of a joint chroma residual sample for training, a prediction sample of a Cb component for training, and a prediction sample of a Cr component for training.

[0270] According to an embodiment of the present disclosure, for the correlation between Cb and Cr, weights may be determined for the residual samples of the Cb component and the residual samples of the Cr component, respectively.

[0271] According to an embodiment of the present disclosure, a neural network may be trained based on first loss information and second loss information, wherein the first loss information corresponds to the difference between the original sample of the Cb component of the original block for training and the reconstructed sample of the Cb component of the reconstructed block for training obtained via the neural network, and the second loss information corresponds to the difference between the original sample of the Cr component of the original block for training and the reconstructed sample of the Cr component of the reconstructed block for training obtained via the neural network.

[0272] The AI-based image processor 1630 may obtain at least one of a residual sample of a Cb component of a current block and a residual sample of a Cr component of a current block by receiving inputs of an initial joint chroma residual sample, a prediction sample of a Cb component, and a prediction sample of a Cr component.

[0273] According to an embodiment of the present disclosure, the input value of the neural network may also include at least one of a quantization step size of the current block, a quantization error of the current block, or a block obtained by downsampling a reconstructed luminance block corresponding to a current chroma block of the current block based on a chroma format.

[0274] According to an embodiment of the present disclosure, the AI-based image processor 1630 can obtain at least one of the residual samples of the Cb component or the residual samples of the Cr component by additionally receiving an input of at least one of the quantization step size of the current block, the quantization error of the current block, or a block obtained by downsampling a reconstructed luminance block corresponding to the current chroma block of the current block based on a chroma format.

[0275] According to an embodiment of the present disclosure, the generator 1610 may further generate Cb cbf information indicating whether the transform coefficient level of the current block regarding the Cb component includes a non-zero Cb component and Cr cbf information indicating whether the transform coefficient level of the current block regarding the Cr component includes a non-zero Cr component. A model of a neural network included in the AI-based image processor 1630 may be determined based on the Cb cbf information and the Cr cbf information.

[0276] According to an embodiment of the present disclosure, a model of a neural network included in the AI-based image processor 1630 may be determined according to at least one of a type of a current block, a type of a slice including the current block, a QP range of a slice, or a Cb component or a Cr component of the current block.

[0277] The adder 1635 may obtain a reconstructed sample of the Cb component and a reconstructed sample of the Cr component by using at least one of the residual sample of the Cb component or the residual sample of the Cr component, the predicted sample of the Cb component, and the predicted sample of the Cr component. For example, the adder 1635 may obtain a reconstructed sample of the Cb component by adding the residual sample of the Cb component to the predicted sample of the Cb component, and may obtain a reconstructed sample of the Cr component by adding the residual sample of the Cr component to the predicted sample of the Cr component.

[0278] The encoder 1640 may encode the joint chroma residual samples of the current block by using the reconstructed samples of the Cb component and the reconstructed samples of the Cr component obtained by the adder 1635 .

[0279] According to an embodiment of the present disclosure, a joint chroma residual sample of a current block may be determined by performing SATD or RDO calculation and may be encoded.

[0280] According to an embodiment of the present disclosure, the encoder 1640 may additionally encode the Cb component prediction information and the Cr component prediction information of the current block obtained by the generator 1610 .

[0281] According to an embodiment of the present disclosure, the AI-based image processor 1630 may apply the obtained scale factors to the residual samples of the Cb component of the current block and the residual samples of the Cr component of the current block, respectively, thereby obtaining the refined residual samples of the Cb component of the current block and the refined residual samples of the Cr component of the current block. In detail, the AI-based image processor 1630 may refine the residual samples of the Cb component by applying the first scale factor to the residual samples of the Cb component, and may refine the residual samples of the Cr component by applying the second scale factor to the residual samples of the Cr component. The adder 1635 may obtain the reconstructed samples of the Cb component by adding the refined residual samples of the Cb component to the predicted samples of the Cb component, and may obtain the reconstructed samples of the Cr component by adding the refined residual samples of the Cr component to the predicted samples of the Cr component. The encoder 1640 may further encode the index of the scale factor indicating the scale factor in the scale factor set by using the reconstructed samples of the Cb component and the reconstructed samples of the Cr component obtained by the adder 1635.

[0282] According to an embodiment of the present disclosure, a scale factor set may be determined from among a plurality of scale factor sets according to at least one of a type of a slice including a current block or a type of the current block.

[0283] According to an embodiment of the present disclosure, the plurality of scale factor sets may include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0284] According to an embodiment of the present disclosure, an AI-based video decoding method may include: obtaining a joint chroma residual sample of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block from a bitstream; obtaining a prediction sample of the Cb component of the current block and a prediction sample of the Cr component of the current block based on the Cb component prediction information and the Cr component prediction information; and reconstructing the current block by applying the joint chroma residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component to a neural network to obtain a reconstructed sample of the Cb component of the current block and a reconstructed sample of the Cr component of the current block.

[0285] According to an embodiment of the present disclosure, the AI-based video decoding method may have the following effects: by using prediction information of the Cb component and prediction information of the Cr component, the Cb component and the Cr component can be more efficiently reconstructed via a neural network trained for a more general, local and specific relationship between the Cb component and the Cr component rather than a fixed correlation between the Cb component and the Cr component.

[0286] According to an embodiment of the present disclosure, an AI-based video decoding method may include: obtaining a joint chroma residual sample of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block from a bitstream; obtaining a prediction sample of the Cb component of the current block and a prediction sample of the Cr component of the current block based on the Cb component prediction information and the Cr component prediction information; obtaining at least one of a residual sample of the Cb component of the current block or a residual sample of the Cr component of the current block by applying the joint chroma residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component to a neural network; and reconstructing the current block by obtaining a reconstructed sample of the Cb component and a reconstructed sample of the Cr component using at least one of the residual sample of the Cb component or the residual sample of the Cr component, the prediction sample of the Cb component, and the prediction sample of the Cr component.

[0287] According to an embodiment of the present disclosure, the AI-based video decoding method may have the following effects: by more accurately obtaining at least one of the residual samples of the Cb component and the residual samples of the Cr component through a neural network, the Cb component and the Cr component are more efficiently reconstructed by using at least one of the residual samples of the Cb component and the residual samples of the Cr component.

[0288] According to an embodiment of the present disclosure, the AI-based video decoding method may include: refining the residual samples of the Cb component by applying a first scaling factor to the residual samples of the Cb component obtained via a neural network, and refining the residual samples of the Cr component by applying a second scaling factor to the residual samples of the Cr component.

[0289] According to an embodiment of the present disclosure, the AI-based video decoding method may have the following effect: before adding the residual samples of the Cb component and the residual samples of the Cr component to the predicted samples of the Cb component and the predicted samples of the Cr component, more refined residual samples are obtained by scaling the residual samples of the Cb component and the residual samples of the Cr component obtained via a neural network.

[0290] According to an embodiment of the present disclosure, the first scale factor and the second scale factor may be determined according to a scale factor value indicated by an index obtained from a bitstream and included in a scale factor set.

[0291] According to an embodiment of the present disclosure, the AI-based video decoding method may have the following effect: the residual samples of the Cb component and the residual samples of the Cr component are more effectively refined by applying an appropriate scaling factor according to an index obtained from a bitstream.

[0292] According to an embodiment of the present disclosure, a scale factor set may be determined from among a plurality of scale factor sets according to at least one of a type of a slice including a current block or a type of the current block.

[0293] According to an embodiment of the present disclosure, the AI-based video decoding method may have the following effects: by determining a scale factor set from a plurality of scale factor sets according to at least one of a slice including a current block or a type of the current block, by using an appropriate scale factor set according to the type of the current block to more effectively refine the residual samples of the Cb component and the residual samples of the Cr component.

[0294] According to an embodiment of the present disclosure, the plurality of scale factor sets may include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0295] According to an embodiment of the present disclosure, the AI-based video decoding method may have the following effect: the residual samples of the Cb component and the residual samples of the Cr component are more effectively refined by using a plurality of scale factor sets.

[0296] According to an embodiment of the present disclosure, a neural network may be trained to determine the correlation between Cb and Cr for each sample of a current block for training by receiving input values ​​of a joint chroma residual sample for training, a prediction sample of a Cb component for training, and a prediction sample of a Cr component for training.

[0297] According to an embodiment of the present disclosure, for the correlation between Cb and Cr, weights may be determined for the residual samples of the Cb component and the residual samples of the Cr component, respectively.

[0298] According to an embodiment of the present disclosure, the AI-based video decoding method may have the following effects: because the neural network learns the correlation between Cb and Cr for each sample of the current block, the values ​​of the residual samples of the Cb component and the residual samples of the Cr component for each sample of the current block are determined more accurately.

[0299] According to an embodiment of the present disclosure, the input value of the neural network may include at least one of a quantization step size of the current block, a quantization error of the current block, or a block obtained by downsampling a reconstructed luminance block corresponding to a current chroma block of the current block based on a chroma format.

[0300] According to an embodiment of the present disclosure, the AI-based video decoding method may have the following effect: a quantization error in a current block is more effectively removed during reconstruction of the current block by additionally using a quantization error or a quantization step size.

[0301] According to an embodiment of the present disclosure, the AI-based video decoding method may have the following effects: by additionally using a block obtained by downsampling a reconstructed luminance block based on a chroma format, the Cb component and Cr component of the current block are more efficiently reconstructed by using information of the reconstructed luminance block.

[0302] According to an embodiment of the present disclosure, a neural network may be trained based on first loss information and second loss information, wherein the first loss information corresponds to the difference between the original sample of the Cb component of the original block for training and the reconstructed sample of the Cb component of the reconstructed block for training obtained via the neural network, and the second loss information corresponds to the difference between the original sample of the Cr component of the original block for training and the reconstructed sample of the Cr component of the reconstructed block for training obtained via the neural network.

[0303] According to an embodiment of the present disclosure, the AI-based video decoding method may have the following effects: because the neural network is trained to minimize multiple loss information corresponding to the difference between the original sample and the reconstructed sample of each of the Cb component and the Cr component, the reconstructed samples of the Cb component and the reconstructed samples of the Cr component obtained via the neural network are reconstructed to be almost the same as the original samples of the Cb component and the original samples of the Cr component.

[0304] According to an embodiment of the present disclosure, a model of a neural network may be determined based on Cb cbf information indicating whether a transform coefficient level of a Cb component of a current block includes a non-zero Cb component and Cr cbf information indicating whether a transform coefficient level of a Cr component of a current block includes a non-zero Cr component, the Cb cbf information and the Cr cbf information being obtained from a bitstream.

[0305] According to an embodiment of the present disclosure, the AI-based video decoding method may have the following effects: because one of the models of the trained neural network is appropriately selected according to the Cb cbf information and the Cr cbf information, the Cb component and the Cr component of the current block are more efficiently reconstructed based on the conditions of the current block (such as the Cb cbf information and the Cr cbf information).

[0306] According to an embodiment of the present disclosure, a model of a neural network may be determined according to at least one of a type of a current block, a type of a slice including the current block, a QP range of a slice, or a Cb component or a Cr component of the current block.

[0307] According to an embodiment of the present disclosure, the AI-based video decoding method may have the following effects: because one of the models of the neural network trained according to various conditions is appropriately selected, the Cb component and the Cr component of the current block are more efficiently reconstructed based on the condition of the current block (such as the type of the current block, the type of the slice including the current block, the QP range of the slice, or whether it is the Cb component or the Cr component of the current block).

[0308] According to an embodiment of the present disclosure, an AI-based video decoding device may include: a memory storing at least one instruction; and at least one processor configured to operate according to at least one instruction. At least one processor may be configured to obtain a joint chroma residual sample of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block from a bitstream. At least one processor may be configured to determine a prediction sample of the Cb component of the current block and a prediction sample of the Cr component of the current block based on the Cb component prediction information and the Cr component prediction information. At least one processor may be configured to reconstruct the current block by applying the joint chroma residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component to a neural network to obtain a reconstructed sample of the Cb component of the current block and a reconstructed sample of the Cr component of the current block.

[0309] According to an embodiment of the present disclosure, an AI-based video decoding device may have the following effects: by using prediction information of the Cb component and prediction information of the Cr component, the Cb component and the Cr component may be more efficiently reconstructed via a neural network trained for a more general, local and specific relationship between the Cb component and the Cr component rather than a fixed correlation between the Cb component and the Cr component.

[0310] According to an embodiment of the present disclosure, at least one processor may be configured to obtain at least one of the residual samples of the Cb component of the current block or the residual samples of the Cr component of the current block by applying the joint chrominance residual samples, the predicted samples of the Cb component, and the predicted samples of the Cr component to a neural network. At least one processor may be configured to reconstruct the current block based on obtaining the reconstructed samples of the Cb component and the reconstructed samples of the Cr component by using at least one of the residual samples of the Cb component or the residual samples of the Cr component, the predicted samples of the Cb component, and the predicted samples of the Cr component.

[0311] According to an embodiment of the present disclosure, an AI-based video decoding device may have the following effects: by more accurately obtaining at least one of the residual samples of the Cb component and the residual samples of the Cr component through a neural network, the Cb component and the Cr component may be more efficiently reconstructed by using at least one of the residual samples of the Cb component and the residual samples of the Cr component.

[0312] According to an embodiment of the present disclosure, at least one processor may be configured to refine the residual samples of the Cb component by applying a first scaling factor to the residual samples of the Cb component obtained via a neural network, and to refine the residual samples of the Cr component by applying a second scaling factor to the residual samples of the Cr component.

[0313] According to an embodiment of the present disclosure, an AI-based video decoding device may have the following effect: before adding the residual samples of the Cb component and the residual samples of the Cr component to the predicted samples of the Cb component and the predicted samples of the Cr component, more refined residual samples are obtained by scaling the residual samples of the Cb component and the residual samples of the Cr component obtained via a neural network.

[0314] According to an embodiment of the present disclosure, the first scale factor and the second scale factor may be determined according to a scale factor value indicated by an index obtained from a bitstream and included in a scale factor set.

[0315] According to an embodiment of the present disclosure, an AI-based video decoding device may have the following effect: a residual sample of a Cb component and a residual sample of a Cr component may be more effectively refined by applying an appropriate scaling factor according to an index obtained from a bitstream.

[0316] According to an embodiment of the present disclosure, a scale factor set may be determined from a plurality of scale factor sets according to at least one of a type of a slice including a current block or a type of the current block.

[0317] According to an embodiment of the present disclosure, an AI-based video decoding device may have the following effects: by determining a scale factor set from a plurality of scale factor sets according to at least one of a slice including a current block or a type of the current block, a residual sample of a Cb component and a residual sample of a Cr component are more efficiently refined by using an appropriate scale factor set according to the type of the current block.

[0318] According to an embodiment of the present disclosure, the plurality of scale factor sets may include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0319] According to an embodiment of the present disclosure, an AI-based video decoding device may have the following effect: a residual sample of a Cb component and a residual sample of a Cr component may be more effectively refined by using a plurality of scale factor sets.

[0320] According to an embodiment of the present disclosure, a neural network may be trained to determine the correlation between Cb and Cr for each sample of a current block for training by receiving input values ​​of a joint chroma residual sample for training, a prediction sample of a Cb component for training, and a prediction sample of a Cr component for training.

[0321] According to an embodiment of the present disclosure, for the correlation between Cb and Cr, weights may be determined for the residual samples of the Cb component and the residual samples of the Cr component, respectively.

[0322] According to an embodiment of the present disclosure, an AI-based video decoding device may have the following effects: because the neural network learns the correlation between Cb and Cr for each sample of the current block, the values ​​of the residual samples of the Cb component and the residual samples of the Cr component for each sample of the current block are determined more accurately.

[0323] According to an embodiment of the present disclosure, the input value of the neural network may include at least one of a quantization step size of the current block, a quantization error of the current block, or a block obtained by downsampling a reconstructed luminance block corresponding to a current chroma block of the current block based on a chroma format.

[0324] According to an embodiment of the present disclosure, an AI-based video decoding device may have the following effect: a quantization error in a current block may be more effectively removed during reconstruction of the current block by additionally using a quantization error or a quantization step size.

[0325] According to an embodiment of the present disclosure, an AI-based video decoding device may have the following effects: by additionally using a block obtained by downsampling a reconstructed luminance block based on a chroma format, the Cb component and Cr component of the current block may be more efficiently reconstructed by using information of the reconstructed luminance block.

[0326] According to an embodiment of the present disclosure, a neural network may be trained based on first loss information and second loss information, wherein the first loss information corresponds to the difference between the original sample of the Cb component of the original block for training and the reconstructed sample of the Cb component of the reconstructed block for training obtained via the neural network, and the second loss information corresponds to the difference between the original sample of the Cr component of the original block for training and the reconstructed sample of the Cr component of the reconstructed block for training obtained via the neural network.

[0327] According to an embodiment of the present disclosure, an AI-based video decoding device may have the following effect: because a neural network is trained to minimize multiple loss information corresponding to the difference between the original sample and the reconstructed sample of each of the Cb component and the Cr component, the reconstructed samples of the Cb component and the reconstructed samples of the Cr component obtained via the neural network are reconstructed to be almost the same as the original samples of the Cb component and the original samples of the Cr component.

[0328] According to an embodiment of the present disclosure, a model of a neural network may be determined based on Cb cbf information indicating whether a transform coefficient level of a Cb component of a current block includes a non-zero Cb component and Cr cbf information indicating whether a transform coefficient level of a Cr component of a current block includes a non-zero Cr component, the Cb cbf information and the Cr cbf information being obtained from a bitstream.

[0329] According to an embodiment of the present disclosure, an AI-based video decoding device may have the following effects: because one of the models of the trained neural network is appropriately selected according to the Cb cbf information and the Cr cbf information, the Cb component and the Cr component of the current block are more efficiently reconstructed based on the conditions of the current block (such as the Cb cbf information and the Cr cbf information).

[0330] According to an embodiment of the present disclosure, a model of a neural network may be determined according to at least one of a type of a current block, a type of a slice including the current block, a QP range of a slice, or a Cb component or a Cr component of the current block.

[0331] According to an embodiment of the present disclosure, an AI-based video decoding device may have the following effects: because one of the models of a neural network trained according to various conditions is appropriately selected, the Cb component and the Cr component of the current block are more efficiently reconstructed based on the condition of the current block (such as the type of the current block, the type of the slice including the current block, the QP range of the slice, or whether the current block is a Cb component or a Cr component).

[0332] According to an embodiment of the present disclosure, an AI-based video encoding method may include: generating initial joint chroma residual samples of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block; determining prediction samples of the Cb component of the current block and prediction samples of the Cr component of the current block based on the Cb component prediction information and the Cr component prediction information; and encoding the joint chroma residual samples of the current block by applying the initial joint chroma residual samples, the prediction samples of the Cb component, and the prediction samples of the Cr component to a neural network to obtain reconstructed samples of the Cb component of the current block and reconstructed samples of the Cr component of the current block.

[0333] According to an embodiment of the present disclosure, an AI-based video encoding method may have the following effects: by using prediction information of the Cb component and prediction information of the Cr component, a joint chrominance residual sample that can more effectively reconstruct the Cb component and the Cr component is encoded via a neural network trained for a more general, local and specific relationship between the Cb component and the Cr component rather than a fixed correlation between the Cb component and the Cr component.

[0334] According to an embodiment of the present disclosure, an AI-based video encoding method may include: generating an initial joint chroma residual sample of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block; determining a prediction sample of the Cb component of the current block and a prediction sample of the Cr component of the current block based on the Cb component prediction information and the Cr component prediction information; obtaining at least one of a residual sample of the Cb component of the current block or a residual sample of the Cr component of the current block by applying the initial joint chroma residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component to a neural network; and encoding the joint chroma residual sample of the current block by obtaining a reconstructed sample of the Cb component and a reconstructed sample of the Cr component by using at least one of the residual sample of the Cb component or the residual sample of the Cr component, the prediction sample of the Cb component, and the prediction sample of the Cr component.

[0335] According to an embodiment of the present disclosure, the AI-based video encoding method may have the following effects: by more accurately obtaining at least one of the residual samples of the Cb component and the residual samples of the Cr component through a neural network, and by using at least one of the residual samples of the Cb component and the residual samples of the Cr component to encode the joint chrominance residual samples that can more effectively reconstruct the Cb component and the Cr component.

[0336] According to an embodiment of the present disclosure, the AI-based video encoding method may include: refining the residual samples of the Cb component by applying a first scaling factor to the residual samples of the Cb component obtained via a neural network, and refining the residual samples of the Cr component by applying a second scaling factor to the residual samples of the Cr component.

[0337] According to an embodiment of the present disclosure, the AI-based video encoding method may have the following effect: before adding the residual samples of the Cb component and the residual samples of the Cr component to the predicted samples of the Cb component and the predicted samples of the Cr component, more refined residual samples are obtained by scaling the residual samples of the Cb component and the residual samples of the Cr component obtained via a neural network.

[0338] According to an embodiment of the present disclosure, the first scale factor and the second scale factor may be encoded as indexes indicating scale factor values ​​included in a scale factor set.

[0339] According to an embodiment of the present disclosure, the AI-based video encoding method may have the following effect: a residual sample of a Cb component and a residual sample of a Cr component may be more efficiently refined by encoding and signaling an index indicating an appropriate scale factor.

[0340] According to an embodiment of the present disclosure, a scale factor set may be determined from a plurality of scale factor sets according to at least one of a type of a slice including a current block or a type of the current block.

[0341] According to an embodiment of the present disclosure, the AI-based video encoding method may have the following effects: by determining a scale factor set from a plurality of scale factor sets according to at least one of a slice including a current block or a type of the current block, by using an appropriate scale factor set according to the type of the current block to more effectively refine the residual samples of the Cb component and the residual samples of the Cr component.

[0342] According to an embodiment of the present disclosure, the plurality of scale factor sets may include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0343] According to an embodiment of the present disclosure, the AI-based video encoding method may have the following effect: a residual sample of a Cb component and a residual sample of a Cr component are more effectively refined by using a plurality of scale factor sets.

[0344] According to an embodiment of the present disclosure, a neural network may be trained to determine the correlation between Cb and Cr for each sample of a current block for training by receiving input values ​​of a joint chroma residual sample for training, a prediction sample of a Cb component for training, and a prediction sample of a Cr component for training.

[0345] According to an embodiment of the present disclosure, for the correlation between Cb and Cr, weights may be determined for the residual samples of the Cb component and the residual samples of the Cr component, respectively.

[0346] According to an embodiment of the present disclosure, the AI-based video encoding method may have the following effects: because the neural network learns the correlation between Cb and Cr for each sample of the current block, the values ​​of the residual samples of the Cb component and the residual samples of the Cr component for each sample of the current block are determined more accurately.

[0347] According to an embodiment of the present disclosure, the input value of the neural network may include at least one of a quantization step size of the current block, a quantization error of the current block, or a block obtained by downsampling a reconstructed luminance block corresponding to a current chroma block of the current block based on a chroma format.

[0348] According to an embodiment of the present disclosure, the AI-based video encoding method may have the following effect: a quantization error in a current block is more effectively removed during reconstruction of the current block by additionally using a quantization error or a quantization step size.

[0349] According to an embodiment of the present disclosure, the AI-based video encoding method may have the following effects: by additionally using a block obtained by downsampling a reconstructed luminance block based on a chroma format, the Cb component and Cr component of the current block are more efficiently reconstructed by using information of the reconstructed luminance block.

[0350] According to an embodiment of the present disclosure, a neural network may be trained based on first loss information and second loss information, wherein the first loss information corresponds to the difference between the original sample of the Cb component of the original block for training and the reconstructed sample of the Cb component of the reconstructed block for training obtained via the neural network, and the second loss information corresponds to the difference between the original sample of the Cr component of the original block for training and the reconstructed sample of the Cr component of the reconstructed block for training obtained via the neural network.

[0351] According to an embodiment of the present disclosure, the AI-based video encoding method may have the following effects: because the neural network is trained to minimize multiple loss information corresponding to the difference between the original sample and the reconstructed sample of each of the Cb component and the Cr component, the reconstructed samples of the Cb component and the reconstructed samples of the Cr component obtained via the neural network are reconstructed to be almost the same as the original samples of the Cb component and the original samples of the Cr component.

[0352] According to an embodiment of the present disclosure, a model of a neural network may be determined based on Cb cbf information indicating whether a transform coefficient level of a Cb component of a current block includes a non-zero Cb component and Cr cbf information indicating whether a transform coefficient level of a Cr component of a current block includes a non-zero Cr component.

[0353] According to an embodiment of the present disclosure, the AI-based video encoding method may have the following effects: because one of the models of the trained neural network is appropriately selected according to the Cb cbf information and the Cr cbf information, the Cb component and the Cr component of the current block are more efficiently reconstructed based on the conditions of the current block (such as the Cb cbf information and the Cr cbf information).

[0354] According to an embodiment of the present disclosure, a model of a neural network may be determined according to at least one of a type of a current block, a type of a slice including the current block, a QP range of a slice, or a Cb component or a Cr component of the current block.

[0355] According to an embodiment of the present disclosure, the AI-based video encoding method may have the following effects: because one of the models of the neural network trained according to various conditions is appropriately selected, the Cb component and the Cr component of the current block are more efficiently reconstructed based on the condition of the current block (such as the type of the current block, the type of the slice including the current block, the QP range of the slice, or whether the Cb component or the Cr component of the current block).

[0356] According to an embodiment of the present disclosure, an AI-based video encoding device may include: a memory storing at least one instruction; and at least one processor configured to operate according to at least one instruction. At least one processor may be configured to generate an initial joint chroma residual sample of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block. At least one processor may be configured to determine the prediction sample of the Cb component of the current block and the prediction sample of the Cr component of the current block based on the Cb component prediction information and the Cr component prediction information. At least one processor may be configured to encode the joint chroma residual sample of the current block by applying the initial joint chroma residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component to a neural network to obtain a reconstructed sample of the Cb component of the current block and a reconstructed sample of the Cr component of the current block.

[0357] According to an embodiment of the present disclosure, an AI-based video encoding device may have the following effects: by using prediction information of the Cb component and prediction information of the Cr component, a joint chrominance residual sample that can more effectively reconstruct the Cb component and the Cr component is encoded via a neural network trained for a more general, local and specific relationship between the Cb component and the Cr component rather than a fixed correlation between the Cb component and the Cr component.

[0358] According to an embodiment of the present disclosure, an AI-based video encoding device may include: a memory storing at least one instruction; and at least one processor configured to operate according to at least one instruction. At least one processor may be configured to generate an initial joint chroma residual sample of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block. At least one processor may be configured to determine the prediction sample of the Cb component of the current block and the prediction sample of the Cr component of the current block based on the Cb component prediction information and the Cr component prediction information. At least one processor may be configured to obtain at least one of the residual samples of the Cb component of the current block or the residual samples of the Cr component of the current block by applying the initial joint chroma residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component to a neural network. At least one processor may be configured to encode the joint chroma residual sample of the current block by obtaining the reconstructed sample of the Cb component and the reconstructed sample of the Cr component using at least one of the residual sample of the Cb component or the residual sample of the Cr component, the prediction sample of the Cb component, and the prediction sample of the Cr component.

[0359] According to an embodiment of the present disclosure, an AI-based video encoding device may have the following effects: by more accurately obtaining at least one of the residual samples of the Cb component and the residual samples of the Cr component through a neural network, and by using at least one of the residual samples of the Cb component and the residual samples of the Cr component to encode a joint chrominance residual sample that can more effectively reconstruct the Cb component and the Cr component.

[0360] According to an embodiment of the present disclosure, at least one processor may be configured to refine the residual samples of the Cb component by applying a first scaling factor to the residual samples of the Cb component obtained via a neural network, and to refine the residual samples of the Cr component by applying a second scaling factor to the residual samples of the Cr component.

[0361] According to an embodiment of the present disclosure, an AI-based video encoding device may have the following effect: before adding the residual samples of the Cb component and the residual samples of the Cr component to the predicted samples of the Cb component and the predicted samples of the Cr component, more refined residual samples are obtained by scaling the residual samples of the Cb component and the residual samples of the Cr component obtained via a neural network.

[0362] According to an embodiment of the present disclosure, the first scale factor and the second scale factor may be encoded as indexes indicating scale factor values ​​included in a scale factor set.

[0363] According to an embodiment of the present disclosure, an AI-based video encoding apparatus may have the effect of more efficiently refining residual samples of a Cb component and residual samples of a Cr component by encoding and signaling an index indicating an appropriate scale factor.

[0364] According to an embodiment of the present disclosure, a scale factor set may be determined from a plurality of scale factor sets according to at least one of a type of a slice including a current block or a type of the current block.

[0365] According to an embodiment of the present disclosure, an AI-based video encoding device may have the following effects: by determining a scale factor set from a plurality of scale factor sets according to at least one of a slice including a current block or a type of the current block, a residual sample of a Cb component and a residual sample of a Cr component may be more efficiently refined by using an appropriate scale factor set according to the type of the current block.

[0366] According to an embodiment of the present disclosure, the plurality of scale factor sets may include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0367] According to an embodiment of the present disclosure, an AI-based video encoding device may have the following effect: a residual sample of a Cb component and a residual sample of a Cr component may be more effectively refined by using a plurality of scale factor sets.

[0368] According to an embodiment of the present disclosure, a neural network may be trained to determine a correlation between Cb and Cr for each sample of a current block for training by receiving input values ​​of a joint chroma residual sample for training, a prediction sample of a Cb component for training, and a prediction sample of a Cr component for training.

[0369] According to an embodiment of the present disclosure, for the correlation between Cb and Cr, weights may be determined for the residual samples of the Cb component and the residual samples of the Cr component, respectively.

[0370] According to an embodiment of the present disclosure, an AI-based video encoding device may have the following effects: because the neural network learns the correlation between Cb and Cr for each sample of the current block, the values ​​of the residual samples of the Cb component and the residual samples of the Cr component for each sample of the current block are determined more accurately.

[0371] According to an embodiment of the present disclosure, the input value of the neural network may include at least one of a quantization step size of the current block, a quantization error of the current block, or a block obtained by downsampling a reconstructed luminance block corresponding to a current chroma block of the current block based on a chroma format.

[0372] According to an embodiment of the present disclosure, an AI-based video encoding device may have the following effect: a quantization error in a current block may be more effectively removed during reconstruction of the current block by additionally using a quantization error or a quantization step size.

[0373] According to an embodiment of the present disclosure, an AI-based video encoding device may have the following effects: by additionally using a block obtained by downsampling a reconstructed luminance block based on a chroma format, the Cb component and Cr component of the current block may be more efficiently reconstructed by using information of the reconstructed luminance block.

[0374] According to an embodiment of the present disclosure, a neural network may be trained based on first loss information and second loss information, wherein the first loss information corresponds to the difference between the original sample of the Cb component of the original block for training and the reconstructed sample of the Cb component of the reconstructed block for training obtained via the neural network, and the second loss information corresponds to the difference between the original sample of the Cr component of the original block for training and the reconstructed sample of the Cr component of the reconstructed block for training obtained via the neural network.

[0375] According to an embodiment of the present disclosure, an AI-based video encoding device may have the following effect: because a neural network is trained to minimize multiple loss information corresponding to the difference between the original sample and the reconstructed sample of each of the Cb component and the Cr component, the reconstructed samples of the Cb component and the reconstructed samples of the Cr component obtained via the neural network are reconstructed to be almost the same as the original samples of the Cb component and the original samples of the Cr component.

[0376] According to an embodiment of the present disclosure, a model of a neural network may be determined based on Cb cbf information indicating whether a transform coefficient level of a Cb component of a current block includes a non-zero Cb component and Cr cbf information indicating whether a transform coefficient level of a Cr component of a current block includes a non-zero Cr component.

[0377] According to an embodiment of the present disclosure, an AI-based video encoding device may have the following effects: because one of the models of the trained neural network is appropriately selected according to the Cb cbf information and the Cr cbf information, the Cb component and the Cr component of the current block are more efficiently reconstructed based on the conditions of the current block (such as the Cb cbf information and the Cr cbf information).

[0378] According to an embodiment of the present disclosure, a model of a neural network may be determined according to at least one of a type of a current block, a type of a slice including the current block, a QP range of a slice, or a Cb component or a Cr component of the current block.

[0379] According to an embodiment of the present disclosure, an AI-based video encoding device may have the following effects: because one of the models of a neural network trained according to various conditions is appropriately selected, the Cb component and the Cr component of the current block are more efficiently reconstructed based on the condition of the current block (such as the type of the current block, the type of the slice including the current block, the QP range of the slice, or whether the Cb component or the Cr component of the current block).

[0380] The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory storage medium" may mean that the storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), and may mean that data may be permanently or temporarily stored in the storage medium. For example, the "non-transitory storage medium" may include a buffer that temporarily stores data.

[0381] According to an embodiment of the present disclosure, the method according to various embodiments of the present disclosure may be included and set in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a CD-ROM), or distributed online (e.g., downloaded or uploaded) through an application store, or distributed directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily generated in a machine-readable storage medium (such as a memory of a manufacturer's server, an application store's server, or a relay server).

Claims

1. A video decoding method based on artificial intelligence AI, comprising: Obtaining a joint chroma residual sample of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block from a bitstream (S1110), wherein the current block includes a Cb component and a Cr component; Determine a prediction sample of the Cb component of the current block based at least on the Cb component prediction information; Determine a prediction sample of the Cr component of the current block based at least on the Cr component prediction information (S1130); as well as The current block is reconstructed by inputting the joint chroma residual samples, the predicted samples of the Cb component and the predicted samples of the Cr component into a neural network, and obtaining the reconstructed samples of the Cb component of the current block and the reconstructed samples of the Cr component of the current block from the output of the neural network (S1150).

2. The AI-based video decoding method according to claim 1, wherein: Reconstructing the current block includes: Obtaining at least one of a residual sample of the Cb component of the current block or a residual sample of the Cr component of the current block by inputting the joint chrominance residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component into the neural network; and The current block is reconstructed by obtaining reconstructed samples of the Cb component and reconstructed samples of the Cr component using at least one of the residual samples of the Cb component or the residual samples of the Cr component, the predicted samples of the Cb component, and the predicted samples of the Cr component.

3. The AI-based video decoding method according to any one of claims 1 to 2, wherein: Obtaining at least one of the residual samples of the Cb component or the residual samples of the Cr component includes: Refining the residual samples of the Cb component by applying a first scaling factor to the residual samples of the Cb component obtained via the neural network, and The residual samples of the Cr component are refined by applying a second scale factor to the residual samples of the Cr component obtained via the neural network.

4. The AI-based video decoding method according to any one of claims 1 to 3, wherein: The first scale factor and the second scale factor are determined according to a scale factor value indicated by an index obtained from the bitstream and included in a scale factor set.

5. The AI-based video decoding method according to any one of claims 1 to 4, wherein: The scale factor set is determined from a plurality of scale factor sets according to at least one of a type of a slice including the current block or a type of the current block.

6. The AI-based video decoding method according to any one of claims 1 to 5, wherein: The plurality of scale factor sets include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

7. The AI-based video decoding method according to any one of claims 1 to 6, wherein: The neural network is trained to determine the correlation between the Cb component and the Cr component for each sample of the current block for training by receiving input values ​​of the joint chroma residual sample for training, the prediction sample of the Cb component for training, and the prediction sample of the Cr component for training.

8. The AI-based video decoding method according to any one of claims 1 to 7, wherein: For the correlation between Cb and Cr, one or more weights are determined for the residual samples of the Cb component and the residual samples of the Cr component, respectively.

9. The AI-based video decoding method according to any one of claims 1 to 8, wherein: The input value of the neural network also includes at least one of the following: a quantization step size of the current block, a quantization error of the current block, or a block obtained by downsampling a reconstructed luminance block corresponding to a current chrominance block of the current block based on a chrominance format.

10. The AI-based video decoding method according to any one of claims 1 to 9, wherein: The neural network is trained according to first loss information and second loss information, wherein the first loss information corresponds to the difference between the original samples of the Cb component of the original block for training and the reconstructed samples of the Cb component of the reconstructed block for training obtained via the neural network, and the second loss information corresponds to the difference between the original samples of the Cr component of the original block for training and the reconstructed samples of the Cr component of the reconstructed block for training obtained via the neural network.

11. The AI-based video decoding method according to any one of claims 1 to 10, wherein: The model of the neural network is determined based on Cb coding block flag cbf information indicating whether the transformation coefficient level of the Cb component of the current block includes a non-zero Cb component and Cr cbf information indicating whether the transformation coefficient level of the Cr component of the current block includes a non-zero Cr component, and the Cb cbf information and the Cr cbf information are obtained from the bitstream.

12. The AI-based video decoding method according to any one of claims 1 to 11, wherein: The model of the neural network is determined according to at least one of the type of the current block, the type of the slice including the current block, the QP range of the slice, or the Cb component or the Cr component of the current block.

13. A video decoding device based on artificial intelligence AI, comprising: A memory storing at least one instruction; as well as At least one processor is configured to operate according to the at least one instruction, wherein the at least one processor is further configured to: obtaining a joint chroma residual sample of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block from a bitstream, wherein the current block includes a Cb component and a Cr component, determining a prediction sample of the Cb component of the current block based at least on the Cb component prediction information, determining a prediction sample of the Cr component of the current block based at least on the Cr component prediction information, and The current block is reconstructed by inputting the joint chrominance residual samples, the predicted samples of the Cb component and the predicted samples of the Cr component into a neural network, and obtaining the reconstructed samples of the Cb component of the current block and the reconstructed samples of the Cr component of the current block from the output of the neural network.

14. The AI-based video decoding device of claim 13, wherein: The at least one processor is further configured to: Obtaining at least one of a residual sample of the Cb component of the current block or a residual sample of the Cr component of the current block by inputting the joint chrominance residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component into the neural network; as well as The current block is reconstructed by obtaining reconstructed samples of the Cb component and reconstructed samples of the Cr component using at least one of the residual samples of the Cb component or the residual samples of the Cr component, the predicted samples of the Cb component, and the predicted samples of the Cr component.

15. A video encoding method based on artificial intelligence AI, comprising: Generate an initial joint chroma residual sample of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block (S1410); Determining a prediction sample of the Cb component of the current block based at least on the Cb component prediction information; Determine a prediction sample of the Cr component of the current block based at least on the Cr component prediction information (S1430); as well as The joint chroma residual samples of the current block are encoded by inputting the initial joint chroma residual samples, the predicted samples of the Cb component and the predicted samples of the Cr component into a neural network, and obtaining the reconstructed samples of the Cb component of the current block and the reconstructed samples of the Cr component of the current block from the output of the neural network (S1450).

Images