AI-based Video Decoding Device and Method for Chroma Component Prediction, and Video Encoding Device and Method

Abstract

Obtain the joint chroma residual samples of the current block, the prediction information of the Cb component of the current block, and the prediction information of the Cr component of the current block from the bitstream, determine the predicted samples of the Cb component of the current block based on at least the prediction information of the Cb component, determine the predicted samples of the Cr component of the current block based on at least the prediction information of the Cr component, input the joint chroma residual samples, the predicted samples of the Cb component, and the predicted samples of the Cr component into a neural network, and obtain the restored samples of the Cb component of the current block and the restored samples of the Cr component of the current block from the output of the neural network, thereby restoring the current block, and provide an AI-based video decoding method and apparatus.

Description

Technical Field

[0001] The present invention relates to a video decoding method and apparatus, and a video encoding method and apparatus. Specifically, it relates to a method and apparatus for performing prediction on Cb and Cr components, which are chroma components of an image, based on AI (Artificial Intelligence).

Background Art

[0002] In a codec such as H.266 Versatile Video Coding (VVC), in order to reduce the bitrate for separate coding of each chroma component, the residual samples of the Cb component and the Cr component are jointly encoded and decoded by joint coding using the similarity or correlation between the Cb component and the Cr component.

[0003] The correlation between the Cb component and the Cr component is adaptively selected for a predetermined number of cases to encode and decode the joint chroma residual samples for the Cb component and the Cr component.

[0004] Although such a rule-based chroma component prediction algorithm has exhibited good performance, an AI-based chroma component prediction algorithm that can flexibly consider the characteristics of an image as the resolution of the image increases and the content of the image diversifies can be implemented.

Summary of the Invention

Means for Solving the Problems

[0005] The AI-based video decoding method according to an embodiment of the present invention obtains the joint chroma residual sample of the current block, the prediction information of the Cb component of the current block, and the prediction information of the Cr component of the current block from a bitstream, where the current block includes a Cb component and a Cr component; determining a predicted sample of the Cb component of the current block based on at least the prediction information of the Cb component; determining a predicted sample of the Cr component of the current block based on at least the prediction information of the Cr component; and inputting the joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component into a neural network, and obtaining a restored sample of the Cb component of the current block and a restored sample of the Cr component of the current block from the output of the neural network, thereby restoring the current block may be included.

[0006] The AI-based video decoding apparatus according to an embodiment of the present invention includes a memory storing one or more instructions; and at least one processor operating according to the one or more instructions. The at least one processor obtains the joint chroma residual sample of the current block, the prediction information of the Cb component of the current block, and the prediction information of the Cr component of the current block from a bitstream, where the current block includes a Cb component and a Cr component. The at least one processor determines a predicted sample of the Cb component of the current block based on at least the prediction information of the Cb component. The at least one processor determines a predicted sample of the Cr component of the current block based on at least the prediction information of the Cr component. The at least one processor may input the joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component into a neural network, and obtain a restored sample of the Cb component of the current block and a restored sample of the Cr component of the current block from the output of the neural network, thereby restoring the current block.

[0007] The AI-based video encoding method according to an embodiment of the present invention includes steps of generating an initial joint chroma residual sample of a current block, prediction information of a Cb component of the current block, and prediction information of a Cr component of the current block; determining a predicted sample of the Cb component of the current block based at least on the prediction information of the Cb component; determining a predicted sample of the Cr component of the current block based at least on the prediction information of the Cr component; and inputting the initial joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component into a neural network, and obtaining a restored sample of the Cb component of the current block and a restored sample of the Cr component of the current block from an output of the neural network, thereby encoding a joint chroma residual sample of the current block.

[0008] The AI-based video encoding apparatus according to an embodiment of the present invention includes a memory storing one or more instructions; and at least one processor operating according to the one or more instructions. The at least one processor generates an initial joint chroma residual sample of a current block, prediction information of a Cb component of the current block, and prediction information of a Cr component of the current block. The at least one processor determines a predicted sample of the Cb component of the current block based at least on the prediction information of the Cb component. The at least one processor determines a predicted sample of the Cr component of the current block based at least on the prediction information of the Cr component. The at least one processor can encode a joint chroma residual sample of the current block by inputting the initial joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component into a neural network, and obtaining a restored sample of the Cb component of the current block and a restored sample of the Cr component of the current block from an output of the neural network.

Brief Description of the Drawings

[0009] The above aspects, other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the description to be described hereinafter together with the accompanying drawings.

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Modes for Carrying Out the Invention

[0011] The present invention can be subjected to various modifications and can have various embodiments. Specific embodiments are illustrated in the drawings and will be described in more detail by detailed description. However, this is not intended to limit the present invention to the embodiments, and the present invention should be understood to include all modifications, equivalents, or alternatives included in the spirit and technical scope of various embodiments.

[0012] When explaining the embodiments of the present invention, if it is determined that a specific description of related known technologies makes the gist of the present invention unnecessarily unclear, the detailed description thereof will be omitted. Also, the numbers (for example, the first, the second, etc.) used in the description process of the present invention are merely identification symbols for distinguishing one component from other components.

[0013] In the present invention, the expression "at least one of a, b, or c" refers to "a", "b", "c", "a and b", "a and c", "b and c", "any of a, b, and c", or a variation thereof.

[0014] Also, in the present invention, when one component is referred to as being "connected" or "connected to" another component, the one component may be directly connected or directly connected to the other component, but unless there is a contrary description, it should be understood that it may also be connected or connected via still other components in between.

[0015] In the present invention, components expressed as "~ part (unit)", "module", etc. may be combined into one component from two or more components, or one component may be further divided into two or more parts according to each subdivided function. Needless to say, each of the components described below may further perform some or all of the functions of other components in addition to the main function it is in charge of, and some of the functions of the main function that each component is in charge of may be exclusively performed by other components.

[0016] In the present invention, "image or picture" means a still image (or frame), a video or a video composed of a plurality of consecutive still images.

[0017] In the present invention, "neural network" is a typical example of an artificial neural network model that mimics the cranial nerves, and is not limited to an artificial neural network model using a specific algorithm. The neural network may be referred to as a deep neural network.

[0018] In the present invention, "weight" is a value used in the calculation process of each layer forming the neural network, and is used, for example, when applying an input value to a predetermined arithmetic expression. The weight is a value set as a result of training, and is updated by separate training data as necessary.

[0019] In the present invention, "current block" means a block that is the current processing target. The current block is a slice, tile, maximum coding unit, coding unit, prediction unit, or conversion unit divided from the current image.

[0020] In the present invention, "sample" is data assigned to a sampling position in data such as an image, a block, a filter kernel, or a feature map, and means data to be processed. For example, a sample includes pixels in a two-dimensional image.

[0021] Hereinafter, in FIGS. 1 and 2, a method for predicting the chroma components without a neural network will be described later.

[0022] FIG. 1 is a drawing for explaining the relationship among the Cb component, the Cr component, and the joint chroma residual according to an embodiment of the present invention.

[0023] Referring to FIG. 1, graph 100 shows the relationship among the actual value 140 of the residual sample of the Cr component, the actual value 110 of the residual sample of the Cb component, and the joint chroma residual sample 120 in the video encoding process. The joint chroma residual sample 120 is defined as the average value of the value 130 obtained by multiplying the actual value 140 of the residual sample of the Cr component by -1 and the actual value 110 of the residual sample of the Cb component.

[0024] Graph 150 shows an example of the residual sample 160 of the Cb component and the residual sample 170 of the Cr component obtained using the joint chroma residual sample 140 in the video decoding process. Specifically, the same value as the joint chroma residual sample 140 is defined as the restored value of the residual sample 160 of the Cb component, and the value generated by multiplying the joint chroma residual sample 140 by -1 is defined as the restored value of the residual sample 170 of the Cr component.

[0025] FIG. 2 is a drawing for explaining the relational expression among the Cb component, the Cr component, and the joint chroma residual according to an embodiment of the present invention.

[0026] Referring to FIG. 2, tu_cbf_cb is a syntax element corresponding to the Cb coded block flag (cbf) information, indicating whether the current transform block contains a Cb component with a non-zero transform coefficient level. tu_cbf_cr is a syntax element corresponding to the Crcbf information, indicating whether the current transform block contains a Cr component with a non-zero transform coefficient level. tu_joint_cbcr_residual_flag is a syntax element corresponding to the joint chroma information, indicating whether one chroma sample is coded to represent the residual samples of the Cb component and the corresponding Cr component in the current transform block.

[0027] In the video decoding process, the sps_joint_cbcr_enabled_flag is obtained from the sequence parameter set. When the sps_joint_cbcr_enabled_flag indicates 1, it means that the joint chroma mode in which one chroma sample is coded to represent the residual samples of the Cb component and the corresponding Cr component is allowed for the blocks included in the current sequence.

[0028] Therefore, first, when the sps_joint_cbcr_enabled_flag allows the joint chroma mode in the current sequence, the prediction type of the coding unit including the current transform block is the intra prediction type, and when the Cr cbf information or the Cb cbf information is 1, tu_joint_cbcr_residual_flag, that is, the joint chroma information, can be obtained. Also, when the prediction type of the coding unit including the current transform block is the inter prediction type, tu_joint_cbcr_residual_flag, that is, the joint chroma information, can be obtained only when both the Cr cbf information and the Cb cbf information are 1.

[0029] Therefore, when the prediction type of the coding unit is the intra prediction type and there is a transform coefficient with a level of 1 in the Cr component or a transform coefficient with a level of 1 in the Cb component in the current block, it is determined whether one chroma sample is to be coded in order to indicate the residual samples of the Cb component and the corresponding Cr component by joint chroma information. When the prediction type of the coding unit is the inter prediction type and there is a transform coefficient with a level of 1 in the Cr component and a transform coefficient with a level of 1 in the Cb component in the current block, it is determined whether one chroma sample is to be coded in order to indicate the residual samples of the Cb component and the corresponding Cr component by joint chroma information. Here, if the obtained joint chroma information indicates 0, it means that the residual samples of the Cb component and the corresponding Cr component can be obtained from the bitstream respectively. On the other hand, if the joint chroma information indicates 1, it means that one chroma sample is obtained from the bitstream, and from this, the residual samples of the Cb component and the corresponding Cr component can be restored.

[0030] If the Cb cbf information (tu_cbf_cb) and the Cr cbf information (tu_cbf_cr) are obtained, weight values for determining the restored value resCb of the residual samples of the Cb component and the restored value resCr of the residual samples of the Cr component are determined according to the Cb cbf information and the Cr cbf information.

[0031] For example, when tu_cbf_cb is 1 and tu_cbf_cr is 0, the restored value resCb of the residual samples of the Cb component is determined to be the same value as the joint chroma residual sample resJointC, and the restored value resCr of the residual samples of the Cr component is determined to be the value obtained by multiplying the joint chroma residual sample resJointC by ±1 / 2 based on the CSign value obtained in the picture parameter set. In this case, the weight value of the Cb component is 1, the weight value of the Cr component is ±1 / 2, and the joint mode index is 1.

[0032] For example, when tu_cbf_cb is 1 and tu_cbf_cr is 1, the restored value resCb of the residual sample of the Cb component is defined as the same value as the joint chroma residual sample resJointC, and the restored value resCr of the residual sample of the Cr component is defined as the value obtained by multiplying the joint chroma residual sample resJointC by ±1 based on the CSign value. In this case, the weighting value of the Cb component is 1, the weighting value of the Cr component is ±1, and the joint mode index is 2.

[0033] For example, when tu_cbf_cb is 0 and tu_cbf_cr is 1, the restored value resCb of the residual sample of the Cb component is defined as the value obtained by multiplying the joint chroma residual sample resJointC by ±1 / 2 based on the CSign value, and the restored value resCr of the residual sample of the Cr component is defined as the same value as the joint chroma residual sample resJointC. In this case, the weighting value of the Cb component is ±1 / 2, the weighting value of the Cr component is 1, and the joint mode index is 3.

[0034] When the prediction type of the coding unit is the intra prediction mode, the joint mode index can be any of 1, 2, and 3. For example, when the prediction type of the coding unit is the intra prediction mode, the weighting value of the Cb component and the weighting value of the Cr component are determined by the joint mode index, respectively.

[0035] When the prediction type of the coding unit is the inter prediction mode, only the joint mode index of 2 is allowed. Therefore, when the prediction type of the coding unit is the inter prediction mode, when the joint mode index is 2, (for example, only when tu_cbf_cb is 1 and tu_cbf_cr is 1) the joint chroma method is allowed, and the weighting value of the Cb component is defined as 1 and the weighting value of the Cr component is defined as ±1.

[0036] Hereinafter, in FIGS. 3 to 16, a method for predicting an AI-based chroma component using a neural network will be described later.

[0037] FIG. 3 is a drawing for explaining a neural network for AI-based chroma component prediction according to an embodiment of the present invention.

[0038] As shown in FIG. 3, the neural network 340 includes a first convolutional layer 341, a first activation layer 342, a second convolutional layer 343, a second activation layer 344, a third convolutional layer, a third activation layer, a fourth convolutional layer, a fourth activation layer, a fifth convolutional layer, a fifth activation layer, a sixth convolutional layer, a sixth activation layer, a seventh convolutional layer 345, a seventh activation layer 346, an eighth convolutional layer 347, an eighth activation layer 348, a ninth convolutional layer 349, and an addition layer 380. The addition layer 380 may be referred to as an adder.

[0039] A prediction sample 310 of the Cb component of the current block, a joint chroma residual sample 320 of the current block, and a prediction sample 330 of the Cr component of the current block are input to the first convolutional layer 341. The prediction sample 310 of the Cb component of the current block, the joint chroma residual sample 320 of the current block, and the prediction sample 330 of the Cr component of the current block are concatenated into one data item and input to the first convolutional layer 341.

[0040] The 3X3 shown in the first convolutional layer 341 exemplifies performing a convolution process on the prediction sample 310 of the Cb component of the current block, the joint chroma residual sample 320 of the current block, and the prediction sample 330 of the Cr component of the current block using a filter kernel of size 3X3. In one embodiment of the present invention, as shown in FIG. 3, there are 16 channels 350. Therefore, according to the convolution process result, a feature map of 16 channels 350 is generated by 16 filter kernels of size 3X3.

[0041] The feature map generated by the first convolutional layer 341 shows the unique characteristics of the prediction sample 310 of the Cb component of the current block, the joint chroma residual sample 320 of the current block, and the prediction sample 330 of the Cr component of the current block. For example, the feature map shows the vertical direction characteristics, horizontal direction characteristics, or edge characteristics, etc. of the prediction sample 310 of the Cb component of the current block, the joint chroma residual sample 320 of the current block, and the prediction sample 330 of the Cr component of the current block. In one embodiment of the present invention, the feature map is generated for each of the prediction sample 310 of the Cb component of the current block, the joint chroma residual sample 320 of the current block, and the prediction sample 330 of the Cr component of the current block.

[0042] The feature map output from the first convolutional layer 341 is input into the first activation layer 342.

[0043] The first activation layer 342 gives non-linear characteristics to the feature map. The non-linear characteristics show one or more non-linear characteristics. The first activation layer 342 includes a sigmoid function, a Tanh function, a ReLU (Rectified Linear Unit) function, etc., or includes any other suitable function known to those skilled in the art.

[0044] The application of the non-linear characteristics by the first activation layer 342 to the feature map means that some sample values of the feature map are changed. At this time, the change is performed by applying the non-linear characteristics.

[0045] The first activation layer 342 determines whether to transmit the sample values of the feature map to the second convolutional layer 343. For example, among the sample values of the feature map, some sample values are activated by the first activation layer 342 and transmitted to the second convolutional layer 343, and some sample values are deactivated by the first activation layer 342 and not transmitted to the second convolutional layer 343. The unique characteristics of the prediction sample 310 of the Cb component of the current block, the joint chroma residual sample 320 of the current block, and the prediction sample 330 of the Cr component of the current block shown by the feature map are emphasized by the first activation layer 342.

[0046] The 16-channel feature map 350 output from the first activation layer 342 is input to the second convolutional layer 343.

[0047] The 3X3 shown in the second convolutional layer 343 exemplifies performing a convolution process on the input feature map using a filter kernel of size 3X3. The output of the second convolutional layer 343 is input to the second activation layer 344. The second activation layer 344 gives non-linear characteristics to the input feature map.

[0048] The feature map output from the second activation layer 344 is repeatedly operated on by convolutional layers and activation layers, and is input and output in the order of the third convolutional layer, the third activation layer, the fourth convolutional layer, the fourth activation layer, the fifth convolutional layer, the fifth activation layer, the sixth convolutional layer, and the sixth activation layer. The output of the sixth activation layer is input to the seventh convolutional layer 345 again.

[0049] The 3X3 shown in the seventh convolutional layer 345 exemplifies performing a convolution process on the input feature map using a filter kernel of size 3X3. The output of the seventh convolutional layer 345 is input to the seventh activation layer 346. The 16-channel feature map 355 output from the seventh activation layer 346 is input to the eighth convolutional layer 347 again, and the output of the eighth convolutional layer 347 is input to the eighth activation layer 348.

[0050] The output of the eighth activation layer 348 is input to the ninth convolutional layer 349. The ninth convolutional layer 349 outputs two feature maps using two filter kernels of size 3X3.

[0051] The two feature maps output from the ninth convolutional layer 349 are respectively added to the predicted sample 310 of the Cb component of the current block and the predicted sample 330 of the Cr component of the current block in the addition layer 380, and the addition results, the restored sample 360 of the Cb component of the current block and the restored sample 370 of the Cr component of the current block are output.

[0052] The two feature maps output from the 9th convolutional layer 349 are respectively added to the predicted sample 310 of the Cb component of the current block and the predicted sample 330 of the Cr component of the current block, so that the restored sample 360 of the Cb component of the current block and the restored sample 370 of the Cr component of the current block are obtained. Therefore, the two feature maps output from the 9th convolutional layer 349 are respectively referred to as the residual sample of the Cb component of the current block and the residual sample of the Cr component of the current block.

[0053] Although the neural network 340 in FIG. 3 is shown to include 9 convolutional layers and 8 activation layers, this is just an example. According to an embodiment of the present invention, the number of convolutional layers and activation layers included in the neural network 340 can be variously changed.

[0054] Also, according to an embodiment of the present invention, the neural network 340 may be implemented via an RNN (recurrent neural network). For example, the CNN structure of the neural network 340 according to an embodiment of the present invention can be changed to an RNN structure.

[0055] Since the neural network 340 in FIG. 3 includes an addition layer 380, the restored sample 360 of the Cb component of the current block and the restored sample 370 of the Cr component of the current block are obtained from the neural network 340. For neural networks that do not include the addition layer 380, it will be described later with reference to FIG. 4.

[0056] FIG. 4 is a drawing for explaining a neural network for AI-based chroma component prediction according to an embodiment of the present invention.

[0057] As shown in FIG. 4, the neural network 440 includes a first convolutional layer 441, a first activation layer 442, a second convolutional layer 443, a second activation layer 444, a third convolutional layer, a third activation layer, a fourth convolutional layer, a fourth activation layer, a fifth convolutional layer, a fifth activation layer, a sixth convolutional layer, a sixth activation layer, a seventh convolutional layer 445, a seventh activation layer 446, an eighth convolutional layer 447, an eighth activation layer 448, and a ninth convolutional layer 449.

[0058] The prediction sample 410 of the Cb component of the current block, the joint chroma residual sample 420 of the current block, and the prediction sample 430 of the Cr component of the current block can be input to the first convolutional layer 441. The prediction sample 410 of the Cb component of the current block, the joint chroma residual sample 420 of the current block, and the prediction sample 430 of the Cr component of the current block can be concatenated into one data item and input to the first convolutional layer 441.

[0059] The 3X3 shown in the first convolutional layer 441 exemplifies performing a convolution process on the prediction sample 410 of the Cb component of the current block, the joint chroma residual sample 420 of the current block, and the prediction sample 430 of the Cr component of the current block using a filter kernel of size 3X3. In one embodiment of the present invention, as shown in FIG. 4, there are 16 channels 450. Therefore, as a result of the convolution process, a feature map of 16 channels 450 can be generated by 16 filter kernels of size 3X3.

[0060] The feature map generated by the first convolutional layer 441 shows the unique characteristics of the prediction sample 410 of the Cb component of the current block, the joint chroma residual sample 420 of the current block, and the prediction sample 430 of the Cr component of the current block. For example, the feature map shows vertical direction characteristics, horizontal direction characteristics, or edge characteristics of the prediction sample 410 of the Cb component of the current block, the joint chroma residual sample 420 of the current block, and the prediction sample 430 of the Cr component of the current block. In one embodiment of the present invention, the feature map is generated for each of the prediction sample 410 of the Cb component of the current block, the joint chroma residual sample 420 of the current block, and the prediction sample 430 of the Cr component of the current block.

[0061] The feature map output from the first convolutional layer 441 can be input to the first activation layer 442.

[0062] The first activation layer 442 imparts non-linear characteristics to the feature map. The non-linear characteristics exhibit one or more non-linear characteristics. The first activation layer 442 includes a sigmoid function, a Tanh function, a ReLU function, etc., or any other suitable function known to those skilled in the art.

[0063] The application of the non-linear characteristics by the first activation layer 442 to the feature map means that some sample values of the feature map are changed. At this time, the change is performed by applying the non-linear characteristics.

[0064] The first activation layer 442 determines whether to transmit the sample values of the feature map to the second convolutional layer 443. For example, among the sample values of the feature map, some sample values are activated by the first activation layer 442 and transmitted to the second convolutional layer 443, and some sample values are deactivated by the first activation layer 442 and not transmitted to the second convolutional layer 443. The inherent characteristics of the predicted sample 410 of the Cb component, the joint chroma residual sample 420 of the current block, and the predicted sample 430 of the Cr component of the current block shown in the feature map can be emphasized by the first activation layer 442.

[0065] The 16-channel feature map 450 output from the first activation layer 442 is input to the second convolutional layer 443.

[0066] The 3X3 shown in the second convolutional layer 443 exemplifies performing a convolution process on the input feature map using a 3X3-sized filter kernel. The output of the second convolutional layer 443 is input to the second activation layer 444. The second activation layer 444 imparts non-linear characteristics to the input feature map.

[0067] The feature map output from the second activation layer 444 can have the operations in the convolutional layer and the activation layer repeated, and be input and output in the order of the third convolutional layer, the third activation layer, the fourth convolutional layer, the fourth activation layer, the fifth convolutional layer, the fifth activation layer, the sixth convolutional layer, and the sixth activation layer. The output of the sixth activation layer is input to the seventh convolutional layer 445 again.

[0068] The 3X3 shown in the 7th convolutional layer 445 exemplifies performing a convolution process on the input feature map using a filter kernel of size 3X3. The output of the 7th convolutional layer 445 is input to the 7th activation layer 446. The 16-channel feature map 455 output from the 7th activation layer 446 is input again to the 8th convolutional layer 447, and the output of the 8th convolutional layer 447 is input to the 8th activation layer 448.

[0069] The output of the 8th activation layer 448 is input to the 9th convolutional layer 449. The 9th convolutional layer 449 outputs two feature maps using two filter kernels of size 3X3.

[0070] The two feature maps output from the 9th convolutional layer 449 can be referred to as the residual sample 460 of the Cb component of the current block and the residual sample 470 of the Cr component of the current block, respectively.

[0071] The neural network 440 in FIG. 4 shows that it includes 9 convolutional layers and 8 activation layers, but this is just an example. According to an embodiment of the present invention, the number of convolutional layers and activation layers included in the neural network 440 can be variously changed.

[0072] Also, according to an embodiment of the present invention, the neural network 440 may be implemented via an RNN (recurrent neural network). For example, the CNN structure of the neural network 440 according to an embodiment of the present invention is changed to an RNN structure.

[0073] Obtain the restored sample 460 of the Cb component of the current block and the restored sample 470 of the Cr component of the current block from the neural network 440.

[0074] FIG. 5 is a drawing for explaining a method of training a neural network for AI-based chroma component prediction according to an embodiment of the present invention.

[0075] FIG. 5 shows a training method of the neural network 520 for AI-based chroma component prediction. The prediction sample 505 of the Cb component of the current block for training shown in FIG. 5 corresponds to the prediction sample of the Cb component of the current block described above, the joint chroma residual sample 510 of the current block for training corresponds to the joint chroma residual sample of the current block described above, and the prediction sample 515 of the Cr component of the current block for training corresponds to the prediction sample of the Cr component of the current block described above.

[0076] According to the training method of the neural network 520 according to the present invention, in one embodiment, the restored sample 535 of the Cb component of the current block for training, which is the sum result of the residual sample 525 of the Cb component of the current block output from the neural network 520 and the prediction sample 505 of the Cb component of the current block for training, is the same as or similar to the original sample 545 of the Cb component of the current block for training, and the neural network 520 is trained so that the restored sample 540 of the Cr component of the current block for training, which is the sum result of the residual sample 530 of the Cr component of the current block output from the neural network 520 and the prediction sample 515 of the Cr component of the current block for training, is the same as or similar to the original sample 550 of the Cb component of the current block for training. For example, the loss information 555 corresponding to the sum of the difference between the restored sample 535 of the Cb component of the current block for training and the original sample 545 of the Cb component of the current block for training, and the difference between the restored sample 540 of the Cr component of the current block for training and the original sample 550 of the Cb component of the current block for training is used for training the neural network 520.

[0077] Specifically, first, the prediction sample 505 of the Cb component of the current block for training, the joint chroma residual sample 510 of the current block for training, and the prediction sample 515 of the Cr component of the current block for training are obtained. For example, the prediction sample 505 of the Cb component is determined based on at least the prediction information of the Cb component obtained from the bit stream, and the prediction sample 515 of the Cr component is determined based on at least the prediction information of the Cr component obtained from the bit stream.

[0078] After the prediction sample 505 of the Cb component of the current block for training, the joint chroma residual sample 510 of the current block for training, and the prediction sample 515 of the Cr component of the current block for training are input into the neural network 520, the residual sample 525 of the Cb component of the current block for training and the residual sample 530 of the Cr component of the current block for training output from the neural network 520 are obtained. The neural network 520 operates according to preset weight values.

[0079] The residual sample 525 of the Cb component of the current block for training is added to the prediction sample 505 of the Cb component of the current block for training to obtain the restored sample 535 of the Cb component of the current block for training. The residual sample 530 of the Cr component of the current block for training is added to the prediction sample 515 of the Cr component of the current block for training to obtain the restored sample 540 of the Cr component of the current block for training.

[0080] The loss information 555 corresponding to the sum of the difference between the restored sample 535 of the Cb component of the current block for training and the original sample 545 of the Cb component of the current block for training and the difference between the restored sample 540 of the Cr component of the current block for training and the original sample 550 of the Cr component of the current block for training is calculated, and the weight values set in the neural network 520 are updated by the loss information 555. The neural network 520 updates the weight values so that the loss information 555 decreases or is minimized.

[0081] The heart shape shown in FIG. 5 corresponds to the current block for training and the exemplar object, and the remaining part corresponds to the background. For the object 526 and the background 527 of the current block for training output from the neural network 520, a weight value of 1 is applied to the residual sample 525 of the Cb component of the current block for training, and a weight value of 1 is applied to the residual sample 530 of the Cr component of the current block for training and the object 531 of the current block for training, and a weight value of 1 / 2 may be applied to the background 532. Based on such a configuration, by the trained model of the neural network, the relationship between the Cb component and the Cr component is further generalized using the predicted samples of the Cb component and the Cr component, and is more local and specific. For example, for the corresponding pixels of the Cb component and the Cr component, instead of a fixed correlation relationship, different correlation relationships between the Cb component and the Cr component for the object, the background, the panorama, etc., or different correlation relationships between the Cb component and the Cr component for other parts of the object may be used to restore the Cb component and the Cr component.

[0082] In one embodiment of the present invention, the loss information 555 includes at least one of the L1-norm value, the L2-norm value, the SSIM (Structural Similarity) value, the PSNR-HVS (Peak Signal-To-Noise Ratio-Human Vision System) value, the MS-SSIM (Multiscale SSIM) value, the VIF (Variance Inflation Factor) value, or the VMAF (Video Multimethod Assessment Fusion) value for the sum of the difference between the restored sample 535 of the Cb component of the current block for training and the original sample 545 of the Cb component of the current block for training and the difference between the restored sample 540 of the Cr component of the current block for training and the original sample 550 of the Cr component of the current block for training.

[0083] The training of the neural network 520 according to an embodiment of the present invention is performed by a training device. The training device is the AI-based video encoding device 1600 or the AI-based video decoding device 1300. According to an embodiment of the present invention, the training device may be an external server. In this case, the neural network 520 and the weight values trained by the external server can be transmitted to the AI-based video encoding device 1600 or the AI-based video decoding device 1300 via a network. Also, the trained neural network can be downloaded on one or more devices.

[0084] According to an embodiment of the present invention, in addition to the prediction sample 505 of the Cb component of the current block for training, the joint chroma residual sample 510 of the current block for training, and the prediction sample 515 of the Cr component of the current block for training, other inputs are added to the neural network 520. For example, other inputs to be added are, when the current block is an intra-block or an inter-block, the quantization step size of the current block, when the current block is an intra-block or an inter-block, the quantization error of the current block, when the current block is an inter-block, the quantization step size or quantization error of the reference frame of the current block, and the restored luma block corresponding to the current chroma block of the current block is a block downsampled based on the chroma format (for example, if the chroma format is 4:2:0, a block downscaled by 2), or any other suitable method.

[0085] The quantization error of the current block includes, as a sample value, the quantization error value calculated from the quantization parameter. This value indicates the quantization error for each sample included in the current block. The size of the quantization error map including the quantization error value targeted for the additional input of the neural network corresponds to the size of the current block.

[0086] The quantization error value indicates the amount of error caused by quantization and inverse quantization applied to the residual sample in the encoding and decoding processes for the original block.

[0087] A large quantization error value means that the difference between the conversion coefficient before quantization and the conversion coefficient after inverse quantization is large. The larger the difference between the conversion coefficient before quantization and the conversion coefficient after inverse quantization, the lower the identity between the original block and the current block obtained by decoding the encoded data.

[0088] Since the error caused by quantization and inverse quantization corresponds to artifacts, it is necessary to perform AI-based video encoding and video decoding considering the quantization error value.

[0089] In one embodiment of the present invention, the quantization error value can be calculated from the following Equation 1. Quantization error value = quantization step size^2 / 12 (Equation 1)

[0090] Referring to Equation 1, the quantization error value is proportional to the value obtained by squaring the quantization step size.

[0091] In one embodiment of the present invention, the quantization step size is a value used for quantization of the conversion coefficient, and the conversion coefficient is quantized by dividing the conversion coefficient by the quantization step size. Conversely, the quantized conversion coefficient is inverse quantized by multiplying the quantized conversion coefficient by the quantization step size.

[0092] The quantization step size is approximated by the following Equation 2. Quantization step size = 2^(quantization parameter / n) / quantization scale[quantization parameter % n] (Equation 2)

[0093] In Equation 2, quantization scale[quantization parameter % n] indicates the scale value indicated by the quantization parameter among n predetermined scale values. In the HEVC codec, since six scale values (26214, 23302, 20560, 18396, 16384, and 14564) are defined, according to the HEVC codec, n is 6.

[0094] Referring to Equation 1 and Equation 2, as the quantization parameter increases, the quantization step size increases and the quantization error value increases.

[0095] By further considering the quantization error or the quantization step size, the neural network can more efficiently remove the quantization error within the current block when restoring the current block.

[0096] In one embodiment of the present invention, the neural network can more efficiently restore the current chroma block using the information of the restored luma block by using the downsampled block based on the chroma format for the restored luma block corresponding to the current chroma block of the current block.

[0097] According to one embodiment of the present invention, there are multiple models of the neural network 520. For example, depending on whether the type of slice containing the current block is an intra-slice or an inter-slice, one of the multiple models can be selected.

[0098] According to one embodiment of the present invention, there are multiple models of the neural network 520. For example, depending on whether the coded block type of the current block is an intra-block or an inter-block, one of the multiple models can be selected.

[0099] According to one embodiment of the present invention, there are multiple models of the neural network 520. For example, depending on the QP range (QP range) of the slice containing the current block, one of the multiple models can be selected. Specifically, if the QP range is 0 or more and less than a, model 1 is selected from the multiple models; if the QP range is a or more and less than b, model 2 is selected from the multiple models; if the QP range is b or more and less than c, model 3 can be selected from the multiple models.

[0100] According to an embodiment of the present invention, there are a plurality of models of the neural network 520. For example, based on the Cb component and the Cr component of the current block, one model can be selected from a plurality of models. Specifically, if one of the residuals of the Cb component and the Cr component is used as it is, a weight value 1 is applied, and for the remaining one of the residuals of the Cb component and the Cr component, a model of the neural network can be applied.

[0101] FIG. 6 is a drawing for explaining a method of training a neural network for AI-based chroma component prediction according to an embodiment of the present invention.

[0102] The neural network 620 in FIG. 6 is different from the neural network 520 in FIG. 5 in that the output of the neural network is a restored sample of the Cb component and a restored sample of the Cr component instead of the residual sample of the Cb component and the residual sample of the Cr component.

[0103] FIG. 6 shows a training method of the neural network 620 for AI-based chroma component prediction. The prediction sample 605 of the Cb component of the current block for training shown in FIG. 6 corresponds to the prediction sample of the Cb component of the current block described above, the joint chroma residual sample 610 of the current block for training corresponds to the joint chroma residual sample of the current block described above, and the prediction sample 615 of the Cr component of the current block for training corresponds to the prediction sample of the Cr component of the current block described above.

[0104] According to the training method of the neural network 620 according to the present invention, the restoration sample 635 of the Cb component of the current block for training output from the neural network 620 according to an embodiment of the present invention is the same as or similar to the original sample 645 of the Cb component of the current block for training, and the neural network 620 is trained so that the restoration sample 640 of the Cr component of the current block for training output from the neural network 620 is the same as or similar to the original sample 650 of the Cr component of the current block for training. For example, loss information 655 corresponding to the sum of the difference between the restoration sample 635 of the Cb component of the current block for training and the original sample 645 of the Cb component of the current block for training, and the difference between the restoration sample 640 of the Cr component of the current block for training and the original sample 650 of the Cr component of the current block for training is used for training the neural network 620.

[0105] Specifically, first, the prediction sample 605 of the Cb component of the current block for training, the joint chroma residual sample 610 of the current block for training, and the prediction sample 615 of the Cr component of the current block for training are obtained. For example, the prediction sample 605 of the Cb component is determined based on at least the Cb component prediction information obtained from the bitstream, and the prediction sample 615 of the Cr component is determined based on at least the Cr component prediction information obtained from the bitstream.

[0106] After the prediction sample 505 of the Cb component of the current block for training, the joint chroma residual sample 510 of the current block for training, and the prediction sample 515 of the Cr component of the current block for training are input into the neural network 500, the restoration sample 635 of the Cb component of the current block for training and the restoration sample 640 of the Cr component of the current block for training output from the neural network 500 are obtained. The neural network 620 operates according to preset weight values.

[0107] Loss information 655 corresponding to the sum of the difference between the restored sample 635 of the Cb component of the current block for training and the original sample 645 of the Cb component of the current block for training, and the difference between the restored sample 640 of the Cr component of the current block for training and the original sample 650 of the Cr component of the current block for training is calculated, and the weighted values set in the neural network 620 are updated by the loss information 655. The neural network 620 updates the weighted values so that the loss information 655 decreases or is minimized.

[0108] The loss information 655 includes at least one of the L1-norm value, L2-norm value, SSIM (Structural Similarity) value, PSNR-HVS (Peak Signal-To-Noise Ratio-Human Vision System) value, MS-SSIM (Multiscale SSIM) value, VIF (Variance Inflation Factor) value, or VMAF (Video Multimethod Assessment Fusion) value with respect to the sum of the difference between the restored sample 635 of the Cb component of the current block for training and the original sample 645 of the Cb component of the current block for training, and the difference between the restored sample 640 of the Cr component of the current block for training and the original sample 650 of the Cr component of the current block for training.

[0109] The training of the neural network 620 according to an embodiment of the present invention is performed by a training device. The training device is the AI-based video encoding device 1500 or the AI-based video decoding device 1200. According to an embodiment of the present invention, the training device may be an external server. In this case, the neural network 620 and the weighted values trained by the external server can be transmitted to the AI-based video encoding device 1500 or the AI-based video decoding device 1200. In an embodiment of the present invention, the trained neural network can be downloaded to one or more devices.

[0110] According to an embodiment of the present invention, in addition to the prediction sample 505 of the Cb component of the current block for training, the joint chroma residual sample 510 of the current block for training, and the prediction sample 515 of the Cr component of the current block for training, other inputs are added to the neural network 620. For example, the other inputs to be added may be the quantization step size of the current block, the quantization error of the current block, if the current block is an inter-block, the quantization step size or quantization error of the reference frame of the current block, and the restored luma block corresponding to the current chroma block of the current block is downsampled based on the chroma format (for example, if the chroma format is 4:2:0, the block downscaled by about 2), or any other suitable method.

[0111] According to an embodiment of the present invention, there are a plurality of models of the neural network 620. For example, depending on whether the type of slice in which the current block is included is an intra-slice or an inter-slice, one of the plurality of models can be selected.

[0112] According to an embodiment of the present invention, there are a plurality of models of the neural network 620. For example, depending on whether the encoding block type of the current block is an intra-block or an inter-block, one of the plurality of models can be selected.

[0113] According to an embodiment of the present invention, there are a plurality of models of the neural network 620. For example, depending on the QP range of the slice in which the current block is included, one of the plurality of models can be selected. Specifically, if the QP range is 0 or more and less than a, model 1 is selected from the plurality of models, if the QP range is a or more and less than b, model 2 is selected from the plurality of models, and if the QP range is b or more and less than c, model 3 can be selected from the plurality of models.

[0114] According to an embodiment of the present invention, there are a plurality of models of the neural network 620. For example, based on the Cb component and the Cr component of the current block, one model can be selected from a plurality of models. Specifically, if one of the residuals of the Cb component and the Cr component is used as it is, a weight value 1 is applied, and for the remaining one of the residuals of the Cb component and the Cr component, a model of the neural network can be applied.

[0115] FIG. 7 is a table showing the correspondence between the model of the neural network for AI-based chroma component prediction and the prediction type according to an embodiment of the present invention. Such a correspondence can be pre-stored in the AI-based video encoding devices 1500 and 1600 or the AI-based video decoding devices 1200 and 1300.

[0116] Referring to FIG. 7, the model of the neural network used may be different depending on the prediction type. If the prediction type is an intra prediction type, one model is used from among models 1, 2, and 3 of the plurality of models, and if the prediction type is an inter prediction type, one model is used from among models 1 and 2 of the plurality of models.

[0117] Also, as shown in FIG. 2 described above, the models used by the Cb cbf information (tu_cbf_cb) and the Cr cbf information (tu_cbf_cr) may be different.

[0118] Specifically, when the Cb cbf information (tu_cbf_cb) and the Cr cbf information (tu_cbf_cr) are obtained, a model for determining the restored value resCb of the residual sample of the Cb component and the restored value resCr of the residual sample of the Cr component is determined by the Cb cbf information and the Cr cbf information.

[0119] For example, when tu_cbf_cb is 1 and tu_cbf_cr is 0, the model for the restored value resCb of the residual sample of the Cb component and the restored value resCr of the residual sample of the Cr component is determined as model 1 among the models of the neural network. In this case, the joint mode index is 1.

[0120] For example, when tu_cbf_cb is 1 and tu_cbf_cr is 1, the models for the restored value resCb of the residual sample of the Cb component and the restored value resCr of the residual sample of the Cr component are defined as Model 2 among the neural network models. In this case, the joint mode index is 2.

[0121] For example, when tu_cbf_cb is 0 and tu_cbf_cr is 1, the models for the restored value resCb of the residual sample of the Cb component and the restored value resCr of the residual sample of the Cr component are defined as Model 3 among the neural network models. In this case, the joint mode index is 3.

[0122] In one embodiment of the present invention, when the prediction type of the coding unit is the intra prediction mode, any of the index values 1, 2, and 3 can be used as the joint mode index. For example, when the prediction type of the coding unit is the intra prediction mode, the models used for the Cb component and the Cr component are determined by the joint mode index.

[0123] In one embodiment of the present invention, when the prediction type of the coding unit is the inter prediction mode, only the joint mode index of 2 is allowed. Therefore, when the prediction type of the coding unit is the inter prediction mode, when the joint mode index is 2, that is, when tu_cbf_cb is 1 and tu_cbf_cr is 1, only the joint chroma method is allowed, and the models for the Cb component and the Cr component are defined as Model 2.

[0124] FIG. 8 is a table showing the correspondence between the neural network models for AI-based chroma component prediction and the prediction types according to one embodiment of the present invention. Such a correspondence can be stored in advance in the AI-based video encoding devices 1500 and 1600 or the AI-based video decoding devices 1200 and 1300, etc.

[0125] Referring to FIG. 8, if the prediction type is an intra prediction type, for the Cb component, it is obtained by multiplying the joint chroma residual by a weighting value of 1, and for the Cr component, it is obtained by applying the joint chroma residual to model 1 of the neural network ([1, model 1]). For both the Cb component and the Cr component, it can also be obtained by applying the joint chroma residual to model 2 of the neural network ([model 2]). For the Cb component, it is obtained by applying the joint chroma residual to model 3 of the neural network, and for the Cr component, it is obtained by multiplying the joint chroma residual by a weighting value of 1 ([model 3, 1]). One of these methods is used. If the prediction type is an inter prediction type, for the Cb component, it is obtained by multiplying the joint chroma residual by a weighting value of 1, and for the Cr component, the method of applying the joint chroma residual to model 2 of the neural network ([1, model 2]) is used.

[0126] Also, as shown in FIG. 2 described above, the models used for the Cb cbf information (tu_cbf_cb) and the Cr cbf information (tu_cbf_cr) may be different.

[0127] Specifically, if the Cb cbf information (tu_cbf_cb) and the Cr cbf information (tu_cbf_cr) are obtained, a model or weighting value for determining the restored value resCb of the residual sample of the Cb component and the restored value resCr of the residual sample of the Cr component is determined based on the Cb cbf information and the Cr cbf information.

[0128] For example, when tu_cbf_cb is 1 and tu_cbf_cr is 0, the weighting value of the restored value resCb of the residual sample of the Cb component is determined to be 1, and the model for the restored value resCr of the residual sample of the Cr component is determined to be model 1. In this case, the joint mode index is 1.

[0129] For example, when tu_cbf_cb is 1 and tu_cbf_cr is 1, the models for the restored value resCb of the residual sample of the Cb component and the restored value resCr of the residual sample of the Cr component are both determined to be model 2 among the neural network models. In this case, the joint mode index is 2.

[0130] For example, when tu_cbf_cb is 0 and tu_cbf_cr is 1, the model for the restored value resCb of the residual sample of the Cb component is defined as model 3 among the neural network models, and the weighting value for the restored value resCr of the residual sample of the Cr component is defined as 1. In this case, the joint mode index is 3.

[0131] In one embodiment of the present invention, when the prediction type of the coding unit is the intra prediction mode, any of the index values 1, 2, and 3 can be used as the joint mode index. For example, when the prediction type of the coding unit is the intra prediction mode, the weighting value or model used for the Cb component and the Cr component is determined by the joint mode index.

[0132] In one embodiment of the present invention, when the prediction type of the coding unit is the inter prediction mode, only the joint mode index 2 is allowed. Therefore, when the prediction type of the coding unit is the inter prediction mode, when the joint mode index is 2, (for example, only when tu_cbf_cb is 1 and tu_cbf_cr is 1) the joint chroma method is allowed, and the models for the Cb component and the Cr component are defined as model 2.

[0133] FIG. 9 is a drawing for explaining a neural network and a scale factor for AI-based chroma component prediction according to an embodiment of the present invention.

[0134] Referring to FIG. 9, the prediction sample 905 of the Cb component of the current block, the joint chroma residual sample 910 of the current block, and the prediction sample 915 of the Cr component of the current block are input to the neural network 920. Before the prediction sample 905 of the Cb component of the current block and the prediction sample 915 of the Cr component of the current block are added to the residual sample 925 of the Cb component and the residual sample 930 of the Cr component output via the neural network 920, further scale factors 945 and 950 (for example, also referred to as the first scale factor 945 and the second scale factor 950) are respectively multiplied.

[0135] Specifically, the residual sample 925 of the Cb component output via the neural network 920 is multiplied by the first scale factor 945, and the residual sample 925 of the Cb component can be refined. Then, by adding the prediction sample 905 of the Cb component of the current block to the refined residual sample of the Cb component, the restored sample 935 of the Cb component of the current block is obtained. The residual sample 930 of the Cr component output via the neural network 920 is multiplied by the second scale factor 950, and the residual sample 930 of the Cr component can be refined. Then, by adding the prediction sample 915 of the Cr component of the current block to the refined residual sample of the Cr component, the restored sample 940 of the Cr component of the current block can be obtained.

[0136] Even if the trained model of the neural network is used, if the residual sample value of the Cb component or the Cr component is different from the original residual value, the residual sample needs to be refined, so a scale factor is further used. An example of the scale factor will be described later with reference to FIG. 10.

[0137] Also, when the restored sample of the Cb component of the current block and the restored sample of the Cr component of the current block are obtained with the prediction sample of the Cb component of the current block, the joint chroma residual sample of the current block, and the prediction sample of the Cr component of the current block as inputs, similar to the neural network in FIG. 6 described above, a scale factor is further used.

[0138] According to an embodiment of the present invention, a prediction sample of the Cb component of the current block is subtracted from a restored sample of the Cb component of the current block, which is one of the outputs of the neural network, to obtain a residual sample of the Cb component of the current block. A prediction sample of the Cr component of the current block is subtracted from a restored sample of the Cr component of the current block, which is another one of the outputs of the neural network, to obtain a residual sample of the Cr component of the current block. The residual sample of the Cb component of the current block is multiplied by a first scale factor to obtain a refined residual sample of the Cb component. The residual sample of the Cr component of the current block is multiplied by the first scale factor to obtain a refined residual sample of the Cr component. By adding the prediction sample of the Cb component of the current block to the refined residual sample of the Cb component again, a restored sample of the refined Cb component can be obtained. By adding the prediction sample of the Cr component of the current block to the refined residual sample of the Cr component again, a restored sample of the refined Cb component can be obtained.

[0139] Even in the case of a neural network trained to output a restored sample immediately, since purification by scaling is necessary, in such a neural network, a prediction sample is subtracted from the restored sample to obtain a residual sample, and a refined sample value can be obtained by applying a scale factor to the residual sample.

[0140] FIG. 10 is a table showing the correspondence between a scale factor set and a prediction type according to an embodiment of the present invention. Such a correspondence can be stored in advance in the AI-based video encoding device 1600 or the AI-based video decoding device 1300 or the like.

[0141] Referring to FIG. 10, if the prediction type is an intra prediction type, the scale factor set includes 1, 1 / 2, 1 / 4. If the prediction type is an inter prediction type, the scale factor set includes 1, 1 / 2.

[0142] For example, one of a plurality of scale factors included in a scale factor set that varies depending on the prediction type is selected and used. An index corresponding to the scale factor value may be signaled for each block or each slice.

[0143] According to one embodiment of the present invention, the same scale factor is used for the Cb component and the Cr component.

[0144] According to one embodiment of the present invention, when the scale factor is signaled for each block or each slice, different scale factors are used for the Cb component and the Cr component.

[0145] The prediction type is the prediction type of the slice or the prediction type of the coded block. That is, the scale factor set can be selected based on the slice type or based on the block type.

[0146] FIG. 11 is a flowchart of an AI-based video decoding method according to one embodiment of the present invention.

[0147] In S1110, the AI-based video decoding apparatuses 1200 and 1300 obtain the joint chroma residual sample of the current block, the Cb component prediction information of the current block, and the Cr component prediction information of the current block from the bitstream.

[0148] According to one embodiment of the present invention, the joint chroma residual sample of the current block is the average value of the actual value of the residual sample of the Cb component of the current block and the actual value of the residual sample of the Cr component of the current block.

[0149] According to one embodiment of the present invention, the joint chroma residual sample of the current block is also the same as the actual value of the residual sample of the Cb component of the current block.

[0150] According to an embodiment of the present invention, the joint chroma residual sample of the current block is also the same as the actual value of the residual sample of the Cr component of the current block.

[0151] According to an embodiment of the present invention, the joint chroma residual sample of the current block is determined by SATD (Sum of Transform Difference) or RDO (Rate Distortion Optimization) calculation on the encoding side and transmitted through the bitstream.

[0152] In S1130, the AI-based video decoding apparatuses 1200 and 1300 determine the predicted sample of the Cb component of the current block and the predicted sample of the Cr component of the current block based on the Cb component prediction information and the Cr component prediction information. For example, the predicted sample of the Cb component is determined based on at least the Cb component prediction information obtained from the bitstream, and the predicted sample of the Cr component is determined based on at least the Cr component prediction information obtained from the bitstream.

[0153] In S1150, the AI-based video decoding apparatuses 1200 and 1300 apply the joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component to the neural network to obtain the restored sample of the Cb component of the current block and the restored sample of the Cr component of the current block, thereby restoring the current block.

[0154] According to an embodiment of the present invention, the AI-based video decoding apparatus 1300 applies the joint chroma residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component to the neural network to obtain at least one of the residual sample of the Cb component of the current block or the residual sample of the Cr component of the current block. The AI-based video decoding apparatus 1300 restores the current block by obtaining the restored sample of the Cb component and the restored sample of the Cr component using at least one of the residual sample of the Cb component or the residual sample of the Cr component, the prediction sample of the Cb component, and the prediction sample of the Cr component.

[0155] According to an embodiment of the present invention, the AI-based video decoding apparatus 1300 applies a first scale factor to the residual sample of the Cb component obtained through the neural network to refine the residual sample of the Cb component, and applies a second scale factor to the residual sample of the Cr component to refine the residual sample of the Cr component.

[0156] According to an embodiment of the present invention, the first scale factor and the second scale factor are included in a scale factor set and are determined by the scale factor value indicated by the index obtained from the bitstream.

[0157] According to an embodiment of the present invention, the scale factor set is determined by at least one of the type of the slice in which the current block is included or the type of the current block among a plurality of scale factor sets.

[0158] According to an embodiment of the present invention, the plurality of scale factor sets include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0159] According to an embodiment of the present invention, the neural network can be trained to determine the correlation between Cb and Cr for each sample of the current block for training, using the joint chroma residual sample for training, the predicted sample of the Cb component for training, and the predicted sample of the Cr component for training as input values.

[0160] According to an embodiment of the present invention, for the correlation between Cb and Cr, a weighting value for the residual sample of the Cb component and a weighting value for the residual sample of the Cr component are respectively determined.

[0161] According to an embodiment of the present invention, the input value of the neural network further includes at least one of the quantization step size of the current block, the quantization error of the current block, or the restored luma block corresponding to the current chroma block of the current block, which is downsampled based on the chroma format.

[0162] According to an embodiment of the present invention, the neural network can be trained by the first loss information corresponding to the difference between the original sample of the Cb component of the original block for training and the restored sample of the Cb component of the restored block for training obtained through the neural network, and the second loss information corresponding to the difference between the original sample of the Cr component of the original block for training and the restored sample of the Cr component of the restored block for training obtained through the neural network.

[0163] According to an embodiment of the present invention, the model of the neural network is determined based on the Cb cbf information indicating whether the current block includes a Cb component with a non-zero transform coefficient level for the Cb component and the Cr cbf information indicating whether the current block includes a Cr component with a non-zero transform coefficient level for the Cr component, obtained from the bitstream.

[0164] According to an embodiment of the present invention, the model of the neural network is determined by at least one of the type of the current block, the type of the slice in which the current block is included, the QP range of the slice, or whether the current block is a Cb component or a Cr component.

[0165] FIG. 12 is a block diagram showing the configuration of an AI-based video decoding apparatus according to an embodiment of the present invention.

[0166] Referring to FIG. 12, the AI-based video decoding apparatus 1200 includes a decoding unit 1210, a prediction unit 1220, an AI-based image processing unit 1230, and a restoration unit 1240.

[0167] The decoding unit 1210, the prediction unit 1220, the AI-based image processing unit 1230, and the restoration unit 1240 may be implemented by one or more processors. The decoding unit 1210, the prediction unit 1220, the AI-based image processing unit 1230, and the restoration unit 1240 operate according to instructions stored in a memory. The instructions cause one or more processors to perform one or more operations.

[0168] Although FIG. 12 shows the decoding unit 1210, the prediction unit 1220, the AI-based image processing unit 1230, and the restoration unit 1240 separately, the decoding unit 1210, the prediction unit 1220, the AI-based image processing unit 1230, and the restoration unit 1240 may be implemented by one processor. For example, the decoding unit 1210, the prediction unit 1220, the AI-based image processing unit 1230, and the restoration unit 1240 may be implemented by a dedicated processor, or may be implemented by a combination of a general-purpose processor such as an AP (application processor), a CPU (central processing unit), or a GPU (graphic processing unit) and software. In the case of a dedicated processor, it includes a memory containing instructions for implementing an embodiment of the present invention, or includes a memory processing unit for using an external memory.

[0169] The decoding unit 1210, the prediction unit 1220, the AI-based image processing unit 1230, and the restoration unit 1240 may be configured by a plurality of processors. In this case, it may be implemented by a combination of dedicated processors, or may be implemented by a combination of a plurality of general-purpose processors such as an AP, a CPU, or a GPU and software. Further, the processor may include a processor dedicated to artificial intelligence. As another example, the processor dedicated to artificial intelligence may be configured as a chip separate from the processor.

[0170] The decoding unit 1210 acquires the joint chroma residual sample of the current block, the prediction information of the Cb component of the current block, and the prediction information of the Cr component of the current block from the bitstream.

[0171] According to an embodiment of the present invention, the joint chroma residual sample of the current block is an average value of the actual value of the residual sample of the Cb component of the current block and the actual value of the residual sample of the Cr component of the current block.

[0172] According to an embodiment of the present invention, the joint chroma residual sample of the current block is also the same as the actual value of the residual sample of the Cb component of the current block.

[0173] According to an embodiment of the present invention, the joint chroma residual sample of the current block is also the same as the actual value of the residual sample of the Cr component of the current block.

[0174] According to an embodiment of the present invention, the joint chroma residual sample of the current block is determined by SATD or RDO calculation on the encoding side and transmitted through the bitstream.

[0175] In an embodiment of the present invention, the decoding unit 1210 transmits the prediction information of the Cb component of the current block and the prediction information of the Cr component of the current block to the prediction unit 1220.

[0176] The prediction unit 1220 determines the predicted sample of the Cb component and the predicted sample of the Cr component of the current block based on the Cb component prediction information and the Cr component prediction information.

[0177] According to an embodiment of the present invention, the AI-based image processing unit 1230 includes the neural network 520 described above with reference to FIG. 5 or the neural network 620 described above with reference to FIG. 6.

[0178] According to an embodiment of the present invention, the neural network included in the AI-based image processing unit 1230 can be trained to determine the correlation between Cb and Cr for each sample of the training current block, using the training joint chroma residual sample, the predicted sample of the Cb component for training, and the predicted sample of the Cr component for training as input values.

[0179] According to an embodiment of the present invention, for the correlation between Cb and Cr, a weighting value for the residual sample of the Cb component and a weighting value for the residual sample of the Cr component are respectively determined.

[0180] According to an embodiment of the present invention, the neural network can be trained by first loss information corresponding to the difference between the original sample of the Cb component of the training original block and the restored sample of the Cb component of the training restored block obtained through the neural network, and second loss information corresponding to the difference between the original sample of the Cr component of the training original block and the restored sample of the Cr component of the training restored block obtained through the neural network.

[0181] The AI-based image processing unit 1230 is input with a joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component to obtain a restored sample of the Cb component and a restored sample of the Cr component.

[0182] According to an embodiment of the present invention, the input value of the neural network further includes at least one of the quantization step size of the current block, the quantization error of the current block, or the restored luma block corresponding to the current chroma block of the current block, which is a block downsampled based on the chroma format.

[0183] According to an embodiment of the present invention, the AI-based image processing unit 1230 is further input with at least one of the quantization step size of the current block, the quantization error of the current block, and the restored luma block corresponding to the current chroma block of the current block, which is a block downsampled based on the chroma format, to obtain a restored sample of the Cb component and a restored sample of the Cr component.

[0184] According to an embodiment of the present invention, the decoding unit 1210 further obtains from the bitstream Cb cbf information indicating whether the transform coefficient level of the current block for the Cb component includes a Cb component that is not 0, and Cr cbf information indicating whether the transform coefficient level of the current block for the Cr component includes a Cr component that is not 0. The model of the neural network included in the AI-based image processing unit 1230 is determined based on the Cb cbf information and the Cr cbf information.

[0185] According to an embodiment of the present invention, the model of the neural network included in the AI-based image processing unit 1230 is determined by at least one of the type of the current block, the type of the slice in which the current block is included, the QP range of the slice, or whether the current block is a Cb component or a Cr component.

[0186] The restoration unit 1240 restores the current block using the restored sample of the Cb component and the restored sample of the Cr component obtained by the AI-based image processing unit 1230.

[0187] The output of the neural network included in the AI-based image processing unit 1230 may be one of the residual samples of the Cb component and the residual samples of the Cr component instead of the restored samples of the Cb component and the restored samples of the Cr component. In such a case, an addition unit for adding the residual samples of the Cb component and the residual samples of the Cr component and the predicted samples of the Cb component and the predicted samples of the Cr component may be further included. This will be described later with reference to the AI-based video decoding apparatus of FIG. 13.

[0188] FIG. 13 is a block diagram showing the configuration of an AI-based video decoding apparatus according to an embodiment of the present invention.

[0189] Referring to FIG. 13, the AI-based video decoding apparatus 1300 includes a decoding unit 1310, a prediction unit 1320, an AI-based image processing unit 1330, an addition unit 1335, and a restoration unit 1340.

[0190] The decoding unit 1310, the prediction unit 1320, the AI-based image processing unit 1330, the addition unit 1335, and the restoration unit 1340 may be implemented by one or more processors. The decoding unit 1310, the prediction unit 1320, the AI-based image processing unit 1330, the addition unit 1335, and the restoration unit 1340 operate according to instructions stored in a memory.

[0191] Although FIG. 13 shows the decoding unit 1310, the prediction unit 1320, the AI-based image processing unit 1330, the addition unit 1335, and the restoration unit 1340 separately, the decoding unit 1310, the prediction unit 1320, the AI-based image processing unit 1330, the addition unit 1335, and the restoration unit 1340 may be implemented by one processor. In this case, the decoding unit 1310, the prediction unit 1320, the AI-based image processing unit 1330, the addition unit 1335, and the restoration unit 1340 may be implemented by a dedicated processor, or may be implemented by a combination of a general-purpose processor such as an AP, a CPU, or a GPU and software. Further, in the case of a dedicated processor, it includes a memory containing instructions for implementing an embodiment of the present invention, or includes a memory processing unit for using an external memory.

[0192] The decoding unit 1310, the prediction unit 1320, the AI-based image processing unit 1330, the summation unit 1335, and the restoration unit 1340 may be configured by a plurality of processors. In this case, it may be implemented by a combination of dedicated processors, or may be implemented by a combination of a plurality of general-purpose processors such as an AP, a CPU, or a GPU and software. Further, the processor may include a processor dedicated to artificial intelligence. As another example, the processor dedicated to artificial intelligence may be configured as a chip separate from the processor.

[0193] The decoding unit 1310 acquires the joint chroma residual sample of the current block, the prediction information of the Cb component of the current block, and the prediction information of the Cr component of the current block from the bitstream.

[0194] According to an embodiment of the present invention, the joint chroma residual sample of the current block is an average value of the actual value of the residual sample of the Cb component of the current block and the actual value of the residual sample of the Cr component of the current block.

[0195] According to an embodiment of the present invention, the joint chroma residual sample of the current block is also the same as the actual value of the residual sample of the Cb component of the current block.

[0196] According to an embodiment of the present invention, the joint chroma residual sample of the current block is also the same as the actual value of the residual sample of the Cr component of the current block.

[0197] According to an embodiment of the present invention, the joint chroma residual sample of the current block is determined by SATD or RDO calculation on the encoding side and transmitted through the bitstream.

[0198] The decoding unit 1310 transmits the prediction information of the Cb component of the current block and the prediction information of the Cr component of the current block to the prediction unit 1320.

[0199] The prediction unit 1320 determines the prediction sample of the Cb component and the prediction sample of the Cr component of the current block based on the Cb component prediction information and the Cr component prediction information.

[0200] The AI-based image processing unit 1330 includes the neural network 520 described above with reference to FIG. 5.

[0201] According to an embodiment of the present invention, the neural network included in the AI-based image processing unit 1330 can be trained to determine the correlation between Cb and Cr for each sample of the current block for training, using the joint chroma residual sample for training, the prediction sample of the Cb component for training, and the prediction sample of the Cr component for training as input values.

[0202] According to an embodiment of the present invention, for the correlation between Cb and Cr, a weighting value for the residual sample of the Cb component and a weighting value for the residual sample of the Cr component are respectively determined.

[0203] According to an embodiment of the present invention, the neural network can be trained by first loss information corresponding to the difference between the original sample of the Cb component of the original block for training and the restored sample of the Cb component of the restored block for training obtained through the neural network, and second loss information corresponding to the difference between the original sample of the Cr component of the original block for training and the restored sample of the Cr component of the restored block for training obtained through the neural network.

[0204] The AI-based image processing unit 1330 is input with a joint chroma residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component, and obtains at least one of the residual sample of the Cb component or the residual sample of the Cr component of the current block.

[0205] According to an embodiment of the present invention, the input value of the neural network further includes at least one of the quantization step size of the current block, the quantization error of the current block, or the restored luma block corresponding to the current chroma block of the current block, which is downsampled based on the chroma format.

[0206] According to an embodiment of the present invention, the AI-based image processing unit 1330 further inputs at least one of the quantization step size of the current block, the quantization error of the current block, or the restored luma block corresponding to the current chroma block of the current block, which is downsampled based on the chroma format, to obtain at least one of the residual samples of the Cb component or the residual samples of the Cr component.

[0207] The adder 1335 uses at least one of the residual samples of the Cb component or the residual samples of the Cr component, the predicted samples of the Cb component, and the predicted samples of the Cr component to obtain the restored samples of the Cb component and the restored samples of the Cr component. For example, the adder 1335 adds the residual samples of the Cb component and the predicted samples of the Cb component to obtain the restored samples of the Cb component, and adds the residual samples of the Cr component and the predicted samples of the Cr component to obtain the restored samples of the Cr component.

[0208] According to an embodiment of the present invention, the decoding unit 1310 further obtains an index indicating a scale factor from the bitstream. The AI-based image processing unit 1330 further obtains an index indicating the scale factor included in the scale factor set from the decoding unit 1310. The AI-based image processing unit 1330 applies the scale factors obtained for the residual samples of the Cb component and the residual samples of the Cr component of the current block respectively, to obtain the refined residual samples of the Cb component and the refined residual samples of the Cr component of the current block. Specifically, the AI-based image processing unit 1330 applies a first scale factor to the residual samples of the Cb component to refine the residual samples of the Cb component, and applies a second scale factor to the residual samples of the Cr component to refine the residual samples of the Cr component. The adding unit 1335 adds the refined residual samples of the Cb component and the predicted samples of the Cb component to obtain a restored sample of the Cb component, and adds the refined residual samples of the Cr component and the predicted samples of the Cr component to obtain a restored sample of the Cr component.

[0209] According to an embodiment of the present invention, the scale factor set is determined by at least one of the type of the slice in which the current block is included or the type of the current block among a plurality of scale factor sets.

[0210] According to an embodiment of the present invention, the plurality of scale factor sets include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0211] According to an embodiment of the present invention, the decoding unit 1310 further obtains from the bitstream Cb cbf information indicating whether the transform coefficient level of the current block for the Cb component includes a Cb component that is not 0, and Cr cbf information indicating whether the transform coefficient level of the current block for the Cr component includes a Cr component that is not 0. The neural network model included in the AI-based image processing unit 1330 is determined based on the Cb cbf information and the Cr cbf information.

[0212] According to an embodiment of the present invention, the model of the neural network included in the AI-based image processing unit 1330 is determined by at least one of the type of the current block, the type of the slice in which the current block is included, the QP range of the slice, or whether it is a Cb component or a Cr component of the current block.

[0213] The restoration unit 1340 restores the current block using the restored samples of the Cb component and the restored samples of the Cr component obtained by the addition unit 1335.

[0214] FIG. 14 is a flowchart of an AI-based video encoding method according to an embodiment of the present invention.

[0215] In S1410, the AI-based video encoding apparatuses 1500 and 1600 generate an initial joint chroma residual sample of the current block, prediction information of the Cb component of the current block, and prediction information of the Cr component of the current block.

[0216] According to an embodiment of the present invention, the initial joint chroma residual sample of the current block is an average value of the actual value of the residual sample of the Cb component of the current block and the actual value of the residual sample of the Cr component of the current block.

[0217] According to an embodiment of the present invention, the initial joint chroma residual sample of the current block is also the same as the actual value of the residual sample of the Cb component of the current block.

[0218] According to an embodiment of the present invention, the initial joint chroma residual sample of the current block is also the same as the actual value of the residual sample of the Cr component of the current block.

[0219] In S1430, the AI-based video encoding devices 1500 and 1600 determine the prediction samples of the Cb component and the prediction samples of the Cr component of the current block based on the Cb component prediction information and the Cr component prediction information. For example, the prediction samples of the Cb component are determined based on at least the Cb component prediction information obtained from the bitstream, and the prediction samples of the Cr component are determined based on at least the Cr component prediction information obtained from the bitstream.

[0220] In S1450, the AI-based video encoding devices 1500 and 1600 apply the initial joint chroma residual samples, the prediction samples of the Cb component, and the prediction samples of the Cr component to the neural network to obtain the restored samples of the Cb component and the restored samples of the Cr component of the current block, thereby encoding the joint chroma residual samples of the current block.

[0221] According to an embodiment of the present invention, the AI-based video encoding device 1600 applies the initial joint chroma residual samples, the prediction samples of the Cb component, and the prediction samples of the Cr component to the neural network to obtain at least one of the residual samples of the Cb component and the residual samples of the Cr component of the current block. The AI-based video encoding device 1600 uses at least one of the residual samples of the Cb component or the residual samples of the Cr component, the prediction samples of the Cb component, and the prediction samples of the Cr component to obtain the restored samples of the Cb component and the restored samples of the Cr component, thereby encoding the joint chroma residual samples of the current block.

[0222] According to an embodiment of the present invention, the joint chroma residual samples of the current block can be determined by SATD (Sum of Transform Difference) or RDO (Rate Distortion Optimization) calculation and encoded.

[0223] According to an embodiment of the present invention, the Cb component prediction information and the Cr component prediction information of the current block are further encoded.

[0224] According to an embodiment of the present invention, the AI-based video encoding device 1600 applies a first scale factor to the residual sample of the Cb component obtained through the neural network to purify the residual sample of the Cb component, and applies a second scale factor to the residual sample of the Cr component to purify the residual sample of the Cb component.

[0225] According to an embodiment of the present invention, the first scale factor and the second scale factor may be encoded into an index indicating a scale factor value included in a scale factor set. That is, an index indicating a scale factor value may be determined by SATD or RDO calculation and included in a bitstream for signaling.

[0226] According to an embodiment of the present invention, the scale factor set is determined by at least one of the type of the slice in which the current block is included or the type of the current block among a plurality of scale factor sets.

[0227] According to an embodiment of the present invention, the plurality of scale factor sets include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0228] According to an embodiment of the present invention, the neural network may be trained to determine the correlation between Cb and Cr for each sample of the training current block, using the joint chroma residual sample for training, the prediction sample of the Cb component for training, and the prediction sample of the Cr component for training as input values.

[0229] According to an embodiment of the present invention, for the correlation between Cb and Cr, a weighting value for the residual sample of the Cb component and a weighting value for the residual sample of the Cr component are respectively determined.

[0230] According to an embodiment of the present invention, the input value of the neural network further includes at least one of the quantization step size of the current block, the quantization error of the current block, or the restored luma block corresponding to the current chroma block of the current block, which is downsampled based on the chroma format.

[0231] According to an embodiment of the present invention, the neural network can be trained by first loss information corresponding to the difference between the original sample of the Cb component of the original block for training and the restored sample of the Cb component of the restored block for training obtained through the neural network, and second loss information corresponding to the difference between the original sample of the Cr component of the original block for training and the restored sample of the Cr component of the restored block for training obtained through the neural network.

[0232] According to an embodiment of the present invention, the model of the neural network is determined based on Cb cbf information indicating whether the current block for the Cb component, obtained from the bitstream, includes a Cb component with a non-zero transform coefficient level, and Cr cbf information indicating whether the current block for the Cr component includes a Cr component with a non-zero transform coefficient level.

[0233] According to an embodiment of the present invention, the model of the neural network is determined by at least one of the type of the current block, the type of the slice in which the current block is included, the QP range of the slice, or whether the current block is a Cb component or a Cr component.

[0234] FIG. 15 is a block diagram showing the configuration of an AI-based video encoding apparatus according to an embodiment of the present invention.

[0235] Referring to FIG. 15, the AI-based video decoding apparatus 1500 includes a generation unit 1510, a prediction unit 1520, an AI-based image processing unit 1530, and an encoding unit 1540.

[0236] The generation unit 1510, prediction unit 1520, AI-based image processing unit 1530, and encoding unit 1540 can be implemented by one or more processors. The generation unit 1510, prediction unit 1520, AI-based image processing unit 1530, and encoding unit 1540 operate according to instructions stored in a memory.

[0237] FIG. 15 separately shows the generation unit 1510, prediction unit 1520, AI-based image processing unit 1530, and encoding unit 1540. However, the generation unit 1510, prediction unit 1520, AI-based image processing unit 1530, and encoding unit 1540 can be implemented by one processor. In this case, the generation unit 1510, prediction unit 1520, AI-based image processing unit 1530, and encoding unit 1540 may be implemented by a dedicated processor, or may be implemented by a combination of a general-purpose processor such as an AP, CPU, or GPU and software. Also, in the case of a dedicated processor, it includes a memory containing instructions for implementing an embodiment of the present invention, or includes a memory processing unit for using an external memory.

[0238] The generation unit 1510, prediction unit 1520, AI-based image processing unit 1530, and encoding unit 1540 may be composed of a plurality of processors. In this case, it may be implemented by a combination of dedicated processors, or may be implemented by a combination of a plurality of general-purpose processors such as an AP, CPU, or GPU and software. Also, the processor may include an artificial intelligence dedicated processor. As another example, the artificial intelligence dedicated processor may be composed of a chip separate from the processor.

[0239] The generation unit 1510 generates an initial joint chroma residual sample of the current block, prediction information of the Cb component of the current block, and prediction information of the Cr component of the current block.

[0240] According to an embodiment of the present invention, the initial joint chroma residual sample of the current block is an average value of the actual value of the residual sample of the Cb component of the current block and the actual value of the residual sample of the Cr component of the current block.

[0241] According to an embodiment of the present invention, the initial joint chroma residual sample of the current block is also the same as the actual value of the residual sample of the Cb component of the current block.

[0242] According to an embodiment of the present invention, the initial joint chroma residual sample of the current block is also the same as the actual value of the residual sample of the Cr component of the current block.

[0243] The generation unit 1510 transmits the Cb component prediction information of the current block and the Cr component prediction information of the current block to the prediction unit 1520.

[0244] The prediction unit 1520 determines the predicted sample of the Cb component of the current block and the predicted sample of the Cr component of the current block based on the Cb component prediction information and the Cr component prediction information.

[0245] According to an embodiment of the present invention, the AI-based image processing unit 1530 includes the neural network 520 described above with reference to FIG. 5 or the neural network 620 described above with reference to FIG. 6.

[0246] According to an embodiment of the present invention, the neural network included in the AI-based image processing unit 1530 can be trained to determine the correlation between Cb and Cr for each sample of the current block for training, using the training joint chroma residual sample, the predicted sample of the Cb component for training, and the predicted sample of the Cr component for training as input values.

[0247] According to an embodiment of the present invention, for the correlation between Cb and Cr, a weighting value for the residual sample of the Cb component and a weighting value for the residual sample of the Cr component are respectively determined.

[0248] According to an embodiment of the present invention, the neural network can be trained by first loss information corresponding to the difference between the original sample of the Cb component of the original block for training and the restored sample of the Cb component of the restored block for training obtained through the neural network, and second loss information corresponding to the difference between the original sample of the Cr component of the original block for training and the restored sample of the Cr component of the restored block for training obtained through the neural network.

[0249] The AI-based image processing unit 1530 receives the initial joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component, and obtains the restored sample of the Cb component and the restored sample of the Cr component.

[0250] According to an embodiment of the present invention, the input value of the neural network further includes at least one of the quantization step size of the current block, the quantization error of the current block, or the restored luma block corresponding to the current chroma block of the current block that is downsampled based on the chroma format.

[0251] According to an embodiment of the present invention, the AI-based image processing unit 1530 further receives at least one of the quantization step size of the current block, the quantization error of the current block, or the restored luma block corresponding to the current chroma block of the current block that is downsampled based on the chroma format, and obtains the restored sample of the Cb component and the restored sample of the Cr component.

[0252] According to an embodiment of the present invention, the generation unit 1510 further generates Cb cbf information indicating whether the current block includes a Cb component whose transform coefficient level for the Cb component is not 0, and Cr cbf information indicating whether the current block includes a Cr component whose transform coefficient level for the Cr component is not 0. The model of the neural network included in the AI-based image processing unit 1530 is determined based on the Cb cbf information and the Cr cbf information.

[0253] According to one embodiment of the present invention, the model of the neural network included in the AI-based image processing unit 1530 is determined by at least one of the type of the current block, the type of the slice in which the current block is included, the QP range of the slice, or whether it is the Cb component or the Cr component of the current block.

[0254] The encoding unit 1540 encodes the joint chroma residual sample of the current block using the restored samples of the Cb component and the restored samples of the Cr component obtained by the AI-based image processing unit 1530.

[0255] According to one embodiment of the present invention, the joint chroma residual sample of the current block can be determined by SATD (Sum of Transform Difference) or RDO (Rate Distortion Optimization) calculation and encoded.

[0256] According to one embodiment of the present invention, the encoding unit 1540 further encodes the Cb component prediction information and the Cr component prediction information of the current block obtained by the generation unit 1510.

[0257] The output of the neural network included in the AI-based image processing unit 1530 may be one of the residual samples of the Cb component and the residual samples of the Cr component instead of the restored samples of the Cb component and the restored samples of the Cr component. In such a case, an addition unit for adding the residual samples of the Cb component and the residual samples of the Cr component and the prediction samples of the Cb component and the prediction samples of the Cr component may be further included. This can be described later in the AI-based video encoding apparatus of FIG. 16.

[0258] FIG. 16 is a block diagram showing the configuration of an AI-based video encoding apparatus according to one embodiment of the present invention.

[0259] Referring to FIG. 16, the AI-based video encoding apparatus 1600 includes a generation unit 1610, a prediction unit 1620, an AI-based image processing unit 1630, an addition unit 1635, and an encoding unit 1640.

[0260] The generation unit 1610, the prediction unit 1620, the AI-based image processing unit 1630, the summation unit 1635, and the encoding unit 1640 can be implemented by one or more processors. The generation unit 1610, the prediction unit 1620, the AI-based image processing unit 1630, the summation unit 1635, and the encoding unit 1640 operate according to instructions stored in a memory.

[0261] FIG. 16 shows the generation unit 1610, the prediction unit 1620, the AI-based image processing unit 1630, the summation unit 1635, and the encoding unit 1640 individually, but the generation unit 1610, the prediction unit 1620, the AI-based image processing unit 1630, the summation unit 1635, and the encoding unit 1640 can be implemented by one processor. In this case, the generation unit 1610, the prediction unit 1620, the AI-based image processing unit 1635, the summation unit 1635, and the encoding unit 1640 may be implemented by a dedicated processor, or may be implemented by a combination of a general-purpose processor such as an AP, a CPU, or a GPU and software. Further, in the case of a dedicated processor, it includes a memory containing instructions for implementing an embodiment of the present invention, or includes a memory processing unit for using an external memory.

[0262] The generation unit 1610, the prediction unit 1620, the AI-based image processing unit 1630, the summation unit 1635, and the encoding unit 1640 may be composed of a plurality of processors. In this case, it may be implemented by a combination of dedicated processors, or may be implemented by a combination of a plurality of general-purpose processors such as an AP, a CPU, or a GPU and software. Further, the processor may include an artificial intelligence dedicated processor. As another example, the artificial intelligence dedicated processor may be composed of a chip separate from the processor.

[0263] The generation unit 1610 generates an initial joint chroma residual sample of the current block, prediction information of the Cb component of the current block, and prediction information of the Cr component of the current block.

[0264] According to an embodiment of the present invention, the initial joint chroma residual sample of the current block is the average value of the actual value of the residual sample of the Cb component of the current block and the actual value of the residual sample of the Cr component of the current block.

[0265] According to an embodiment of the present invention, the initial joint chroma residual sample of the current block is also the same as the actual value of the residual sample of the Cb component of the current block.

[0266] According to an embodiment of the present invention, the initial joint chroma residual sample of the current block is also the same as the actual value of the residual sample of the Cr component of the current block.

[0267] The generation unit 1610 transmits the Cb component prediction information of the current block and the Cr component prediction information of the current block to the prediction unit 1620.

[0268] The prediction unit 1620 determines the predicted sample of the Cb component of the current block and the predicted sample of the Cr component of the current block based on the Cb component prediction information and the Cr component prediction information.

[0269] According to an embodiment of the present invention, the AI-based image processing unit 1630 includes the neural network 520 described above with reference to FIG. 5.

[0270] According to an embodiment of the present invention, the neural network included in the AI-based image processing unit 1630 can be trained to determine the correlation between Cb and Cr for each sample of the current block for training, using the joint chroma residual sample for training, the predicted sample of the Cb component for training, and the predicted sample of the Cr component for training as input values.

[0271] According to an embodiment of the present invention, for the correlation between Cb and Cr, a weighting value for the residual sample of the Cb component and a weighting value for the residual sample of the Cr component are respectively determined.

[0272] According to an embodiment of the present invention, the neural network can be trained by first loss information corresponding to the difference between the original sample of the Cb component of the original block for training and the restored sample of the Cb component of the restored block for training obtained through the neural network, and second loss information corresponding to the difference between the original sample of the Cr component of the original block for training and the restored sample of the Cr component of the restored block for training obtained through the neural network.

[0273] The AI-based image processing unit 1630 receives the initial joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component, and obtains at least one of the residual sample of the Cb component of the current block or the residual sample of the Cr component of the current block.

[0274] According to an embodiment of the present invention, the input value of the neural network further includes at least one of the quantization step size of the current block, the quantization error of the current block, or the restored luma block corresponding to the current chroma block of the current block that is downsampled based on the chroma format.

[0275] According to an embodiment of the present invention, the AI-based image processing unit 1630 further receives at least one of the quantization step size of the current block, the quantization error of the current block, or the restored luma block corresponding to the current chroma block of the current block that is downsampled based on the chroma format, and obtains at least one of the residual sample of the Cb component or the residual sample of the Cr component.

[0276] According to an embodiment of the present invention, the generation unit 1610 further generates Cb cbf information indicating whether the current block's transform coefficient level for the Cb component includes a Cb component that is not 0, and Cr cbf information indicating whether the current block's transform coefficient level for the Cr component includes a Cr component that is not 0. The neural network model included in the AI-based image processing unit 1630 is determined based on the Cb cbf information and the Cr cbf information.

[0277] According to an embodiment of the present invention, the neural network model included in the AI-based image processing unit 1630 is determined by at least one of the type of the current block, the type of the slice in which the current block is included, the QP range of the slice, or whether the current block is a Cb component or a Cr component.

[0278] The addition unit 1635 uses at least one of the residual samples of the Cb component or the Cr component, the prediction samples of the Cb component, and the prediction samples of the Cr component to obtain the restored samples of the Cb component and the restored samples of the Cr component. For example, the addition unit 1635 adds the residual sample of the Cb component and the prediction sample of the Cb component to obtain the restored sample of the Cb component, and adds the residual sample of the Cr component and the prediction sample of the Cr component to obtain the restored sample of the Cr component.

[0279] The encoding unit 1640 encodes the joint chroma residual sample of the current block using the restored samples of the Cb component and the Cr component obtained by the addition unit 1635.

[0280] According to an embodiment of the present invention, the joint chroma residual sample of the current block can be determined by SATD (Sum of Transform Difference) or RDO (Rate Distortion Optimization) calculation and encoded.

[0281] According to an embodiment of the present invention, the encoding unit 1640 further encodes the Cb component prediction information and the Cr component prediction information of the current block obtained by the generation unit 1610.

[0282] According to an embodiment of the present invention, the AI-based image processing unit 1630 applies the scale factors obtained for the residual samples of the Cb component of the current block and the residual samples of the Cr component of the current block, respectively, to obtain the refined residual samples of the Cb component of the current block and the refined residual samples of the Cr component of the current block. Specifically, the AI-based image processing unit 1630 applies a first scale factor to the residual samples of the Cb component to refine the residual samples of the Cb component, and applies a second scale factor to the residual samples of the Cr component to refine the residual samples of the Cr component. The addition unit 1635 adds the refined residual samples of the Cb component and the predicted samples of the Cb component to obtain a restored sample of the Cb component, and adds the refined residual samples of the Cr component and the predicted samples of the Cr component to obtain a restored sample of the Cr component. The encoding unit 1640 further encodes the index indicating the scale factor among the scale factor sets using the restored samples of the Cb component and the restored samples of the Cr component obtained by the addition unit 1635.

[0283] According to an embodiment of the present invention, the scale factor set is determined by at least one of the type of the slice in which the current block is included or the type of the current block among a plurality of scale factor sets.

[0284] According to an embodiment of the present invention, the plurality of scale factor sets include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0285] According to an embodiment of the present invention, an AI-based video decoding method includes the steps of: obtaining a joint chroma residual sample of a current block, prediction information of a Cb component of the current block, and prediction information of a Cr component of the current block from a bitstream; obtaining a predicted sample of the Cb component of the current block and a predicted sample of the Cr component of the current block based on the prediction information of the Cb component and the prediction information of the Cr component; and applying the joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component to a neural network to obtain a restored sample of the Cb component of the current block and a restored sample of the Cr component of the current block, thereby restoring the current block.

[0286] According to an embodiment of the present invention, an AI-based video decoding method uses prediction information of a Cb component and a Cr component, and through a neural network that generalizes more and trains a more local and specific relationship between the Cb component and the Cr component instead of a fixed correlation between the Cb component and the Cr component, has the effect of more efficiently restoring the Cb component and the Cr component.

[0287] According to an embodiment of the present invention, an AI-based video decoding method includes the steps of: obtaining a joint chroma residual sample of a current block, prediction information of a Cb component of the current block, and prediction information of a Cr component of the current block from a bitstream; obtaining a predicted sample of the Cb component of the current block and a predicted sample of the Cr component of the current block based on the prediction information of the Cb component and the prediction information of the Cr component; applying the joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component to the neural network to obtain at least one of a residual sample of the Cb component of the current block or a residual sample of the Cr component of the current block; and using at least one of the residual sample of the Cb component or the residual sample of the Cr component, the predicted sample of the Cb component, and the predicted sample of the Cr component to obtain a restored sample of the Cb component and a restored sample of the Cr component, thereby restoring the current block.

[0288] According to an embodiment of the present invention, the AI-based video decoding method can more accurately obtain at least one of the residual samples of the Cb component and the Cr component through a neural network, and has the effect of more efficiently restoring the Cb component and the Cr component by using at least one of the residual samples of the Cb component and the Cr component.

[0289] According to an embodiment of the present invention, the AI-based video decoding method includes applying a first scale factor to the residual sample of the Cb component obtained through the neural network to purify the residual sample of the Cb component, and applying a second scale factor to the residual sample of the Cr component to purify the residual sample of the Cr component.

[0290] According to an embodiment of the present invention, the AI-based video decoding method has the effect of obtaining a further purified residual sample by scaling using a scale factor before adding the residual samples of the Cb component and the Cr component obtained through the neural network to the prediction samples of the Cb component and the Cr component.

[0291] According to an embodiment of the present invention, the first scale factor and the second scale factor are included in a scale factor set and are determined by the scale factor value indicated by the index obtained from the bitstream.

[0292] According to an embodiment of the present invention, the AI-based video decoding method has the effect that the residual samples of the Cb component and the Cr component are more effectively purified by applying a suitable scale factor according to the index obtained from the bitstream.

[0293] According to an embodiment of the present invention, the scale factor set is determined by at least one of the type of the slice in which the current block is included or the type of the current block among a plurality of scale factor sets.

[0294] According to an embodiment of the present invention, in the AI-based video decoding method, since the scale factor set is determined by at least one of the slice in which the current block is included among a plurality of scale factor sets or the type of the current block, a suitable scale factor set is used according to the type of the current block, and the residual samples of the Cb component and the Cr component are more effectively purified.

[0295] According to an embodiment of the present invention, the plurality of scale factor sets include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0296] According to an embodiment of the present invention, in the AI-based video decoding method, a plurality of scale factor sets are used, and the residual samples of the Cb component and the Cr component are more effectively purified.

[0297] According to an embodiment of the present invention, the neural network can be trained with the joint chroma residual samples for training, the prediction samples of the Cb component for training, and the prediction samples of the Cr component for training as input values to determine the correlation between Cb and Cr for each sample of the current block for training.

[0298] According to an embodiment of the present invention, for the correlation between Cb and Cr, a weighting value for the residual sample of the Cb component and a weighting value for the residual sample of the Cr component are respectively determined.

[0299] According to an embodiment of the present invention, in the AI-based video decoding method, since the neural network learns the correlation between Cb and Cr for each sample of the current block, the values of the residual samples of the Cb component and the Cr component for each sample of the current block are more accurately determined.

[0300] According to an embodiment of the present invention, the input value of the neural network includes at least one of the quantization step size of the current block, the quantization error of the current block, or the restored luma block corresponding to the current chroma block of the current block, which is downsampled based on the chroma format.

[0301] The AI-based video decoding method according to an embodiment of the present invention has an effect that the quantization error in the current block is more efficiently removed when restoring the current block by further using the quantization error or the quantization step size.

[0302] The AI-based video decoding method according to an embodiment of the present invention has an effect that the Cb component and the Cr component of the current block are more efficiently restored using the information of the restored luma block by further using the block in which the restored luma block is downsampled based on the chroma format.

[0303] According to an embodiment of the present invention, the neural network can be trained by the first loss information corresponding to the difference between the original sample of the Cb component of the original block for training and the restored sample of the Cb component of the restored block for training obtained through the neural network, and the second loss information corresponding to the difference between the original sample of the Cr component of the original block for training and the restored sample of the Cr component of the restored block for training obtained through the neural network.

[0304] The AI-based video decoding method according to an embodiment of the present invention has an effect that the restored samples of the Cb component and the Cr component obtained through the neural network are restored to be almost the same as the original samples of the Cb component and the Cr component by training the neural network to minimize the loss information corresponding to the difference between the original samples and the restored samples of the Cb component and the Cr component, respectively.

[0305] According to an embodiment of the present invention, the model of the neural network is determined based on Cb cbf information indicating whether the conversion coefficient level of the current block for the Cb component, which is obtained from the bitstream and is not 0, includes a Cb component, and Cr cbf information indicating whether the conversion coefficient level of the current block for the Cr component, which is not 0, includes a Cr component.

[0306] In the AI-based video decoding method according to an embodiment of the present invention, one of the models of the neural network trained by the Cb cbf information and the Cr cbf information is preferably selected, and based on the conditions of the current block such as the Cb cbf information and the Cr cbf information, the Cb component and the Cr component of the current block are restored more efficiently.

[0307] According to an embodiment of the present invention, the model of the neural network is determined by at least one of the type of the current block, the type of the slice in which the current block is included, the QP range of the slice, or whether the current block is a Cb component or a Cr component.

[0308] In the AI-based video decoding method according to an embodiment of the present invention, one of the models of the neural network trained by various conditions is preferably selected, and based on the conditions of the current block such as the type of the current block, the slice type including the current block, the QP range of the slice, or whether the current block is a Cb component or a Cr component, the Cb component and the Cr component of the current block are restored more efficiently.

[0309] An AI-based video decoding device according to an embodiment of the present invention includes a memory that stores one or more instructions, and at least one processor that operates according to the one or more instructions. The at least one processor obtains joint chroma residual samples of a current block, prediction information of a Cb component of the current block, and prediction information of a Cr component of the current block from a bitstream. The at least one processor determines predicted samples of the Cb component of the current block and predicted samples of the Cr component of the current block based on the prediction information of the Cb component and the prediction information of the Cr component. The at least one processor applies the joint chroma residual samples, the predicted samples of the Cb component, and the predicted samples of the Cr component to a neural network to obtain restored samples of the Cb component of the current block and restored samples of the Cr component of the current block, thereby restoring the current block.

[0310] The AI-based video decoding device according to an embodiment of the present invention uses prediction information of the Cb component and the Cr component, and has the effect of more efficiently restoring the Cb component and the Cr component through a neural network that trains a more generalized and more local and specific relationship between the Cb component and the Cr component instead of a fixed correlation between the Cb component and the Cr component.

[0311] According to an embodiment of the present invention, the at least one processor applies the joint chroma residual samples, the predicted samples of the Cb component, and the predicted samples of the Cr component to a neural network to obtain at least one of the residual samples of the Cb component of the current block or the residual samples of the Cr component of the current block. The at least one processor restores the current block by using at least one of the residual samples of the Cb component or the residual samples of the Cr component, the predicted samples of the Cb component, and the predicted samples of the Cr component to obtain the restored samples of the Cb component and the restored samples of the Cr component.

[0312] The AI-based video decoding device according to an embodiment of the present invention has an effect of more accurately obtaining at least one of the residual samples of the Cb component and the Cr component through a neural network and more efficiently restoring the Cb component and the Cr component by using at least one of the residual samples of the Cb component and the Cr component.

[0313] According to an embodiment of the present invention, at least one processor applies a first scale factor to the residual sample of the Cb component obtained through the neural network to purify the residual sample of the Cb component, and applies a second scale factor to the residual sample of the Cr component to purify the residual sample of the Cr component.

[0314] The AI-based video decoding device according to an embodiment of the present invention has an effect of obtaining a further purified residual sample by scaling using a scale factor before adding the residual samples of the Cb component and the Cr component obtained through the neural network to the prediction samples of the Cb component and the Cr component.

[0315] According to an embodiment of the present invention, the first scale factor and the second scale factor are included in a scale factor set and are determined by the scale factor value indicated by an index obtained from a bitstream.

[0316] The AI-based video decoding device according to an embodiment of the present invention has an effect that the residual samples of the Cb component and the Cr component are more effectively purified by applying a suitable scale factor according to an index obtained from a bitstream.

[0317] According to an embodiment of the present invention, the scale factor set is determined by at least one of the type of the slice in which the current block is included or the type of the current block among a plurality of scale factor sets.

[0318] According to an embodiment of the present invention, in the AI-based video decoding apparatus, the scale factor set is determined by at least one of the slice in which the current block is included or the type of the current block among a plurality of scale factor sets, so that a scale factor set suitable for the type of the current block is used, and the residual samples of the Cb component and the Cr component are more effectively refined.

[0319] According to an embodiment of the present invention, the plurality of scale factor sets include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0320] According to an embodiment of the present invention, in the AI-based video decoding apparatus, a plurality of scale factor sets are used, and the residual samples of the Cb component and the Cr component are more effectively refined.

[0321] According to an embodiment of the present invention, the neural network can be trained to determine the correlation between Cb and Cr for each sample of the current block for training, using the joint chroma residual sample for training, the predicted sample of the Cb component for training, and the predicted sample of the Cr component for training as input values.

[0322] According to an embodiment of the present invention, for the correlation between Cb and Cr, a weighting value for the residual sample of the Cb component and a weighting value for the residual sample of the Cr component are respectively determined.

[0323] According to an embodiment of the present invention, in the AI-based video decoding apparatus, since the neural network learns the correlation between Cb and Cr for each sample of the current block, the values of the residual samples of the Cb component and the Cr component for each sample of the current block can be determined more accurately.

[0324] According to an embodiment of the present invention, the input value of the neural network includes at least one of the quantization step size of the current block, the quantization error of the current block, or the restored luma block corresponding to the current chroma block of the current block, which is downsampled based on the chroma format.

[0325] The AI-based video decoding device according to an embodiment of the present invention further uses a quantization error or a quantization step size, thereby having an effect of more efficiently removing the quantization error in the current block when restoring the current block.

[0326] The AI-based video decoding device according to an embodiment of the present invention further uses a block in which the restored luma block is downsampled based on the chroma format, thereby having an effect of more efficiently restoring the Cb component and the Cr component of the current block using the information of the restored luma block.

[0327] According to an embodiment of the present invention, the neural network can be trained by first loss information corresponding to the difference between the original sample of the Cb component of the original block for training and the restored sample of the Cb component of the restored block for training obtained through the neural network, and second loss information corresponding to the difference between the original sample of the Cr component of the original block for training and the restored sample of the Cr component of the restored block for training obtained through the neural network.

[0328] The AI-based video decoding device according to an embodiment of the present invention has an effect that the restored samples of the Cb component and the Cr component obtained through the neural network are restored to be almost the same as the original samples of the Cb component and the Cr component because the neural network is trained to minimize the loss information corresponding to the difference between the original sample and the restored sample of each of the Cb component and the Cr component.

[0329] According to an embodiment of the present invention, the model of the neural network is determined based on Cb cbf information indicating whether the conversion coefficient level of the current block for the Cb component, which is obtained from the bitstream and is not 0, includes a Cb component, and Cr cbf information indicating whether the conversion coefficient level of the current block for the Cr component, which is not 0, includes a Cr component.

[0330] In an AI-based video decoding apparatus according to an embodiment of the present invention, one of the models of the neural network trained by the Cb cbf information and the Cr cbf information is preferably selected, and based on the conditions of the current block such as the Cb cbf information and the Cr cbf information, the Cb component and the Cr component of the current block are restored more efficiently.

[0331] According to an embodiment of the present invention, the model of the neural network is determined by at least one of the type of the current block, the type of the slice in which the current block is included, the QP range of the slice, or whether the current block is a Cb component or a Cr component.

[0332] In an AI-based video decoding apparatus according to an embodiment of the present invention, one of the models of the neural network trained by various conditions is preferably selected, and based on the conditions of the current block such as the type of the current block, the slice type including the current block, the QP range of the slice, or whether the current block is a Cb component or a Cr component, the Cb component and the Cr component of the current block are restored more efficiently.

[0333] The AI-based video encoding method according to an embodiment of the present invention includes steps of generating an initial joint chroma residual sample of a current block, prediction information of a Cb component of the current block, and prediction information of a Cr component of the current block; determining a predicted sample of the Cb component of the current block and a predicted sample of the Cr component of the current block based on the prediction information of the Cb component and the prediction information of the Cr component; and applying the initial joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component to a neural network to obtain a restored sample of the Cb component of the current block and a restored sample of the Cr component of the current block, thereby encoding the joint chroma residual sample of the current block.

[0334] The AI-based video encoding method according to an embodiment of the present invention has an effect of encoding a joint chroma residual sample that can more efficiently restore a Cb component and a Cr component through a neural network that uses prediction information of the Cb component and the Cr component to train a more generalized and more local and specific relationship between the Cb component and the Cr component instead of a fixed correlation between the Cb component and the Cr component.

[0335] The AI-based video encoding method according to an embodiment of the present invention includes steps of generating an initial joint chroma residual sample of a current block, prediction information of a Cb component of the current block, and prediction information of a Cr component of the current block; determining a predicted sample of the Cb component of the current block and a predicted sample of the Cr component of the current block based on the prediction information of the Cb component and the prediction information of the Cr component; applying the initial joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component to the neural network to obtain at least one of a residual sample of the Cb component of the current block or a residual sample of the Cr component of the current block; and encoding the joint chroma residual sample of the current block by obtaining a restored sample of the Cb component and a restored sample of the Cr component using at least one of the residual sample of the Cb component or the residual sample of the Cr component, the predicted sample of the Cb component, and the predicted sample of the Cr component.

[0336] The AI-based video decoding method according to an embodiment of the present invention has an effect of encoding a joint chroma residual sample that can more accurately obtain at least one of residual samples of Cb and Cr components through a neural network and more efficiently restore the Cb and Cr components using at least one of the residual samples of the Cb and Cr components.

[0337] The AI-based video encoding method according to an embodiment of the present invention includes steps of applying a first scale factor to the residual sample of the Cb component obtained through the neural network to purify the residual sample of the Cb component, and applying a second scale factor to the residual sample of the Cr component to purify the residual sample of the Cr component.

[0338] The AI-based video encoding method according to an embodiment of the present invention has an effect of obtaining a further purified residual sample by scaling using a scale factor before adding the residual samples of the Cb and Cr components obtained through the neural network to the predicted samples of the Cb and Cr components.

[0339] According to an embodiment of the present invention, the first scale factor and the second scale factor may be encoded into an index indicating a scale factor included in a scale factor set.

[0340] The AI-based video encoding method according to an embodiment of the present invention has an effect that the residual samples of the Cb component and the Cr component are more effectively refined by encoding and signaling an index indicating a suitable scale factor.

[0341] According to an embodiment of the present invention, the scale factor set is determined by at least one of the type of the slice in which the current block is included or the type of the current block among a plurality of scale factor sets.

[0342] The AI-based video encoding method according to an embodiment of the present invention has an effect that a suitable scale factor set is used according to the type of the current block because the scale factor set is determined by at least one of the slice in which the current block is included or the type of the current block among a plurality of scale factor sets, and the residual samples of the Cb component and the Cr component are more effectively refined.

[0343] According to an embodiment of the present invention, the plurality of scale factor sets include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0344] The AI-based video encoding method according to an embodiment of the present invention has an effect that the residual samples of the Cb component and the Cr component are more effectively refined by using a plurality of scale factor sets.

[0345] According to an embodiment of the present invention, the neural network can be trained to determine the correlation between Cb and Cr for each sample of the current block for training, using the joint chroma residual sample for training, the predicted sample of the Cb component for training, and the predicted sample of the Cr component for training as input values.

[0346] According to an embodiment of the present invention, for the correlation between Cb and Cr, a weighting value for the residual sample of the Cb component and a weighting value for the residual sample of the Cr component are respectively determined.

[0347] The AI-based video encoding method according to an embodiment of the present invention has an effect that the values of the residual samples of the Cb component and the Cr component for each sample of the current block are more accurately determined by the neural network learning the correlation between Cb and Cr for each sample of the current block.

[0348] According to an embodiment of the present invention, the input value of the neural network includes at least one of the quantization step size of the current block, the quantization error of the current block, or the restored luma block corresponding to the current chroma block of the current block, which is downsampled based on the chroma format.

[0349] The AI-based video encoding method according to an embodiment of the present invention has an effect that the quantization error in the current block is more efficiently removed during restoration of the current block by further using the quantization error or the quantization step size.

[0350] The AI-based video encoding method according to an embodiment of the present invention has an effect that the Cb component and the Cr component of the current block are more efficiently restored using the information of the restored luma block by further using the block in which the restored luma block is downsampled based on the chroma format.

[0351] According to an embodiment of the present invention, the neural network can be trained by the first loss information corresponding to the difference between the original sample of the Cb component of the original block for training and the restored sample of the Cb component of the restored block for training obtained through the neural network, and the second loss information corresponding to the difference between the original sample of the Cr component of the original block for training and the restored sample of the Cr component of the restored block for training obtained through the neural network.

[0352] According to an embodiment of the present invention, in the AI-based video encoding method, a neural network is trained to minimize loss information corresponding to the difference between the original sample and the restored sample of each of the Cb component and the Cr component, so that the restored samples of the Cb component and the Cr component obtained through the neural network are restored to be substantially the same as the original samples of the Cb component and the Cr component.

[0353] According to an embodiment of the present invention, the model of the neural network is determined based on the Cb cbf information indicating whether the transform coefficient level of the current block for the Cb component is non-zero and includes the Cb component, and the Cr cbf information indicating whether the transform coefficient level of the current block for the Cr component is non-zero and includes the Cr component.

[0354] According to an embodiment of the present invention, in the AI-based video encoding method, one of the models of the neural network trained by the Cb cbf information and the Cr cbf information is preferably selected, and based on the conditions of the current block such as the Cb cbf information and the Cr cbf information, the Cb component and the Cr component of the current block are restored more efficiently.

[0355] According to an embodiment of the present invention, the model of the neural network is determined by at least one of the type of the current block, the type of the slice in which the current block is included, the QP range of the slice, or whether the current block is the Cb component or the Cr component.

[0356] According to an embodiment of the present invention, in the AI-based video encoding method, one of the models of the neural network trained by various conditions is preferably selected, and based on the conditions of the current block such as the type of the current block, the slice type including the current block, the QP range of the slice, or whether the current block is the Cb component or the Cr component, the Cb component and the Cr component of the current block are restored more efficiently.

[0357] An AI-based video encoding device according to an embodiment of the present invention includes a memory for storing one or more instructions and at least one processor operating according to the one or more instructions. The at least one processor generates an initial joint chroma residual sample of a current block, prediction information of a Cb component of the current block, and prediction information of a Cr component of the current block. The at least one processor determines a predicted sample of the Cb component of the current block and a predicted sample of the Cr component of the current block based on the prediction information of the Cb component and the prediction information of the Cr component. The at least one processor applies the initial joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component to a neural network to obtain a restored sample of the Cb component of the current block and a restored sample of the Cr component of the current block, thereby encoding the joint chroma residual sample of the current block.

[0358] The AI-based video encoding device according to an embodiment of the present invention has an effect of encoding a joint chroma residual sample that can more efficiently restore the Cb component and the Cr component through a neural network that uses prediction information of the Cb component and the Cr component to train a more generalized and more local and specific relationship between the Cb component and the Cr component instead of a fixed correlation between the Cb component and the Cr component.

[0359] An AI-based video encoding device according to an embodiment of the present invention includes a memory that stores one or more instructions, and at least one processor that operates according to the one or more instructions. The at least one processor generates an initial joint chroma residual sample of a current block, prediction information of a Cb component of the current block, and prediction information of a Cr component of the current block. The at least one processor determines a predicted sample of the Cb component of the current block and a predicted sample of the Cr component of the current block based on the prediction information of the Cb component and the prediction information of the Cr component. The at least one processor applies the initial joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component to a neural network to obtain at least one of a residual sample of the Cb component of the current block or a residual sample of the Cr component of the current block. The at least one processor uses at least one of the residual sample of the Cb component or the residual sample of the Cr component, the predicted sample of the Cb component, and the predicted sample of the Cr component to obtain a restored sample of the Cb component and a restored sample of the Cr component, thereby encoding the joint chroma residual sample of the current block.

[0360] An AI-based video decoding device according to an embodiment of the present invention has an effect of encoding a joint chroma residual sample that can more accurately obtain at least one of residual samples of the Cb component and the Cr component through a neural network and more efficiently restore the Cb component and the Cr component using at least one of the residual samples of the Cb component and the Cr component.

[0361] According to an embodiment of the present invention, the at least one processor applies a first scale factor to the residual sample of the Cb component obtained through the neural network to refine the residual sample of the Cb component, and applies a second scale factor to the residual sample of the Cr component to refine the residual sample of the Cr component.

[0362] According to an embodiment of the present invention, an AI-based video encoding device has the effect of obtaining a further refined residual sample by scaling, using a scale factor, the residual samples of the Cb and Cr components obtained through a neural network before adding them to the predicted samples of the Cb and Cr components.

[0363] According to an embodiment of the present invention, the first scale factor and the second scale factor may be encoded into an index indicating a scale factor included in a scale factor set.

[0364] According to an embodiment of the present invention, an AI-based video encoding device has the effect that the residual samples of the Cb and Cr components are more effectively refined by encoding and signaling an index indicating a suitable scale factor.

[0365] According to an embodiment of the present invention, a scale factor set is determined by at least one of the type of the slice in which the current block is included or the type of the current block among a plurality of scale factor sets.

[0366] According to an embodiment of the present invention, an AI-based video encoding device has the effect that a suitable scale factor set is used according to the type of the current block because the scale factor set is determined by at least one of the slice in which the current block is included or the type of the current block among a plurality of scale factor sets, and the residual samples of the Cb and Cr components are more effectively refined.

[0367] According to an embodiment of the present invention, a plurality of scale factor sets include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

[0368] According to an embodiment of the present invention, an AI-based video encoding device has the effect that a plurality of scale factor sets are used and the residual samples of the Cb and Cr components are more effectively refined.

[0369] According to an embodiment of the present invention, the neural network can be trained to determine the correlation between Cb and Cr for each sample of the current block for training, using the joint chroma residual sample for training, the predicted sample of the Cb component for training, and the predicted sample of the Cr component for training as input values.

[0370] According to an embodiment of the present invention, for the correlation between Cb and Cr, a weighting value for the residual sample of the Cb component and a weighting value for the residual sample of the Cr component are respectively determined.

[0371] The AI-based video encoding device according to an embodiment of the present invention has the effect that the values of the residual samples of the Cb component and the Cr component for each sample of the current block can be determined more accurately by the neural network learning the correlation between Cb and Cr for each sample of the current block.

[0372] According to an embodiment of the present invention, the input value of the neural network includes at least one of the quantization step size of the current block, the quantization error of the current block, or the restored luma block corresponding to the current chroma block of the current block, which is downsampled based on the chroma format.

[0373] The AI-based video encoding device according to an embodiment of the present invention has the effect that the quantization error within the current block is more efficiently removed during the restoration of the current block by further using the quantization error or the quantization step size.

[0374] The AI-based video encoding device according to an embodiment of the present invention has the effect that the Cb component and the Cr component of the current block can be restored more efficiently using the information of the restored luma block by further using the block in which the restored luma block is downsampled based on the chroma format.

[0375] According to an embodiment of the present invention, the neural network can be trained by first loss information corresponding to the difference between the original sample of the Cb component of the original block for training and the restored sample of the Cb component of the restored block for training obtained through the neural network, and second loss information corresponding to the difference between the original sample of the Cr component of the original block for training and the restored sample of the Cr component of the restored block for training obtained through the neural network.

[0376] In the AI-based video encoding device according to an embodiment of the present invention, the neural network is trained to minimize the loss information corresponding to the difference between the original sample and the restored sample of each of the Cb component and the Cr component, so that the restored samples of the Cb component and the Cr component obtained through the neural network are restored almost the same as the original samples of the Cb component and the Cr component.

[0377] According to an embodiment of the present invention, the model of the neural network is determined based on the Cb cbf information indicating whether the transform coefficient level of the current block for the Cb component is not zero and includes the Cb component, and the Cr cbf information indicating whether the transform coefficient level of the current block for the Cr component is not zero and includes the Cr component.

[0378] In the AI-based video encoding device according to an embodiment of the present invention, one of the models of the neural network trained by the Cb cbf information and the Cr cbf information is preferably selected, and based on the conditions of the current block such as the Cb cbf information and the Cr cbf information, the Cb component and the Cr component of the current block are restored more efficiently.

[0379] According to an embodiment of the present invention, the model of the neural network is determined by at least one of the type of the current block, the type of the slice in which the current block is included, the QP range of the slice, or whether the current block is the Cb component or the Cr component.

[0380] According to an embodiment of the present invention, in the AI-based video encoding device, one of the neural network models trained under various conditions is preferably selected, and based on the conditions of the current block, such as the type of the current block, the slice type including the current block, the QP range of the slice, or whether it is the Cb component or the Cr component of the current block, the Cb component and the Cr component of the current block can be restored more efficiently.

[0381] The machine-readable recording medium can be provided in the form of a non-transitory recording medium. Here, the "non-transitory recording medium" is a tangible device and only means that it does not include signals (e.g., electromagnetic waves), and this term does not distinguish between the case where data is semi-permanently stored in the recording medium and the case where it is temporarily stored. For example, the "non-transitory recording medium" may include a buffer in which data is temporarily stored.

[0382] According to an embodiment, the methods according to various embodiments disclosed in the present invention may be provided included in a computer program product. The computer program product can be traded as a commodity between a seller and a purchaser. The computer program product is distributed in the form of a machine-readable recording medium (e.g., compact disc read only memory (CD-ROM)), or through an application store, or can be directly distributed online (e.g., downloaded or uploaded) between two user devices (e.g., smartphones). In the case of online distribution, at least a part of the computer program product (e.g., a downloadable app) can be at least temporarily stored or generated in a machine-readable recording medium such as the memory of the manufacturer's server, the application store's server, or the relay server.

Claims

1. Obtain the joint chroma residual samples of the current block, the prediction information of the Cb component of the current block, and the prediction information of the Cr component of the current block from the bitstream (S1110), where the current block includes steps of the Cb component and the Cr component, and Determine the predicted samples of the Cb component of the current block based on at least the prediction information of the Cb component; Determine (S1130) the predicted samples of the Cr component of the current block based on at least the prediction information of the Cr component; Input the joint chroma residual samples, the predicted samples of the Cb component, and the predicted samples of the Cr component into a neural network, and obtain the restored samples of the Cb component and the restored samples of the Cr component of the current block from the output of the neural network, thereby restoring (S1150) the current block, including an AI-based video decoding method.

2. The step of restoring the current block includes Inputting the joint chroma residual samples, the predicted samples of the Cb component, and the predicted samples of the Cr component into a neural network to obtain at least one of the residual samples of the Cb component or the residual samples of the Cr component of the current block; Using at least one of the residual samples of the Cb component or the residual samples of the Cr component, the predicted samples of the Cb component, and the predicted samples of the Cr component to obtain the restored samples of the Cb component and the restored samples of the Cr component, thereby restoring the current block, further including the AI-based video decoding method according to claim 1.

3. The step of obtaining at least one of the residual samples of the Cb component or the residual samples of the Cr component includes Applying a first scale factor to the residual samples of the Cb component obtained through the neural network to purify the residual samples of the Cb component, and applying a second scale factor to the residual samples of the Cr component to purify the residual samples of the Cr component, further including the AI-based video decoding method according to any one of claims 1 to 2.

4. The AI-based video decoding method according to any one of claims 1 to 3, wherein the first scale factor and the second scale factor are included in a scale factor set and are determined by scale factor values indicated by indexes obtained from a bitstream.

5. The AI-based video decoding method according to any one of claims 1 to 4, wherein the scale factor set is determined by at least one of the type of the slice in which the current block is included or the type of the current block among a plurality of scale factor sets.

6. The AI-based video decoding method according to any one of claims 1 to 5, wherein the plurality of scale factor sets include [1, 1 / 2, 1 / 4] and [1, 1 / 2].

7. The AI-based video decoding method according to any one of claims 1 to 6, wherein the neural network is trained to determine the correlation between the Cb component and the Cr component for each sample of the current block for training, using the joint chroma residual sample for training, the prediction sample of the Cb component for training, and the prediction sample of the Cr component for training as input values.

8. The AI-based video decoding method according to any one of claims 1 to 7, wherein for the correlation between Cb and Cr, a weighting value for the residual sample of the Cb component and one or more weighting values for the residual sample of the Cr component are respectively determined.

9. The AI-based video decoding method according to any one of claims 1 to 8, wherein the input values of the neural network further include at least one of the quantization step size of the current block, the quantization error of the current block, or the restored luma block corresponding to the current chroma block of the current block, which is downsampled based on the chroma format.

10. The neural network is The AI-based video decoding method according to any one of claims 1 to 9, which is trained by first loss information corresponding to the difference between the original sample of the Cb component of the original block for training and the restored sample of the Cb component of the restored block for training obtained through the neural network, and second loss information corresponding to the difference between the original sample of the Cr component of the original block for training and the restored sample of the Cr component of the restored block for training obtained through the neural network.

11. The model of the neural network is determined based on Cb cbf (coded block flag) information indicating whether the current block for the Cb component, obtained from the bitstream, includes a Cb component with a non-zero transform coefficient level, and Cr cbf information indicating whether the current block for the Cr component includes a Cr component with a non-zero transform coefficient level, the AI-based video decoding method according to any one of claims 1 to 10.

12. The model of the neural network is determined by at least one of the type of the current block, the type of the slice in which the current block is included, the QP range of the slice, or whether the current block is a Cb component or a Cr component, the AI-based video decoding method according to any one of claims 1 to 11.

13. A memory for storing one or more instructions, At least one processor operating according to the one or more instructions, comprising: The at least one processor: Obtains the joint chroma residual sample of the current block, the Cb component prediction information of the current block, and the Cr component prediction information of the current block from the bitstream, where the current block includes a Cb component and a Cr component. Determines a predicted sample of the Cb component of the current block based on at least the Cb component prediction information. Determines a predicted sample of the Cr component of the current block based on at least the Cr component prediction information. An AI-based video decoding apparatus that restores the current block by inputting the joint chroma residual sample, the predicted sample of the Cb component, and the predicted sample of the Cr component into a neural network and obtaining a restored sample of the Cb component of the current block and a restored sample of the Cr component of the current block from the output of the neural network.

14. The at least one processor restoring the current block comprises: inputting the joint chroma residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component into a neural network to obtain at least one of the residual sample of the Cb component or the residual sample of the Cr component of the current block; using at least one of the residual sample of the Cb component or the residual sample of the Cr component, the prediction sample of the Cb component, and the prediction sample of the Cr component to obtain the restored sample of the Cb component and the restored sample of the Cr component, thereby restoring the current block. The AI-based video decoding apparatus according to claim 13.

15. generating (S1410) an initial joint chroma residual sample of the current block, prediction information of the Cb component of the current block, and prediction information of the Cr component of the current block; determining a prediction sample of the Cb component of the current block based at least on the prediction information of the Cb component; determining (S1430) a prediction sample of the Cr component of the current block based at least on the prediction information of the Cr component; encoding (S1450) the joint chroma residual sample of the current block by inputting the initial joint chroma residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component into a neural network and obtaining the restored sample of the Cb component and the restored sample of the Cr component of the current block from the output of the neural network. An AI-based video encoding method comprising the steps.

Images