Video image encoder and video image decoder

Abstract

To provide a video image encoder and video image decoder which applicably choose between an NN filter part and a ccALF processing part.SOLUTION: In a video image transmission system, a loop filter of a video image decoder produces a reconstruction image from the sum of a prediction image and a residual image; includes an NN filter part which carries out filter processing using a product-sum operation of two or more layers from a brightness and a color difference signal, and a ccALF processing part which derives a color difference signal from the brightness and the color difference signal. Either of the NN filter part or the ccALF processing part exclusively is adapted to the reconstruction image.SELECTED DRAWING: Figure 14

Description

[Technical field]

[0001] An embodiment of the present invention relates to a video encoding device and a video decoding device. [Background technology]

[0002] In order to efficiently transmit or record moving images, a moving image encoding device is used that generates encoded data by encoding moving images, and a moving image decoding device is used that generates a decoded image by decoding the encoded data.

[0003] Specific examples of video encoding methods include H.264 / AVC and H.265 / HEVC (High-Efficiency Video Coding).

[0004] In such a video coding method, images (pictures) constituting a video are managed in a hierarchical structure consisting of slices obtained by dividing images, coding tree units (CTUs) obtained by dividing slices, coding units (sometimes called coding units: CUs) obtained by dividing coding tree units, and transform units (TUs) obtained by dividing coding units, and are coded / decoded for each CU.

[0005] In such a video coding method, a predicted image is usually generated based on a locally decoded image obtained by encoding / decoding an input image, and a prediction error (sometimes called a "difference image" or "residual image") obtained by subtracting the predicted image from the input image (original image) is coded. Methods for generating a predicted image include inter-prediction and intra-prediction. In addition, an in-loop filter technique is defined, which reduces coding noise generated by quantization processing in the coding loop and prevents image quality degradation from propagating between frames.

[0006] Moreover, Non-Patent Document 1 can be cited as a recent example of a video encoding and decoding technique.

[0007] Non-Patent Document 1 discloses a method in which a neural network filter is placed between the SAO and ALF of an in-loop filter, and a network model is adaptively selected for each CTU based on the input. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Z. Dai, Y. Yu, H. Yu, K. Sato, L. Xu, Z. Xie, D. Wang, "[AHG11] Neural Network-based Adaptive Model Selection for CNN In-Loop Filtering", JVET-X0126, Oct. 2021. Summary of the Invention [Problem to be solved by the invention]

[0009] In Non-Patent Document 1, the entire network parameters are adaptively switched according to the input image. However, when switching the entire network parameters, it is difficult to switch parameters in small block units such as CTU. In addition, in Non-Patent Document 1, NN (Neural Network) filtering is performed before ALF, but there is a problem that the processing time is long if ALF class classification and filtering are performed sequentially after applying the NN filter. [Means for solving the problem]

[0010] A video decoding device according to one embodiment of the present invention includes a header decoding unit that decodes an NN model index from encoded data in a parameter set, a picture header, or a slice header, a CT information decoding unit 3021 that decodes an NN layer index from encoded data on a CTU basis, an NN filter unit that performs filtering using a neural network composed of multiple layers, and an NN layer selection unit that switches parameters of a specific layer among the multiple layers of a neural network model, wherein the NN filter unit selects a model to be used in the NN filter unit based on the NN model index, and the NN layer selection unit switches the parameters of the specific layer based on the NN layer index.

[0011] The system also includes a CT decoding unit that decodes nn_alf_ctb_flag in the CTU, an ALF processing unit that applies ALF processing to the decoded image, an NN filter unit that applies filter processing to the decoded image using a neural network, and an ALF padding unit, and the ALF unit includes an ALF class classification unit that performs ALF class classification, and an ALF class filter unit that performs ALF class filtering, and the ALF class classification unit performs class classification from the image before processing by the NN filter unit, and the ALF class filter unit performs class filtering from the image after processing by the NN filter unit. Effect of the Invention

[0012] By selecting a NN model in a first domain and then selecting a fine-tuned network layer in a second domain that is smaller than the first domain, it is possible to adaptively select a NN model even in small block units without the overhead required to switch the entire NN model.

[0013] In addition, by performing ALF class classification on the image before applying the NN filter, and then performing ALF processing using the image after NN filter processing and the ALF class information, it is possible to perform NN filter processing and ALF class classification in parallel. [Brief description of the drawings]

[0014] [Figure 1] 1 is a schematic diagram showing a configuration of a video transmission system according to an embodiment of the present invention. [Diagram 2] 1 is a diagram showing the configuration of a transmitting device equipped with a video encoding device according to the present embodiment, and a receiving device equipped with a video decoding device, in which PROD_A indicates the transmitting device equipped with the video encoding device, and PROD_B indicates the receiving device equipped with the video decoding device. [Diagram 3] 1 is a diagram showing the configuration of a recording device equipped with a video encoding device according to the present embodiment, and a playback device equipped with a video decoding device, in which PROD_C indicates a recording device equipped with a video encoding device, and PROD_D indicates a playback device equipped with a video decoding device. [Figure 4] FIG. 2 is a diagram showing a hierarchical structure of encoded data. [Diagram 5] 1 is a conceptual diagram of an image to be processed in the video transmission system according to the present embodiment. [Figure 6] FIG. 2 is a conceptual diagram showing an example of a reference picture and a reference picture list. [Figure 7] FIG. 1 is a schematic diagram showing a configuration of an image decoding device. [Figure 8] 11 is a flowchart illustrating a schematic operation of an image decoding device. [Figure 9] FIG. 13 is a schematic diagram showing a configuration of an inter-prediction parameter derivation unit. [Figure 10] FIG. 13 is a schematic diagram showing a configuration of an inter-prediction image generating unit. [Figure 11] FIG. 1 is a block diagram showing a configuration of an image encoding device. [Figure 12] FIG. 2 illustrates a deblocking filter. [Figure 13] FIG. 13 is a diagram showing the configuration of a neural network of an NN filter unit 611. [Figure 14] 13 is a diagram showing exclusive switching processing between ccALF and the NN filter in the loop filter 305. FIG. [Figure 15]FIG. 13 is a diagram showing a process for obtaining ccNN from color difference in the loop filter 305. [Figure 16] FIG. 13 is a diagram illustrating a process performed by an NN filter unit 611d in the loop filter 305 using each of the Y, Cb, and Cr components of an input image. [Figure 17] FIG. 13 is a diagram showing syntax for exclusively switching the ccALF and NN filter flags in a slice header. [Figure 18] A diagram showing syntax for exclusively switching the ccALF flag and the NN scale index in the CTU. [Figure 19] A diagram showing the syntax for exclusively switching the ccALF flag and the NN flag in the CTU. [Figure 20] 13 is a diagram illustrating exclusive switching processing between the ALF and the NN filter in the loop filter 305. FIG. [Figure 21] FIG. 13 is a diagram showing syntax for exclusively switching the chroma ALF and NN filter flags in a slice header. [Figure 22] A diagram showing syntax for exclusively switching the ALF flag and NN scale index in a CTU. [Figure 23] A diagram showing the syntax for exclusively switching the ALF flag and the NN flag in the CTU. [Figure 24] FIG. 13 is a diagram showing NN model switching in a slice and partial model switching in a CTU. [Diagram 25] A diagram showing decoding of the NN model index in the slice header. [Figure 26] A diagram showing decoding of NN partial model index in a CTU. [Figure 27] FIG. 13 is a diagram showing a process of deriving bS by using a bS calculation parameter used in an NN filter and a DF (Deblocking Filter). [Figure 28]FIG. 13 is a diagram showing a process of deriving bS by inputting a bS calculation parameter used in DF as a difference from a bS calculation parameter used in an NN filter. [Figure 29] FIG. 13 is a diagram showing syntax related to decoding of parameters for calculating bS of an NN filter. [Diagram 30] 1 is a derivation table of parameters beta' and tC' based on the quantization parameters. [Diagram 31] FIG. 13 is a diagram showing ALF class classification processing using an image before NN filtering and ALF class filtering processing using an image after NN filtering. [Diagram 32] FIG. 13 is a diagram illustrating an ALF padding process. [Diagram 33] FIG. 13 is a diagram showing syntax regarding a flag that enables the application of an NN filter only when a slice is rectangular. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] (First embodiment) Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0016] FIG. 1 is a schematic diagram showing the configuration of a video transmission system according to this embodiment.

[0017] The video transmission system 1 is a system that transmits coded data obtained by coding an image with a different resolution after resolution conversion, decodes the transmitted coded data, and converts the image back to the original resolution for display. The video transmission system 1 is made up of a video coding device 10, a network 21, a video decoding device 30, and an image display device 41.

[0018] The video encoding device 10 includes a preprocessing device (preprocessing unit) 51, an image encoding device (image encoding unit) 11, and a synthesis information creating device (synthetic information creating unit) 71.

[0019] The video decoding device 30 includes an image decoding device (image decoding unit) 31 and a post-processing unit (post-processing unit) 61.

[0020] The pre-processing device 51 converts the resolution of an image T included in the video as necessary, and supplies a variable resolution video T2 including images with different resolutions to the image coding device 11. The pre-processing device 51 may also supply filter information indicating whether or not the resolution of an image is to be converted to the image coding device 11. If the information indicates resolution conversion, the video coding device 10 sets ref_pic_resampling_enabled_flag, which will be described later, to 1. Then, the video coding device 10 performs coding in a sequence parameter set SPS (Sequence Parameter Set) of the coded data Te.

[0021] The synthesis information generating device 71 generates filter information based on an image T1 included in the video, and sends the filter information to the image encoding device 11. A variable resolution image T2 is input to the image encoding device 11. The image encoding device 11 encodes image size information of the input image in PPS units using the RPR framework, and transmits the encoded image size information to the image decoding device 31.

[0022] The network 21 transmits the encoded filter information and the encoded data Te to the image decoding device 31. A part or all of the encoded filter information may be included in the encoded data Te as supplemental extension information SEI. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 21 is not necessarily limited to a bidirectional communication network, and may be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. The network 21 may also be replaced by a storage medium on which the encoded data Te is recorded, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blue-ray Disc: registered trademark).

[0023] The image decoding device 31 decodes each of the encoded data Te transmitted by the network 21 , generates a variable resolution decoded image, and supplies it to the post-processing device 61 .

[0024] When the filter information indicates resolution conversion, the post-processing device 61 performs super-resolution processing using model parameters for super-resolution based on image size information included in the encoded data. Then, the post-processing device 61 generates a decoded image of the original size by inversely converting the resolution-converted image. When the filter information does not indicate resolution conversion, the post-processing device 61 performs image restoration processing using model parameters for image restoration. By performing the image restoration processing, a decoded image with reduced encoding noise is generated.

[0025] The image display device 41 displays all or part of one or more decoded images Td2 input from the post-processing device 61. The image display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. The display may be in the form of a stationary display, a mobile display, an HMD, or the like. When the image decoding device 31 has high processing power, it displays a high-quality image, and when it has only low processing power, it displays an image that does not require high processing power or display power.

[0026] Fig. 5 is a conceptual diagram of an image to be processed in the video transmission system shown in Fig. 1, showing the change in resolution of the image over time. However, in Fig. 5, no distinction is made as to whether the image is encoded or not. Fig. 5 shows an example in which an image is transmitted to the image decoding device 31 with a reduced resolution during processing in the video transmission system. As shown in Fig. 5, the image preprocessing device 51 usually performs conversion to reduce the resolution of the image in order to reduce the amount of information to be transmitted.

[0027] <operator> The operators used in this specification are listed below.

[0028] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is the OR assignment operator, and || indicates logical OR.

[0029] x?y:z is a ternary operator that takes y when x is true (non-zero) and z when x is false (0).

[0030] Clip3(a, b, c) is a function that clips c to a value between a and b (inclusive). If c < a, it returns a; if c > b, it returns b; otherwise, it returns c (assuming a <= b).

[0031] abs(a) is a function that returns the absolute value of a.

[0032] Int(a) is a function that returns the integer value of a.

[0033] floor(a) is a function that returns the largest integer less than or equal to a.

[0034] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0035] a / d represents the division of a by d, rounded down to the nearest integer.

[0036] a^b represents a to the power of b. When a = 2 and b is an integer, 2^b = 1 << b.

[0037] <Structure of the Encoded Data Te> Prior to the detailed description of the image encoding device 11 and the image decoding device 31 according to this embodiment, the data structure of the encoded data Te generated by the image encoding device 11 and decoded by the image decoding device 31 will be described.

[0038] Fig. 4 is a diagram showing a hierarchical structure of data in the coded data Te. The coded data Te illustratively includes a sequence and a plurality of pictures constituting the sequence. Fig. 4 shows a diagram showing a coded video sequence that defines the sequence SEQ, a coded picture that defines the picture PICT, a coded slice that defines the slice S, coded slice data that defines the slice data, a coding tree unit included in the coded slice data, and a coding unit included in the coding tree unit.

[0039] (Coded Video Sequence) The coded video sequence defines a set of data that the image decoding device 31 refers to in order to decode the sequence SEQ to be processed. As shown in Fig. 4, the sequence SEQ includes a video parameter set VPS (Video Parameter Set), a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), an adaptation parameter set (APS), a picture PICT, and supplemental enhancement information SEI (Supplemental Enhancement Information).

[0040] The video parameter set VPS specifies a set of coding parameters common to multiple videos composed of multiple layers, as well as a set of coding parameters related to multiple layers and each individual layer included in the video.

[0041] The sequence parameter set SPS specifies a set of coding parameters that the image decoding device 31 refers to in order to decode the target sequence. For example, the width and height of a picture are specified. Note that there may be multiple SPSs. In that case, one of the multiple SPSs is selected from the PPS.

[0042] (Encoded Picture) The coded picture defines a set of data to be referenced by the image decoding device 31 in order to decode the picture PICT to be processed. The picture PICT is, as shown in FIG. The picture PICT includes slices 0 to NS-1 (NS is the total number of slices included in the picture PICT).

[0043] Hereinafter, when there is no need to distinguish between slices 0 to NS-1, the subscripts of the symbols may be omitted. The same applies to other data that are included in the coded data Te described below and have subscripts.

[0044] (Coded Slice) A coded slice defines a set of data to be referenced by the image decoding device 31 in order to decode a target slice S. As shown in Fig. 4, a slice includes a slice header and slice data.

[0045] The slice header includes a group of coding parameters to be referred to by the image decoding device 31 in order to determine a decoding method for the current slice. Slice type designation information (slice_type) that designates a slice type is an example of a coding parameter included in the slice header.

[0046] Slice types that can be specified by the slice type specification information include (1) an I slice that uses only intra prediction when encoding, (2) a P slice that uses uni-prediction (L0 prediction) or intra prediction when encoding, and (3) a B slice that uses uni-prediction (L0 prediction or L1 prediction), bi-prediction, or intra prediction when encoding. Note that inter prediction is not limited to uni-prediction or bi-prediction, and a predicted image may be generated using more reference pictures. Hereinafter, when referring to P or B slice, it refers to a slice including a block that can use inter prediction.

[0047] In addition, the slice header may include a reference to a picture parameter set PPS (pic_parameter_set_id).

[0048] (Encoded slice data) The coded slice data specifies a set of data that the image decoding device 31 refers to in order to decode the slice data to be processed. As shown in the coded slice header in Fig. 4, the slice data includes a coding tree unit (CTU). A CTU is a block of a fixed size (e.g., 64x64) that constitutes a slice, and is also called a largest coding unit (LCU) or a coding tree block (CTB).

[0049] (coding tree unit) 4 specifies a set of data that the image decoding device 31 refers to in order to decode a CTU to be processed. The CTU is divided into coding units CU, which are basic units of encoding processing, by recursive quad tree division (QT (Quad Tree) division), binary tree division (BT (Binary Tree) division), or ternary tree division (TT (Ternary Tree) division). BT division and TT division are collectively called multi tree division (MT (Multi Tree) division). A node of a tree structure obtained by recursive quad tree division is called a coding node. Intermediate nodes of a quad tree, binary tree, and ternary tree are coding nodes, and the CTU itself is specified as the top coding node.

[0050] CT includes, as CT information, a CU split flag (split_cu_flag) indicating whether CT splitting is performed, a QT split flag (qt_split_cu_flag) indicating whether QT splitting is performed, an MT split direction (mtt_split_cu_vertical_flag) indicating the split direction of MT splitting, and an MT split type (mtt_split_cu_binary_flag) indicating the split type of MT splitting. split_cu_flag, qt_split_cu_flag, mtt_split_cu_vertical_flag, and mtt_split_cu_binary_flag are transmitted for each encoding node.

[0051] (Encoding Unit) 4 specifies a set of data to be referred to by the image decoding device 31 in order to decode a coding unit to be processed. Specifically, a CU is composed of a CU header CUH, prediction parameters, transformation parameters, quantization transformation coefficients, etc. The CU header specifies a prediction mode, etc.

[0052] The prediction process may be performed on a CU basis, or on a sub-CU basis by further dividing a CU. If the size of a CU and a sub-CU are the same, there is one sub-CU in the CU. If the size of a CU is larger than that of a sub-CU, the CU is divided into sub-CUs. For example, if the CU is 8x8 and the sub-CU is 4x4, the CU is divided into 2 parts horizontally and 2 parts vertically, into 4 sub-CUs.

[0053] There are two types of prediction (prediction modes): intra prediction and inter prediction. Intra prediction is a prediction within the same picture, while inter prediction refers to a prediction process performed between different pictures (for example, between display times or between layer images).

[0054] The transform and quantization processes are performed in units of CUs, but the quantized transform coefficients may be entropy coded in units of sub-blocks such as 4x4.

[0055] (Prediction parameters) The predicted image is derived from prediction parameters associated with the block, which include intra-prediction and inter-prediction parameters.

[0056] Hereinafter, prediction parameters of inter prediction will be described. Inter prediction parameters are composed of prediction list use flags predFlagL0 and predFlagL1, reference picture indexes refIdxL0 and refIdxL1, and motion vectors mvL0 and mvL1. predFlagL0 and predFlagL1 are flags indicating whether or not a reference picture list (L0 list, L1 list) is used, and when the value is 1, the corresponding reference picture list is used. Note that in this specification, when "a flag indicating whether or not XX" is written, a flag other than 0 (for example, 1) is XX, 0 is not XX, and 1 is treated as true and 0 is treated as false in logical negation, logical product, etc. (similarly below). However, in an actual device or method, other values ​​can be used as true and false values.

[0057] Syntax elements for deriving inter-prediction parameters include, for example, an affine flag affine_flag used in merge mode, a merge flag merge_flag, a merge index merge_idx, an MMVD flag mmvd_flag, an inter-prediction identifier inter_pred_idc for selecting a reference picture to be used in AMVP mode, a reference picture index refIdxLX, a prediction vector index mvp_LX_idx for deriving a motion vector, a difference vector mvdLX, and a motion vector precision mode amvr_mode.

[0058] (Reference picture list) The reference picture list is a list of reference pictures stored in the reference picture memory 306. Figure 6 is a conceptual diagram showing an example of a reference picture and a reference picture list. In the conceptual diagram showing an example of a reference picture in Figure 6, a rectangle represents a picture, an arrow represents a reference relationship between pictures, the horizontal axis represents time, I, P, and B in the rectangle represent an intra picture, a uni-predictive picture, and a bi-predictive picture, respectively, and the numbers in the rectangle represent a decoding order. As shown in the figure, the decoding order of pictures is I0, P1, B2, B3, and B4, and the display order is I0, B3, B2, B4, and P1.

[0059] (Merge prediction and AMVP prediction) The prediction parameter decoding (encoding) method includes a merge prediction mode and an AMVP (Advanced Motion Vector Prediction) mode, and merge_flag is a flag for identifying these. The merge prediction mode is a mode in which the prediction list usage flag predFlagLX, the reference picture index refIdxLX, and the motion vector mvLX are not included in the encoded data, but are derived from the prediction parameters of the neighboring blocks that have already been processed. The AMVP mode is a mode in which inter_pred_idc, refIdxLX, and mvLX are included in the encoded data. Note that mvLX is encoded as mvp_LX_idx, which identifies the prediction vector mvpLX, and the difference vector mvdLX. In addition to the merge prediction mode, there may be an affine prediction mode and an MMVD prediction mode.

[0060] inter_pred_idc is a value indicating the type and number of reference pictures, and takes one of the values ​​PRED_L0, PRED_L1, and PRED_BI. PRED_L0 and PRED_L1 indicate uni-prediction using one reference picture managed in the L0 list and L1 list, respectively. PRED_BI indicates bi-prediction using two reference pictures managed in the L0 list and L1 list.

[0061] The merge_idx is an index indicating which prediction parameter is to be used as the prediction parameter of the current block among prediction parameter candidates (merge candidates) derived from blocks for which processing has been completed.

[0062] (Motion Vector) mvLX indicates the amount of shift between blocks on two different pictures. A prediction vector and a difference vector related to mvLX are called mvpLX and mvdLX, respectively.

[0063] (Inter prediction identifier inter_pred_idc and prediction list usage flag predFlagLX) The relationship between inter_pred_idc, predFlagL0, and predFlagL1 is as follows, and they are mutually convertible.

[0064] inter_pred_idc=(predFlagL1<<1)+predFlagL0 predFlagL0=inter_pred_idc&1 predFlagL1=inter_pred_idc>>1 (Configuration of an image decoding device) The configuration of an image decoding device 31 (FIG. 7) according to this embodiment will be described.

[0065] The image decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit (prediction image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a prediction image generating unit (prediction image generating device) 308, an inverse quantization and inverse transform unit 311, an adder 312, and a prediction parameter derivation unit 320. Note that, in accordance with the image encoding device 11 described below, the image decoding device 31 may also be configured not to include the loop filter 305.

[0066] The parameter decoding unit 302 further includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit), and the CU decoding unit 3022 further includes a TU decoding unit 3024. These may be collectively referred to as a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, PPS, and APS, a picture header, and a slice header (slice information) from the encoded data. The CT information decoding unit 3021 decodes the CT from the encoded data. The CU decoding unit 3022 decodes the CU from the encoded data. The TU decoding unit 3024 decodes QP update information (quantization correction value) and quantization prediction error (residual_coding) from the encoded data when a prediction error is included in the TU.

[0067] In cases other than the skip mode (skip_mode==0), the TU decoding unit 3024 decodes the QP update information and the quantized prediction error from the encoded data. More specifically, in cases of skip_mode==0, the TU decoding unit 3024 decodes a flag cu_cbp indicating whether or not the current block includes a quantized prediction error, and decodes the quantized prediction error when cu_cbp is 1. When cu_cbp does not exist in the encoded data, it derives 0.

[0068] The predicted image generating unit 308 includes an inter predicted image generating unit 309 and an intra predicted image generating unit 310 .

[0069] The prediction parameter derivation unit 320 is configured to include an inter prediction parameter derivation unit 303 and an intra prediction parameter derivation unit 304 .

[0070] In addition, although an example in which CTU and CU are used as processing units will be described below, the present invention is not limited to this example, and processing may be performed in sub-CU units. Alternatively, CTU and CU may be read as blocks, and sub-CU as sub-blocks, and processing may be performed in block or sub-block units.

[0071] The entropy decoding unit 301 performs entropy decoding on the encoded data Te input from the outside, and decodes each code (syntax element).

[0072] The entropy decoding unit 301 outputs the decoded code to the parameter decoding unit 302. The decoded code is, for example, a prediction mode predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX, amvr_mode, etc. Control of which code to decode is performed based on an instruction from the parameter decoding unit 302.

[0073] (Basic flow) FIG. 8 is a flowchart illustrating a schematic operation of the image decoding device 31.

[0074] (S1100: Decode Parameter Set Information) The header decoding unit 3020 decodes parameter set information such as the VPS, SPS, and PPS from the encoded data.

[0075] (S1200: Decode slice information) The header decoding unit 3020 decodes the slice header (slice information) from the encoded data.

[0076] Thereafter, the image decoding device 31 repeats the processes from S1300 to S5000 for each CTU included in the target picture to derive a decoded image of each CTU.

[0077] (S1300: Decode CTU Information) The CT information decoding unit 3021 decodes the CTU from the encoded data.

[0078] (S1400: Decode CT Information) The CT information decoding unit 3021 decodes the CT from the encoded data.

[0079] (S1500: CU Decoding) The CU decoding unit 3022 performs S1510 and S1520 to decode the CU from the encoded data.

[0080] (S1510: Decoding CU information) The CU decoding unit 3022 decodes the CU information, prediction information, and CU residual flags tu_cb_coded_flag, tu_cr_coded_flag, tu_y_coded_flag, and the like from the encoded data.

[0081] (S1520: TU information decoding) When a prediction error is included in a TU, the TU decoding unit 3024 decodes the QP update information, the quantization prediction error, and the transform index mts_idx from the encoded data. Note that the QP update information is a difference value from the quantization parameter predicted value qPpred, which is a predicted value of the quantization parameter QP.

[0082] (S2000: Generation of predicted image) The predicted image generation unit 308 generates a predicted image for each block included in the current CU based on prediction information.

[0083] (S3000: Inverse Quantization and Inverse Transformation) The inverse quantization and inverse transform unit 311 executes inverse quantization and inverse transform processing for each TU included in the target CU.

[0084] (S4000: Generate decoded image) The adder 312 adds the predicted image supplied from the predicted image generation unit 308 and the prediction error supplied from the inverse quantization and inverse transform unit 311 to generate a decoded image of the current CU.

[0085] (S5000: Loop Filter) The loop filter 305 applies a loop filter such as a deblocking filter, SAO, or ALF to the decoded image to generate a decoded image.

[0086] (Configuration of inter-prediction parameter derivation unit) 9 is a schematic diagram showing the configuration of the inter prediction parameter derivation unit 303 according to this embodiment. The inter prediction parameter derivation unit 303 derives inter prediction parameters by referring to the prediction parameters stored in the prediction parameter memory 307 based on the syntax elements input from the parameter decoding unit 302. In addition, the inter prediction parameters are output to the inter prediction image generation unit 309 and the prediction parameter memory 307. The inter prediction parameter derivation unit 303 and its internal elements, the AMVP prediction parameter derivation unit 3032, the merge prediction parameter derivation unit 3036, the affine prediction unit 30372, the MMVD prediction unit 30373, the GPM unit 30377, the DMVR unit 30537, and the MV addition unit 3038, are means common to the image encoding device and the image decoding device, and therefore may be collectively referred to as a motion vector derivation unit (motion vector derivation device).

[0087] When affine_flag is 1, that is, when affine prediction mode is indicated, the affine prediction unit 30372 derives inter prediction parameters on a sub-block basis.

[0088] When mmvd_flag is 1, that is, when it indicates the MMVD prediction mode, the MMVD predictor 30373 derives inter prediction parameters from the merge candidates and difference vectors derived by the merge prediction parameter derivation unit 3036.

[0089] If gpm_flag is 1, i.e., indicates Geometric Partitioning Mode, the GPM unit 30377 derives the GPM parameters.

[0090] When merge_flag is 1, that is, when it indicates the merge prediction mode, merge_idx is derived and output to the merge prediction parameter derivation unit 3036 .

[0091] When merge_flag is 0, that is, when it indicates the AMVP prediction mode, the AMVP prediction parameter derivation unit 3032 derives mvpLX from inter_pred_idc, refIdxLX, or mvp_LX_idx.

[0092] (MV addition section) The MV adder 3038 adds the derived mvpLX and mvdLX to derive mvLX.

[0093] (Affine prediction part) The affine prediction unit 30372 1) derives motion vectors of two control points CP0 and CP1, or three control points CP0, CP1, and CP2 of the target block, 2) derives affine prediction parameters for the target block, and 3) derives motion vectors of each sub-block from the affine prediction parameters.

[0094] (Merge prediction) The merge prediction parameter derivation unit 3036 derives prediction parameters of the current block using prediction parameters (mvLX, refIdxLX, etc.) of spatially neighboring blocks or temporally neighboring blocks of the current block.

[0095] (DMVR) Next, the DMVR unit 30375 performs a DMVR (Decoder side Motion Vector Refinement) process. When merge_flag is 1 or the skip flag skip_flag is 1 for the target CU, the DMVR unit 30375 corrects the motion vector mvLX of the target CU. Specifically, when the prediction parameters derived by the merge prediction unit 30374 are bi-predictive, the DMVR unit 30375 corrects the mvLX using two reference pictures and a predicted image derived from the motion vector. The corrected mvLX is supplied to the inter-prediction image generation unit 309.

[0096] (AMVP Prediction) The AMVP prediction parameter derivation unit 3032 selects the motion vector mvpListLX[mvp_LX_idx] indicated by mvp_LX_idx from among the prediction vector candidates as mvpLX, and outputs the selected motion vector to the MV addition unit 3038.

[0097] (MV addition section) The MV addition unit 3038 calculates mvLX by adding the mvpLX input from the AMVP prediction parameter derivation unit 3032 and the decoded mvdLX. The addition unit 3038 outputs the calculated mvLX to the inter predicted image generation unit 309 and the prediction parameter memory 307.

[0098] mvLX[0]=mvpLX[0]+mvdLX[0] mvLX[1]=mvpLX[1]+mvdLX[1] The loop filter 305 is a filter provided in the encoding loop, which removes block distortion and ringing distortion to improve image quality. The loop filter 305 applies a filter such as a deblocking filter, a sample adaptive offset (SAO), an adaptive loop filter (ALF), a cross-component ALF (ccALF), a neural network filter (NN), or a cross-component neural network filter (ccNN) to the decoded image of the CU generated by the adder 312.

[0099] The SAO unit adds an offset to the post-DF image according to the SAO class. There are two types of SAO class classification: band offset and edge offset. Band offset divides pixel values ​​into 32 bands. Edge offset classifies the target pixel based on the value of the internal division point of the pixel value change between two adjacent pixels in each of the four directions: up / down, left / right, and diagonal. Band offset adds an offset value set for each band to pixels whose pixel values ​​belong to four consecutive bands. Edge offset also adds an offset for each class to the target pixel value and outputs the processed image.

[0100] The ALF unit 312 divides the image after DF or SAO processing into 4x4 small regions. The luminance image (Y image) is classified into a plurality of classes (e.g., 25) based on the directionality (e.g., 5 directional classes) and activity (e.g., 5 levels) for each small region. Then, a filter based on the classification is used to perform a filter process and output the processed image. A 7x7 diamond-shaped filter may be applied to the Y image, and a 5x5 diamond-shaped filter may be applied to the chrominance image (Cb, Cr images). The ALF filter coefficients may be fixed filters provided in advance in the video decoding device and video encoding device, or filter coefficients decoded from the encoded data.

[0101] The ccALF unit 313 adds the filter processing results for the pixel positions of Cb and Cr and the surrounding pixels (e.g., 8 pixels) of the Y image corresponding to the same positions to the pixel values ​​of Cb and Cr for the Y image after DF or SAO processing. The ccALF filter coefficients are also decoded and encoded by the encoded data.

[0102] The reference picture memory 306 stores the decoded image of the CU in a location that is determined in advance for each current picture and current CU.

[0103] The prediction parameter memory 307 stores prediction parameters at a predetermined location for each CTU or CU. The meter and the parameters derived by the prediction parameter derivation unit 320 are stored.

[0104] The predicted image generating unit 308 receives parameters derived by the prediction parameter derivation unit 320. The predicted image generating unit 308 also reads a reference picture from the reference picture memory 306. The predicted image generating unit 308 generates a predicted image of a block or sub-block using the parameters and a reference picture (reference picture block) in a prediction mode indicated by predMode. Here, the reference picture block is a set of pixels on the reference picture (usually rectangular, so called a block), and is an area referenced to generate a predicted image.

[0105] (Inter-prediction image generation unit 309) When predMode indicates inter prediction mode, the inter prediction image generation unit 309 generates a prediction image of a block or sub-block by inter prediction using the inter prediction parameters and reference picture input from the inter prediction parameter derivation unit 303.

[0106] 10 is a schematic diagram showing a configuration of an inter-prediction image generation unit 309 included in the prediction image generation unit 308 according to this embodiment. The inter-prediction image generation unit 309 includes a motion compensation unit (prediction image generation device) 3091 and a synthesis unit 3095. The synthesis unit 3095 includes an IntraInter synthesis unit 30951, a GPM synthesis unit 30952, a BDOF unit 30954, and a weighted prediction unit 3094.

[0107] (Motion Compensation) The motion compensation unit 3091 (interpolated image generation unit 3091) generates an interpolated image (motion compensated image) by reading a reference block from the reference picture memory 306 based on the inter prediction parameters (predFlagLX, refIdxLX, mvLX) input from the inter prediction parameter derivation unit 303. The reference block is a block at a position shifted by mvLX from the position of the target block on the reference picture RefPicLX specified by refIdxLX. Here, if mvLX is not integer precision, a filter for generating pixels at decimal positions called a motion compensation filter is applied to generate an interpolated image.

[0108] In addition, the motion compensation unit 3091 has the function of scaling the interpolated image according to the horizontal scaling ratio RefPicScale[i][j][0] of the reference picture derived by the scale parameter derivation unit 30378 and the vertical scaling ratio RefPicScale[i][j][1] of the reference picture.

[0109] The synthesis unit 3095 includes an IntraInter synthesis unit 30951, a GPM synthesis unit 30952, a weighted prediction unit 3094, and a BDOF unit 30954.

[0110] (IntraInter synthesis processing) The IntraInter synthesis unit 30951 generates a predicted image by a weighted sum of an inter predicted image and an intra predicted image.

[0111] (GPM synthesis processing) The GPM synthesis unit 30952 generates a predicted image using the above-mentioned GPM.

[0112] (BDOF prediction) In bi-prediction mode, the BDOF unit 30954 generates a predicted image by referring to two predicted images (a first predicted image and a second predicted image) and a gradient correction term.

[0113] (Weighted prediction) The weighted prediction unit 3094 performs weighted prediction from the interpolated image PredLX to generate a predicted image pbSamples of the block.

[0114] When predMode indicates an intra prediction mode, the intra prediction image generation unit 310 performs intra prediction using the intra prediction parameters input from the intra prediction parameter derivation unit 304 and reference pixels read from the reference picture memory 306 .

[0115] The inverse quantization and inverse transform unit 311 inverse quantizes the quantized transform coefficients input from the parameter decoding unit 302 to obtain transform coefficients.

[0116] The adder 312 generates a decoded image of the block by adding, for each pixel, the predicted image of the block input from the predicted image generation unit 308 and the prediction error input from the inverse quantization and inverse transform unit 311. The adder 312 stores the decoded image of the block in the reference picture memory 306, and also outputs it to the loop filter 305.

[0117] The inverse quantization and inverse transform unit 311 inverse quantizes the quantized transform coefficients input from the parameter decoding unit 302 to obtain transform coefficients.

[0118] The adder 312 generates a decoded image of the block by adding, for each pixel, the predicted image of the block input from the predicted image generation unit 308 and the prediction error input from the inverse quantization and inverse transform unit 311. The adder 312 stores the decoded image of the block in the reference picture memory 306, and also outputs it to the loop filter 305.

[0119] (Configuration of Image Encoding Device) Next, the configuration of the image encoding device 11 according to this embodiment will be described. Fig. 11 is a block diagram showing the configuration of the image encoding device 11 according to this embodiment. The image encoding device 11 includes a prediction image generating unit 101, a subtraction unit 102, a transformation and quantization unit 103, an inverse quantization and inverse transformation unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference image storage unit, frame memory) 109, an encoding parameter determining unit 110, a parameter encoding unit 111, a prediction parameter derivation unit 120, and an entropy encoding unit 104.

[0120] The predicted image generation unit 101 generates a predicted image for each CU. The predicted image generation unit 101 includes the inter predicted image generation unit 309 and the intra predicted image generation unit 310, which have already been described, and therefore a description thereof will be omitted.

[0121] The subtraction unit 102 generates a prediction error by subtracting the pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 from the pixel values ​​of the image T. The subtraction unit 102 outputs the prediction error to the transformation and quantization unit 103.

[0122] The transform / quantization unit 103 calculates transform coefficients by frequency transforming the prediction errors input from the subtraction unit 102, and derives quantized transform coefficients by quantizing the prediction errors. The transform / quantization unit 103 outputs the quantized transform coefficients to the parameter coding unit 111 and the inverse quantization / inverse transform unit 105.

[0123] The inverse quantization and inverse transform unit 105 is the same as the inverse quantization and inverse transform unit 311 (FIG. 7) in the image decoding device 31, and a description thereof will be omitted. The calculated prediction error is output to the addition unit .

[0124] The parameter coding unit 111 includes a header coding unit 1110, a CT information coding unit 1111, and a CU coding unit 1112 (prediction mode coding unit). The CU coding unit 1112 further includes a TU coding unit 1114. The following describes an outline of the operation of each module.

[0125] The header encoding unit 1110 performs encoding processing of parameters such as filter information, header information, division information, prediction information, and quantized transform coefficients.

[0126] The CT information encoding unit 1111 encodes the QT, MT (BT, TT) division information and the like.

[0127] The CU encoding unit 1112 encodes the CU information, prediction information, division information, and so on.

[0128] When a prediction error is included in a TU, the TU encoding unit 1114 encodes the QP update information and the quantized prediction error.

[0129] The CT information encoding unit 1111 and the CU encoding unit 1112 supply syntax elements such as inter prediction parameters (predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX), intra prediction parameters (intra_luma_mpm_flag, intra_luma_mpm_idx, intra_luma_mpm_reminder, intra_chroma_pred_mode), and quantized transform coefficients to the parameter encoding unit 111.

[0130] The entropy coding unit 104 receives the quantized transform coefficients and coding parameters (division information, prediction parameters) from the parameter coding unit 111. The entropy coding unit 104 entropy codes these to generate and output coded data Te.

[0131] The prediction parameter derivation unit 120 is a means including the inter prediction parameter encoding unit 112 and the intra prediction parameter encoding unit 113, and derives intra prediction parameters and intra prediction parameters from the parameters input from the encoding parameter determination unit 110.

[0132] The adder 106 generates a decoded image by adding, for each pixel, the pixel value of the predicted block input from the predicted image generation unit 101 and the prediction error input from the inverse quantization and inverse transform unit 105. The adder 106 stores the generated decoded image in a reference picture memory 109.

[0133] The loop filter 107 performs deblocking filtering, SAO, and ALF on the decoded image generated by the adder 106. Note that the loop filter 107 does not necessarily have to include the above three types of filters, and may be configured, for example, as only a deblocking filter.

[0134] The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in a predetermined location for each current picture and CU.

[0135] The reference picture memory 109 stores the decoded image generated by the loop filter 107 at a predetermined position for each current picture and CU.

[0136] The coding parameter determination unit 110 selects one set from among a plurality of sets of coding parameters. The coding parameters are the above-mentioned QT, BT or TT division information, prediction parameters, or parameters to be coded that are generated in relation to these. The predicted image generation unit 101 generates a predicted image using these coding parameters.

[0137] The coding parameter determination unit 110 calculates an RD cost value indicating the magnitude of the information amount and the coding error for each of the multiple sets. The RD cost value is, for example, the sum of the code amount and the value obtained by multiplying the squared error by a coefficient λ. The code amount is the information amount of the coded data Te obtained by entropy coding the quantization error and the coding parameters. The squared error is the sum of the squares of the prediction errors calculated in the subtraction unit 102. The coefficient λ is a real number greater than zero that is set in advance. The coding parameter determination unit 110 selects a set of coding parameters that minimizes the calculated cost value. The coding parameter determination unit 110 outputs the determined coding parameters to the parameter coding unit 111 and the prediction parameter derivation unit 120.

[0138] In addition, a part of the image encoding device 11 and the image decoding device 31 in the above-mentioned embodiment, for example, the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the predicted image generating unit 308, the inverse quantization and inverse transform unit 311, the addition unit 312, the prediction parameter derivation unit 320, the predicted image generating unit 101, the subtraction unit 102, the transform and quantization unit 103, the entropy coding unit 104, the inverse quantization and inverse transform unit 105, the loop filter 107, the coding parameter determination unit 110, the parameter coding unit 111, and the prediction parameter derivation unit 120 may be realized by a computer. In this case, a program for realizing this control function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into and executed by a computer system. In addition, the "computer system" referred to here is a computer system built into either the image encoding device 11 or the image decoding device 31, and includes hardware such as an OS and peripheral devices. In addition, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" may also include devices that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and devices that hold a program for a certain period of time, such as volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0139] In addition, a part or the whole of the image encoding device 11 and the image decoding device 31 in the above-mentioned embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image encoding device 11 and the image decoding device 31 may be individually made into a processor, or a part or the whole may be integrated into a processor. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the progress of semiconductor technology, an integrated circuit based on that technology may be used.

[0140] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention. (ALF section 312) The ALF unit 312 receives the reconstructed luminance image recPictureL and the reconstructed chrominance images recPictureCb and recPictureCr as input, and outputs processed reconstructed picture sample arrays alfPictureL, AlfPictureCb, and AlfPictureCr. Note that, hereinafter, an image is also referred to as a sample array, and an image of one pixel is also referred to as a sample.

[0141] Each value in alfPictureL and each value in alfPictureCb and alfPictureCr are initialized with the value of recPictureL and the values ​​of recPictureCb and recPictureCr, respectively. Below, alf_ctb_flag[cIdx][rx][ry] is a flag that indicates whether to apply ALF to the image of the color component indicated by cIdx of the target area (e.g., CTU) whose upper left coordinate is indicated by (rx, ry).

[0142] The ALF unit 312 performs the following processes.

[0143] For each CTU position (rx, ry), where rx = 0..PicWidthInCtbsY-1, ry = 0..PicHeightInCtbsY-1, do the following, where PicWidthInCtbsY and PicHeightInCtbsY are the number of CTBs in the vertical and horizontal directions in the picture, respectively.

[0144] When alf_ctb_flag[0][rx][ry] is 1, recPictureL, alfPictureL, and the brightness control The filter performs CTB filtering on the luminance samples using the ctb position (xCtb, yCtb) of the component, and outputs alfPictureL.

[0145] where xCtb is rx< <CtbLog2SizeY、yCtbはry<<CtbLog2SizeYである。

[0146] If sps_chroma_format_idc is not 0 and alf_ctb_flag[1][rx][ry] is 1, CTB filtering is performed on the chrominance samples using recPicture, alfPicture, the chrominance CTB position (xCtbC, yCtbC), and the alternative chrominance filter index alf_ctb_filter_alt_idx[1][rx][ry], and alfPictureCb is output.

[0147] Here, recPicture is set to recPictureCb, and alfPicture is set to alfPictureCb. Also, xCtbC is (rx< <CtbLog2SizeY) / SubWidthC、yCtbCは(ry<<CtbLog2SizeY) / SubHeightC If sps_chroma_format_idc is not 0 and alf_ctb_flag[2][rx][ry] is 1, CTB filtering is performed on the chrominance samples using recPicture, alfPicture, the chrominance CTB position (xCtbC, yCtbC), and the filter index alf_ctb_filter_alt_idx[2][rx][ry], and alfPictureCr is output.

[0148] Here, recPicture is set to recPictureCr, and alfPicture is set to alfPictureCr.

[0149] If sps_chroma_format_idc is not 0, ccAlfPictureCb and ccAlfPictureCr are set to alfPictureCb and alfPictureCr, respectively.

[0150] At the CTB position (rx, ry) of each luminance component, perform the following processing, where rx = 0..PicWidthInCtbsY-1, ry = 0..PicHeightInCtbsY-1.

[0151] If sps_chroma_format_idc is not 0 and alf_ctb_cc_cb_idc[rx][ry] is not 0, the ccALF unit 313 performs filtering on the Cb, Cr pixel positions of the Y image and the surrounding pixels (e.g., 8 pixels) of the Y image corresponding to the same positions, and outputs ccAlfPictureCb.

[0152] If sps_chroma_format_idc is not 0 and alf_ctb_cc_cr_idc[rx][ry] is not 0, the ccALF unit 313 performs the above processing and outputs ccAlfPictureCr.

[0153] (ccALF part 313) The ccALF unit 313 takes as input the reconstructed luminance image recPictureL before ALF processing, the reconstructed chrominance image alfPictureC after ALF, the top left position of the chrominance CTB corresponding to the top left sample position of the current picture (xCtbC, yCtbC), the width of the CTB in the chrominance samples ccAlfWidth, the height of the CTB in the chrominance samples ccAlfHeight, and the ccAF filter coefficients CcAlfCoeff[j] (j=0..6), and outputs the reconstructed chrominance image ccalfPictureC after processing. In deriving the filtered ccAlfPicture[xCtbC+x][yCtbC+y], the reconstructed chrominance samples in each current chrominance block of alfPictureC[xCtbC+x][yCtbC+y] are derived as follows: Here, x = 0.. ccAlfWidth-1, y = 0.. ccAlfHeight-1.

[0154] The luminance position (xL, yL) corresponding to the current chrominance position (xCtbC+x, yCtbC+y) is calculated as follows:

[0155] xL=(xCtbC+x)*SubWidthC yL=(yCtbC+y)*SubHeightC clipLeftPos, clipRightPos, clipTopPos, clipBottomPos, clipTopLeftFlag and clipBot RightFlag is derived by invoking the ALF boundary derivation process with (xCtbC*SubWidthC, yCtbC*SubHeightC), (x*SubWidthC, y*SubHeightC) and vbOffset as inputs, where vbOffset is set to 4. clipLeftPos, clipRightPos, clipTopPos, clipBottomPos are the left, right, top, and bottom ALF reference region boundary positions, respectively. The ALF reference region boundary is a virtual boundary if it signals a virtual bounday boundary. Otherwise, the ALF reference region boundary is a CTB boundary if the respective boundary of the current coding tree block is a tile, slice, or subpicture boundary, and in-loop filtering operations across tile, slice, or subpicture boundaries are invalid. For a given CTU, the virtual boundary is a row that is four pixels above the top row of the CTU below in the case of a luminance component, and a row that is two pixels above the top row of the CTU below in the case of a chrominance component. clipTopLeftFlag is a binary flag that is set to 1 if the CTB containing the luminance position (xCtb, yCtb) is different from the CTB covering the luminance position (xCtb-CtbSizeY, yCtb-CtbSizeY), and set to 0 otherwise. clipBotRightFlag is a binary flag that is set to 1 if the CTB containing the luminance position (xCtb, yCtb) is different from the CTB covering the luminance position (xCtb+CtbSizeY, yCtb+CtbSizeY), and set to 0 otherwise. Here, CtbSizeY is the size of the luminance CTB.

[0156] The luminance position (h_{x+i,y+j},v_{y+j}) (i=-1..1,j=-1..2) in recPictureL is derived as follows:

[0157] h_{x+i,y+j}=Clip3(0,pps_pic_width_in_luma_samples-1,xL+i) v_{y+j}=Clip3(0,pps_pic_height_in_luma_samples-1,yL+j) (h_{x+i,y+j},v_{y+j}) is updated by performing ALF sample padding processing using (xCtbC*SubWidthC,yCtbC*SubHeightC),(hx+i,y+j,vy+j),isChroma,clipLeftPos,clipRightPos,clipTopPos,clipBottomPos,clipTopLeftFlag and clipBotRightFlag. Here, isChroma is a flag indicating whether the component to be processed is a chrominance component, and is set to 0 here.

[0158] applyAlfLineBufBoundary is a variable that determines the vertical sample position offset and is derived as follows:

[0159] If the bottom boundary of the current CTB is the bottom boundary of the current picture and pps_pic_height_in_luma_samples-yCtbC*SubHeightC<=-4 applyAlfLineBufBoundary=0 Otherwise applyAlfLineBufBoundary=1 The vertical sample position offsets yP1 and yP2 are calculated using the vertical luminance sample position (y*subHeightC) and applyAlfLineBufBoundary as follows:

[0160] If (y*subHeightC==CtbSizeY-5 || y*subHeightC==CtbSizeY-4) && applyAlfLineBufBoundary==1 yP1=0 yP2=0 ·(y*subHeightC==CtbSizeY-6 || y*subHeightC==CtbSizeY-3) && applyAlfLineBufBoundary==1 yP1=1 yP2=1 Otherwise yP1=1 yP2=2 The pixel value curr of the target pixel is calculated as follows:

[0161] curr=alfPictureC[xCtbC+x][yCtbC+y] The cross-component filter coefficient array f[j] (j=0..6) is derived as follows:

[0162] f[j] = CcAlfCoeff[j] ·sum is derived as follows:

[0163] sum = f[0]*(recPictureL[h_{x,y-yP1}][v_{y-yP1}]-recPictureL[h_{x,y}][v_{y}])+ f[1]*(recPictureL[h_{x-1,y}][v_{y}]-recPictureL[h_{x,y}][v_{y}])+ f[2]*(recPictureL[h_{x+1,y}][v_{y}]-recPictureL[h_{x,y}][v_{y}])+ f[3]*(recPictureL[h_{x-1,y+yP1}][v_{y+yP1}]-recPictureL[h_{x,y}][v_{y}])+ f[4]*(recPictureL[h_{x,y+yP1}][v_{y+yP1}]-recPictureL[h_{x,y}][v_{y}])+ f[5]*(recPictureL[h_{x+1,y+yP1}][v_{y+yP1}]-recPictureL[h_{x,y}][v_{y}])+ f[6]*(recPictureL[h_{x,y+yP2}][v_{y+yP2}]-recPictureL[h_{x,y}][v_{y}]) scaledSum = Clip3(-(1<<(BitDepth-1)),(1<<(BitDepth-1))-1,(sum+64)>>7) sum = (SubHeightC==1 && (y==CtbSizeY-3 || y==CtbSizeY-4)) ? curr : curr+scaledSum ccAlfPicture[xCtbC+x][yCtbC+y] is derived as follows:

[0164] ccAlfPicture[xCtbC+x][yCtbC+y] = Clip3(0,(1< <BitDepth)-1,sum) (bS derivation unit 331) The bS derivation unit 331 derives the edge degree edgeIdc and the maximum filter length maxFilterLength of the deblocking filter. edgeIdc indicates whether the input image resPicture before NN filter processing (before processing by the NN filter unit 601) has a partition division boundary, a prediction block boundary, or a transform block boundary. Furthermore, the strength bS of the deblocking filter is derived from edgeIdc, the transform block boundary, and the coding parameters. The coding parameters are, for example, the prediction mode CuPredMode, the BDPCM prediction mode intra_bdpcm_luma_flag, a flag indicating whether the IBC prediction mode is used, a motion vector, a reference picture, and flags tu_y_coded_flag and tu_u_coded_flag indicating whether a non-zero coefficient exists in the transform block. edgeIdc and bS may take values ​​of 0, 1, and 2, or may have other values. Specifically, the bS derivation unit 331 inputs (xCb, yCb), the width nCbW and height cCbH of the target block, edgeType, the color component cIdx of the target block, and a two-dimensional array edgeIdc of size nCbWxnCbH, and outputs a two-dimensional array bS of size nCbWxnCbH. (xCb, yCb) are coordinates based on the upper left coordinates of the target image. EdgeType is a variable that indicates whether to specify a vertical edge (EDGE_VER) or horizontal edge (EDGE_HOR) filter.

[0165] The bS derivation unit 331 derives the maximum filter length maxFilterLength of the deblocking filter according to the transform block size. When edgeType==EDGE_VER, if the width of the transform block is 4 or less, maxFilterLengthP=1 is set, if the width of the transform block is 32 or more, maxFilterLengthP=7 is set, and otherwise maxFilterLengthP=3 is set. When edgeType==EDGE_HOR, the same process is performed by replacing the above width with height.

[0166] The bS derivation unit 331 performs the following process by referring to edgeType: Calculates gridSize, scaleWidth, and scaleHeight as follows.

[0167] gridSize = cIdx == 0 ? 4 : 8 scaleWidth = (cIdx==0) ? 1 : SubWidthC scaleHeight = (cIdx==0) ? 1 : SubHeightC In addition, xN, yN, xDi (i=0..xN), and yDj (j=0..yN) are calculated as follows:

[0168] If edgeType is EDGE_VER, xN = Max(0,(nCbW / gridSize)-1) yN = cIdx == 0 ? (nCbH / 4)-1 : (nCbH / 2)-1 xDi = (i*gridSize) yDj = cIdx == 0 ? (j<<2) : (j<<1) If edgeType is EDGE_HOR, xN = cIdx == 0 ? (nCbW / 4)-1 : (nCbW / 2)-1 yN = Max(0,(nCbH / gridSize)-1) xDi = cIdx == 0 ? (i<<2) : (i<<1) yDj = j*gridSize For i=0..xN, j=0..yN, the following processing is performed. (If it is not a CU / TU / PU boundary, or if the boundary is specified as loop filter off in VirtualBoundary, bS==0) bS[xDi][yDj]=0 if any of the following conditions are met: edgeIdc[xDi][yDj]==0 edgeType==EDGE_VER, and VirtualBoundariesPresentFlag==1, and (xCb+xDi)==VirtualBoundaryPosX[n] / scaleWidth edgeType==EDGE_HOR, and VirtualBoundariesPresentFlag==1, and (yCb+yDi)==VirtualBoundaryPosY[n] / scaleheight If none of the above conditions are met, the following process is carried out: Two pixels p0 and q0 of the blocks on either side of the boundary are found as follows.

[0169] If edgeType==EDGE_VER, p0=recPicture[xCb+xDi-1][yCb+yDj] q0=recPicture[xCb+xDi][yCb+yDj] If edgeType==EDGE_HOR, p0=recPicture[xCb+xDi][yCb+yDj-1] q0=recPicture[xCb+xDi][yCb+yDj] bS[xDi][yDj] is calculated as follows: If cIdx==0 and both coding blocks including p0 and q0 have intra_bdpcm_luma_flag==1, or if cIdx>0 and both coding blocks including p0 and q0 have intra_bdpcm_chroma_flag==1, bS[xDi][yDj]=0 · If CuPredMode[cIdx==0?0:1][x0][y0]==MODE_INTRA, or if CuPredMode[cIdx==0?0:1][x1][y1]==MODE_INTRA, or if p0 or q0 is a coding block and both ciip_flag==1, then bS[xDi][yDj]=2 If the block boundary is also a transformation block boundary, bS[xDi][yDj]=1 if the following condition is met: ··cIdx==0 and tu_y_coded_flag[x0][y0]+tu_y_coded_flag[x1][y1]>0 where (x0,y0) and (x1,y1) are the luminance positions of the top left sample of the luminance transformation block containing samples p0 and q0, respectively. ··cIdx==1 and tu_cb_coded_flag[x0][y0]+tu_joint_cbcr_residual_flag[x0][y0]+tu_cb_coded_flag[x1][y1]+tu_joint_cbcr_residual_flag[x1][y1]>0 where (x0,y0) and (x1,y1) are the luminance positions corresponding to the top left sample of the Cb transform block containing chrominance samples p0 and q0, respectively. ··cIdx==2 and tu_cr_coded_flag[x0][y0]+tu_joint_cbcr_residual_flag[x0][y0]+tu_cr_coded_flag[x1][y1]+tu_joint_cbcr_residual_flag[x1][y1]>0 where (x0,y0), (x1,y1) are the luminance positions corresponding to the top left sample of the Cr transform block containing chrominance samples p0, q0, respectively. When cIdx==0 and edgeIdc[xDi][yDj]==2, if any of the following conditions is met, bS[xDi][yDj]=1 The prediction mode of the coding block containing sample p0 is different from the prediction mode of the coding block containing sample q0. The subblock containing sample p0 and the subblock containing sample q0 are both coded in IBC prediction mode, and the absolute value of the difference between the horizontal or vertical components of the motion vectors used to predict the two subblocks is equal to or greater than a predetermined threshold MVTH (e.g., MVTH=8 for luminance samples in 1 / 16 units). ··A different reference picture or a different number of motion vectors is used for predicting the subblock containing sample p0 than for predicting the subblock containing sample q0. The number of motion vectors used for predicting the subblock of the top-left sample (xSb,ySb) is equal to PredFlagL0[xSb][ySb]+PredFlagL1[xSb][ySb]. The reference pictures and motion vectors used for predicting the subblock containing the sample located at (xS,yS) refer to the values ​​of RefPicList[X][RefIdxLX[xS][yS]] and MvLX[xS][yS]. One motion vector is used to predict the subblock containing p0, and one motion vector is used to predict the subblock containing q0, and the absolute value of the difference between the horizontal and vertical components of the motion vectors is equal to or greater than MVTH. Two motion vectors and two reference pictures are used to predict the subblock containing p0, and two motion vectors of the same two reference pictures are used to predict the subblock containing q0, and the absolute value of the difference between the horizontal or vertical components of the two motion vectors used to predict the two subblocks of the same reference picture is equal to or greater than MVTH. If the subblock containing p0 is predicted using two motion vectors from the same two reference images, the subblock containing q0 is predicted using two motion vectors from the same two reference images, and the following condition is satisfied: The absolute value of the difference between the horizontal or vertical components of the motion vectors in list 0 used to predict two subblocks is equal to or greater than MVTH, or the absolute value of the difference between the horizontal or vertical components of the motion vectors in list 1 used to predict two subblocks is equal to or greater than MVTH. The absolute value of the difference between the horizontal or vertical component of the motion vector of list 0 used to predict the subblock containing p0 and the horizontal or vertical component of the motion vector of list 1 used to predict the subblock containing q0 is equal to or greater than MVTH, or the absolute value of the difference between the horizontal or vertical component of the motion vector of list 1 used to predict the subblock containing p0 and the horizontal or vertical component of the motion vector of list 0 used to predict the subblock containing q0 is equal to or greater than MVTH. Otherwise, bS[xDi][yDj]=0. The bS derivation unit 331 further derives an edge determination parameter dE used for switching the deblocking filter in the block edge determination unit. The bS derivation unit 331 derives Q from the already derived bS value and the parameter sh_luma_tc_offset_div2 transmitted in the slice header. sh_luma_tc_offset_div2 is the offset (divided by 2) of the deblocking parameter tC applied to the luminance component of the current slice. Q=Clip3(0,65,qP+2*(bS-1)+(sh_luma_tc_offset_div2<<1)) FIG. 30 is a derivation table of the parameters beta′ and tC′ based on the quantization parameters.

[0170] 30 for deriving tc from Q. Furthermore, the bS derivation unit 331 shifts tc' according to BitDepth to derive tc.

[0171] When BitDepth is less than 10 tC=(tC'+(1<<(9-BitDepth)))>>(10-BitDepth) Otherwise tC=tC'*(1<<(BitDepth-10)) The bS derivation unit 331 derives the following: dp0=abs(p[2,0]-2*p[1,0]+p[0,0]) dq0=abs(q[2,0]-2*q[1,0]+q[0,0]) dpq0=dp0+dq0 dp3=abs(p[2,3],0-2*p[1,3]+p[0,3]) dq3=abs(q[2,3]-2*q[1,3]+q[0,3]) dpq3=dp3+dq3 d=dq0+dpq3 where q[j][k] = edgeType == EDGE_VER? recPicture[xCb + xBl + j][yCb + yBl + k] : recPicture[xCb + xBl + k][yCb + yBl + j] p[j][k] = edgeType == EDGE_VER? recPicture[xCb + xBl - I - 1][yCb + yBl + k] : recPicture[xCb + xBl + k][yCb + yBl - l - 1] Furthermore, when dpq < sThr1, sp + pq < sThr2, and spq < (5 * tc + 1) >> 1, set dSam to 1; otherwise, set it to 0. Here, when the block size is large, sThr1 is derived by sThr1 = (3 * beta) >> 5 and sThr2 = beta >> 4; otherwise, sThr1 is derived by sThr1 = beta >> 3 and sThr2 = beta >> 2. For the case of (xCb + xBl, yCb + yBl), set dSam to dSam0.

[0172] When edgeType == EDGE_VER, and (xCb + xBl, yCb + yBl + 3), and edgeType == EDGE_HOR, and (xCb + xBl + 3, yCb + yBl), set dSam to dSam3. The bS derivation unit 331 sets dE = 3 when both dSam0 and dSam3 are 1. When d < beta in other cases, set dE = 1. In other cases, set dE = 0.

[0173] FIG. 12 is a diagram for explaining an example of a deblocking filter.

[0174] The DF unit 603 determines that block distortion exists when the difference in pixel values of adjacent pixels along the block (CTU / CU / TU) boundary of the input image is within a predetermined range. Then, by performing deblocking processing on the block boundary in the input image, smoothing of the image near the block boundary is performed. The image on which deblocking processing has been performed by the deblocking filter is the deblocked decoded image P_DB.

[0175] When the value of dE is other than 0 and other than 3, the DF unit 603 performs the following process as a short tap filter. The on / off determination of the deblocking filter is performed using the following equation.

[0176] abs(p20-2*p10+p00)+abs(p23-2*p13+p03)+abs(q20-2*q10+q00)+abs(q23-2*q13+q03)< beta (Equation DB-1) Here, p2k, p1k, p0k, q0k, q1k, q2k are columns or rows of pixels whose distances from the block boundary are 2, 1, 0, 0, 1, and 2, respectively. p2k, p1k, p0k are pixels contained in block P of blocks P and Q adjacent to the boundary, and q0k, q1k, q2k are pixels contained in block Q. k indicates the pixel number in the direction of the block boundary, and k>=0. beta is a threshold derived from the average value QPavg of the quantization parameters of blocks P and Q, and pps_beta_offset_div2 and slice_beta_offset_div2 notified in the PPS or slice header SH. If (Equation DB-1) is satisfied, the deblocking filter is turned on (implemented) for the boundary between blocks P and Q. In addition, filter processing is performed using the following equation.

[0177] p2'=Clip3(p2-2*tc,p2+2*tc,(2*p3+3*p2+p1+p0+q0+4)>>3) (Formula DB-2) p1'=Clip3(p1-2*tc,p1+2*tc,(p2+p1+p0+q0+2)>>2) p0'=Clip3(p0-2*tc,p0+2*tc,(p2+2*p1+2*p0+2*q0+q1+4)>>3) q0'=Clip3(q0-2*tc,q0+2*tc,(p1+2*p0+2*q0+2*q1+q2+4)>>3) q1'=Clip3(q1-2*tc,q1+2*tc,(p0+q0+q1+q2+2)>>2) q2'=Clip3(q2-2*tc,q2+2*tc,(p0+q0+q1+3*q2+2*q3+4)>>3) Since (Equation DB-2) is a common process for each k (k>=0) in FIG. 12, k is omitted. 2, p1, p0, q0, q1, q2, and q3 are pixel columns or rows whose distances from the block boundary are 3, 2, 1, 0, 0, 1, 2, and 3, respectively. tc is a variable for suppressing filtering, and is a threshold derived from the average value QPavg of the quantization parameters of blocks P and Q, and pps_tc_offset_div2, slice_tc_offset_div2, etc. notified in the PPS or slice header SH. It is derived using the following formula.

[0178] When the value of dE is 3, the DF unit 603 derives the intermediate pixel value refMiddle and pixel values ​​refP and refQ that depend on maxFilterLength as a long tap filter.

[0179] refMiddle=(p4+p3+2*(p2+p1+p0+q0+q0+q2)+q3+q4+8)>>4 refP=(pmaxFilterLengthP+pmaxFilterLengthP-1)>>1 refQ=(qmaxFilterLengthQ+pmaxFilterLengthQ-1)>>1 The DF unit 603 derives a filtered pixel value using the target pixel pi, refMiddle, and refP (qi, refMiddle, refQ). pi'=Clip3(pi-(tC*tCPDi>>1),pi+(tC*tCPDi>>1),(refMiddle*fi+refP*(64-fi)+32)>>6) qj'=Clip3(qj-(tC*tCQDj>>1),qj+(tC*tCQDj>>1),(refMiddle*gj+refQ*(64-gj)+32)>>6) Here, tCPDi and tCQDj are predetermined values ​​determined by maxFilerLengthP and maxFilterLengthQ, respectively.

[0180] (Example of the configuration of the NN filter unit 611) 13 is a diagram showing the configuration of the neural network filter unit (NN filter unit 611) in the loop filter 305. The NN filter unit 611 is a means for performing filtering processing on an input image using a neural network model. At the same time, reduction / enlargement may be performed at the same magnification or a rational number magnification.

[0181] Here, the neural network model (hereafter referred to as NN model) refers to the elements and connections (topology) of a neural network, and the parameters (weights, biases) of the neural network. Note that the topology may be fixed, and only the parameters of the neural network model may be switched.

[0182] The loop filter 305 of the video decoding device (loop filter 107 of the video encoding device) includes an NN filter unit 611. The NN filter unit 611 filters an image in the reference picture memory 306 / 106 and stores the image in the reference picture memory 306 / 106. As already described, the loop filter may include a DF, ALF, SAO, bilateral filter, or the like.

[0183] (Details of the NN filter unit 611) The NN filter uses the input image inSamples and input parameters (e.g., QP, bS, etc.) to perform filtering using a neural network model. The input image may be an image for each component, or may be an image in which images of multiple components are used as different channels. Also, the input parameters may be used as channels different from the images.

[0184] The neural network may iteratively apply the following process.

[0185] As shown in the following formula, inSamples is convolved with kernel k[m][i][j] and bias is added to derive output image outSamples, where nn=0..n-1, xx=0..width-1, yy=0..height-1.

[0186] outSamples[nn][xx][yy]=ΣΣΣ(k[mm][i][j]*inSamples[mm][xx+i-of][yy+j-of]+bias[nn]) For 1x1 Conv, Σ represents the sum of mm=0..m-1, i=0, j=0. In this case, of=0 is set. For 3x3 Conv, Σ represents the sum of mm=0..m-1, i=0..2, j=0..2. In this case, of=1 is set. n is the number of channels in outSamples, m is the number of channels in inSamples, width is the width of inSamples and outSamples, and height is the height of inSamples and outSamples. of is the size of the padding placed around inSamples to make the sizes of inSamples and outSamples the same. In the following, when the output of the NN filter is a correction value, it is indicated as corrNN instead of outSamples.

[0187] Alternatively, a process called Depth wise Conv may be performed, which is expressed by the following formula: Here, nn=0..n-1, xx=0..width-1, and yy=0..height-1.

[0188] outSamples[nn][xx][yy]=ΣΣ(k[nn][i][j]*inSamples[nn][xx+i-of][yy+j-of]+bias[nn]) Σ represents the sum for i and j, respectively. n is the number of channels in outSamples and inSamples, width is the width of inSamples and outSamples, and height is the height of inSamples and outSamples.

[0189] Also, a nonlinear process called Activate, for example, ReLU, may be used. ReLU(x) = x >= 0 ? x : 0 Alternatively, leakyReLU shown in the following formula may be used. leakyReLU(x) = x >= 0 ? x : a * x Here, a is a predetermined value, for example, 0.1 or 0.125. In order to perform integer arithmetic, all the values ​​of k, bias, and a above may be integers, and a right shift may be performed after Conv.

[0190] With ReLU, 0 is always output for values ​​less than 0, and the input value is output as is for values ​​greater than or equal to 0. On the other hand, with leakyReLU, linear processing is performed for values ​​less than 0 with the gradient set by a. With ReLU, the gradient for values ​​less than 0 disappears, which can make it difficult for learning to progress. With leakyReLU, the gradient for values ​​less than 0 is retained, making the above problem less likely to occur. Of the above leakyReLU(x), PReLU, which uses a parameterized value of a, may also be used.

[0191] (Exclusive processing between ccALF and ccNN filters) FIG. 14 is a diagram showing the exclusive processing between the ccALF unit 313 and the ccNN filter unit 611b in the loop filter 305. In FIG.

[0192] The CT information decoding unit 3201 decodes from the encoded data a flag alf_ctb_cc_cb_flag indicating whether to apply ccALF to the Cb image of the target region (e.g., CTB), a flag alf_ctb_cc_cr_flag indicating whether to apply ccALF to the Cr image of the target region, and a parameter nn_ctb_scale_idx[cIdx] indicating whether to apply a ccNN filter to the image of the cIdx component of the target region.

[0193] The loop filter 305 of this configuration includes an ALF unit 312 , a ccALF unit 313 , an NN filter unit 611 a , a ccNN filter unit 611 b , and an exclusive switch unit 314 .

[0194] The ALF unit 312 includes Y_ALF, Cb_ALF, and Cr_ALF filters, and each filter performs processing using a Y image (Y_ALF), a Cb image (Cb_ALF), and a Cr image (Cr_ALF), and outputs the processed image.

[0195] The ccNN filter unit 611b uses a neural network model to derive a correction value corr that corrects errors using correlation between color components. Here, the correction value corr is derived by inputting the Y component other than the output Cb and Cr, so it is called a cross-component filter. The input image of the ccNN filter unit 611b may be the Y component image recSamples below.

[0196] inSamples[0][x][y] = recSamples[0][x][y] Furthermore, images qpMap and bS of QP may be input.

[0197] The ccNN filter unit 611b derives the color difference scale value NNScaleVal from nn_ctb_scale_idx. The correspondence between nn_ctb_scale_idx and the color difference scale value NNScaleVal is determined in advance. For example, it is derived as follows. NNScaleVal = 0 (nn_ctb_scale_idx == 0) NNScaleVal = 1 (nn_ctb_scale_idx == 1) NNScaleVal = 2 (nn_ctb_scale_idx == 2) Alternatively, for example, it may be derived as follows using a table NNScaleTbl that holds an array of NNScaleVal with nn_ctb_scale_idx as an index. NNScaleVal = NNScaleTbl[nn_ctb_scale_idx] If nn_ctb_scale_idx is 0, no processing is performed in the ccNN filter section.

[0198] The ccNN filter unit 611b may obtain chrominance decoded image samples recSamples by scaling corrNN (corrRescale) using NNScaleVal and then adding it to the input image of the ccNN filter unit 611 (eg, the post-ALF image resSamplesALF).

[0199] corrRescale[x][y] = (corrNN[x][y]*NNScaleVal + (1<<(shift-1))>>shift recSamples[x][y] = resSamplesAlf[x][y]+corrRescale[x][y] shift=1 is also acceptable.

[0200] In addition, in a configuration that uses a binary nn_ctb_flag instead of a ternary or more value nn_ctb_scale_idx, the following processing is performed.

[0201] recSamples[x][y] = nn_ctb_flag[cIdx]==0 ? recSamples[cIdx][x][y]+corrNN[x][y] : recSamples[0][x][y] Or the following would be fine: if nn_ctb_flag[cIdx]==0 then corrNN[x][y] = 0 recSamples[x][y] = recSamples[cIdx][x][y] + corrNN[x][y] The ccNN filter unit 611b outputs the result of processing the Y image using a neural network model. After that, the output of the ccNN filter unit 611b is added to each of the Cb and Cr ALF processed images.

[0202] When the exclusive switching unit 314 applies the ccNN filter (nn_ctb_scale_idx is 1 or more), it does not apply ccALF (alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag are set to 0), and applies the ccNN filter unit 611b to the input image. Conversely, when the exclusive switching unit 314 applies ccALF (alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag are 1), it does not apply ccNN (nn_ctb_scale_idx is set to 0), and applies the ccALF unit 313 to the input image.

[0203] The exclusive switching unit 314 may be realized as the following operation by the CT information decoding unit 3201. If ccALF indicates application (alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag are 1), nn_ctb_scale_idx[1] and nn_ctb_scale_idx[2] are not decoded from the encoded data, and nn_ctb_scale_idx[1] and nn_ctb_scale_idx[2] are set to 0. If they do not appear in the encoded data, nn_ctb_scale_idx[1] and nn_ctb_scale_idx[2] are set to 0. If alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag are 0, the ccNN indexes nn_ctb_scale_idx[1] and nn_ctb_scale_idx[2] are decoded from the encoded data.

[0204] The exclusive switching unit 314 may be realized as the following operation by the CT information decoding unit 3201. Indicates that the ccNN filter is applied (nn_ctb_scale_idx[1], nn_ctb_scale_idx[2] are 1). If nn_ctb_scale_idx[1] and nn_ctb_scale_idx[2] are 0, alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag are decoded from the encoded data, and alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag are set to 0. If they do not appear in the encoded data, alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag are set to 0. If nn_ctb_scale_idx[1] and nn_ctb_scale_idx[2] are 0, alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag are decoded from the encoded data.

[0205] In the above, for the ccNN flag nn_ctb_flag and the ccALF flags alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag indicating the application of ccALF to Cb and Cr, exclusive processing is achieved by decodeing only either nn_ctb_flag or alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag, and if one is 1 or greater, the other is set to 0. Here, nn_ctb_flag is a flag that controls the application of ccNN and has two values, 0 and 1. The nn_ctb_flag header decoder 3020 decodes it.

[0206] The above configuration has the advantage that it is possible to select only the more effective filter out of the ccALF and ccNN filters for each region and perform processing. Since they are exclusive, the amount of processing can be reduced.

[0207] (Exclusive processing between ccALF and NN filter) FIG. 15 is a diagram showing a process in which the loop filter 305 uses an NN filter for color difference.

[0208] The NN filter unit 611c receives at least the Cb and Cr images, performs processing using the correlation between the components, and outputs the Cb and Cr images. The input image inSamples of the NN filter unit 611c may be the following color component image recSamples, and the output may be corrNN.

[0209] inSamples[0][x][y] = recSamples[1][x][y] inSamples[1][x][y] = recSamples[2][x][y] inSamples[2][x][y] = bS[x][y] The ALF unit 312 performs filtering on the output image from the NN filter unit 611a for the Y component, and performs filtering on the output image from the color difference NN filter unit 611c for the Cb and Cr components.

[0210] The ALF unit 312 applies ALF processing to an input image having a color component cIdx where alf_ctb_flag[cIdx] is other than 0.

[0211] The ccALF selection unit 314 determines whether to add the output of the ccALF unit 313 to the ALF output image of Cb, Cr based on the values ​​of alf_ctb_cc_cb_flag, alf_ctb_cc_cr_flag, and nn_ctb_scale_idx. When the value of nn_ctb_scale_idx is 1 or more, the ccALF selection unit 314 sets the values ​​of alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag to 0 and does not add the output from the ccALF unit 313. When the values ​​of alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag are 1, the ccALF selection unit 314 sets the value of nn_ctb_scale_idx to 0 and adds the output from the ccALF unit 313.

[0212] FIG. 16 is a diagram showing a process performed by the NN filter unit 611d in the loop filter 305 using each of the Y, Cb, and Cr components of an input image.

[0213] The NN filter unit 611d uses the input image and outputs an image that has been processed using a neural network model using each component.

[0214] The NN filter unit 611d outputs the results of processing the input image resSamples containing Y, Cb, and Cr using a neural network model as the correction value corrNN[cIdx], and outputs nn_ctb_s Scaling is performed using cale_idx[cIdx] and added to the Cb or Cr input image.

[0215] NNScaleVal = NNScaleTbl[nn_ctb_scale_idx[cIdx]] corrRescale[x][y] = (corrNN[x][y]*NNScaleVal + (1<<(shift-1))>>shift recSamples[cIdx][x][y] = resSamples[cIdx][x][y]+corrRescale[x][y] shift=1, cIdx=0..2 are also acceptable.

[0216] The input image inSamples of the NN filter unit 611e may be the following color component image recSamples[cIdx], and the output may be corrNN[cIdx]. cIdx=0..2 inSamples[0][x][y] = recSamples[0][x][y] inSamples[1][x][y] = recSamples[1][x][y] inSamples[2][x][y] = recSamples[2][x][y] Furthermore, images qpMap and bS of QP may be input.

[0217] inSamples[3][x][y] = qpMap[x][y] inSamples[4][x][y] = bS[x][y] The ALF unit 312 outputs an image that has been subjected to the ALF process using the image processed by the NN filter unit 611d for each component.

[0218] The ccALF selection unit 314 determines whether or not to add the output from the ccALF unit 313 to the ALF output image of Cb, Cr, based on the values ​​of alf_ctb_cc_cb_flag, alf_ctb_cc_cr_flag, and nn_ctb_scale_idx decoded by the CT information decoding unit 3201. When the value of nn_ctb_scale_idx is 1 or greater, the ccALF selection unit 314 sets the values ​​of alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag to 0 and does not add the output from the ccALF unit 313; when the values ​​of alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag are 1, the ccALF selection unit 314 sets the value of nn_ctb_scale_idx to 0 and adds the output from the ccALF unit 313.

[0219] <Another configuration example 1> FIG. 17 is a diagram showing syntax for exclusively switching the flags of ccALF and the NN filter in a target picture (target slice).

[0220] The header decoding unit 3020 decodes, from the coded data, a flag sh_nn_enabled_flag indicating whether an NN filter is applied to a target picture (target slice) when a flag (NN flag) sps_nn_enabled_flag indicating whether an NN filter is available is 1 in SYN0001. When sh_nn_enabled_flag is 1 and a chrominance component exists in the input image, the header decoding unit 3020 decodes a flag sh_nn_chroma_enabled_flag indicating whether an NN filter is applied to a chrominance image (Cb, Cr image) of the target picture (target slice).

[0221] In SYN0003, if sh_nn_chroma_enabled_flag is 0 and the flag sps_ccalf_enabled_flag indicating whether ccALF is available is 1 (available), the header decoding unit 3020 decodes flags sh_alf_cc_cb_enabled_flag and sh_alf_cc_cr_enabled_flag indicating whether ccALF is applied to the target picture (target slice). Otherwise (if sh_alf_cc_cb_enabled_flag does not appear), the header decoding unit 3020 derives 0 (not applied) for each of sh_alf_cc_cb_enabled_flag and sh_alf_cc_cr_enabled_flag (sh_alf_cc_cr_enabled_flag).

[0222] If sh_alf_cc_cb_enabled_flag and sh_alf_cc_cr_enabled_flag are each 1, the header decoding unit 3020 further decodes the indexes sh_alf_cc_cb_aps_id and sh_alf_cc_cr_aps_id indicating the ccALF filter coefficients for the Cb image and the Cr image, respectively.

[0223] <Another configuration example 2> FIG. 18 is a diagram showing the syntax for exclusively switching between the ccALF and the NN filter in the target region (here, the CTU).

[0224] (SYN0011) If sh_nn_enabled_flag decoded from the encoded data (e.g., slice header) is 1, nn_ctb_flag[0] is decoded, and if nn_ctb_flag[0] is other than 0, index nn_luma_fixed_filter_idx indicating the neural network model to be used for the luminance NN filter is decoded. nn_luma_fixed_filter_idx is used by the NN model selection unit 323. The NN model selection unit 323 switches the neural network model (neural network parameters) to be used in the NN filter unit 611 to the parameters specified by nn_luma_fixed_filter_idx.

[0225] (SYN0012) When sh_nn_chroma_enabled_flag decoded by the header decoding unit 3020 is 1, the CT information decoding unit 3021 decodes the scale indexes nn_ctb_scale_idx[1] and nn_ctb_scale_idx[2] indicating the degree of application of the NN filter to the Cb image and Cr image in the target region (CtbAddrX, CtbAddrY). nn_ctb_scale_idx is a ternary or greater parameter having values ​​0, 1, .., N-1, and scales the chrominance pixel values ​​with a scale corresponding to each index value. When the index value is 0, no processing is performed using the NN filter.

[0226] (SYN0013) When an NN filter is applied to the target area (when nn_ctb_scale_idx[1] and nn_ctb_scale_idx[2] are values ​​indicating application and are greater than 0), the CT information decoding unit 3021 decodes the indices nn_cb_fixed_filter_idx and nn_cr_fixed_filter_idx indicating the NN model to be used for the Cb and Cr NN filters, respectively.

[0227] (SYN0014A) The CT information decoding unit 3021 applies ccALF to the target picture / slice but does not apply NN to the Cb image of the target region (sh_alf_cc_cb_enabled_flag is 1 and nn_ctb_scale_idx[1] is 0), the CT information decoding unit 3021 decodes the flag alf_ctb_cc_cb_flag indicating whether to apply ccALF to the Cb image of the target region. Otherwise (alf_ctb_cc_cb_flag does not appear), the CT information decoding unit 3021 estimates 0 (not applied) to alf_ctb_cc_cb_flag.

[0228] (SYN0015A) If alf_ctb_cc_cb_flag is 1 (a value indicating application), the CT information decoding unit 3021 decodes the filter index alf_ctb_cc_cb_idc of ccALF in Cb.

[0229] (SYN0014B) The CT information decoding unit 3021 decodes the flag alf_ctb_cc_cr_flag indicating whether to apply ccALF to the Cr image of the target region when sh_alf_cc_cb_enabled_flag is 1 and nn_ctb_scale_idx[1] is 0. In other cases (when alf_ctb_cc_cr_flag does not appear), the CT information decoding unit 3021 infers 0 (not applied) to alf_ctb_cc_cr_flag.

[0230] (SYN0015B) If alf_ctb_cc_cr_flag is 1, the CT information decoding unit 3021 decodes the filter index alf_ctb_cc_cr_idc of the ccALF of Cr.

[0231] <Another configuration example 3> Fig. 19 is a diagram showing syntax for exclusively switching between ccALF and NN filter in a target region (e.g., CTU). The NN flag nn_ctb_flag[0] indicating whether to apply an NN filter to the Y image of the target region is decoded. (SYN0022) When sh_nn_chroma_enabled_flag decoded by the header decoding unit 3020 is 1, the CT information decoding unit 3021 decodes the NN flags nn_ctb_flag[1] and nn_ctb_flag[2] indicating whether to apply an NN filter to the Cb and Cr images of the target region. nn_ctb_flag is a two-valued flag, and when the value is 1, an NN filter is performed at (CtbAddrX, CtbAddrY), and when the value is 0, NN filtering is not performed.

[0232] (SYN0024) The CT information decoding unit 3021 has detected that sh_alf_cc_cb_enabled_flag is 1 and N If the N flag nn_ctb_flag[1] is 0, decode the flag alf_ctb_cc_cb_flag indicating whether to apply ccALF to the Cb image of the target region. Otherwise (if alf_ctb_cc_cb_flag does not appear), infer alf_ctb_cc_cb_flag to be 0 (not applied).

[0233] (SYN0025) If alf_ctb_cc_cb_flag is 1, the CT information decoding unit 3021 decodes the filter index alf_ctb_cc_cb_idc of ccALF in Cb.

[0234] (SYN0024) The CT information decoding unit 3021 decodes the flag alf_ctb_cc_cr_flag indicating whether to apply ccALF to the Cr image of the target region if sh_alf_cc_cr_enabled_flag is 1 and the NN flag nn_ctb_scale_idx[2] of Cr is 0. In other cases (if alf_ctb_cc_cr_flag does not appear), the CT information decoding unit 3021 infers 0 (not applied) to alf_ctb_cc_cr_flag.

[0235] (SYN0025) If alf_ctb_cc_cr_flag is 1, the CT information decoding unit 3021 decodes the filter index alf_ctb_cc_cr_idc of the ccALF of Cr.

[0236] (Exclusive switching between ALF and NN filters) FIG. 20 is a diagram showing exclusive switching processing between the ALF unit 312 and the NN filter unit 611e in a target region (eg, CTU).

[0237] The loop filter 305 of this configuration has an ALF unit 312, a ccALF unit 313, an NN filter unit 611e, and an exclusive switching unit 314, and performs processing using unprocessed images for each of the Y, Cb, and Cr color components, alf_ctb_flag, and nn_ctb_scale_idx, and outputs processed images for each of the Y, Cb, and Cr color components.

[0238] The ALF section 312 and the ccALF section 313 are similar to those described above, and therefore their explanation will be omitted.

[0239] The NN filter unit 611e outputs the results of processing the Cb (cIdx=1) or Cr (cIdx=2) input image resSamples using a neural network model as a correction value corrNN, performs scaling using nn_ctb_scale_idx[cIdx], and adds it to the Cb or Cr input image. The NN filter unit 611e includes a Cb_NN filter and a Cr_NN filter, and the Cb_NN filter outputs an image processed using the Cb image, and the Cr_NN filter outputs an image processed using the Cr image.

[0240] NNScaleVal = NNScaleTbl[nn_ctb_scale_idx[cIdx]] corrRescale[x][y] = (corrNN[Idx][x][y]*NNScaleVal + (1<<(shift-1))>>shift recSamples[cIdx][x][y] = resSamples[cIdx][x][y]+corrRescale[x][y] shift=1, cIdx=1, 2 is also acceptable.

[0241] The input image inSamples of the NN filter unit 611e may be the following color component image recSamples, and the output may be corrNN.

[0242] inSamples[0][x][y] = recSamples[1][x][y] or inSamples[0][x][y] = recSamples[2][x][y] Using the input image and the values ​​of alf_ctb_flag and nn_ctb_scale_idx, the exclusive switching unit 314 applies the NN filter unit 611e to the input image if nn_ctb_scale_idx is 1 or greater, and otherwise (nn_ctb_scale_idx==0) and alf_ctb_flag is 1, applies the ALF unit 312 to the input image.

[0243] In addition, when alf_ctb_flag is 1, the exclusive switching unit 314 applies the ALF unit 312 to the input image, and In the case other than (alf_ctb_flag == 0) and nn_ctb_scale_idx is 1 or more, the NN filter unit 611e is applied to the input image.

[0244] FIG. 21 is a diagram showing syntax for exclusively switching the chroma ALF and NN filter flags in a slice header.

[0245] In SYN0031, the header decoding unit 3020 decodes sh_nn_enabled_flag when the flag sps_nn_enabled_flag indicating whether to apply the NN filter is 1. When sh_nn_enabled_flag is 1, nn_ctb_scale_idx[0] is decoded.

[0246] In SYN0032, if sh_nn_enabled_flag is 1 and a chrominance component exists in the input image, the header decoding unit 3020 decodes a flag sh_nn_chroma_enabled_flag indicating whether to apply an NN filter for chrominance.

[0247] In SYN0033, if sh_nn_chroma_enabled_flag is 0 and a chrominance component exists in the input image, the header decoding unit 3020 decodes flags sh_alf_cb_enabled_flag and sh_alf_cr_enabled_flag indicating whether to apply ALF to the Cb and Cr components, respectively.

[0248] FIG. 22 is a diagram showing syntax for exclusively switching between alf_ctb_flag and nn_ctb_scaleidx in a CTU. When sh_nn_enabled_flag is 1, the CT information decoding unit 3021 decodes nn_ctb_scale_idx[0].

[0249] In SYN0041, if sh_nn_enabled_flag decoded by the header decoding unit 3020 is 1 and nn_ctb_scale_idx[0] is other than 0, the CT information decoding unit 3021 decodes the NN model index nn_luma_fixed_filter_idx of luminance.

[0250] In SYN0042, if sh_nn_chroma_enabled_flag decoded by the header decoding unit 302 is 1, the CT information decoding unit 3021 decodes the Cb and Cr scale indexes nn_ctb_scale_idx[1] and nn_ctb_scale_idx[2] at the CTU position (CtbAddrX, CtbAddrY). nn_ctb_scale_idx is a ternary or greater parameter with values ​​0, 1, .., N-1, and scales the chrominance pixel values ​​with a scale corresponding to each index value. If the index value is 0, no processing is performed using the NN filter.

[0251] In SYN0043, if nn_ctb_scale_idx[1] and nn_ctb_scale_idx[2] are each greater than 0, The indices nn_cb_fixed_filter_idx and nn_cr_fixed_filter_idx of the NN filters for Cb and Cr are decoded, respectively.

[0252] In SYN0044, if sh_alf_cb_enabled_flag decoded by the header decoding unit 3020 is 1 and nn_ctb_scale_idx[1] is 0, the CT information decoding unit 3021 decodes the ALF flag alf_ctb_flag[1] of Cb at the position (CtbAddrX, CtbAddrY) of the CTU. Otherwise (if nn_ctb_scale_idx[1] is other than 0), alf_ctb_flag[1] is not decoded but set to 0. In other words, if alf_ctb_flag[1] does not appear, it is assumed to be 0.

[0253] In SYN0045, if alf_ctb_flag[1] is 1, the CT information decoding unit 3021 decodes the ALF filter index alf_ctb_filter_alt_idx[1] of Cb.

[0254] In SYN0044, the CT information decoder 3021 decodes the sh_alf_cr_ena If bled_flag is 1, the ALF flag alf_ctb_flag[2] of Cr at the CTU position (CtbAddrX, CtbAddrY) is decoded. Otherwise (if nn_ctb_scale_idx[2] is not 0), alf_ctb_flag[2] is not decoded and is set to 0. In other words, if alf_ctb_flag[2] does not appear, it is assumed to be 0.

[0255] In SYN0045, if alf_ctb_flag[2] is 1, the CT information decoding unit 3021 decodes the ALF filter index alf_ctb_filter_alt_idx[2] of Cr.

[0256] FIG. 23 is a diagram showing syntax for exclusively switching the ALF flag and the NN flag in the CTU.

[0257] The difference from the syntax in FIG. 22 is that nn_ctb_flag is decoded instead of nn_ctb_scale_idx. If sh_nn_enabled_flag is 1, the CT information decoding unit 3021 decodes nn_ctb_flag[0].

[0258] In SYN0052, the CT information decoding unit 3021 decodes the NN flags nn_ctb_flag[1] and nn_ctb_flag[2] of Cb and Cr at the position (CtbAddrX, CtbAddrY) of the CTU if the sh_nn_chroma_enabled_flag decoded by the header decoding unit 3020 is 1. nn_ctb_flag is a binary flag having values ​​of 0 and 1. If the value is 1, NN filtering is performed at (CtbAddrX, CtbAddrY), and if the value is 0, NN filtering is not performed.

[0259] In SYN0054, the CT information decoding unit 3021 decodes alf_ctb_flag[1], which is the ALF flag of Cb at the position (CtbAddrX, CtbAddrY) of the CTU, if the sh_alf_cb_enabled_flag decoded by the header decoding unit 3020 is 1 and nn_ctb_flag[1] is 0. Otherwise, if alf_ctb_flag[1] does not appear, it is assumed to be 0.

[0260] In SYN0055, if alf_ctb_flag[1] is 1, the CT information decoding unit 3021 decodes the filter index alf_ctb_filter_alt_idx[1] of the ALF of Cb.

[0261] In SYN0054, the CT information decoding unit 3021 decodes alf_ctb_flag[2], which is the ALF flag of Cr at the position (CtbAddrX, CtbAddrY) of the CTU, if sh_alf_cr_enabled_flag decoded by the header decoding unit 3020 is 1 and nn_ctb_flag[2] is 0. Otherwise, if alf_ctb_flag[2] does not appear, it is assumed to be 0.

[0262] In SYN0055, if alf_ctb_flag[2] is 1, the CT information decoding unit 3021 decodes the filter index alf_ctb_filter_alt_idx[2] of the ALF of Cr.

[0263] (NN model switching with different models and layers) Switching the NN model used in the NN filter unit according to the sequence, picture, or local features of the picture allows effective filtering with a relatively small NN model, but switching the NN model requires memory transfer, and it is difficult to switch the entire large-scale NN model. Below, a configuration is described in which the entire NN model or many layers are switched at a relatively large first level (granularity, range, image region, encoded data region), and a part of the NN model (some layers) is switched at a smaller second level. Below, an example is described in which the first level is the slice level and the second level is the CTU level, but this is not limiting. The first level may be a sequence, a picture, or a sub-picture, and the second level may be a CT or CU that is a further division of the CTU.

[0264] FIG. 25 is a diagram illustrating an example of a syntax table of the NN model index in the slice header.

[0265] When the flag sps_nn_enabled_flag, which is decoded by the header decoding unit 3020 and indicates whether to apply the NN filter, is 1 in SYN0101, the header decoding unit 3020 decodes sh_nn_enabled_flag.

[0266] In SYN0102, if sh_nn_enabled_flag is 1, the header decoding unit 3020 decodes the index sh_nn_luma_fixed_filter_idx of the luminance model selected by the NN model selection unit 323.

[0267] Furthermore, in SYN0103, if a chrominance component exists in the input image, the header decoding unit 3020 decodes a flag sh_nn_chroma_enabled_flag indicating whether to apply an NN filter for chrominance. In SYN0104, if sh_nn_chroma_enabled_flag is 1, the header decoding unit 3020 decodes the indices sh_nn_cb_fixed_filter_idx and sh_nn_cr_fixed_filter_idx of the chrominance models selected by the NN model selection unit 323.

[0268] FIG. 26 is a diagram illustrating an example of a syntax table of the NN partial model index in the CTU.

[0269] When the flag sh_nn_enabled_flag, which is decoded in the slice header and indicates whether the NN filter is available, is 1 in SYN0111, the CT information decoding unit 3021 decodes the index nn_part_filter_idx[0] of the partial model of the luminance model at the position (CtbAddrX, CtbAddrY) of the CTU.

[0270] Also, in SYN0112, when the flag sh_nn_chroma_enabled_flag, which is decoded by the header decoding unit 3020 and indicates whether the chrominance NN filter is usable, is 1, the CT information decoding unit 3021 decodes the indices nn_part_filter_idx[1] and nn_part_filter_idx[2] of the partial models of the Cb and Cr models at the CTU position (CtbAddrX, CtbAddrY).

[0271] FIG. 24 is a diagram showing NN model switching in a first region (eg, a sequence, a group of pictures, a picture, a slice) and partial model switching in a second region (eg, a CTU) smaller than the first region.

[0272] The NN filter unit 611 includes an NN model processing unit 6111 , an NN partial model selection unit 322 , and an NN model selection unit 323 .

[0273] The NN model processing unit 6111 performs filtering on the input image using a specified neural network model. The header decoding unit 3020 decodes the NN model index nn_luma_fixed_filter_idx for the first region (eg, picture, slice) from the coded data (eg, picture header or slice header).

[0274] The NN model selection unit 323 selects one NN model from among multiple NN model candidates prepared in advance based on the NN model index. The NN filter unit 611 performs filtering using the selected NN model. Note that the switching may be performed on both the topology and parameters (weights, biases) of the neural network, or only on the parameters of the neural network.

[0275] For example, the NN model selection unit 323 may derive parameters of a neural network.

[0276] NNModel[i]=NNFixedModel[nn_luma_fixed_filter_idx][i] Here, NNModel is the parameter of the neural network used in the NN filter section, NNFixedModel is the parameter of the neural network specified by nn_luma_fixed_filter_idx, and i = 0..number of parameters-1.

[0277] nn_luma_fixed_filter_idx is a non-negative integer value in the range 0..N-1 (where N is the total number of model candidates), and each number is associated with a corresponding individual NN model.

[0278] The CT information decoding unit 3021 decodes the NN partial model index nn_luma_part_filter_idx in the CTU. The NN partial model selection unit 322 replaces some (some layers) of parameters (hereinafter, partial model) of the NN model based on the NN partial model index. For example, only a specific part of parameters is re-learned (finetuned) according to a specific genre, quantization parameter, or bit rate, and the specific partial parameters are stored in the video decoding device and the video encoding device as selectable candidate NNSubFixedModel. The NN partial model selection unit 322 derives (replaces) a specific value of a parameter of the NNModel using the NNSubFixedModel selected by nn_luma_part_filter_idx.

[0279] NNModel[j] = NNSubFixedModel[nn_luma_part_filter_idx][j] Here, j = nnStartIdx..nnEndIdx. Here, nnStartIdx and nnEndIdx indicate the start and end positions of the parameters to be switched among the parameters of NNModel. However, nnStartIdx>=0, nnEndIdx<=number of parameters-1.

[0280] Furthermore, the range of parameters of the NN model to be replaced may be the input layer, intermediate layer, or output layer.

[0281] The NN submodel index is a non-negative integer value in the range 0..M-1 (where M is the total number of layer candidates), and each number is associated with a corresponding individual submodel candidate.

[0282] This process enables the load time of small blocks (second region) of neural network parameters related to parameter switching to be reduced by adaptively switching models in small block units without switching the entire network parameters.

[0283] (A configuration that allows the block strength coefficient to be set separately for the NN filter and DF) By using information bS related to block boundaries (bS is, for example, a value derived from CU / TU / PU boundaries, TU residuals, PU mode, the PU image itself, pixel values ​​of block boundaries, and motion vectors) in the NN filter, the degree of application of the NN filter can be changed according to the bS value. In DF, the degree of application of DF is also controlled according to the bS value.

[0284] In the following, an example will be described in which the parameters (beta, tC) that control the derivation of bS are decoded using the encoded data, and the degree to which NN and DF are applied is further controlled by the encoder. In this embodiment, separate tC offsets and tC offsets are transmitted by NN and DF, and are decoded individually by the header decoder 3020.

[0285] FIG. 27 is a diagram showing a process of deriving bS using the bS calculation parameters used in the NN filter and DF.

[0286] The bS derivation unit 331a calculates bS for NN and bS for DF using the parameters beta and tC decoded from the encoded data, and inputs them to the NN filter unit 611f and the DF unit 332, respectively.

[0287] The parameter beta is a threshold of the filtering application range from the block boundary in the bS derivation. The parameter tC is a maximum change amount after filtering for a pixel value in the bS derivation, and is used to clip the pixel value after filtering.

[0288] beta,tC increases with QP, and the filtering effect is stronger for high QP values ​​compared to low QP values.

[0289] The bS derivation unit 331a derives beta1 and tC1 using the parameters sh_nn_luma_beta_offset_div2 and sh_nn_luma_tc2_offset_div2 for the NN filter as the slice_luma_beta_offset_div2 and slice_luma_tc2_offset_div2 described above, and derives bS1 using the beta1 and tC1. The derived bS1 is used as the bS value of the NN filter unit 611f. The DF unit 332 derives beta2 and tC2 using parameters slice_beta_offset_div2 and slice_beta_tc2_div2 for the DF, which are different from the parameters for the NN filter, as sliceBetaOffsetDiv2 described below, and calculates bS (hereinafter bS2) using the beta2, tC2, and the input image. The derived bS2 is used as the bS value of the DF unit 332. In the case of color difference, beta and tC may be derived as follows. For the NN filter unit 611f, sliceBetaOffsetDiv2=(cIdx==1?sh_nn_cb_beta_offset_div2:sh_nn_cr_beta_offset_div2) In other cases (DF section 332) sliceBetaOffsetDiv2=(cIdx==1?slice_cb_beta_offset_div2: slice_cr_beta_offset_div2) Q=Clip3(0,63,QpC+(sliceBetaOffsetDiv2<<1)) The parameter beta is derived as follows:

[0290] beta=beta'*(1<<(BitDepth-8)) The processing of the NN filter unit 611f includes the following steps: sliceTcOffsetDiv2=(cIdx==1?sh_nn_cb_tc_offset_div2:sh_nn_cr_tc_offset_div2) Other than that, the processing of the DF unit 332 is sliceTcOffsetDiv2=(cIdx==1?slice_cb_tc_offset_div2: slice_cr_tc_offset_div2) Q=Clip3(0,65,QpC+2*(bS-1)+(sliceTcOffsetDiv2<<1)) The parameter tC is derived as follows.

[0291] When BitDepth is less than 10 tC=(tC'+(1<<(9-BitDepth)))>>(10-BitDepth) Otherwise tC=tC'*(1<<(BitDepth-10)) The DF unit 332 is a filter that performs filtering according to the bS value derived by the bS derivation unit 331a, and has the effect of suppressing deblocking noise.

[0292] The NN filter unit 611f is an NN filter that processes the input image before DF according to the bS value derived by the bS derivation unit 331a, and has the effect of suppressing deblocking noise that occurs at prediction and transform block boundaries.

[0293] The NN filter unit 611f may perform neural network processing by inputting the output parameter bS[][] of the bS derivation unit 331. Alternatively, bS may be used as a part of the input image (one channel) of the NN filter unit 611. The output bS of the bS derivation unit 331 may also be used as an image to be input to the NN filter unit 611. For example, it may be used as one of the channels of the image. In other words, if the top left coordinates of the target block are (xCb, yCb), the width is width, and the height is height, then it may be derived as follows, for example, where x=xCb..xCb+width-1 and y=yCb..yCb+height-1.

[0294] inSamples[0][x][y] = recSamples[cIdx][x][y] inSamples[1][x][y] = bS[x][y] Furthermore, in the above, Y, Cb, and Cr are processed independently, but the NN filter unit 611f may input all of Y, Cb, and Cr and perform filtering as follows.

[0295] inSamples[0][x][y] = recSamples[0][x][y] inSamples[1][x][y] = recSamples[1][x][y] inSamples[2][x][y] = recSamples[2][x][y] inSamples[3][x][y] = bS[x][y] The NN filter unit 611f may input Cb and Cr and perform filtering.

[0296] inSamples[0][x][y] = recSamples[1][x][y] inSamples[1][x][y] = recSamples[2][x][y] inSamples [][x][y] = bS[x][y] Additionally, an image qpMap of the QP may be input. Here, cIdx is a color component index. inSamples is an input image to the NN filter unit 611. The bS image may be used in a configuration in which it is added (concat) to the feature amount of the intermediate layer.

[0297] 29 is a diagram showing the syntax of the bS calculation parameters of the NN filter. These are transmitted separately from the parameters of the DF filter unit 332 (slice_luna_beta_offset_div2, slice_luna_tc_offset_div2, etc.).

[0298] When the flag sps_nn_enabled_flag, which is decoded from sps in SYN0201 and indicates whether the NN filter is available, is 1, the header decoding unit 3020 decodes sh_nn_enabled_flag.

[0299] When sh_nn_enabled_flag is 1 in SYN0202, the header decoding unit 3020 decodes the luminance parameters sh_nn_luma_beta_offset_div2 and sh_nn_luma_tc_offset_div2 for calculating bS used in the NN filter unit 611f.

[0300] In addition, in SYN0203, if a chrominance component exists in the input image, the header decoding unit 3020 decodes a flag sh_nn_chroma_enabled_flag indicating whether the chrominance NN filter is available from the encoded data. In SYN0204, if sh_nn_chroma_enabled_flag is 1, the header decoding unit 3020 decodes the Cb and Cr parameters sh_nn_cb_beta_offset_div2, sh_nn_cb_tc_offset_div2, sh_nn_cr_beta_offset_div2, and sh_nn_cr_tc_offset_div2 for calculating bS used in the NN filter unit.

[0301] (Configuration for sending differential values) 28 is a diagram showing a process of inputting a bS calculation parameter used in DF as a difference with a bS calculation parameter used in an NN filter to derive bS. In this example, when transmitting slice_beta_offset_div2 for DF and slice_beta_offset_div2 for an NN filter, the parameter of one filter is transmitted using the differential syntax of the parameter of the other filter.

[0302] The parameter decoding unit 3020 (or the bS derivation unit 331b) derives bS1 using the parameters beta and tC for the NN filter, and inputs it to the NN filter unit 611f. In addition, the parameter difference beta_diff and tC_diff for DF are used to calculate the parameters for DF from the difference between the parameters for the NN filter and the DF, and then calculates bS2 and sends it to the DF unit 332.

[0303] for example When processing in the NN filter unit 611f, sliceLumaBetaOffsetDiv2 is derived as follows, and beta1 and tC1 for the NN filter are derived using sliceLumaBetaOffsetDiv2 and sliceLumaTcOffsetDiv2. sliceLumaBetaOffsetDiv2 = slice_luma_beta_offset_div2 + beta_diff sliceTcBetaOffsetDiv2 = slice_luma_tc_offset_div2 + tc_diff This may involve performing the following process to derive beta1 and tC1 for the NN filter in the manner described above. sh_nn_luma_beta_offset_div2 = slice_luma_beta_offset_div2 + beta_diff sh_nn_luma_tc_offset_div2 = slice_luma_tc_offset_div2 + tc_diff Of course, the parameters for the DF may be derived from the parameters for the NN filter. slice_luma_beta_offset_div2 = sh_nn_beta_offset_div2 + beta_diff slice_luma_tc_offset_div2 = sh_nn_tc_offset_div2 + tc_diff (NN filter and ALF filter configured in parallel) The ALF (Adaptive Loop Filter) consists of a process of deriving a class classification filtIdx by analyzing an image on a block-by-block basis, and a process of selecting a filter coefficient and performing a filtering process using the obtained classification filtIdx.

[0304] Hereinafter, a configuration will be described in which the same input image is used for ALF class classification and the NN filter, and the ALF filtering process is performed using the NN filtered image.

[0305] FIG. 31 is a diagram showing ALF class classification processing using an input image and ALF filtering processing using an NN-filtered image.

[0306] The NN filter unit 611g performs filtering processing using an NN model on the input image, and outputs an NN-filtered image.

[0307] The ALF classification unit 341 uses the input image to output a classification result filtIdx.

[0308] The switching unit 342 uses the parameter nn_alf_ctb_flag decoded by the header decoding unit 3020, the NN_ALF output image, and the ALF output image to output either the NN_ALF output image or the ALF output image according to the value of nn_alf_ctb_flag.

[0309] The nn_alf_ctb_flag is a variable having two values, 0 or 1, and is a variable that determines an image to be output by the switching unit 342. The nn_alf_ctb_flag is decoded by the CT information decoding unit 3021.

[0310] In each CTB, when nn_alf_ctb_flag is 1, the switching unit 342 outputs the NN filtered image, and when nn_alf_ctb_flag is 0, the switching unit 342 outputs the input image. That is, depending on the value of nn_alf_ctb_flag, the image to be input to the filter processing unit 343 is switched between the NN filtered image and the input image.

[0311] The filtering unit 343 uses the decoded image and the classification filtIdx obtained by the ALF classifying unit 341 to output an image after ALF filtering. The images used by the filter processing unit 343 are the NN filtered image and the input image. The filter processing unit 343 processes the image and outputs the image after ALF processing.

[0312] FIG. 32 is a diagram illustrating the ALF padding process.

[0313] The ALF padding unit 344 uses the input image and outputs the image after padding processing.

[0314] The padding process is a process in which pixels below a virtual boundary, which is a line virtually set within the target CTU (a boundary along the horizontal direction indicated by the Y coordinate), are treated as reference pixels. In this process, the reference image is generated by padding nearby pixels instead of using the original image. For a given CTU, the virtual boundary is the row four pixels above the top row of the CTU below (vbOffset=4) in the case of luminance, and the row two pixels above the top row of the CTU below (vbOffset=2) in the case of chrominance.

[0315] The NN filter unit 611g performs processing using the image after the padding process. The ALF unit 312 also performs processing using the image of the NN filter unit 611g that has been processed using the image after the padding process as an input image, or the image after the padding process.

[0316] As described above, there is an effect of reducing line memory by performing processing that takes virtual boundaries into consideration in both the NN filter unit 611g and the ALF unit 312. In particular, it is preferable for a filter that performs processing before the ALF unit 312 to perform processing that uses the same virtual boundaries as the ALF unit 312 (the same processing by the ALF padding unit 344).

[0317] (Configuration that limits the applicability of NN filtering depending on slice shape) The process of applying the NN filter only to rectangular slices will be described below. An example of applying the NN filter to a configuration in which rectangular slices (hereinafter referred to as rectangular slices) and non-rectangular slices (hereinafter referred to as ) are used as slices to divide a picture within a screen will be described. Slices may be in units of CTUs, or in units of 16x16 or 32x32 less than CTUs. To determine whether a slice is rectangular or non-rectangular, pps_rect_slice_flag is decoded from the PPS of the encoded data, and if pps_rect_slice_flag=1, the width (here, pps_slice_width_in_tiles_minus1[ i ], which is the number of horizontal directions in tile width units minus 1) and height (here, pps_slice_height_in_tiles_minus1[ i ], which is the number of vertical directions in tile height units minus 1) of the slice are decoded from the encoded data. The data of the slices obtained by dividing the picture by the decoded width and height is decoded. If pps_rect_slice_flag=0, the number of CTUs in the slice minus 1 (sh_num_tiles_in_slice_minus1) is decoded in slice units.

[0318] FIG. 33 is a diagram showing syntax related to a flag that enables the application of an NN filter only when a slice is rectangular.

[0319] In SYN0401, if the flag pps_rect_slice_flag, which indicates whether the slice decoded by the header decoding unit 3020 is rectangular or not, is 1, and the flag sps_nn_enabled_flag, which indicates whether the NN filter decoded by the header decoding unit 3020 is usable or not, is 1, the header decoding unit 3020 decodes the flag sh_nn_enabled_flag, which indicates whether the NN filter is to be used or not.

[0320] Furthermore, in SYN0402, if sps_nn_enabled_flag is 1 and a chrominance component exists in the input image, the header decoding unit 3020 decodes a flag sh_nn_enabled_flag indicating whether or not to use an NN filter for the chrominance component.

[0321] [Application example] The above-mentioned video encoding device 10 and video decoding device 30 can be mounted and used in various devices that transmit, receive, record, and play back videos. Note that the video may be a natural video captured by a camera or the like, or an artificial video (including CG and GUI) generated by a computer or the like.

[0322] First, it will be described with reference to FIG. 2 that the above-mentioned video encoding device 10 and video decoding device 30 can be used for transmitting and receiving videos.

[0323] PROD_A in FIG. 2 is a block diagram showing the configuration of a transmission device PROD_A equipped with a video encoding device 10. As shown in the figure, the transmitting device PROD_A includes an encoding unit PROD_A1 that obtains encoded data by encoding a moving image, a modulation unit PROD_A2 that obtains a modulated signal by modulating a carrier wave with the encoded data obtained by the encoding unit PROD_A1, and a transmitting unit PROD_A3 that transmits the modulated signal obtained by the modulation unit PROD_A2. The above-mentioned moving image encoding device 10 is used as this encoding unit PROD_A1.

[0324] The transmitting device PROD_A may further include a camera PROD_A4 for capturing moving images, a recording medium PROD_A5 for recording moving images, an input terminal PROD_A6 for inputting moving images from the outside, and an image processing unit A7 for generating or processing images, as a source of moving images to be input to the encoding unit PROD_A1. In the figure, the transmitting device PROD_A is illustrated as having all of these components, but some of them may be omitted.

[0325] The recording medium PROD_A5 may record unencoded video, or may record video encoded by an encoding method for recording that is different from the encoding method for transmission. In the latter case, a decoding unit (not shown) that decodes the encoded data read from the recording medium PROD_A5 according to the encoding method for recording may be interposed between the recording medium PROD_A5 and the encoding unit PROD_A1.

[0326] PROD_B in Fig. 2 is a block diagram showing the configuration of a receiving device PROD_B equipped with a video decoding device 30. As shown in the figure, the receiving device PROD_B includes a receiving unit PROD_B1 that receives a modulated signal, a demodulation unit PROD_B2 that obtains coded data by demodulating the modulated signal received by the receiving unit PROD_B1, and a decoding unit PROD_B3 that obtains a video by decoding the coded data obtained by the demodulation unit PROD_B2. The above-mentioned video decoding device 30 is used as this decoding unit PROD_B3.

[0327] The receiving device PROD_B may further include, as destinations of the moving image output by the decoding unit PROD_B3, a display PROD_B4 for displaying the moving image, a recording medium PROD_B5 for recording the moving image, and an output terminal PROD_B6 for outputting the moving image to the outside. In the figure, the receiving device PROD_B is illustrated as having all of these components, but some of them may be omitted.

[0328] The recording medium PROD_B5 may be for recording unencoded video, or may be encoded by an encoding method for recording that is different from the encoding method for transmission. In the latter case, it is preferable to interpose an encoding unit (not shown) between the decoding unit PROD_B3 and the recording medium PROD_B5, which encodes the video acquired from the decoding unit PROD_B3 according to the encoding method for recording.

[0329] The transmission medium for transmitting the modulated signal may be wireless or wired. The transmission mode for transmitting the modulated signal may be broadcast (here, this refers to a transmission mode in which the destination is not specified in advance) or communication (here, this refers to a transmission mode in which the destination is specified in advance). In other words, the transmission of the modulated signal may be realized by any of wireless broadcast, wired broadcast, wireless communication, and wired communication.

[0330] For example, a broadcasting station (such as a broadcasting facility) / receiving station (such as a television receiver) for terrestrial digital broadcasting is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals by wireless broadcasting. Also, a broadcasting station (such as a broadcasting facility) / receiving station (such as a television receiver) for cable television broadcasting is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals by cable broadcasting.

[0331] In addition, the servers (workstations, etc.) and clients (television receivers, personal computers, etc.) of Internet-based VOD (Video On Demand) services and video sharing services are also A personal computer (e.g., a mobile computer, a smartphone, etc.) is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals by communication (usually, in a LAN, either wireless or wired is used as a transmission medium, and in a WAN, wired is used as a transmission medium). Here, personal computers include desktop PCs, laptop PCs, and tablet PCs. Smartphones also include multi-function mobile phone terminals.

[0332] The client of the video hosting service has a function to decode the encoded data downloaded from the server and display it on a display, as well as a function to encode the video captured by the camera and upload it to the server. That is, the client of the video hosting service functions as both the transmitting device PROD_A and the receiving device PROD_B.

[0333] Next, it will be described with reference to FIG. 3 that the above-mentioned video encoding device 10 and video decoding device 30 can be used for recording and reproducing videos.

[0334] PROD_C in Fig. 3 is a block diagram showing the configuration of a recording device PROD_C equipped with the above-mentioned video encoding device 10. As shown in the figure, the recording device PROD_C includes an encoding unit PROD_C1 that obtains encoded data by encoding a video, and a writing unit PROD_C2 that writes the encoded data obtained by the encoding unit PROD_C1 onto a recording medium PROD_M. The above-mentioned video encoding device 10 is used as this encoding unit PROD_C1.

[0335] The recording medium PROD_M may be (1) a type built into the recording device PROD_C, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), (2) a type connected to the recording device PROD_C, such as an SD memory card or USB (Universal Serial Bus) flash memory, or (3) a type loaded into a drive device (not shown) built into the recording device PROD_C, such as a DVD (Digital Versatile Disc: registered trademark) or BD (Blu-ray Disc: registered trademark).

[0336] The recording device PROD_C may further include a camera PROD_C3 for capturing moving images, an input terminal PROD_C4 for inputting moving images from the outside, a receiving unit PROD_C5 for receiving moving images, and an image processing unit PROD_C6 for generating or processing images, as a supply source of moving images to be input to the encoding unit PROD_C1. Although the figure illustrates a configuration in which the recording device PROD_C includes all of these, some of them may be omitted.

[0337] The receiving unit PROD_C5 may receive unencoded video, or may receive encoded data encoded by an encoding method for transmission that is different from the encoding method for recording. In the latter case, a transmission decoding unit (not shown) that decodes the encoded data encoded by the encoding method for transmission may be interposed between the receiving unit PROD_C5 and the encoding unit PROD_C1.

[0338] Examples of such a recording device PROD_C include a DVD recorder, a BD recorder, and an HDD (Hard Disk Drive) recorder (in this case, the input terminal PROD_C4 or the receiving unit PROD_C5 is the main source of the moving image). Other examples of such a recording device PROD_C include a camcorder (in this case, the camera PROD_C3 is the main source of the moving image), a personal computer (in this case, the receiving unit PROD_C5 or the image processing unit C6 is the main source of the moving image), and a smartphone (in this case, the camera PROD_C3 or the receiving unit PROD_C5 is the main source of the moving image).

[0339] 3 is a block diagram showing the configuration of a playback device PROD_D that includes the above-mentioned video decoding device 30. As shown in the figure, the playback device PROD_D includes a reading unit PROD_D1 that reads out coded data written on a recording medium PROD_M, and a decoding unit PROD_D2 that decodes the coded data read by the reading unit PROD_D1. and a decoding unit PROD_D2 that obtains a video by decoding the video. The above-mentioned video decoding device 30 is used as this decoding unit PROD_D2.

[0340] The recording medium PROD_M may be (1) a type that is built into the playback device PROD_D, such as an HDD or SSD, (2) a type that is connected to the playback device PROD_D, such as an SD memory card or USB flash memory, or (3) a type that is loaded into a drive device (not shown) built into the playback device PROD_D, such as a DVD or BD.

[0341] The playback device PROD_D may further include, as destinations of the video output by the decoding unit PROD_D2, a display PROD_D3 for displaying the video, an output terminal PROD_D4 for outputting the video to the outside, and a transmission unit PROD_D5 for transmitting the video. In the figure, the playback device PROD_D is illustrated as having all of these components, but some of them may be omitted.

[0342] The transmission unit PROD_D5 may transmit unencoded video, or may transmit encoded data encoded by an encoding method for transmission different from the encoding method for recording. In the latter case, it is preferable to interpose an encoding unit (not shown) between the decoding unit PROD_D2 and the transmission unit PROD_D5, which encodes the video by the encoding method for transmission.

[0343] Examples of such a playback device PROD_D include a DVD player, a BD player, and an HDD player (in this case, the output terminal PROD_D4 to which a television receiver or the like is connected is the main destination of the video image). Other examples of such a playback device PROD_D include a television receiver (in this case, the display PROD_D3 is the main destination of the video image), a digital signage (also called an electronic billboard or electronic bulletin board, and the display PROD_D3 or the transmission unit PROD_D5 is the main destination of the video image), a desktop PC (in this case, the output terminal PROD_D4 or the transmission unit PROD_D5 is the main destination of the video image), a laptop or tablet PC (in this case, the display PROD_D3 or the transmission unit PROD_D5 is the main destination of the video image), and a smartphone (in this case, the display PROD_D3 or the transmission unit PROD_D5 is the main destination of the video image).

[0344] (Hardware and Software Realizations) Furthermore, each block of the above-mentioned video decoding device 30 and video encoding device 10 may be realized in hardware by a logic circuit formed on an integrated circuit (IC chip), or in software by using a CPU (Central Processing Unit).

[0345] In the latter case, each of the above devices includes a CPU that executes the commands of a program that realizes each function, a ROM (Read Only Memory) that stores the above program, and a RAM (Random Access Memory) that expands the above program. The present invention also includes a storage device (recording medium) such as a memory for storing the above-mentioned programs and various data, and the like. The object of the embodiment of the present invention can also be achieved by supplying each of the above-mentioned devices with a recording medium on which the program code (executable program, intermediate code program, source program) of the control program of each of the above-mentioned devices, which is software for realizing the above-mentioned functions, is recorded in a computer-readable manner, and the computer (or CPU or MPU) reads out and executes the program code recorded on the recording medium.

[0346] Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy disks (registered trademark) and hard disks, and CD-ROMs (Compact Disc Read-Only Memory), MO disks (Magneto-Optical Discs), MDs (Mini Discs), DVDs (Digital Versatile Discs: registered trademark), CD-Rs (CD Recordable), and Blu-ray discs (Blu-ray Discs, including optical discs such as Disc: registered trademark, IC cards (including memory cards) Cards such as optical cards, semiconductor memories such as mask ROM / EPROM (Erasable Programmable Read-Only Memory) / EEPROM (Electrically Erasable and Programmable Read-Only Memory: registered trademark) / flash ROM, or logic circuits such as PLD (Programmable Logic Device) and FPGA (Field Programmable Gate Array) can be used.

[0347] Moreover, each of the above devices may be configured to be connectable to a communication network, and the above program code may be supplied via the communication network. This communication network is not particularly limited as long as it is capable of transmitting the program code. For example, the Internet, an intranet, an extranet, a LAN (Local Area Network), an ISDN (Integrated Services Digital Network), a VAN (Value-Added Network), a CATV (Community Antenna / Cable Television) communication network, a virtual private network, a telephone line network, a mobile communication network, a satellite communication network, etc. may be used. Furthermore, the transmission medium constituting this communication network is not limited to a specific configuration or type as long as it is a medium capable of transmitting the program code. For example, it can be used in wired communication such as IEEE (Institute of Electrical and Electronic Engineers) 1394, USB, power line carrier, cable TV line, telephone line, ADSL (Asymmetric Digital Subscriber Line) line, etc., or in wireless communication such as infrared such as IrDA (Infrared Data Association) or a remote control, BlueTooth (registered trademark), IEEE802.11 wireless, HDR (High Data Rate), NFC (Near Field Communication), DLNA (Digital Living Network Alliance: registered trademark), mobile phone network, satellite line, terrestrial digital broadcasting network, etc. The embodiment of the present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

[0348] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. In other words, the technical scope of the present invention also includes embodiments obtained by combining technical means that are appropriately modified within the scope of the claims. [Industrial Applicability]

[0349] The embodiments of the present invention can be suitably applied to a video decoding device that decodes coded data in which image data is coded, and a video coding device that generates coded data in which image data is coded, and can also be suitably applied to the data structure of coded data that is generated by a video coding device and referenced by the video decoding device. [Explanation of symbols]

[0350] 1. Video transmission system 30 Video Decoding Device 31 Image Decoding Device 301 Entropy Decoding Unit 302 Parameter Decoding Unit 303 Inter-prediction parameter derivation unit 304 Intra prediction parameter derivation unit 305, 107 Loop Filter 306, 109 Reference Picture Memory 307, 108 Prediction parameter memory 308, 101 Prediction image generation unit 309 Inter-prediction image generation unit 310 Intra-prediction image generation unit 311, 105 Inverse quantization and inverse transformation unit 312, 106 Addition section 320 Prediction Parameter Derivation Unit 10 Video Encoding Device 11 Image encoding device 102 Subtraction section 103 Transformation and Quantization Section 104 Entropy coding unit 110 Encoding parameter determination unit 111 Parameter Encoding Unit 112 Inter-prediction parameter coding unit 113 Intra prediction parameter coding unit 120 Prediction parameter derivation part 71 Filter information creation device

Claims

1. A reconstructed image is generated from the sum of the predicted image and the residual image. A video decoding device comprising an NN filter unit that performs filtering processing using two or more layers of product-sum operations on luminance and chrominance signals, and a ccALF processing unit that derives a chrominance signal from the luminance and chrominance signals, wherein either the NN filter unit or the ccALF processing unit is exclusively applied to a reconstructed image.

2. The video decoding device of claim 1, further comprising a header decoding unit that decodes sh_nn_enabled_flag indicating whether the processing of the NN filter unit is available in an SPS, picture header, or slice header, and sh_alf_cc_cb_enabled_flag and sh_alf_cc_cr_enabled_flag indicating whether the processing of the ccALF processing unit is available from encoded data, wherein when sh_nn_enabled_flag has a value of available, the header decoding unit sets sh_alf_cc_cb_enabled_flag and sh_alf_cc_cr_enabled_flag to values ​​indicating that the processing is not available without decoding from the encoded data, or when sh_alf_cc_cb_enabled_flag and sh_alf_cc_cr_enabled_flag have a value of available, the header decoding unit sets sh_nn_enabled_flag to a value indicating that the processing is not available without decoding from the encoded data.

3. The video decoding device according to claim 1, characterized in that it comprises a header decoding unit that decodes sh_nn_enabled_flag, sh_alf_cc_cb_enabled_flag, and sh_alf_cc_cr_enabled_flag in the slice header from the encoded data, a CT information decoding unit 3021 that decodes nn_ctb_scale_idx for each CTU, an NN filter unit that performs filtering from the luminance and chrominance signals according to the nn_ctb_scale_idx, and a ccALF processing unit that derives a chrominance signal from the luminance and chrominance signals when alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag are 1, and decodes only either nn_ctb_scale_idx or alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag, and if one is 1 or greater, the other is set to 0.

4. 2. The video decoding device according to claim 1, wherein, when nn_ctb_scale_idx is 1 or more, the CT information decoding unit derives 0 without decoding alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag.

5. 2. The video decoding device according to claim 1, wherein the CT information decoding unit 3021 derives 0 without decoding nn_ctb_scale_idx when alf_ctb_cc_cb_flag and alf_ctb_cc_cr_flag are 1.

6. A video decoding device comprising a header decoding unit which decodes sh_nn_enabled_flag and sh_alf_enabled_flag in the slice header, a CT information decoding unit 3021 which decodes nn_ctb_scale_idx from the NN flag for each CTU, an NN filter unit which performs filtering from luminance and color difference signals when nn_ctb_scale_idx is 1 or greater, and an ALF processing unit which applies filtering while switching between different filtering processes depending on edge strength when alf_ctb_flag is 1, and which decodes only one of nn_ctb_scale_idx or alf_ctb_flag, and when one is 1 or greater, the other is set to 0.

7. The present invention includes a header decoding unit that decodes an NN model index from the encoded data in a parameter set, a picture header, or a slice header, a CT information decoding unit 3021 that decodes an NN partial model index from the encoded data in units of CTUs, an NN filter unit that performs filtering using a neural network composed of a plurality of partial models, an NN model selection unit that switches parameters of the above-mentioned models of the neural network model, and an NN partial model selection unit that switches parameters of a specific partial model among the above-mentioned plurality of partial models of the neural network model, The NN filter unit selects a model to be used in the NN filter unit based on the NN model index, and the NN partial model selection unit selects a model to be used in the NN filter unit based on the NN partial model index. and switching parameters of the specific partial model.

8. The video decoding device according to claim 7, characterized in that the model selection unit switches models in units of a first region, and the partial model selection unit switches partial models in units of a second region smaller than the first region.

9. A video decoding device including a header decoding unit that decodes filter parameters from encoded data in a slice header, a bS derivation unit that derives a deblocking filter strength bS, a DF unit that applies a deblocking filter to a decoded image, an NN filter unit that applies a filter process to the decoded image using a neural network, and an image synthesis unit, the bS derivation unit derives the bS in response to the filter parameters; The header decoding unit decodes, as the filter parameters, a first filter parameter for use in an NN filter and a second filter parameter for use in a DF; the bS derivation unit derives a first bS value (bS1) from the first filter parameter and a second bS value (bS2) from the second filter parameter; The NN filter unit performs a filtering process using a first bS value; The video decoding device, wherein the DF unit performs a filter process using a second bS value.

10. A video decoding device including a header decoding unit that decodes filter parameters from encoded data in a slice header, a bS derivation unit that derives a deblocking filter strength bS, a DF unit that applies a deblocking filter to a decoded image, an NN filter unit that applies a filter process to the decoded image using a neural network, and an image synthesis unit, the bS derivation unit derives the bS in response to the filter parameters; The header decoding unit decodes, as the filter parameters, NN filter parameters for use in the NN filter and differential parameters which are differences between the filter parameters for use in the DF and the NN filter parameters; the bS derivation unit derives a first bS value (bS1) from the NN filter parameter, and derives a second bS value (bS2) from the NN filter parameter and the difference parameter; The NN filter unit performs a filtering process using a first bS value; The video decoding device, wherein the DF unit performs a filter process using a second bS value.

11. The video decoding device according to claim 10, characterized in that the filter parameters are beta and tC, and the bS derivation unit uses beta to control the filtering application range from the block boundary, and uses tC to clip the amount of change after filtering for a certain pixel value.

12. The apparatus includes a CT decoding unit that decodes alf_ctb_flag in a CTU, an ALF processing unit that applies ALF processing to the decoded image, an NN filter unit that applies filtering to the decoded image by a neural network, and an ALF padding unit, The ALF unit includes an ALF class classification unit that performs ALF class classification, and an ALF class filter unit that performs ALF class filtering, The ALF classifier performs class classification on the image before processing by the NN filter. The video decoding device is characterized in that the ALF class filter unit performs class filtering on the image processed by the NN filter unit.

13. 13. The video decoding device according to claim 12, wherein the NN filter unit and the ALF class classification unit perform processing in parallel.

14. The ALF padding unit performs padding processing using neighboring pixels instead of referring to the target pixel for pixels below the virtual boundary and for that pixel and the origin. The video decoding device according to claim 12, characterized in that the NN filter unit and the ALF unit apply the processing using the image after the padding processing.

15. A reconstructed image is generated by adding the predicted image and the residual image. A video coding device comprising an NN filter unit that performs filtering processing using two or more layers of product-sum operations on luminance and chrominance signals, and a ccALF processing unit that derives a chrominance signal from the luminance and chrominance signals, wherein either the NN filter unit or the ccALF processing unit is exclusively applied to a reconstructed image.

Images