Moving image encoding device and moving image decoding device

Abstract

In the case where a model parameter suitable for an input moving image is selected from a set of predetermined model parameters and is subjected to resolution inverse-conversion to be applied, a moving image that is not suitable for a plurality of model parameters may result in exhibiting low-quality. The present invention comprises: a neural network for performing scaling with a rational number multiple; and a prediction unit provided with an interpolation unit that interpolates the rational number multiple. A first scaling magnification rate by the neural network and a second scaling magnification rate by the interpolation unit are derived from the actual width and height of a reference image and the actual width and height of a target image, and an interpolation image is derived by using first scaling by the neural network and second scaling by the interpolation unit.

Description

Video encoding device, video decoding device

[0001] The present invention relates to a video encoding device and a video decoding device.

[0002] In order to efficiently transmit or record moving images, a moving image encoding device is used that generates encoded data by encoding moving images, and a moving image decoding device is used that generates decoded images by decoding the encoded data.

[0003] Specific examples of video encoding methods include H.264 / AVC and H.265 / HEVC (High-Efficiency Video Coding).

[0004] In such a video coding method, images (pictures) constituting a video are managed in a hierarchical structure consisting of slices obtained by dividing images, coding tree units (CTUs) obtained by dividing slices, coding units (sometimes called coding units (CUs)) obtained by dividing coding tree units, and transform units (TUs) obtained by dividing coding units, and are coded / decoded for each CU.

[0005] In such video coding methods, a predicted image is typically generated based on a locally decoded image obtained by encoding / decoding an input image, and the predicted image is subtracted from the input image (original image) to obtain a prediction error (sometimes called a "difference image" or "residual image"), which is then coded. Methods for generating predicted images include inter-frame prediction (inter-prediction) and intra-frame prediction (intra-prediction).

[0006] Furthermore, Non-Patent Document 1 is an example of a recent video encoding and decoding technique.

[0007] Non-Patent Document 1 defines the RPR (Reference Picture Re-sampling) technology that enables encoding and decoding of variable image resolutions. Furthermore, Annex D of Non-Patent Document 1 defines supplemental enhancement information (SEI) for transmitting image properties, display methods, timing, etc. simultaneously with encoded data.

[0008] Non-Patent Documents 2 and 3 disclose a method for enlarging to an arbitrary magnification, which involves performing regression using a neural network (NN) based on scale information for coordinate information, and a method for performing bicubic interpolation in the feature layer of the NN.

[0009] ITU-T Recommendation H.266 (08 / 20) 2020-08-29Meta-SR: A Magnification-Arbitrary Network for Super-Resolution, https: / / arxiv.org / pdf / 1903.00875.pdf, 2019ASDN: A Deep Convolutional Network for Arbitrary Scale ImageSuper-Resolution, https: / / arxiv.org / pdf / 2010.02414.pdf, 2020

[0010] However, the RPR in Non-Patent Document 1 is capable of scaling by a rational number with an accuracy of 1 / 16834, and can select two coefficients according to the scaling factor, but it is only capable of selecting a linear filter, which results in low performance.

[0011] Furthermore, the methods disclosed in Non-Patent Document 2 and Non-Patent Document 3 have problems in that the processing and image quality are insufficient when used as an image encoding device or an image decoding device. In particular, the method in Non-Patent Document 2 is based on point-based processing, which makes the processing complex and unsuitable for picture- or block-based processing. The method in Non-Patent Document 3 is dependent on the size of the feature layer, which limits the magnification that can be achieved. Both methods have insufficient processing according to the magnification and insufficient performance, and also have problems with handling motion vectors when applied to reference images.

[0012] A video decoding device according to one aspect of the present invention includes a neural network that performs scaling by a rational number factor and a prediction unit that includes an interpolation unit that performs interpolation by a rational number factor, and derives a first scaling factor by the neural network and a second scaling factor by the interpolation unit from the actual width and height of a reference image and the actual width and height of a target image, and derives an interpolated image using the first scaling by the neural network and the second scaling by the interpolation unit.

[0013] This configuration has the advantage of being able to output a suitable image by switching the neural network parameters according to the enlargement ratio. In addition, by switching the parameters after enlarging the conversion size, the memory size for switching parameters can be kept to a minimum.

[0014] FIG. 1 is a schematic diagram showing the configuration of a video transmission system according to the present embodiment. FIG. 1 is a diagram showing the configuration of a transmitting device equipped with a video encoding device according to the present embodiment, and a receiving device equipped with a video decoding device. PROD_A indicates a transmitting device equipped with a video encoding device, and PROD_B indicates a receiving device equipped with a video decoding device. FIG. 2 is a diagram showing the configuration of a recording device equipped with a video encoding device according to the present embodiment, and a playback device equipped with a video decoding device. PROD_C indicates a recording device equipped with a video encoding device, and PROD_D indicates a playback device equipped with a video decoding device. FIG. 3 is a diagram showing the hierarchical structure of encoded data. FIG. 4 is a conceptual diagram of an image to be processed in the video transmission system according to the present embodiment. FIG. 5 is a conceptual diagram showing an example of a reference picture and a reference picture list. FIG. 6 is a schematic diagram showing the configuration of an image decoding device. FIG. 7 is a flowchart explaining the general operation of the image decoding device. FIG. 8 is a schematic diagram showing the configuration of an inter-prediction parameter derivation unit. FIG. 9 is a schematic diagram showing the configuration of an inter-prediction image generation unit. FIG. 10 is a block diagram showing the configuration of an image encoding device. FIG. 11 is a diagram showing the configuration of a neural network of an NN filter unit 611. FIG. 12 is a diagram showing an example of the configuration of the NN filter unit 611. FIG. 1 is a diagram showing an example of the configuration of an NN filter unit 611. FIG. 2 is a diagram showing an example of the configuration of an integer resolution conversion unit 6112I. FIG. 3 is a diagram showing an example of the configuration of a rational resolution conversion unit 6112R. FIG. 4 is a diagram showing processing units of the NN filter unit 611 and the rational resolution conversion unit 6112R. FIG. 5 is a diagram showing an example of the configuration of the NN filter unit 611. FIG. 6 is a diagram showing an example of the input configuration of a motion vector of the NN filter unit 611. FIG. 7 is a diagram showing an example of the configuration of the NN filter unit 611.

[0015] First Embodiment Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0016] FIG. 1 is a schematic diagram showing the configuration of a video transmission system according to this embodiment.

[0017] The video transmission system 1 is a system that transmits coded data obtained by encoding images with different resolutions after resolution conversion, decodes the transmitted coded data, and converts the images back to the original resolution for display. The video transmission system 1 comprises a video encoding device 10, a network 21, a video decoding device 30, and an image display device 41.

[0018] The video encoding device 10 comprises a preprocessing device (preprocessing unit) 51 , an image encoding device (image encoding unit) 11 , and a synthesis information creating device (synthetic information creating unit) 71 .

[0019] The video decoding device 30 includes an image decoding device (image decoding unit) 31 and a post-processing unit (post-processing unit) 61 .

[0020] The preprocessing device 51 converts the resolution of an image T included in the video as necessary, and supplies a variable resolution video T2 including images with different resolutions to the image coding device 11. The preprocessing device 51 may also supply filter information indicating whether or not the resolution of an image is to be converted to the image coding device 11. If the information indicates resolution conversion, the video coding device 10 sets ref_pic_resampling_enabled_flag, which will be described later, to 1. Then, the video coding device 10 performs coding using the sequence parameter set SPS (Sequence Parameter Set) of the coded data Te.

[0021] The synthesis information creation device 71 creates filter information based on an image T1 included in the video and sends it to the image encoding device 11. A variable resolution image T2 is input to the image encoding device 11. The image encoding device 11 encodes image size information of the input image in PPS units using the RPR framework and sends it to the image decoding device 31.

[0022] The network 21 transmits the encoded filter information and the encoded data Te to the image decoding device 31. Some or all of the encoded filter information may be included in the encoded data Te as supplemental enhancement information SEI. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. The network 21 is not necessarily limited to a bidirectional communication network, but may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. Furthermore, the network 21 may be replaced by a storage medium on which the encoded data Te is recorded, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blu-ray Disc: registered trademark).

[0023] The image decoding device 31 decodes each of the coded data Te transmitted over the network 21 , generates a variable resolution decoded image, and supplies it to the post-processing device 61 .

[0024] If the filter information indicates resolution conversion, the post-processing device 61 performs super-resolution processing using model parameters for super-resolution based on image size information included in the encoded data.Then, the post-processing device 61 generates a decoded image of the original size by inversely converting the resolution-converted image.If the filter information does not indicate resolution conversion, the post-processing device 61 performs image restoration processing using model parameters for image restoration.By performing the image restoration processing, a decoded image with reduced coding noise is generated.

[0025] The image display device 41 displays all or part of one or more decoded images Td2 input from the post-processing device 61. The image display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. The display may be in the form of a stationary display, a mobile display, an HMD, or the like. Furthermore, if the image decoding device 31 has high processing power, it displays high-quality images, and if it has only low processing power, it displays images that do not require high processing power or display power.

[0026] FIG. 5 is a conceptual diagram of an image to be processed in the moving image transmission system shown in FIG. 1, and is a diagram showing the change in the resolution of the image as time elapses. However, in FIG. 5, it does not distinguish whether the image is encoded or not. FIG. 5 shows an example of reducing the resolution and transmitting the image to the image decoder 31 in the processing process of the moving image transmission system. As shown in FIG. 5, usually, the image pre-processor 51 performs a conversion to reduce the resolution of the image in order to reduce the amount of information of the transmitted information.

[0027] <Operator> The operators used in this specification are described below.

[0028] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is an OR assignment operator, and || represents a logical OR.

[0029] x? y : z is a ternary operator that takes y when x is true (non-zero) and z when x is false (0).

[0030] Clip3(a, b, c) is a function that clips c to a value between a and b. When c < a, it returns a; when c > b, it returns b; and in other cases, it returns c (where a <= b).

[0031] abs(a) is a function that returns the absolute value of a.

[0032] Int(a) is a function that returns the integer value of a. <b

[0033] floor(a) is a function that returns the largest integer less than or equal to a.

[0034] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0035] a / d represents the division of a by d (rounding down the decimal part).

[0036] a^b represents the b-th power of a. When a = 2 and b is an integer, 2^b = 1<<b. <

[0037] <Structure of encoded data Te> Before providing a detailed description of the image encoding device 11 and image decoding device 31 according to this embodiment, the data structure of the encoded data Te generated by the image encoding device 11 and decoded by the image decoding device 31 will be described.

[0038] 4 is a diagram showing a hierarchical structure of data in coded data Te. The coded data Te illustratively includes a sequence and multiple pictures constituting the sequence. Fig. 4 shows a coded video sequence that defines the sequence SEQ, coded pictures that define pictures PICT, coded slices that define slices S, coded slice data that define slice data, coding tree units included in the coded slice data, and coding units included in the coding tree units.

[0039] (Encoded Video Sequence) The encoded video sequence defines a set of data that the image decoding device 31 refers to in order to decode the sequence SEQ to be processed. As shown in Fig. 4, the sequence SEQ includes a video parameter set VPS (Video Parameter Set), a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), an adaptation parameter set (APS), a picture PICT, and supplemental enhancement information SEI (Supplemental Enhancement Information).

[0040] The video parameter set VPS specifies a set of coding parameters common to multiple videos composed of multiple layers, as well as a set of coding parameters related to multiple layers included in the video and each individual layer.

[0041] The sequence parameter set SPS defines a set of coding parameters that the image decoding device 31 references to decode the target sequence. For example, the width and height of a picture are defined. Note that there may be multiple SPSs. In this case, one of the multiple SPSs is selected from the PPS.

[0042] The sequence parameter set SPS includes the following syntax elements: ref_pic_resampling_enabled_flag: This flag specifies whether to use the resolution-varying function (resampling) when decoding each image included in a single sequence that references the target SPS. In other words, this flag indicates that the size of the reference picture referenced in generating a predicted image varies between images included in a single sequence. A value of 1 indicates that resampling is enabled; a value of 0 indicates that resampling is not enabled. pic_width_max_in_luma_samples: This syntax element specifies the width, in luminance block units, of the image with the largest width among the images included in a single sequence. The value of this syntax element must be a non-zero integer multiple of Max(8, MinCbSizeY), where MinCbSizeY is a value determined by the minimum size of a luminance block. pic_height_max_in_luma_samples: A syntax element that specifies the height of the image with the maximum height among the images in a single sequence, in units of luminance blocks. The value of this syntax element is required to be a non-zero integer multiple of Max(8, MinCbSizeY). sps_temporal_mvp_enabled_flag: A flag that specifies whether to use temporal motion vector prediction when decoding the target sequence. If the value of this flag is 1, temporal motion vector prediction is used; if the value is 0, temporal motion vector prediction is not used. Specifying this flag can prevent misalignment of the reference coordinate position when referencing a reference picture with a different resolution, for example.

[0043] The picture parameter set PPS defines a set of coding parameters that the image decoding device 31 references to decode each picture in the target sequence. For example, the picture parameter set PPS includes a reference value of the quantization width used in decoding the picture (pic_init_qp_minus26) and a flag indicating the application of weighted prediction (weighted_pred_flag). Note that there may be multiple PPSs. In this case, one of the multiple PPSs is selected for each picture in the target sequence.

[0044] Here, the picture parameter set PPS includes the following syntax elements: pic_width_in_luma_samples: This syntax element specifies the width of the target picture. The value of this syntax element must be a non-zero, integral multiple of Max(8, MinCbSizeY), and equal to or less than pic_width_max_in_luma_samples. pic_height_in_luma_samples: This syntax element specifies the height of the target picture. The value of this syntax element must be a non-zero, integral multiple of Max(8, MinCbSizeY), and equal to or less than pic_height_max_in_luma_samples. conformance_window_flag: This flag indicates whether a conformance (cropping) window offset parameter will be subsequently signaled. The conformance window offset parameter indicates the location where the conformance window is displayed. If this flag is 1, the parameter is signaled; if it is 0, the parameter is not present. conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, conf_win_bottom_offset: Offset values ​​for specifying the left, right, top, and bottom positions of the picture output by the decoding process, relative to the rectangular area specified by the output picture coordinates. Furthermore, if the value of conformance_window_flag is 0, the values ​​of conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset are presumed to be 0. scaling_window_flag: A flag indicating whether a scaling window offset parameter exists in the target PPS, and is a flag related to the specification of the output image size. When this flag is 1, it indicates that the parameter exists in the PPS, and when this flag is 0, it indicates that the parameter does not exist in the PPS.Also, if the value of ref_pic_resampling_enabled_flag is 0, the value of scaling_window_flag is also required to be 0. scaling_win_left_offset, scaling_win_right_offset, scaling_win_top_offset, scaling_win_bottom_offset: Syntax elements that specify the offsets, in luma samples, to be applied to the image size for scaling ratio calculation at the left, right, top, and bottom positions of the target picture, respectively. Also, if the value of scaling_window_flag is 0, the values ​​of scaling_win_left_offset, scaling_win_right_offset, scaling_win_top_offset, and scaling_win_bottom_offset are inferred to be 0. Additionally, the value of scaling_win_left_offset + scaling_win_right_offset is required to be less than pic_width_in_luma_samples, and the value of scaling_win_top_offset + scaling_win_bottom_offset is required to be less than pic_height_in_luma_samples.

[0045] The width PicOutputWidthL and height PicOutputHeightL of the output picture are derived as follows:

[0046] PicOutputWidthL = pic_width_in_luma_samples - (scaling_win_right_offset + scaling_win_left_offset) PicOutputHeightL = pic_height_in_luma_samples - (scaling_win_bottom_offset + scaling_win_top_offset) (Coded Picture) A coded picture defines a set of data that the image decoding device 31 references in order to decode a picture PICT to be processed. As shown in Fig. 4, the picture PICT includes a picture header PH and slices 0 to NS-1 (NS is the total number of slices included in the picture PICT).

[0047] Hereinafter, when there is no need to distinguish between slices 0 to NS-1, the subscripts of the symbols may be omitted. This also applies to other data that are included in the coded data Te described below and have subscripts.

[0048] A picture header includes the following syntax elements: pic_temporal_mvp_enabled_flag: A flag that specifies whether temporal motion vector prediction is used for inter prediction of a slice associated with the picture header. If the value of this flag is 0, the syntax elements of the slice associated with the picture header are restricted so that temporal motion vector prediction is not used in decoding the slice. If the value of this flag is 1, it indicates that temporal motion vector prediction is used in decoding the slice associated with the picture header. If this flag is not specified, its value is assumed to be 0.

[0049] (Encoded Slice) An encoded slice defines a set of data to be referenced by the image decoding device 31 in order to decode a slice S to be processed. As shown in Fig. 4, a slice includes a slice header and slice data.

[0050] The slice header includes a group of coding parameters that the image decoding device 31 refers to in order to determine a decoding method for the current slice. Slice type specification information (slice_type) that specifies the slice type is an example of a coding parameter included in the slice header.

[0051] Slice types that can be specified by the slice type specification information include (1) an I slice that uses only intra prediction during encoding, (2) a P slice that uses uni-prediction (L0 prediction) or intra prediction during encoding, and (3) a B slice that uses uni-prediction (L0 prediction or L1 prediction), bi-prediction, or intra prediction during encoding. Note that inter prediction is not limited to uni-prediction or bi-prediction, and a predicted image may be generated using more reference pictures. Hereinafter, the terms P and B slices refer to slices including blocks that can use inter prediction.

[0052] Note that the slice header may include a reference to the picture parameter set PPS (pic_parameter_set_id).

[0053] (Coded Slice Data) The coded slice data defines a set of data that the image decoding device 31 references in order to decode the slice data to be processed. The slice data includes a CTU, as shown in the coded slice header in Fig. 4. A CTU is a block of a fixed size (e.g., 64x64) that constitutes a slice, and is also called a Largest Coding Unit (LCU).

[0054] (Coding Tree Unit) Figure 4 defines a set of data that the image decoding device 31 references to decode the target CTU. The CTU is divided into coding units (CUs), which are basic units of encoding processing, by recursive quad tree (QT) division, binary tree (BT) division, or ternary tree (TT) division. BT division and TT division are collectively called multi-tree (MT) division. A node in a tree structure obtained by recursive quad tree division is called a coding node. The intermediate nodes of a quad tree, binary tree, and ternary tree are coding nodes, and the CTU itself is defined as the top-level coding node.

[0055] The CT includes, as CT information, a CU split flag (split_cu_flag) indicating whether CT splitting is performed, a QT split flag (qt_split_cu_flag) indicating whether QT splitting is performed, an MT split direction (mtt_split_cu_vertical_flag) indicating the split direction of MT splitting, and an MT split type (mtt_split_cu_binary_flag) indicating the split type of MT splitting. split_cu_flag, qt_split_cu_flag, mtt_split_cu_vertical_flag, and mtt_split_cu_binary_flag are transmitted for each encoding node.

[0056] (Coding Unit) Fig. 4 defines a set of data that the image decoding device 31 references in order to decode a coding unit to be processed. Specifically, a CU is composed of a CU header CUH, prediction parameters, transformation parameters, quantization transformation coefficients, etc. The CU header defines a prediction mode, etc.

[0057] The prediction process may be performed in units of CUs, or in units of sub-CUs obtained by further dividing a CU. When the sizes of a CU and a sub-CU are the same, there is one sub-CU in the CU. When the size of a CU is larger than the size of a sub-CU, the CU is divided into sub-CUs. For example, when a CU is 8x8 and a sub-CU is 4x4, the CU is divided into four sub-CUs, divided horizontally in half and vertically in half.

[0058] There are two types of prediction (prediction modes): intra prediction and inter prediction. Intra prediction is a prediction within the same picture, while inter prediction refers to a prediction process performed between different pictures (for example, between display times or between layer images).

[0059] The transformation and quantization processes are performed in units of CUs, but the quantized transformation coefficients may be entropy coded in units of sub-blocks such as 4x4.

[0060] (Prediction Parameters) A predicted image is derived from prediction parameters associated with a block. Prediction parameters include intra-prediction and inter-prediction parameters.

[0061] The following describes prediction parameters for inter prediction. The inter prediction parameters are composed of prediction list usage flags predFlagL0 and predFlagL1, reference picture indices refIdxL0 and refIdxL1, and motion vectors mvL0 and mvL1. predFlagL0 and predFlagL1 are flags indicating whether a reference picture list (L0 list, L1 list) is used, and when the value is 1, the corresponding reference picture list is used. Note that in this specification, when a "flag indicating whether XX is true" is used, a flag other than 0 (e.g., 1) is considered to be XX, and 0 is considered to be not XX, and in logical negation, logical product, etc., 1 is treated as true and 0 is treated as false (the same applies below). However, in actual devices and methods, other values ​​may be used as true and false values.

[0062] Syntax elements for deriving inter-prediction parameters include, for example, an affine flag affine_flag used in merge mode, a merge flag merge_flag, a merge index merge_idx, an MMVD flag mmvd_flag, an inter-prediction identifier inter_pred_idc for selecting a reference picture to be used in AMVP mode, a reference picture index refIdxLX, a prediction vector index mvp_LX_idx for deriving a motion vector, a difference vector mvdLX, and a motion vector precision mode amvr_mode.

[0063] (Reference Picture List) A reference picture list is a list of reference pictures stored in the reference picture memory 306. FIG. 6 is a conceptual diagram showing an example of a reference picture and a reference picture list. In the conceptual diagram showing an example of a reference picture in FIG. 6, each rectangle represents a picture, each arrow represents the reference relationship between pictures, the horizontal axis represents time, I, P, and B in the rectangle represent intra-pictures, uni-predictive pictures, and bi-predictive pictures, respectively, and the numbers in the rectangle represent the decoding order. As shown in the diagram, the decoding order of pictures is I0, P1, B2, B3, and B4, and the display order is I0, B3, B2, B4, and P1. FIG. 6 shows an example of a reference picture list for picture B3 (target picture). A reference picture list is a list representing candidate reference pictures, and one picture (slice) may have one or more reference picture lists. In the example shown in the figure, the current picture B3 has two reference picture lists: an L0 list RefPicList0 and an L1 list RefPicList1. For each CU, refIdxLX specifies which picture in the reference picture list RefPicListX (X=0 or 1) to actually reference. The figure shows an example where refIdxL0=2 and refIdxL1=0. Note that LX is a notation method used when there is no distinction between L0 prediction and L1 prediction; hereinafter, parameters for the L0 list and parameters for the L1 list will be distinguished by replacing LX with L0 or L1.

[0064] (Merge Prediction and AMVP Prediction) Prediction parameter decoding (encoding) methods include merge prediction mode and AMVP (Advanced Motion Vector Prediction, adaptive motion vector prediction) mode, and merge_flag is a flag for distinguishing between them. Merge prediction mode is a mode in which the prediction list usage flag predFlagLX, reference picture index refIdxLX, and motion vector mvLX are not included in the encoded data, but are derived from prediction parameters of already processed neighboring blocks, etc. AMVP mode is a mode in which inter_pred_idc, refIdxLX, and mvLX are included in the encoded data. Note that mvLX is encoded as mvp_LX_idx, which identifies the prediction vector mvpLX, and a difference vector mvdLX. In addition to merge prediction mode, affine prediction mode and MMVD prediction mode may also be used.

[0065] inter_pred_idc is a value indicating the type and number of reference pictures, and takes one of the values ​​PRED_L0, PRED_L1, or PRED_BI. PRED_L0 and PRED_L1 indicate uni-prediction using one reference picture managed in the L0 list and L1 list, respectively. PRED_BI indicates bi-prediction using two reference pictures managed in the L0 list and L1 list.

[0066] The merge_idx is an index indicating which prediction parameter from among prediction parameter candidates (merge candidates) derived from blocks for which processing has been completed is to be used as the prediction parameter for the current block.

[0067] (Motion Vector) mvLX indicates the amount of shift between blocks on two different pictures. The predicted vector and differential vector related to mvLX are called mvpLX and mvdLX, respectively.

[0068] (Inter prediction identifier inter_pred_idc and prediction list use flag predFlagLX) The relationship between inter_pred_idc, predFlagL0, and predFlagL1 is as follows, and they can be converted into each other.

[0069] inter_pred_idc = (predFlagL1 << 1) + predFlagL0 predFlagL0 = inter_pred_idc & 1 predFlagL1 = inter_pred_idc >> 1 (Configuration of Image Decoding Apparatus) The configuration of an image decoding apparatus 31 (FIG. 7) according to this embodiment will be described.

[0070] The image decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit (prediction image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a prediction image generation unit (prediction image generation device) 308, an inverse quantization and inverse transform unit 311, an adder 312, and a prediction parameter derivation unit 320. Note that, in accordance with the image coding device 11 described below, the image decoding device 31 may also be configured without including the loop filter 305.

[0071] The parameter decoding unit 302 further includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit), and the CU decoding unit 3022 further includes a TU decoding unit 3024. These may be collectively referred to as a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, PPS, and APS, and slice headers (slice information) from the coded data. The CT information decoding unit 3021 decodes the CT from the coded data. The CU decoding unit 3022 decodes the CU from the coded data. When a prediction error is included in the TU, the TU decoding unit 3024 decodes QP update information (quantization correction value) and quantized prediction error (residual_coding) from the coded data.

[0072] The TU decoding unit 3024 decodes the QP update information and the quantized prediction error from the coded data when the mode is other than the skip mode (skip_mode==0). More specifically, when skip_mode==0, the TU decoding unit 3024 decodes the flag cu_cbp indicating whether or not the current block includes a quantized prediction error, and decodes the quantized prediction error when cu_cbp is 1. When cu_cbp does not exist in the coded data, it is derived as 0.

[0073] The predicted image generating unit 308 includes an inter predicted image generating unit 309 and an intra predicted image generating unit 310 .

[0074] The prediction parameter derivation unit 320 includes an inter prediction parameter derivation unit 303 and an intra prediction parameter derivation unit 304 .

[0075] In the following, an example will be described in which CTUs and CUs are used as processing units, but this is not limiting and processing may be performed in sub-CU units. Alternatively, CTUs and CUs may be read as blocks, and sub-CUs as sub-blocks, and processing may be performed in block or sub-block units.

[0076] The entropy decoding unit 301 performs entropy decoding on the coded data Te input from the outside, and decodes each code (syntax element).

[0077] The entropy decoding unit 301 outputs the decoded code to the parameter decoding unit 302. The decoded code is, for example, a prediction mode predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX, amvr_mode, etc. Control of which code to decode is performed based on an instruction from the parameter decoding unit 302.

[0078] (Basic Flow) FIG. 8 is a flowchart illustrating the general operation of the image decoding device 31.

[0079] (S1100: Decode parameter set information) The header decoder 3020 decodes parameter set information such as VPS, SPS, and PPS from the coded data.

[0080] (S1200: Decode slice information) The header decoding unit 3020 decodes the slice header (slice information) from the coded data.

[0081] Thereafter, the image decoding device 31 repeats the processes from S1300 to S5000 for each CTU included in the current picture, thereby deriving a decoded image of each CTU.

[0082] (S1300: Decode CTU information) The CT information decoding unit 3021 decodes the CTU from the encoded data.

[0083] (S1400: Decode CT information) The CT information decoding unit 3021 decodes the CT from the encoded data.

[0084] (S1500: CU Decoding) The CU decoding unit 3022 performs S1510 and S1520 to decode the CU from the encoded data.

[0085] (S1510: Decode CU information) The CU decoding unit 3022 decodes CU information, prediction information, the TU split flag split_transform_flag, the CU residual flags cbf_cb, cbf_cr, cbf_luma, and the like from the encoded data.

[0086] (S1520: TU information decoding) When a TU includes a prediction error, the TU decoding unit 3024 decodes the QP update information, the quantization prediction error, and the transform index mts_idx from the coded data. Note that the QP update information is a difference value from the quantization parameter predicted value qPpred, which is a predicted value of the quantization parameter QP.

[0087] (S2000: Generate predicted image) The predicted image generation unit 308 generates a predicted image for each block included in the current CU based on prediction information.

[0088] (S3000: Inverse quantization and inverse transform) The inverse quantization and inverse transform unit 311 executes inverse quantization and inverse transform processing on each TU included in the target CU.

[0089] (S4000: Decoded image generation) The addition unit 312 generates a decoded image of the target CU by adding the predicted image supplied from the predicted image generation unit 308 and the prediction error supplied from the inverse quantization and inverse transform unit 311.

[0090] (S5000: Loop filter) The loop filter 305 applies a loop filter such as a deblocking filter, SAO, or ALF to the decoded image to generate a decoded image.

[0091] (Configuration of Inter Prediction Parameter Derivation Unit) FIG. 9 is a schematic diagram showing the configuration of the inter prediction parameter derivation unit 303 according to this embodiment. The inter prediction parameter derivation unit 303 derives inter prediction parameters by referring to prediction parameters stored in the prediction parameter memory 307, based on the syntax elements input from the parameter decoding unit 302. The inter prediction parameter derivation unit 303 also outputs the inter prediction parameters to the inter prediction image generation unit 309 and the prediction parameter memory 307. The inter prediction parameter derivation unit 303 and its internal elements, namely, the AMVP prediction parameter derivation unit 3032, the merge prediction parameter derivation unit 3036, the affine prediction unit 30372, the MMVD prediction unit 30373, the GPM unit 30377, the DMVR unit 30537, and the MV addition unit 3038, are means common to the image encoding device and the image decoding device, and may therefore be collectively referred to as a motion vector derivation unit (motion vector derivation device).

[0092] The scale parameter derivation unit 30378 derives the horizontal scaling ratio of the reference picture, RefPicScale[i][j][0], the vertical scaling ratio of the reference picture, RefPicScale[i][j][1], and RefPicIsScaled[i][j] indicating whether the reference picture has been scaled, as follows, where i indicates whether the reference picture list is an L0 list or an L1 list, and j is the value of the L0 reference picture list or the L1 reference picture list:

[0093] RefPicScale[i][j][0] = ((fRefWidth << 14)+(PicOutputWidthL >> 1)) / PicOutputWidthL RefPicScale[ i ][ j ][ 1 ] = ((fRefHeight << 14)+(PicOutputHeightL >> 1)) / PicOutputHeightL RefPicIsScaled[i][j] = (RefPicScale[i][j][0] != (1<<14)) || (RefPicScale[i][j][1] != (1<<14)) Here, the variable PicOutputWidthL is the value used when calculating the horizontal scaling ratio when a picture to be coded is referenced, and is calculated by subtracting the left and right offset values ​​from the number of horizontal luminance pixels of the picture to be coded. The variable PicOutputHeightL is a value used when calculating the vertical scaling ratio when a picture to be coded is referenced, and is calculated by subtracting the top and bottom offset values ​​from the vertical luminance pixel count of the picture to be coded. The variable fRefWidth is the value of PicOutputWidthL of the reference picture with reference list value j in list i, and the variable fRefHight is the value of PicOutputHeightL of the reference picture with reference picture list value j in list i.

[0094] The scaling ratio (RefPicScale) is smaller than the reference scaling ratio of 1x (2^14 = 16384) when enlarging, and larger than the 1x scaling ratio when reducing. Therefore, it is the intuitive reciprocal of the scale. It expresses, with an accuracy of 2^14, how many pixels of movement on the input image (fRefWidth, fRefHeight) corresponds to a movement of one pixel on the output image (PicOutputWidthL, PicOutputHeightL).

[0095] In this specification, the scale of output image size / input image size is called a scaling factor (scaleFactor) scale_factor, and the scale of input image size / output image size is called a scaling ratio (scalingRatio) RefPicScale.

[0096] If the precision of scaleFactor is 1 and the precision of scalingFactor is 1 / (2^14), then the following relationship holds:

[0097] scaleFactor=(1<<14) / scalingRatio scalingRatio=(1<<14) / scaleFactor When affine_flag is 1, that is, when the affine prediction mode is indicated, the affine prediction unit 30372 derives inter prediction parameters in units of subblocks.

[0098] When mmvd_flag is 1, that is, when the MMVD prediction mode is indicated, the MMVD predictor 30373 derives inter prediction parameters from the merge candidates and difference vectors derived by the merge prediction parameter derivation unit 3036 .

[0099] If gpm_flag is 1, that is, indicates Geometric Partitioning Mode, the GPM unit 30377 derives the GPM parameters.

[0100] If merge_flag is 1, that is, if it indicates merge prediction mode, merge_idx is derived and output to the merge prediction parameter derivation unit 3036 .

[0101] When merge_flag is 0, that is, when it indicates the AMVP prediction mode, the AMVP prediction parameter derivation unit 3032 derives mvpLX from inter_pred_idc, refIdxLX, or mvp_LX_idx.

[0102] (MV Addition Unit) The MV addition unit 3038 adds the derived mvpLX and mvdLX to derive mvLX.

[0103] (Affine prediction unit) The affine prediction unit 30372 1) derives motion vectors for two control points CP0 and CP1 or three control points CP0, CP1, and CP2 of the target block, 2) derives affine prediction parameters for the target block, and 3) derives motion vectors for each sub-block from the affine prediction parameters.

[0104] (Merge Prediction) The merge prediction parameter derivation unit 3036 derives prediction parameters of the current block using prediction parameters (mvLX, refIdxLX, etc.) of blocks spatially or temporally neighboring the current block.

[0105] (DMVR) Next, the DMVR unit 30375 performs DMVR (Decoder-side Motion Vector Refinement) processing. When merge_flag is 1 or the skip flag skip_flag is 1 for the target CU, the DMVR unit 30375 corrects the motion vector mvLX of the target CU. Specifically, when the prediction parameters derived by the merge prediction unit 30374 are bi-predictive, the DMVR unit 30375 corrects the mvLX using a predicted image derived from two reference pictures and the motion vector. The corrected mvLX is supplied to the inter-prediction image generation unit 309.

[0106] (AMVP Prediction) The AMVP prediction parameter derivation unit 3032 selects the motion vector mvpListLX[mvp_LX_idx] indicated by mvp_LX_idx from among the prediction vector candidates as mvpLX, and outputs it to the MV addition unit 3038.

[0107] (MV Addition Unit) The MV addition unit 3038 calculates mvLX by adding the mvpLX input from the AMVP prediction parameter derivation unit 3032 and the decoded mvdLX. The addition unit 3038 outputs the calculated mvLX to the inter predicted image generation unit 309 and the prediction parameter memory 307.

[0108] mvLX[0] = mvpLX[0] + mvdLX[0] mvLX[1] = mvpLX[1] + mvdLX[1] The loop filter 305 is a filter provided in the encoding loop that removes blocking noise and ringing noise and improves image quality. The loop filter 305 applies filters such as a deblocking filter, sample adaptive offset (SAO), and adaptive loop filter (ALF) to the decoded image of the CU generated by the adder 312.

[0109] The reference picture memory 306 stores the decoded image of the CU in a predetermined location for each current picture and current CU.

[0110] The prediction parameter memory 307 stores prediction parameters at a predetermined location for each CTU or CU. Specifically, the prediction parameter memory 307 stores parameters decoded by the parameter decoding unit 302, parameters derived by the prediction parameter derivation unit 320, and the like.

[0111] The predicted image generation unit 308 receives as input the parameters derived by the prediction parameter derivation unit 320. The predicted image generation unit 308 also reads a reference picture from the reference picture memory 306. The predicted image generation unit 308 generates a predicted image of a block or sub-block using the parameters and a reference picture (reference picture block) in the prediction mode indicated by predMode. Here, a reference picture block is a set of pixels on a reference picture (usually rectangular, and therefore referred to as a block), and is an area referenced to generate a predicted image.

[0112] (Inter-prediction image generation unit 309) When predMode indicates inter-prediction mode, the inter-prediction image generation unit 309 generates a prediction image of a block or sub-block by inter-prediction using the inter-prediction parameters and reference picture input from the inter-prediction parameter derivation unit 303.

[0113] 10 is a schematic diagram showing the configuration of an inter-prediction image generation unit 309 included in the prediction image generation unit 308 according to this embodiment. The inter-prediction image generation unit 309 includes a motion compensation unit (prediction image generation device) 3091 and a synthesis unit 3095. The synthesis unit 3095 includes an intra-inter synthesis unit 30951, a GPM synthesis unit 30952, a BDOF unit 30954, and a weighted prediction unit 3094.

[0114] (Motion Compensation) The motion compensation unit 3091 (interpolated image generation unit 3091) generates an interpolated image (motion-compensated image) by reading a reference block from the reference picture memory 306 based on the inter-prediction parameters (predFlagLX, refIdxLX, mvLX) input from the inter-prediction parameter derivation unit 303. The reference block is a block located at a position shifted by mvLX from the position of the current block on the reference picture RefPicLX specified by refIdxLX. Here, if mvLX does not have integer precision, an interpolated image is generated by applying a filter called a motion compensation filter, which is used to generate pixels at decimal positions.

[0115] The motion compensation unit 3091 has a function of scaling the interpolated image according to the horizontal scaling ratio RefPicScale[i][j][0] and the vertical scaling ratio RefPicScale[i][j][1] of the reference picture derived by the scale parameter derivation unit 30378. The above processing may be performed by the NN filter unit 611 included in the motion compensation unit 3091, and further by the interpolation unit 6114.

[0116] The synthesis unit 3095 includes an IntraInter synthesis unit 30951, a GPM synthesis unit 30952, a weighted prediction unit 3094, and a BDOF unit 30954.

[0117] (IntraInter Synthesis Processing) The intrainter synthesis unit 30951 generates a predicted image by weighting the inter predicted image and the intra predicted image.

[0118] (GPM Synthesis Processing) The GPM synthesis unit 30952 generates a predicted image using the above-mentioned GPM.

[0119] (BDOF Prediction) In bi-prediction mode, the BDOF unit 30954 generates a predicted image by referring to two predicted images (a first predicted image and a second predicted image) and a gradient correction term.

[0120] (Weighted Prediction) The weighted prediction unit 3094 performs weighted prediction from the interpolated image PredLX to generate a predicted image pbSamples of the block.

[0121] When predMode indicates an intra prediction mode, the intra prediction image generation unit 310 performs intra prediction using the intra prediction parameters input from the intra prediction parameter derivation unit 304 and reference pixels read from the reference picture memory 306 .

[0122] The inverse quantization and inverse transform unit 311 inverse quantizes the quantized transform coefficients input from the parameter decoding unit 302 to obtain transform coefficients.

[0123] The adder 312 generates a decoded image for the block by adding, for each pixel, the predicted image for the block input from the predicted image generation unit 308 and the prediction error input from the inverse quantization and inverse transform unit 311. The adder 312 stores the decoded image for the block in the reference picture memory 306 and also outputs it to the loop filter 305.

[0124] The inverse quantization and inverse transform unit 311 inverse quantizes the quantized transform coefficients input from the parameter decoding unit 302 to obtain transform coefficients.

[0125] The adder 312 generates a decoded image for the block by adding, for each pixel, the predicted image for the block input from the predicted image generation unit 308 and the prediction error input from the inverse quantization and inverse transform unit 311. The adder 312 stores the decoded image for the block in the reference picture memory 306 and also outputs it to the loop filter 305.

[0126] (Configuration of Image Encoding Device) Next, the configuration of the image encoding device 11 according to this embodiment will be described. Fig. 11 is a block diagram showing the configuration of the image encoding device 11 according to this embodiment. The image encoding device 11 includes a prediction image generation unit 101, a subtraction unit 102, a transformation / quantization unit 103, an inverse quantization / inverse transformation unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference image storage unit, frame memory) 109, a coding parameter determination unit 110, a parameter encoding unit 111, a prediction parameter derivation unit 120, and an entropy encoding unit 104.

[0127] The predicted image generation unit 101 generates a predicted image for each CU. The predicted image generation unit 101 includes the inter predicted image generation unit 309 and the intra predicted image generation unit 310, which have already been described, and therefore further description thereof will be omitted.

[0128] The subtraction unit 102 generates a prediction error by subtracting the pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 from the pixel values ​​of the image T. The subtraction unit 102 outputs the prediction error to the transformation / quantization unit 103.

[0129] The transform / quantization unit 103 calculates transform coefficients by frequency transforming the prediction errors input from the subtraction unit 102, and derives quantized transform coefficients by quantizing them. The transform / quantization unit 103 outputs the quantized transform coefficients to the parameter coding unit 111 and the inverse quantization / inverse transform unit 105.

[0130] The inverse quantization and inverse transform unit 105 is the same as the inverse quantization and inverse transform unit 311 (FIG. 7) in the image decoding device 31, and therefore a description thereof will be omitted. The calculated prediction error is output to the adder .

[0131] The parameter coding unit 111 includes a header coding unit 1110, a CT information coding unit 1111, and a CU coding unit 1112 (prediction mode coding unit). The CU coding unit 1112 further includes a TU coding unit 1114. The following describes an outline of the operation of each module.

[0132] The header encoding unit 1110 performs encoding processing of parameters such as filter information, header information, division information, prediction information, and quantized transform coefficients.

[0133] The CT information encoding unit 1111 encodes QT, MT (BT, TT) division information and the like.

[0134] The CU encoding unit 1112 encodes CU information, prediction information, division information, and the like.

[0135] When a prediction error is included in a TU, the TU encoding unit 1114 encodes the QP update information and the quantized prediction error.

[0136] The CT information encoding unit 1111 and the CU encoding unit 1112 supply syntax elements such as inter-prediction parameters (predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX), intra-prediction parameters (intra_luma_mpm_flag, intra_luma_mpm_idx, intra_luma_mpm_reminder, intra_chroma_pred_mode), and quantized transform coefficients to the parameter encoding unit 111.

[0137] The entropy coding unit 104 receives the quantized transform coefficients and coding parameters (division information, prediction parameters) from the parameter coding unit 111. The entropy coding unit 104 entropy codes these to generate and output coded data Te.

[0138] The prediction parameter derivation unit 120 is a means including an inter-prediction parameter coding unit 112 and an intra-prediction parameter coding unit 113, and derives intra-prediction parameters and intra-prediction parameters from the parameters input from the coding parameter determination unit 110. The derived intra-prediction parameters and intra-prediction parameters are output to the parameter coding unit 111.

[0139] (Configuration of Inter Prediction Parameter Encoding Unit) The inter prediction parameter encoding unit 112 includes a parameter encoding control unit 1121 and an inter prediction parameter derivation unit 303. The inter prediction parameter derivation unit 303 has the same configuration as the image decoding device. The parameter encoding control unit 1121 includes a merge index derivation unit 11211 and a vector candidate index derivation unit 11212.

[0140] The merge index derivation unit 11211 derives merge candidates and the like, and outputs them to the inter prediction parameter derivation unit 303. The vector candidate index derivation unit 11212 derives predicted vector candidates and the like, and outputs them to the inter prediction parameter derivation unit 303 and the parameter coding unit 111.

[0141] (Configuration of Intra Prediction Parameter Encoding Unit 113) The intra prediction parameter encoding unit 113 includes a parameter encoding control unit 1131 and an intra prediction parameter derivation unit 304. The intra prediction parameter derivation unit 304 has the same configuration as the image decoding device.

[0142] The parameter coding control unit 1131 derives IntraPredModeY and IntraPredModeC. Furthermore, it determines intra_luma_mpm_flag by referring to mpmCandList[ ]. These prediction parameters are output to the intra prediction parameter derivation unit 304 and the parameter coding unit 111.

[0143] However, unlike the image decoding device, the inputs to the inter prediction parameter derivation unit 303 and the intra prediction parameter derivation unit 304 are the coding parameter determination unit 110 and the prediction parameter memory 108 , and they are output to the parameter coding unit 111 .

[0144] The adder 106 generates a decoded image by adding, for each pixel, the pixel values ​​of the predicted block input from the predicted image generation unit 101 and the prediction errors input from the inverse quantization and inverse transform unit 105. The adder 106 stores the generated decoded image in a reference picture memory 109.

[0145] The loop filter 107 performs deblocking filtering, SAO, and ALF on the decoded image generated by the adder 106. Note that the loop filter 107 does not necessarily have to include the above three types of filters, and may be configured with only a deblocking filter, for example.

[0146] The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in a predetermined location for each current picture and CU.

[0147] The reference picture memory 109 stores the decoded image generated by the loop filter 107 at a predetermined location for each current picture and CU.

[0148] The coding parameter determination unit 110 selects one set of coding parameters from among multiple sets of coding parameters. The coding parameters are the above-mentioned QT, BT, or TT division information, prediction parameters, or parameters to be coded that are generated in relation to these. The predicted image generation unit 101 generates a predicted image using these coding parameters.

[0149] The coding parameter determination unit 110 calculates an RD cost value indicating the magnitude of the information amount and the coding error for each of the multiple sets. The RD cost value is, for example, the sum of the code amount and the value obtained by multiplying the squared error by a coefficient λ. The code amount is the information amount of the coded data Te obtained by entropy coding the quantization error and the coding parameters. The squared error is the sum of the squares of the prediction errors calculated by the subtraction unit 102. The coefficient λ is a preset real number greater than zero. The coding parameter determination unit 110 selects the set of coding parameters that minimizes the calculated cost value. The coding parameter determination unit 110 outputs the determined coding parameters to the parameter coding unit 111 and the prediction parameter derivation unit 120.

[0150] Note that a portion of the image encoding device 11 and the image decoding device 31 in the above-described embodiments, such as the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the predicted image generation unit 308, the inverse quantization / inverse transform unit 311, the adder 312, the prediction parameter derivation unit 320, the predicted image generation unit 101, the subtractor 102, the transform / quantization unit 103, the entropy encoding unit 104, the inverse quantization / inverse transform unit 105, the loop filter 107, the encoding parameter determination unit 110, the parameter encoding unit 111, and the prediction parameter derivation unit 120, may be implemented by a computer. In this case, a program for implementing this control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read and executed by a computer system. Note that the "computer system" referred to here refers to a computer system built into either the image encoding device 11 or the image decoding device 31, and includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0151] Furthermore, part or all of the image encoding device 11 and the image decoding device 31 in the above-described embodiments may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image encoding device 11 and the image decoding device 31 may be individually implemented as a processor, or part or all of them may be integrated into a processor. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0152] One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like are possible within the scope that does not deviate from the gist of the present invention.

[0153] (Configuration Example of NN Filter Unit) Fig. 12 is a diagram showing the configuration of the neural network filter unit (NN filter unit 611). The NN filter unit 611 is a means for performing filtering processing on an input image using a neural network, and simultaneously performs reduction / enlargement at a constant magnification or a rational number magnification. The NN filter unit may be used as a loop filter applied to a reference image, a process for generating a predicted image from a reference image, or a post-filter for an output image.

[0154] 12(a) shows an example of the configuration of a loop filter. The loop filter 305 of the video decoding device (loop filter 107 of the video encoding device) includes an NN filter unit 611. The NN filter unit 611 filters an image stored in the reference picture memory 306 / 106 and stores the filtered image in the reference picture memory 306 / 106. As already described, the loop filter may include a DF, ALF, SAO, bilateral filter, etc.

[0155] 12(b) shows an example configuration of a predicted image generation unit. The inter-prediction image generation unit 309 of a video decoding device or a video encoding device includes an NN filter unit 611. The NN filter unit 611 reads an image from the reference picture memory 306 / 106 and filters it to generate a predicted image. The predicted image may be further used for CIIP prediction, GPM prediction, weighted prediction, or BDOF in a synthesis unit 3095, or may be output directly to an adder 312 (or a subtractor 102 in an encoding device).

[0156] 12(c) shows an example of the configuration of a post-filter. The post-processing unit 61 after the video decoding device includes an NN filter unit 611. When outputting an image from the reference picture memory 306, the NN filter unit 611 processes the image and outputs it to the outside. The output image may be displayed, written to a file, re-encoded (transcoded), transmitted, etc.

[0157] (Specific example of NN filter unit) Figure 13 shows an example of the neural network structure of the NN filter unit 611. The NN filter unit 611 is composed of a neural network (feature extraction structure) 1700 and a neural network OutputBlock 1701. The neural network 1700 performs processing at full size. The OutputBlock 1701 may also perform resolution conversion. The neural network 1700 further comprises an InputBlock 1702 and multiple Residual Blocks (ResBlock) 1703. The InputBlock 1702 is composed of a convolution layer (Conv) and an activation layer (Act). The ResBlock 1703 is composed of Conv and Act. Skip connections are a structure that learns the difference between feature vectors, avoiding gradient vanishing in deeper layers and improving learning efficiency. The OutputBlock 1701 is composed of Conv, a resampling unit (Resample) 1704, and a feature extraction layer 1705 (Act, Conv). Resample 1704 may convert the width and height of the feature vector output from feature extraction structure 1700 to the output width and height. Resample 1704 may be bilinear, bicubic, PixelShuffle, deconvolution, etc. Note that Resample 1704 may change the enlargement rate using scale_factor. For example, when using PixelShuffler, after increasing the number of channels by scale_factor*scale_factor, the channels are multiplied by 1 / (scale_factor*scale_factor) and the width and height are multiplied by scale_factor. When using TransposedConvolution, it is possible to enlarge by scale_factor as follows:

[0158] TransposedConvolution (kernel = scale_factor, stride = scale_factor) Furthermore, the feature extraction layer 1705 generates an output image from the enlarged feature vector that has the same channels, width, and height as the input image.

[0159] (Regarding phase and motion vector by NN filter unit) For example, when the top left coordinate of the block to be processed (picture to be processed) is (xSb, ySb), the NN filter unit 611 (interpolation unit 6114) may derive the top left coordinate (xRef, yRef) of the input image using the following formula:

[0160] xRef = xSb * scalingRatio[0] yRef = ySb * scalingRatio[1] Furthermore, the integer coordinates (xIntL, yIntL) and the phase (xFracL, yFracL), which is the decimal coordinate, may be derived as follows.

[0161] xIntL = xRef >> 14 yIntL = yRef >> 14 xFracL = xRef & ((1<<14)-1) yFracL = yRef & ((1<<14)-1) Also, when using 1 / 16 precision motion vectors (refMxLX[0], refMvLX[1]) in the NN filter unit 611, it is possible to derive intermediate coordinates (refxL, refyL) in 1 / 16 units and then derive the integer coordinates and phase below.

[0162] refxSbL = ((xSb<<4) + refMvLX[0]) * scalingRatio[0] refySbL = ((ySb<<4) + refMvLX[1]) * scalingRatio[1] refxL = refxSbL >>14 refyL = refySbL >>14 xIntL = refxL >> 4 yIntL = refxL >> 4 xFracL = refxL & 15 yFracL = refyL & 15 Furthermore, using the intra-block coordinates (xL, yL), the intermediate coordinates may be derived as follows:

[0163] 14 , the motion vectors (refMxLx[0], refMvLx[1]) are shifted by the NN filter unit 611, so that the predicted image generation unit 309 can also generate a predicted image by taking into account the motion of the object on the reference picture. refMvLX[1]) or the phase component of the motion vector (xFracL, yFracL). The phase component is also called the fractional component.

[0164] (Example of Processing According to Magnification Ratio) Figure 14 shows an example of the configuration of the NN filter unit 611. The NN filter unit 611 is composed of a first equal resolution processing unit 6111, a resolution conversion processing unit 6112, and a second equal resolution processing unit 6113. The first equal resolution processing unit 6111 is a neural network that outputs an image of output size (W, H) for an image of input size (W, H). It is configured using, for example, Conv, Act, and addition for difference. Here, "equal resolution" means that the width and height of the input and output are equal, and the number of channels may be changed. Furthermore, a so-called UNET structure (a structure with paths for shrinking and expanding in addition to a fixed-size structure) may be used to use intermediate networks with widths and heights smaller than those of the input and output in a branched network. Note that the expression C x H x W in the figure indicates a tensor (three-dimensional array) with the number of channels C, height H, and width W. Furthermore, the expression B x C x H x W indicates a tensor (four-dimensional array) with the number of batches B, the number of channels C, height H, and width W. It is preferable that the number of input channels C1 and the number of output channels C2 of the first equal resolution processor 6111 are C2>C1, and the number of channels is increased to obtain good characteristics.

[0165] The resolution conversion processing unit 6112 is a neural network that outputs an image of output size (m*W, n*H) obtained by multiplying an image of input size (W, H) by m times in width and n times in height.

[0166] The second equal resolution processor 6113 is a neural network that outputs an image of output size (m*W, n*H) for an image of input size (m*W, n*H). It is preferable that the number of input channels C3 and the number of output channels C4 of the second equal resolution processor 6113 are C3>C4.

[0167] (Example of Scale Adaptation Configuration) The NN filter unit 611 may switch the parameters of the neural network used in the second equal resolution processing unit 6113, such as weights and biases, according to the values ​​of the magnification ratios m and n. For example, scaleIdx may be derived according to m and n, and the base model parameters BaseModel[scaleIdx][i] may be copied to Model according to scaleIdx and index i. Then, the derived model may be used to perform neural network processing.

[0168] if (m >= 2) scaleIdxM = 2 else if (m >= 1) scaleIdxM = 1 else scaleIdxM = 0 if (n >= 2) scaleIdxN = 2 else if (n >= 1) scaleIdxN = 1 else scaleIdxN = 0 for (i in set of parameter indices of second equal resolution processing unit 6113) Model[i] = BaseModel[scaleIdxM][scaleIdxN][i] Furthermore, the horizontal scaling factor m and the vertical scaling factor n may be combined and derived as follows:

[0169] if (m >= 2 || n >= 2) scaleIdx = 2 else if (m >= 1 || n >= 1) scaleIdx = 1 else scaleIdx = 0 for (i in set of parameter indexes of the second equal resolution processor 6113) Model[i] = BaseModel[scaleIdx][i] Alternatively, a configuration in which only the bias is switched may be used. Below, Bias and BaseBias refer to the bias portion of the NN parameters.

[0170] if (m >= 2 || n >= 2) scaleIdx = 2 else if (m >= 1 || n >= 1) scaleIdx = 1 else scaleIdx = 0 for (i in set of bias indices of the second equal resolution processor 6113) Bias[i] = BaseBias[scaleIdx][i] In the above, parameters were switched using m and n, that is, scalingFactor, but it is also possible to switch using scaleRatio, which is a type of inverse of scalingFactor. Note that the layers for which scaling is adapted do not need to be all layers of the NN filter. In the above, the parameters of the second equal resolution processor are adapted.

[0171] The above configuration has the advantage of being able to output a suitable image by switching neural network parameters according to the enlargement ratio. In addition, by switching parameters after enlarging the conversion size, the memory size for switching parameters can be kept to a minimum.

[0172] (Another example of a scale adaptive configuration) Furthermore, the scales m and n (NNScaleFactor[0], NNScaleFactor[1]) or (NNScalingRatio[0], NNScalingRatio[1]) may be input as input to the second equal resolution processing unit 6113, and adaptive processing may be performed. For example, a configuration may be used in which m and n are input as one of the channels of the input image. Also, a configuration may be used in which the scales are concatenated (concat) to a channel of an intermediate layer.

[0173] (Example of Phase Adaptation Configuration) The NN filter unit 611 may switch the parameters of the neural network used in the second equal resolution processing unit 6113, such as the weight and bias, depending on the phase, that is, the values ​​of xFracL and yFracL. For example, scaleIdx may be derived depending on xFracL and yFracL, and the base model parameters BaseModel[scaleIdx][i] may be copied to Model depending on scaleIdx and index i. Then, the derived model may be used to perform neural network processing.

[0174] for (i in set of parameter indexes of the second equal resolution processor 6113) Model[i] = BaseModel[xFracL][yFracL][i] Alternatively, a configuration in which only the bias is switched may be used. Below, Bias and BaseBias refer to the bias portion of the NN parameters.

[0175] for (i in set of bias indexes of the second equal resolution processing unit 6113) Bias[i] = BaseBias[xFracL][yFracL][i] With the above configuration, the parameters of the neural network are switched depending on the phase, which has the effect of outputting a suitable image even if the phase of the upper left coordinate of the input image is not an integer position due to a motion vector, scaling factor, etc. Furthermore, by switching the parameters after enlarging the transformation size, the memory size for switching parameters can be kept to a minimum.

[0176] (Another example of a phase adaptive configuration) Furthermore, the xFracL and yFracL may be input as inputs to the second equal resolution processing unit 6113 for adaptive processing. For example, xFracL and yFracL may be input as channels of the input image. Alternatively, the xFracL and yFracL may be concatenated into a channel of the intermediate layer. As described above, by inputting the phase (xFracL, yFracL) into the network, it is possible to generate an appropriate resolution-converted image even when the upper left coordinate is not an integer position. Here, the pixel position where (xFrac, yFracL) = (0, 0) is called an integer position.

[0177] (Resolution Conversion Using Both NN and Interpolation Unit) Fig. 15 is a diagram showing an example of the configuration of the NN filter unit 611. The NN filter unit 611 is made up of a first equal resolution processing unit 6111, an interpolation unit 6114, and a second equal resolution processing unit 6113. Since 6111 and 6113 have already been explained, their explanation will be omitted.

[0178] The interpolation unit 6114 (motion compensation unit 3091, interpolated image generation unit 3091) performs scaling processing using a linear filter. It may also use a motion vector to determine pixel values ​​at sub-pel positions. The number of taps available for the linear filter may be 2 (bi-linear), 3 (bi-cubic, Lanczos), 4, 6, 8, 10, 12, or the like. The precision of the sub-pel positions may also be 1 / 8 or 1 / 16.

[0179] As shown by the dotted lines in FIG. 15 , the interpolation unit 6114 may receive a motion vector (refMxLX[0], refMvLX[1]) or a phase component (xFracL, yFracL) of the motion vector. The phase component is also called a decimal point component. As already explained, the interpolation unit 6113 may switch the parameters of the neural network according to the scale (scaling magnification or scaling ratio) or the phase component (decimal point component) of the upper left coordinate of the input image. Alternatively, the scale or phase component may be input to the neural network.

[0180] The interpolator 6114 may support rational scaling factors as follows:

[0181] For example, the interpolation unit 6114 derives the pixel position (refXL, refYL) with 1 / 16 pixel accuracy when the upper left coordinates of the block to be processed are (xSb, ySb) and the motion vector is (refMxLX[0], refMvLX[1]) using the following formula: refxSbL = (((xSb - (SubWidthC * pps_scaling_win_left_offset)) << 4) + refMvLX[0]) * scalingRatio[0] refxL = ((Sign(refxSbL) * ((Abs(refxSbL) + 128) >> 8) + xL * ((scalingRatio[0] + 8) >> 4)) + fRefLeftOffset + 32) >> 6 refySbL = (((ySb - (SubHeightC * pps_scaling_win_top_offset)) << 4) + refMvLX[1]) * scalingRatio[1] refyL = ((Sign(refySbL) * ((Abs(refySbL) + 128) >> 8) + yL * ((scalingRatio[1] + 8) >> 4)) + fRefTopOffset + 32) >> 6 The integer position (xIntL, yIntL) and deviation from the integer position (xFracL, yFracL) are calculated as follows.

[0182] xIntL = refxL >> 4 yIntL = refyL >> 4 xFracL = refxL & 15 yFracL = refyL & 15 Next, the interpolation unit 6114 performs the following process depending on whether xFracL and yFracL indicate integer positions.

[0183] If (xFracL==0 and yFracL=0) and (scalingRatio[0]<20481 and scalingRatio[1]<20481), predSampleLXL is derived by the following process. This is the case when the phase is an integer position and the scale factor exceeds a predetermined factor (e.g., 1.25).

[0184] predSampleLXL = refPicLXL[xInt(OFT)][yInt(OFT)] << shift3 Otherwise, if yFracL==0 and scalingRatio[1]<20481, derive predSampleLXL using the following horizontal filter: This is the case when the vertical phase is an integer position and exceeds the specified scaling factor.

[0185] predSampleLXL = fLH[xFracL][i]*refPicLX[xInt(i)][yInt(i)] >> shift1 Otherwise (non-integer position or less than a specified scale factor), if xFracL==0 and scalingRatio[0]<20481, derive predSampleLXL using the following vertical filter: This is the case when the vertical phase is an integer position.

[0186] predSampleLXL = ΣfLH[yFracL][i]*refPicLX[xInt(OFT)][yInt(OFT)] >> shift1 Otherwise (non-integer position), horizontal filtering is performed using the interpolation filter coefficients selected by xFracL and (xIntL, yIntL) and its surrounding pixels. The intermediate image temp is derived by convolution of the filter coefficients and pixels. Furthermore, vertical filtering is performed using the intermediate image and the interpolation filter coefficients selected by yFracL to derive the image predSamplesLXL.

[0187] temp[n] = ΣfLH[yFracL][i]*refPicLX[xInt(i)][yInt(i)] >> shift1 predSampleLXL = ΣfLH[yFracL][i]*temp[i] >> shift2 Here, fLH is the linear filter coefficient, and Σ represents the sum for i=0..NTAP-1 (the number of taps can also be NTAP=8). shift1=Min(4, BitDepth-8), shift2=6, shift3=Max(2, 14-BitDepth), xInt(i)=xIntL+i-OFT, yInt(i)=yIntL+i-OFT. OFT=NTAP / 2-1. If NTAP=8, OFT=3. Note that even in cases where the number of taps is 2 or 3, such as bilinear or bicubic, you can set the coefficients farthest from the center to 0, prepare fLH for 8 taps, and process as NTAP=8. Alternatively, the luminance may be set to NTAP=8 and the color difference may be set to NTAP=4.

[0188] In the above process, scalingRatio[0] and scalingRatio[1] are variables that indicate the horizontal and vertical scales, respectively, and specify the above-mentioned RefPicScale[i][j][0] and RefPicScale[i][j][1]. The scaling factors are 2^14 / scalingRatio[0] and 2^14 / scalingRatio[1].

[0189] As described above, the NN filter unit 611 performs interpolation by linear processing that references four points using the interpolation unit 6114 (resolution conversion processing), and further, the output of the interpolation unit 6114 may be output as an image by the neural network of the second equal resolution processing unit 6113. This has the effect of easily enabling filter processing at an arbitrary magnification (rational magnification) even with a neural network that has difficulty in enlarging or reducing at an arbitrary magnification.

[0190] Furthermore, the feature image obtained by the first uniform resolution processing unit 6111 may be subjected to resolution conversion using a linear filter in the interpolation unit 6114, and then further processed by the second uniform resolution processing unit 6113. The first uniform resolution processing unit 6111 and the second uniform resolution processing unit 6113 perform neural network processing at uniform magnification. By providing the first uniform resolution processing unit 6111 before the interpolation unit 6114, it is possible to achieve an arbitrary magnification while achieving even higher image quality.

[0191] Furthermore, as described above, the parameters of the neural network of the second equal resolution processing unit 6113 are switched in accordance with the rational magnification, which has the effect of enabling a suitable image to be output. Also, by switching the parameters after enlarging the conversion size, the memory size for switching parameters can be kept to a minimum.

[0192] Furthermore, in the above example, the interpolation unit 6114 obtains an image at a position shifted by the motion vector (refMxLx[0], refMvLX[1]) and derives a position on the input image. This allows the predicted image generation unit 309 to generate a predicted image taking into account the movement of an object on the reference picture. Also, as a post-filter, it has the effect of converting the resolution of only an area shifted by a specific position in consideration of the so-called region of interest (Region of Interest) and outputting it. Furthermore, by providing a first equal resolution processing unit 6111 before the interpolation unit 6114, it has the effect of improving image quality.

[0193] 21 is a diagram showing an example of the configuration of the NN filter unit 611. The NN filter unit 611 is made up of a first equal resolution processing unit 6111, a horizontal resolution conversion unit 6111W, a horizontal second equal resolution processing unit 6113W, a vertical interpolation unit 6114H, and a vertical resolution conversion unit 6111H. 6111 and 6113 have already been explained, so their explanation will be omitted.

[0194] The horizontal resolution conversion unit 6111W and the vertical resolution conversion unit 6111H perform only horizontal resolution conversion or only vertical resolution conversion.

[0195] The horizontal second equal resolution processor 6113W and the vertical second equal resolution processor 6113H may use different network parameters (weighting coefficients, biases) depending on the scale (scaleFactor or scalingRatio), as in the second equal resolution processor 6113 already described in the scale adaptive configuration. However, the horizontal second equal resolution processor 6113W switches its network parameters depending on scalingRatio[0], and the vertical second equal resolution processor 6113H switches its network parameters depending on scalingRatio[1]. Alternatively, as shown in another example of the scale adaptive configuration, the horizontal scale scaleFactor[0] or scalingRatio[0] may be input to the horizontal second equal resolution processor 6113W, and the vertical scale scaleFactor[1] or scalingRatio[1] may be input to the vertical second equal resolution processor 6113W for adaptive processing.

[0196] The above configuration achieves the effect of switching or adapting the resolution conversion process according to the scaling factor, thereby achieving optimal resolution conversion processing according to the scaling factor. At the same time, it also achieves the effect of simplifying processing because there is no need to prepare network parameters for scaling Ratio[0] and scaling Ratio[1]. The horizontal second equal resolution processing unit 6113W and the vertical second equal resolution processing unit 6113H may share the same network parameters. Specifically, the image may be transposed vertically and horizontally (swapped horizontally and vertically) before the vertical second equal resolution processing unit 6113H, and then transposed again (swapped horizontally and vertically) after processing by 6113H. Furthermore, similar swapping may be performed before and after the vertical second equal resolution processing unit 6113H. In this way, using transposition allows optimal vertical and horizontal resolution conversion processing to be performed using the same network parameters.

[0197] 22 is a diagram showing an example of the configuration of the NN filter unit 611. The NN filter unit 611 is made up of a first equal resolution processing unit 6111, a horizontal interpolation unit 6114W, a horizontal second equal resolution processing unit 6113W, a vertical interpolation unit 6114H, and a vertical second equal resolution processing unit 6113H. Since 6111 and 6113 have already been explained, their explanation will be omitted.

[0198] The horizontal interpolation section 6114W and the vertical interpolation section 6114H are the separate type interpolation section 6114 already described, which perform only horizontal resolution conversion or only vertical resolution conversion.

[0199] The other processes are as explained in FIG.

[0200] According to the above, since the scaling is switched according to scalingRatio[0] or scalingRatio[1], the effect of performing the appropriate resolution conversion processing according to the scaling factor is achieved, but the effect of simplifying the processing is also achieved because there is no need to prepare network parameters for scalingRatio[0] and scalingRatio[1]. Similarly, by using transposition, it is possible to perform the appropriate vertical and horizontal resolution conversion processing using the same network parameters.

[0201] (Examples of integer resolution conversion and rational number resolution conversion processing) The resolution conversion processing unit 6112 already explained may be composed of an integer resolution conversion unit 6112I that performs scaling by an integer multiple, and a rational number resolution conversion unit 6112R that performs scaling by a rational number multiple. Specific examples of neural networks will be explained below.

[0202] FIG. 16 shows an example of the configuration of the integer resolution converter 6112I. 6112I enlarges an image of size (W, H) to an image of size (m*W, n*H). Here, m and n are the horizontal and vertical scale factors, respectively, and are integers equal to or greater than 1. 6112I is composed of a first reshape process, a first permute process, and a second reshape process. Both reshape and permute are also called pixel shufflers because they rearrange pixels. This process is also called channel-to-space because an image that has been multiplied by an integer (e.g., four times) in the channel direction is used for spatial expansion (e.g., two times horizontally and two times horizontally).

[0203] The first Reshape process converts a (B, n*m*C, H, W)-dimensional image SampleA into a (B, C, n, m, H, W)-dimensional image SampleB. This process corresponds to performing the following process on i0=0..B-1, i1=0..C-1, i2=0..n-1, i3=0..m-1, i4=0..H-1, i5=0..W-1.

[0204] SampleB[i0][i1][i2][i3][i4][i5] = SampleA[i0][i3*(C*n)+i2*(C)+i1][i4][i5] The first Permute process rearranges the order of the indices of the multidimensional array of the (B, C, n, m, H, W)-dimensional image SampleB to 0, 1, 4, 2, 5, 3. Since the indices correspond to (B=0, C=1, n=2, m=3, H=4, W=5), the order is 0=B, 1=C, 4=H, 2=n, 5=W, 3=m, that is, B, C, H, n, W, m. That is, this processing corresponds to performing the following processing for i0=0..B-1, i1=0..C-1, i2=0..n-1, i3=0..m-1, i4=0..H-1, i5=0..W-1.

[0205] SampleC[i0][i1][i4][i2][i5][i3] = SampleB[i0][i1][i2][i3][i4][i5] The second Reshape process converts the (B, C, H, n, W, m)-dimensional image SampleC into a (B, C, n*H, m*W)-dimensional image SampleD. This process corresponds to performing the following process on i0=0..B-1, i1=0..C-1, i2=0..H-1, i3=0..n, i4=0..W-1, i5=0..m-1.

[0206] SampleD[i0][i1][n*i2+i3][m*i4+i5] = SampleC[i0][i1][i2][i3][i4][i5] According to the above processing, integer extension processing can be easily performed by just rearranging.

[0207] 14, it is possible to provide a neural network including a first equal resolution processing unit 6111 that does not change the width and height, and a resolution conversion process that outputs width and height (mW, nH) from an input image whose width and height are (W, H) for integers W, H, m, and n. Therefore, the resolution conversion process can convert the number of channels of m*n*C into the number of channels C.

[0208] FIG. 17 is a diagram showing an example of the configuration of the rational resolution converter 6112R. The processing performed by the integer resolution converter 6112I is simple, but because it only uses an integer multiple of the number of channels for horizontal and / or vertical scaling, it has the problem of only being able to handle integer-multiple scaling. The configuration of 6112R supports rational-multiple scaling. 6112R converts an image of size (M*W, N*H) into an image of size (m*W, n*H). Here, m / M and n / N are the horizontal and vertical scale factors, respectively, and m, n, M, and N are integers greater than or equal to 1.

[0209] The resolution conversion unit 6112R includes a first resolution conversion process 6112D (reducing resolution conversion process) and a second resolution conversion process 6112I (enlarging resolution conversion process). The first resolution conversion process 6112D converts the width and height of an image from (MW, NH) to (W, H). The second resolution conversion process 6112I converts the width and height of the output of the resolution conversion unit 6112D from (W, H) to (mW, nH). Note that the second resolution conversion process 6112I may be the same as the integer resolution conversion unit 6112I.

[0210] 14, it is possible to provide a first equal resolution processing unit 6111 that does not change the width and height, and 6112R that includes resolution conversion processing that outputs width and height (mw, nh) from an input image whose width and height are (MW, NH) for integers M, N, W, H, m, and n. Therefore, the resolution conversion processing of 6112R above changes the number of channels from m*n*C to M*N*C. With this configuration, the high image quality achieved by 6111 and the easy processing by 6112R have the effect of enabling high-image-quality and easy resolution conversion by a rational number factor.

[0211] The first resolution conversion processing unit 6112D is made up of a third reshape processing unit, a second permute processing unit, and a fourth reshape processing unit. This processing unit is also called SpaceToChannel because it uses expansion in the space direction for expansion in the channel direction.

[0212] The third Reshape process converts the (B, n*m*C, N-H, M-W)-dimensional image SampleD_A into a (B, n*m*C, H, N, W, M)-dimensional image SampleD_B. This process corresponds to performing the following process on i0=0..B-1, i1=0..n*m*C-1, i2=0..N-1, i3=0..M-1, i4=0..H-1, i5=0..W-1:

[0213] SampleD_B[i0][i1][i4][i2][i5][i3] = SampleD_A[i0][i1][i4*N+i2][i5*M+i3] The second Permute process rearranges the order of the indices of the (B, n*m*C, H, N, W, M)-dimensional image SampleD_B to 0, 1, 3, 5, 2, 4. Since the indices correspond to (B=0, n*m*C=1, H=2, N=3, W=4, M=5), they are rearranged to B, n*m*C, N, M, H, W. This process corresponds to performing the following process on i0=B-1, i1=n*m*C-1, i2=H-1, i3=N-1, i4=W-1, i5=M-1.

[0214] SampleD_C[i0][i1][i3][i5][i2][i4] = SampleD_B[i0][i1][i2][i3][i4][i5] The fourth Reshape process converts the (B, n*m*C, N, M, H, W)-dimensional image Sample D_C into the (B, n*m*N*M*C, H, W)-dimensional image Sample D_D. This process corresponds to performing the following process on i0=0..B-1, i1=0..n*m*C-1, i2=0..N-1, i3=0..M-1, i4=0..H-1, i5=0..W-1.

[0215] SampleD_D[i0][i1][H*i2+i3][W*i4+i5] = SampleD_C[i0][i1][i2][i3][i4][i5] The above processing has the effect of making it possible to easily perform rational number scaling processing simply by rearranging the data.

[0216] The second resolution conversion processing unit 6112I has the same configuration as the integer resolution conversion unit 6112I already described, and therefore its description will be omitted. As described above, the first resolution conversion process includes a process of multiplying the number of channels by M*N, and the second resolution conversion process includes a process of multiplying the number of channels by 1 / (m*n).

[0217] The above process has the effect of easily performing scaling by a rational number by simply rearranging the data.

[0218] (Configuration that Prevents Phase Changes at Processing Block Boundaries) FIG. 18 shows the processing units of the NN filter unit 611 and rational resolution conversion unit 6112R. The processing units of the NN filter unit 611 and rational resolution conversion unit 6112R are block units (tile units), but if the blocks are set to an arbitrary size, a problem occurs in that the phase (non-integer components of the input image) changes. Specifically, if an image is enlarged by 1.5 times and processed in units of multiples of 64 at the output destination, the coordinates of the output destination are 0, 64, 128, while the coordinates of the input source are 0, 64 / 1.5 = 42.6 and 128 / 1.5 = 85.3. It is possible to process the input image using the coordinates of 0, 43, 85 in the NN filter unit 611, but a non-negligible change occurs at the block boundaries due to the phase shift.

[0219] When the scaleFactor is m / M or n / N, the NN filter unit 611 processes the X and Y coordinates on the input side as multiples of M and N. Similarly, the NN filter unit 611 processes the coordinates on the output side as multiples of m and multiples of n. Here, the top left coordinates (xS, yS) of the input image may be derived as follows:

[0220] xS = M*SX + SOX yS = N*SY + SOY where SX and SY are predetermined integers, and SOX and SOY are predetermined offsets.

[0221] Also, (xS, yS) may be derived using the top left coordinates (xD, yD) of the output as follows:

[0222] xS = m*DX + DOX yS = n*DY + DOY where DX and DY are predetermined integers, and DOX and DOY are predetermined offsets.

[0223] Furthermore, the input size (wS, hS) and output size (wD, hD) of the block to be processed may be derived as follows.

[0224] wS = M * DW hS = N * DH wD = M * DW hD = N * DH Also, when the NN filter unit 611 processes a block with (xC, yC) as the upper-left coordinates, the following equations may be used to derive the upper-left coordinates (xS, yS) of the input image and the upper-left coordinates (xD, yD) of the output image.

[0225] xS = ((xC * NNScalingRatio[0]) >> 14) / M * M yS = ((yC * NNScalingRatio[1]) >> 14) / N * N xD = xC / m * m yD = xC / n * n Here, the following (m, n) and (M, N) described later may be used.

[0226] m = NNScaleFactor[0] = Max(Ceil((PicOutputWidth << KK) / fRefWidth, 1)) n = NNScaleFactor[1] = Max(Ceil((PicOutputHeight << KK) / fRefHeight, 1)) M = 1 << KK N = 1 << KK As described above, by deriving the output coordinates of the processing block or the input image according to the scaleFactor or scalingRatio, it is possible to avoid inappropriate image changes that may occur in units of processing blocks.

[0227] (Configuration of Linear Filter and NN Resolution Conversion) FIG. 19 is a diagram showing an example of the configuration of the NN filter unit 611. Although the configuration of synthesizing linear processing and neural network processing has been described above, a more detailed example will be described here.

[0228] FIG. 19(a) is an example in which the NN filter unit 611 includes an interpolation unit 6114 and a rational resolution conversion unit 6112R. The interpolation unit 6114 can perform resolution conversion of a rational multiple, but by combining it with the rational resolution conversion unit 6112R, higher-quality conversion becomes possible. The scales of the interpolation unit 6114 and the rational resolution conversion unit 6112R can be derived as follows.

[0229] When the resolution of the target input image of the NN filter unit 611 is (fRefWidth, fRefHeight) and the resolution of the output image is (PicOutputWidthL, PicOutputHeightL), the scaleFactor(m, n) of the rational resolution conversion unit 6112R may be derived by the following formula. Here, by using Ceil, the decimal point precision is rounded up in the derivation of NNScaleFactor.

[0230] m = NNScaleFactor[0] = Max(Ceil((PicOutputWidth<<KK) / fRefWidth, 1) n = NNScaleFactor[1] = Max(Ceil((PicOutputHeight<<KK) / fRefHeight, 1) Or, it may be derived as follows.

[0231] m = NNScaleFactor[0] = Max(((PicOutputWidth<<KK) + fRefWidth - 1) / fRefWidth, 1) n = NNScaleFactor[1] = Max(((PicOutputHeight<<KK) + fRefHeight - 1) / fRefHeight, 1) M = 1<<KK N = 1< <kk kkは0以上の整数、0,1,2,3,4,..mであってもよい。kkは有理数解像度変換部6112rがサポートする有理数倍率の精度を示す。たとえば整数倍の場合、kk="0に相当し、有理数倍ではKK">It is equal to 1. When the scaleFactor is in steps of 1 / 2, 1 / 4, 1 / (1 << M), KK = 1, 2, M respectively.

[0232] Furthermore, the scalingRatio of the interpolation unit 6114 may be derived by the following formula.

[0233] InterpolateScalingRatio[0] = ((fRefWidth << (14 + KK)) + ((PicOutputWidth * NNPicFactor[0]) >> 1)) / (PicOutputWidth * NNPicFactor[0]) InterpolateScalingRatio[1] = ((fRefHeight << (14 + KK)) + ((PicOutputHeight * NNPicFactor[1]) >> 1)) / (PicOutputHeight * NNPicFactor[1]) In the above, the refPicScale[] which is the scalingRatio of the combined conversion of the rational resolution conversion unit 6112R and the interpolation unit 6114 is derived to satisfy the following.

[0234] RefPicScale[0] = ((fRefWidth << 14) + (PicOutputWidth >> 1)) / (PicOutputWidth) RefPicScale[1] = ((fRefHeight << 14) + (PicOutputHeight >> 1)) / (PicOutputHeight) Note that the scale of the interpolation unit 6114 may also be derived as follows as the scalingRatio instead of the scalingFactor. Here, the decimal point precision is truncated.

[0235] NNScalingRatio[0] = (fRefWidth / PicOutputWidth) << 14 NNScalingRatio[1] = (fRefHeight / PicOutputHeight) << 14 m = NNScaleFactor[0] = (1 << (14 + KK)) / NNScalingRatio[0] m = NNScaleFactor[1] = (1 << (14 + KK)) / NNScalingRatio[1] In this case, the scalingRatio of the interpolator 6114 is derived as follows:

[0236] InterpolateScalingRatio[0] = ((fRefWidth*NNScaleFactor[0]) + ((PicOutputWidth<<KK) > >1)) / ((PicOutputWidth< <KK)) InterpolateScalingRatio[1] = ((fRefHeight*NNScaleFactor[1]) + ((PicOutputHeight<<KK) > >1)) / PicOutputHeight) As described above, in a configuration in which the NN filter unit 611 includes two resolution conversion units (the interpolation unit 6114 and the rational resolution conversion unit 6112R), the scale obtained by combining the individual scales is the ratio between the input image size (fRefWidth, fRefHeight) and the output image size (PicOutputWidth, PicOutputHeight). Therefore, it is possible to derive the accuracy required for each component.

[0237] In the above configuration, the first scaling factor (NNScaleFactor) by the NN (rational number resolution conversion unit 6112R) and the second scaling factor (InterPolateScaleFactor) by the interpolation unit 6114 have the relationship of first scalingFactor>second scalingFactor, as shown in the following equation.

[0238] NNScaleFactor[0] >= InterPolateScaleFactor[0] = (1 << 14) / InterPolateScalingRatio[0] NNScaleFactor[1] >= InterPolateScaleFactor[1] = (1 << 14) / InterPolateScalingRatio[1] This is because the first scalingFactor is derived by rounding up (the first scalingRatio is rounded down), and since a relatively large image is generated by the NN, it has the effect of achieving higher image quality. Specifically, to achieve 5 / 4x scaling, the first scalingFactor is 2x and the second scalingFactor is 5 / 8x.

[0239] In the above configuration, the second scaling factor when performing the processing using the neural network is always 1 or less.

[0240] InterPolateScaleFactor[0] = (1 << 14) / InterPolateScalingRatio[0] < 1 InterPolateScaleFactor[1] = (1 << 14) / InterPolateScalingRatio[1] < 1 Figure 19(a2) shows an example in which the NN filter unit 611 includes an interpolation unit 6114 and an integer resolution conversion unit 6112I. The interpolation unit 6114 is capable of resolution conversion by a rational number multiple, but by combining it with the integer resolution conversion unit 6112I, even higher image quality conversion is possible. The scales (m, n) of the interpolation unit 6114 and the integer resolution conversion unit 6112I can be derived as follows. This corresponds to the case where KK = 0 in the above explanation.

[0241] For example, if the resolution of the input image to be processed by the NN filter unit 611 is (fRefWidth, fRefHeight) and the resolution of the output image is (PicOutputWidthL, PicOutputHeightL), the scaleFactor(m, n) of the integer resolution conversion unit 6112I may be derived using the following formula.

[0242] m = NNScaleFactor[0] = Max(Ceil( PicOutputWidth / fRefWidth, 1) n = NNScaleFactor[1] = Max(Ceil( PicOutputHeight / fRefHeight, 1) ) Alternatively, you can derive it as follows:

[0243] Alternatively, the scaling factor may be derived as follows: m = NNScaleFactor[0] = Max( (PicOutputWidth + fRefWidth - 1) / fRefWidth, 1) n = NNScaleFactor[1] = Max( (PicOutputHeight + fRefHeight- 1) / fRefHeight, 1). Furthermore, the scaling ratio of the interpolator 6114 may be derived using the following formula.

[0244] InterpolateScalingRatio[0] = ((fRefWidth << 14) + ((PicOutputWidth * NNScaleFactor[0]) >> 1)) / (PicOutputWidth * NNScaleFactor[0]) InterpolateScalingRatio[1] = ((fRefHeight << 14) + ((PicOutputHeight * NNScaleFactor[1]) >> 1)) / PicOutputHeight * NNScaleFactor[1]) Note that the scale of the integer resolution conversion unit 6112I may also be derived as scalingRatio as follows.

[0245] NNScalingRatio[0] = (fRefWidth / PicOutputWidth)<<14 NNScalingRatio[1] = (fRefHeight / PicOutputHeight)<<14 m = NNScaleFactor[0] = (2<<14) / NNScalingRatio[0] n = NNScaleFactor[1] = (2<<14) / NNScalingRatio[1] In this case, the scalingRatio of the interpolator 6114 is derived as follows:

[0246] InterpolateScalingRatio[0] = ((fRefWidth*NNScaleFactor[0]) + (PicOutputWidth>>1)) / (PicOutputWidth) InterpolateScalingRatio[1] = ((fRefHeight*NNScaleFactor[1]) + (PicOutputHeight>>1)) / PicOutputHeight) As described above, in a configuration in which the NN filter unit 611 includes two resolution conversion units (the interpolation unit 6114 and the integer resolution conversion unit 6112I), the scale obtained by combining the individual scales is set to the ratio between the input image size (fRefWidth, fRefHeight) and the output image size (PicOutputWidth, PicOutputHeight). This makes it possible to derive the precision required for each component.

[0247] 19(b) is an example in which the NN filter unit 611 includes a rational number resolution conversion unit 6112R and an interpolation unit 6114. The interpolation unit 6114 is capable of rational number multiple resolution conversion, and by combining it with the rational number resolution conversion unit 6112R, conversion with even higher image quality is possible. Since the order of the rational number resolution conversion unit 6112 and the interpolation unit 6114 is reversed from that of FIG. 19(a) already described, the basic operation will be omitted. It should be noted that the interpolation unit 6114 needs to correct the motion vector, which indicates a position change.

[0248] 19(b2) is an example in which the NN filter unit 611 includes an integer resolution conversion unit 6112I and an interpolation unit 6114. The interpolation unit 6114 is capable of resolution conversion by a rational number multiple, and by combining it with the integer resolution conversion unit 6112I, even higher image quality conversion is possible. Since the order of the integer resolution conversion unit 6112I and the interpolation unit 6114 is reversed from that of the already explained FIG. 19(a2), the basic operation will be omitted. It should be noted that the interpolation unit 6114 needs to correct motion vectors, which indicate position changes.

[0249] (Motion Vector Correction) FIG. 20 is a diagram showing an example of the input configuration of the motion vectors of the NN filter unit 611. In FIG.

[0250] 20(a1), 6114 included in the NN filter unit 611 may acquire a block from a position shifted by (refMxLx[0], refMvLX[1]). As already explained, interpolation is performed on the block at the integer position (xIntL, yIntL) derived from (refMxLx[0], refMvLX[1]) using the filter coefficients corresponding to the decimal point position (xFracL, xFracL).

[0251] 20(b1), 6114 included in the NN filter unit 611 may similarly acquire a block from a position shifted by (refMxLx[0], refMvLX[1]). As already explained, interpolation processing is performed on the block of integer position (xIntL, yIntL) derived from (refMxLx[0], refMvLX[1]) using the filter coefficients corresponding to (xFracL, xFracL), which corresponds to the decimal point position.

[0252] 20(b2), 6112R included in the NN filter unit 611 may acquire a block from a position shifted by an integer vector from (refMxLx[0], refMvLX[1]). For example, resolution conversion processing is performed using a block at an integer position (xIntL, yIntL) derived from (refMxLx[0], refMvLX[1]).

[0253] refxSbL = (((xSb - (SubWidthC * pps_scaling_win_left_offset)) << 4) + refMvLX[0]) * NNscalingRatio[0] refxL = ((Sign(refxSbL) * ((Abs(refxSbL) + 128) >> 8) + xL * ((NNscalingRatio[0] + 8) >> 4)) + fRefLeftOffset + 32) >> 6 refySbL = (((ySb - (SubHeightC * pps_scaling_win_top_offset)) << 4) + refMvLX[1]) * NNscalingRatio[1] refyL = ((Sign(refySbL) * ((Abs(refySbL) + 128) >> 8) + yL * ((NNscalingRatio[1] + 8) >> 4)) + fRefTopOffset + 32) >> 6 The integer position (xIntL, yIntL) and deviation from the integer position (xFracL, yFracL) are calculated as follows.

[0254] xIntL = refxL >> 4 yIntL = refyL >> 4 xFracL = 0 yFracL = 0 The interpolation unit 6114 may only perform processing corresponding to the decimal position of the motion vector xIntL = 0 yIntL = 0 xFracL = refxL & 15 yFracL = refyL & 15 According to the above processing, integer precision shifts are processed in the neural network unit and decimal precision shifts are processed in the interpolation unit, which has the effect of enabling efficient resolution conversion processing including motion vectors.

[0255] (Post-processing device 61) The post-processing device 61 of this embodiment can be used for pre-processing and post-processing, including resolution conversion. An example of this will be described here. Fig. 1 is a block diagram showing how video generated by a pre-processing device 51 is coded by an image coding device 11, and the video decoded by an image decoding device 31 is processed by the post-processing device 61.

[0256] The video encoding device 10 inputs an input image T1 to a synthesis information creation device 71, which creates filter information for deriving first model parameters. The filter information is then sent to the image encoding device 11. The synthesis information creation device 71 creates the filter information from statistical information on the pixel values ​​of the input image T1. The image encoding device 11 encodes the filter information and a reduced image T2, which is obtained by reducing the resolution of the input image T1 using a preprocessing device 51 (referred to as an encoded image), using the reduced image T2 and the filter information. The filter information and the encoded image are then sent to the network 21 as encoded data Te.

[0257] The video decoding device 30 decodes the coded data Te including the coded image and filter information by the image decoding device 31 and sends the decoded data to the post-processing device 61 .

[0258] The image decoding device 31 (header decoding unit 3020) decodes the filter information from the coded data Te acquired via the network 21 based on the syntax of FIG. 13 or 15, and sends the decoded result to the post-processing device 61.

[0259] The post-processing device 61 uses the filter information to derive first model parameters by the processing shown in Fig. 14 or 16. Then, the image decoding device 31 generates a decoded image Td2 by performing inverse resolution conversion on the image Td1 using the decoded image Td1 and the first model parameters.

[0260] [Application Example] The above-described video encoding device 10 and video decoding device 30 can be installed and used in various devices that transmit, receive, record, and play back video. Note that the video may be natural video captured by a camera or the like, or artificial video (including CG and GUI) generated by a computer or the like.

[0261] First, it will be explained with reference to FIG. 2 that the above-described video encoding device 10 and video decoding device 30 can be used for transmitting and receiving videos.

[0262] 2 is a block diagram showing the configuration of a transmitting device PROD_A equipped with a video encoding device 10. As shown in the figure, the transmitting device PROD_A includes an encoding unit PROD_A1 that encodes video to obtain encoded data, a modulation unit PROD_A2 that modulates a carrier wave with the encoded data obtained by the encoding unit PROD_A1 to obtain a modulated signal, and a transmitting unit PROD_A3 that transmits the modulated signal obtained by the modulation unit PROD_A2. The above-described video encoding device 10 is used as this encoding unit PROD_A1.

[0263] The transmitting device PROD_A may further include, as a source of moving images to be input to the encoding unit PROD_A1, a camera PROD_A4 that captures moving images, a recording medium PROD_A5 on which moving images are recorded, an input terminal PROD_A6 for inputting moving images from the outside, and an image processing unit A7 that generates or processes images. In the figure, the transmitting device PROD_A is shown as having all of these components, but some of them may be omitted.

[0264] The recording medium PROD_A5 may record unencoded video, or may record video encoded using a recording encoding method that is different from the transmission encoding method. In the latter case, a decoding unit (not shown) that decodes the encoded data read from the recording medium PROD_A5 in accordance with the recording encoding method may be interposed between the recording medium PROD_A5 and the encoding unit PROD_A1.

[0265] 2 is a block diagram showing the configuration of a receiving device PROD_B equipped with a video decoding device 30. As shown in the figure, the receiving device PROD_B includes a receiving unit PROD_B1 that receives a modulated signal, a demodulating unit PROD_B2 that obtains coded data by demodulating the modulated signal received by the receiving unit PROD_B1, and a decoding unit PROD_B3 that obtains video by decoding the coded data obtained by the demodulating unit PROD_B2. The above-described video decoding device 30 is used as this decoding unit PROD_B3.

[0266] The receiving device PROD_B may further include, as destinations of the moving images output by the decoding unit PROD_B3, a display PROD_B4 for displaying the moving images, a recording medium PROD_B5 for recording the moving images, and an output terminal PROD_B6 for outputting the moving images to the outside. In the figure, the receiving device PROD_B is shown as having all of these components, but some of them may be omitted.

[0267] The recording medium PROD_B5 may be for recording unencoded video, or may be encoded using an encoding method for recording that is different from the encoding method for transmission. In the latter case, it is preferable to interpose an encoding unit (not shown) between the decoding unit PROD_B3 and the recording medium PROD_B5, which encodes the video acquired from the decoding unit PROD_B3 according to the encoding method for recording.

[0268] The transmission medium for transmitting the modulated signal may be wireless or wired. The transmission mode for transmitting the modulated signal may be broadcast (here, this refers to a transmission mode in which the destination is not specified in advance) or communication (here, this refers to a transmission mode in which the destination is specified in advance). In other words, the transmission of the modulated signal may be realized by any of wireless broadcasting, wired broadcasting, wireless communication, and wired communication.

[0269] For example, a broadcasting station (such as a broadcasting facility) and a receiving station (such as a television receiver) for terrestrial digital broadcasting are an example of a transmitting device PROD_A and a receiving device PROD_B that transmit and receive modulated signals via wireless broadcasting, and a broadcasting station (such as a broadcasting facility) and a receiving station (such as a television receiver) for cable television broadcasting are an example of a transmitting device PROD_A and a receiving device PROD_B that transmit and receive modulated signals via cable broadcasting.

[0270] In addition, a server (e.g., a workstation) and a client (e.g., a television receiver, a personal computer, a smartphone) of an Internet-based video-on-demand (VOD) service or video sharing service are examples of a transmitting device PROD_A and a receiving device PROD_B that transmit and receive modulated signals via communication (typically, a LAN uses either a wireless or wired transmission medium, while a WAN uses a wired transmission medium). Here, personal computers include desktop PCs, laptop PCs, and tablet PCs. Smartphones also include multi-function mobile phone terminals.

[0271] The client of the video hosting service has the function of decoding the encoded data downloaded from the server and displaying it on a display, as well as the function of encoding the video images captured by a camera and uploading them to the server. In other words, the client of the video hosting service functions as both the transmitting device PROD_A and the receiving device PROD_B.

[0272] Next, it will be explained with reference to FIG. 3 that the above-described video encoding device 10 and video decoding device 30 can be used for recording and reproducing video.

[0273] PROD_C in Fig. 3 is a block diagram showing the configuration of a recording device PROD_C equipped with the above-mentioned video encoding device 10. As shown in the figure, the recording device PROD_C includes an encoding unit PROD_C1 that obtains encoded data by encoding video, and a writing unit PROD_C2 that writes the encoded data obtained by the encoding unit PROD_C1 onto a recording medium PROD_M. The above-mentioned video encoding device 10 is used as this encoding unit PROD_C1.

[0274] The recording medium PROD_M may be (1) a type that is built into the recording device PROD_C, such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), (2) a type that is connected to the recording device PROD_C, such as an SD memory card or a USB (Universal Serial Bus) flash memory, or (3) a type that is loaded into a drive device (not shown) built into the recording device PROD_C, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blu-ray Disc: registered trademark).

[0275] The recording device PROD_C may further include a camera PROD_C3 that captures moving images, an input terminal PROD_C4 for inputting moving images from an external device, a receiving unit PROD_C5 for receiving moving images, and an image processing unit PROD_C6 that generates or processes images, as sources of moving images to be input to the encoding unit PROD_C1. Although the figure illustrates a configuration in which the recording device PROD_C includes all of these components, some of them may be omitted.

[0276] The receiving unit PROD_C5 may receive unencoded video, or may receive encoded data encoded by a transmission encoding method different from the recording encoding method. In the latter case, a transmission decoding unit (not shown) that decodes the encoded data encoded by the transmission encoding method may be interposed between the receiving unit PROD_C5 and the encoding unit PROD_C1.

[0277] Examples of such recording devices PROD_C include DVD recorders, BD recorders, and HDD (Hard Disk Drive) recorders (in this case, the input terminal PROD_C4 or the receiving unit PROD_C5 is the main source of moving images). Other examples of such recording devices PROD_C include camcorders (in this case, the camera PROD_C3 is the main source of moving images), personal computers (in this case, the receiving unit PROD_C5 or the image processing unit C6 is the main source of moving images), and smartphones (in this case, the camera PROD_C3 or the receiving unit PROD_C5 is the main source of moving images).

[0278] 3PROD_D is a block diagram showing the configuration of a playback device PROD_D equipped with the above-described video decoding device 30. As shown in the figure, the playback device PROD_D includes a reading unit PROD_D1 that reads coded data written to a recording medium PROD_M, and a decoding unit PROD_D2 that obtains video by decoding the coded data read by the reading unit PROD_D1. The above-described video decoding device 30 is used as this decoding unit PROD_D2.

[0279] The recording medium PROD_M may be (1) a type that is built into the playback device PROD_D, such as an HDD or SSD, (2) a type that is connected to the playback device PROD_D, such as an SD memory card or USB flash memory, or (3) a type that is loaded into a drive device (not shown) built into the playback device PROD_D, such as a DVD or BD.

[0280] The playback device PROD_D may further include, as destinations of the video output by the decoding unit PROD_D2, a display PROD_D3 that displays the video, an output terminal PROD_D4 that outputs the video to the outside, and a transmission unit PROD_D5 that transmits the video. Although the figure shows an example of a configuration in which the playback device PROD_D includes all of these, some of them may be omitted.

[0281] The transmitting unit PROD_D5 may transmit unencoded video, or may transmit encoded data encoded by a transmission encoding method different from the recording encoding method. In the latter case, it is preferable to interpose an encoding unit (not shown) between the decoding unit PROD_D2 and the transmitting unit PROD_D5, which encodes the video by the transmission encoding method.

[0282] Examples of such a playback device PROD_D include a DVD player, a BD player, and an HDD player (in this case, the output terminal PROD_D4 to which a television receiver or the like is connected is the main destination of the video image). Other examples of such a playback device PROD_D include a television receiver (in this case, the display PROD_D3 is the main destination of the video image), a digital signage (also called an electronic billboard or electronic bulletin board, and the display PROD_D3 or the transmitter PROD_D5 is the main destination of the video image), a desktop PC (in this case, the output terminal PROD_D4 or the transmitter PROD_D5 is the main destination of the video image), a laptop or tablet PC (in this case, the display PROD_D3 or the transmitter PROD_D5 is the main destination of the video image), and a smartphone (in this case, the display PROD_D3 or the transmitter PROD_D5 is the main destination of the video image).

[0283] (Hardware and software implementation) Furthermore, each block of the above-mentioned video decoding device 30 and video encoding device 10 may be implemented in hardware using a logic circuit formed on an integrated circuit (IC chip), or in software using a CPU (Central Processing Unit).

[0284] In the latter case, each of the above devices includes a CPU that executes instructions of a program that realizes each function, a ROM (Read Only Memory) that stores the program, a RAM (Random Access Memory) that expands the program, a storage device (recording medium) such as a memory that stores the program and various data, etc. The object of the embodiment of the present invention can also be achieved by supplying each of the above devices with a recording medium on which program code (executable program, intermediate code program, source program) of a control program for each of the above devices, which is software that realizes the above-mentioned functions, is recorded in a computer-readable manner, and having the computer (or CPU or MPU) read and execute the program code recorded on the recording medium.

[0285] Examples of the recording medium that can be used include tapes such as magnetic tape and cassette tape, magnetic disks such as floppy disks (registered trademark) and hard disks, disks including optical disks such as CD-ROMs (Compact Disc Read-Only Memory), MO disks (Magneto-Optical discs), MDs (Mini Discs), DVDs (Digital Versatile Discs: registered trademark), CD-Rs (CD Recordable), and Blu-ray Discs (registered trademark), cards such as IC cards (including memory cards) and optical cards, semiconductor memories such as mask ROMs, EPROMs (Erasable Programmable Read-Only Memory), EEPROMs (Electrically Erasable and Programmable Read-Only Memory: registered trademark), and flash ROMs, and logic circuits such as PLDs (Programmable logic devices) and FPGAs (Field Programmable Gate Arrays).

[0286] Furthermore, each of the above devices may be configured to be connectable to a communications network, and the program code may be supplied via the communications network. This communications network may be any network capable of transmitting the program code. For example, the Internet, an intranet, an extranet, a local area network (LAN), an integrated services digital network (ISDN), a value-added network (VAN), a community antenna television / cable television (CATV) communications network, a virtual private network, a telephone line network, a mobile communications network, a satellite communications network, etc. Furthermore, the transmission media constituting this communications network may be any network capable of transmitting the program code, and are not limited to any particular configuration or type. For example, the present invention can be applied to wired networks such as IEEE (Institute of Electrical and Electronic Engineers) 1394, USB, power line carriers, cable TV lines, telephone lines, and ADSL (Asymmetric Digital Subscriber Line) lines, as well as wireless networks such as IrDA (Infrared Data Association), infrared networks like those used in remote controls, Bluetooth (registered trademark), IEEE 802.11 wireless networks, HDR (High Data Rate), NFC (Near Field Communication), DLNA (Digital Living Network Alliance: registered trademark), mobile phone networks, satellite networks, and terrestrial digital broadcasting networks. Note that the present invention can also be implemented in the form of a computer data signal embedded in a carrier wave, in which the program code is embodied by electronic transmission.

[0287] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. In other words, embodiments obtained by combining technical means modified appropriately within the scope of the claims are also included in the technical scope of the present invention.

[0288] Embodiments of the present invention can be suitably applied to a video decoding device that decodes coded data in which image data is coded, and a video coding device that generates coded data in which image data is coded. They can also be suitably applied to the data structure of coded data generated by a video coding device and referenced by a video decoding device. (Cross-reference to related applications) This application claims the benefit of priority to Japanese Patent Application No. 2021-101344, filed on June 18, 2021, the entire contents of which are incorporated herein by reference.

[0289] 1 Video transmission system 30 Video decoding device 31 Image decoding device 301 Entropy decoding unit 302 Parameter decoding unit 303 Inter prediction parameter derivation unit 304 Intra prediction parameter derivation unit 305, 107 Loop filter 306, 109 Reference picture memory 307, 108 Prediction parameter memory 308, 101 Prediction image generation unit 309 Inter prediction image generation unit 310 Intra prediction image generation unit 311, 105 Inverse quantization / inverse transform unit 312, 106 Addition unit 320 Prediction parameter derivation unit 10 Video coding device 11 Image coding device 102 Subtraction unit 103 Transform / quantization unit 104 Entropy coding unit 110 Coding parameter determination unit 111 Parameter coding unit 112 Inter prediction parameter coding unit 113 Intra prediction parameter encoding unit 120 Prediction parameter derivation unit 71 Filter information creation device< / kk>

Claims

1. A prediction unit having a neural network that performs scaling by a rational number factor and an interpolation unit that performs interpolation by a rational number factor, deriving a first scaling factor by the neural network and a second scaling factor by the interpolation unit from the actual width and height of the reference image and the actual width and height of the target image; a first scaling by the neural network and a second scaling by the interpolation unit in series to derive an interpolated image;

2. The video decoding device according to claim 1, characterized in that the neural network outputs a resolution-converted image using a first scaling factor for the reference image, and the interpolation unit derives an interpolated image using a second scaling factor for the resolution-converted image.

3. A header decoder for deriving a scale, a resolution equalization processing unit that does not change the width and height of the input and output images; The present invention is provided with a neural network including a resolution conversion process for converting an input image having a width and height of (M*W, N*H) to a width and height of (m*W, n*H) for integers M, N, W, H, m, and n; The video decoding device is characterized in that the resolution conversion process converts the number of channels from m*n*C to M*N*C.

4. 4. The video decoding device according to claim 3, characterized in that the resolution conversion process includes a first resolution conversion process for converting the width and height from (M*W, N*H) to (W, H), and a second resolution conversion process for converting the width and height from (W, H) to (m*W, n*H) for the output of the first resolution conversion process.

5. The video decoding device according to claim 4, characterized in that the first resolution conversion process includes a process of multiplying the number of channels by M*N, and the second resolution conversion process includes a process of multiplying the number of channels by 1 / (m*n).

6. a header decoder for independently deriving scales for horizontal and vertical components; A neural network that scales the image horizontally and vertically by a rational factor. A prediction unit including an interpolation unit that performs rational number multiplication, A video decoding device which derives parameters of the neural network according to a scale, which is a ratio of resolutions of an input image and an output image, or a decimal point position of the upper left coordinate of the input image.

7. A video decoding device comprising a neural network that performs scaling by a rational number factor in the horizontal direction and a neural network that performs scaling by a rational number factor in the vertical direction, and deriving parameters of the neural network based on the horizontal scale or the vertical scale.