Video encoding device and video decoding device

Abstract

To provide a video encoding device and a video decoding device that use information on annotated regions to perform post-filtering using a neural network for each partial region of a picture.SOLUTION: In a video transmission system 1 consisting of a video encoding device 10, a network 21, a video decoding device 30, and an image display device 41, the video decoding device 30 has an image decoding device 31, a post-filter processing device 61, and an auxiliary extension information decoding device 91, and decodes auxiliary extension information having information indicating what kind of post-filter processing is performed and an identification number for identifying what processing is identified, information on whether or not to apply post-filter processing for each picture, auxiliary extension information having an identification number indicating which post-filter processing is applied, auxiliary extension information indicating a region of interest in a decoded image, and auxiliary extension information having an identification number indicating which post-filter processing is applied to the region of interest in the decoded image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a video encoding device and a video decoding device. [Background technology]

[0002] In order to efficiently transmit or record moving images, a moving image encoding device is used that generates encoded data by encoding moving images, and a moving image decoding device is used that generates a decoded image by decoding the encoded data.

[0003] Specific examples of video encoding methods include H.264 / AVC and H.265 / HEVC (High-Efficiency Video Coding).

[0004] In such a video coding method, images (pictures) constituting a video are managed in a hierarchical structure consisting of slices obtained by dividing images, coding tree units (CTUs) obtained by dividing slices, coding units (sometimes called coding units: CUs) obtained by dividing coding tree units, and transform units (TUs) obtained by dividing coding units, and are coded / decoded for each CU.

[0005] In such a video coding method, a predicted image is usually generated based on a locally decoded image obtained by encoding / decoding an input image, and a prediction error (sometimes called a "difference image" or "residual image") obtained by subtracting the predicted image from the input image (original image) is coded. Methods for generating a predicted image include inter-prediction and intra-prediction.

[0006] Non-Patent Document 1 is an example of a recent video encoding and decoding technology. Non-Patent Document 1 specifies a supplemental enhancement information (SEI) message for transmitting image properties, display method, timing, and the like simultaneously with encoded data. It also discloses an SEI that transmits the topology and parameters of a neural network filter used as a post-filter in units that allow random access. It also discloses an SEI that transmits whether or not to perform post-filter processing on a picture-by-picture basis.

[0007] In addition, non-patent document 2 discloses an SEI that transmits annotated region information with auxiliary enhancement information being parameters that identify the annotated region using a rectangular bounding box that represents the size and position of an identified object in an image (picture). [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Additional SEI messages for VSEI (Draft 4), JVET-AD2006 [Non-Patent Document 2] ITU-T H.274 (05 / 22) Versatile supplemental enhancement information messages for coded video bitstreams Summary of the Invention [Problem to be solved by the invention]

[0009] In the method disclosed in Non-Patent Document 1, post-finishing is performed using a neural network. However, there is a problem that it is not possible to apply filtering to a part of a picture.

[0010] In addition, the method disclosed in Non-Patent Document 2 can indicate annotated regions, but has the problem that it does not show how to use the region when applying it to post-filter processing. [Means for solving the problem]

[0011] A video decoding device according to one aspect of the present invention includes: an image decoding device that decodes encoded data to generate a decoded image; a postfiltering device that performs postfiltering on the decoded image; a supplemental enhancement information decoding device for decoding a plurality of supplemental enhancement information used in the post-filter processing device; The auxiliary extension information includes information indicating what kind of post-filter processing is to be performed and an identification number for identifying the type of processing, Supplementary enhancement information having information on whether or not post-filtering is to be applied for each picture and an identification number indicating which post-filtering is to be applied; Supplementary enhancement information indicating a region of interest of a decoded image; The method further comprises decoding auxiliary extension information having an identification number indicating which post-filter processing is to be applied to the region of interest of the decoded image. Further, the auxiliary extension information indicating the region of interest of the decoded image includes reliability information of the region of interest, The method is characterized in that auxiliary extension information including an identification number indicating which post-filter processing is to be applied to the region of interest of the decoded image and threshold information for reliability information of the region of interest is decoded.

[0012] Further, the auxiliary extension information indicating the attention area of ​​the decoded image includes label information of the attention information, The method is characterized in that it decodes auxiliary extension information including an identification number indicating which post-filter processing is to be applied to the region of interest of the decoded image, and label information of the region of interest.

[0013] A video encoding device according to one aspect of the present invention comprises: an image encoding device for encoding an input image; an auxiliary enhancement information generating device for generating a plurality of auxiliary enhancement information for post-filtering; An auxiliary extension coding device for coding the plurality of auxiliary extension information, In the auxiliary extension generating device, information indicating what kind of post-filter processing is to be performed and auxiliary extension information having an identification number for identifying what kind of processing is to be performed; Supplementary enhancement information having information on whether or not post-filtering is to be applied for each picture and an identification number indicating which post-filtering is to be applied; Supplementary enhancement information indicating a region of interest of a decoded image; The present invention is characterized in that auxiliary extension information having an identification number indicating which post-filter processing is to be applied to the region of interest of the decoded image is generated, and is coded by the auxiliary extension coding device. Effect of the Invention

[0014] This configuration solves the problem of performing post-filtering using a neural network for each partial region of a picture using information on annotated regions. [Brief description of the drawings]

[0015] [Figure 1] 1 is a schematic diagram showing a configuration of a video transmission system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing a hierarchical structure of encoded data. [Diagram 3] FIG. 13 is a diagram showing a hierarchical structure of encoded data in units of PUs. [Figure 4] FIG. 1 is a schematic diagram showing a configuration of an image decoding device. [Diagram 5] 11 is a flowchart illustrating a schematic operation of an image decoding device. [Figure 6] FIG. 1 is a block diagram showing a configuration of an image encoding device. [Figure 7]A diagram showing the syntax of a portion of the Neural Network Post Filter Characteristics (NNPFC) SEI message. [Figure 8] FIG. 13 shows the syntax of the Neural Network Post Filter Activation (NNPFA) SEI message. [Figure 9] A diagram showing the syntax of the annotated region SEI message. [Figure 10] A diagram showing the syntax of the annotated region SEI message. [Figure 11] A diagram showing the syntax of the Neural Network Post Filter Domain SEI (NNPFR) SEI message. [Figure 12] A diagram showing the syntax of an SEI payload, which is a container for an SEI message. [Figure 13] FIG. 13 is a flowchart showing the processing of the post-filter processor 61. [Figure 14] FIG. 1 is a diagram showing an NN encoding device and a decoding device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] (First embodiment) Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0017] FIG. 1 is a schematic diagram showing the configuration of another moving image transmission system according to the present embodiment.

[0018] The video transmission system 1 is a system that transmits coded data obtained by coding an image, and decodes and displays the coded data. The video transmission system 1 is made up of a video coding device 10, a network 21, a video decoding device 30, and an image display device 41.

[0019] The video encoding device 10 is composed of an image encoding device (image encoding unit) 11, an auxiliary extension information creation device (auxiliary extension information creation unit) 71, an auxiliary extension information encoding device (auxiliary extension information encoding unit) 81, and a pre-filter processing device (pre-filter processing unit) 51.

[0020] The video encoding device 10 creates a prefiltered image T2 from an input video T1 using a prefilter processing device 51, compresses and encodes the image using an image encoding device 11, and analyzes the input video T1 and a locally decoded image T3 from the image encoding device 11 to generate auxiliary extension information to be input to a postfilter processing device 61 using an auxiliary extension information creation device 51. The auxiliary extension information is then encoded using an auxiliary extension information encoding device 81 to generate encoded data Te, which is then sent to a network 21.

[0021] The video decoding device 30 includes an image decoding device (image decoding unit) 31, an auxiliary extension information decoding device (auxiliary extension information decoding unit) 91, and a post-filter processing device (host filter processing unit) 61.

[0022] The video decoding device 30 decodes the encoded data Te received from the network 21 using the image decoding device 31 and the auxiliary extension information decoding device 91, and performs post-filter processing on the decoded image Td1 using the auxiliary extension information in the post-filter processing device 61, and outputs the post-filter decoded image Td2 to the image display device 41.

[0023] The postfilter processor 61 may output the decoded image Td1 as is without performing postfiltering on the auxiliary enhancement information.

[0024] The image display device 41 receives the post-filter image Td2 output from the post-filter processing device 61. The image display device 41 displays all or a part of the above. The image display device 41 includes a display device such as a liquid crystal display, an organic EL (Electro-luminescence) display, etc. The display may be in the form of a stationary, mobile, HMD, etc. When the image decoding device 31 has high processing power, it displays high quality images, and when it has only low processing power, it displays images that do not require high processing power or display power.

[0025] The network 21 transmits the encoded auxiliary extension information and the encoded data Te to the image decoding device 31. A part or all of the encoded auxiliary extension information may be included in the encoded data Te as auxiliary extension information SEI. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 21 is not necessarily limited to a bidirectional communication network, and may be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. The network 21 may also be used for transmitting digital media such as DVDs (Digital Versatile Discs: registered trademark), BDs (Blue-ray The encoded data Te may be replaced by a storage medium such as a Disc (registered trademark) on which the encoded data Te is recorded.

[0026] As an example of a specific embodiment, the pre-filtering process reduces an input image, and the post-filtering process enlarges a decoded image by super-resolution processing based on a neural network. The auxiliary extension information encoding device and the auxiliary extension information decoding device may encode and decode auxiliary extension information for these neural network processes.

[0027] As another example of a specific embodiment, in the pre-filter processing, no particular processing is performed, and in the post-filter processing, neural network processing is performed to restore the decoded image to the size of the input moving image by image restoration processing based on a neural network. In the auxiliary extension information encoding device and the auxiliary extension information decoding device, the auxiliary extension information for these neural network processes may be encoded and decoded.

[0028] In such a configuration, a framework is provided that enables efficient encoding and decoding of auxiliary extension information.

[0029] <Operator> The operators used in this specification are described below.

[0030] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is an OR assignment operator, and || represents a logical OR.

[0031] x? y : z is a ternary operator that takes y when x is true (non-zero) and z when x is false (0).

[0032] Clip3(a, b, c) is a function that clips c to a value between a and b (inclusive). If c < a, it returns a; if c > b, it returns b; otherwise, it returns c (where a <= b).

[0033] abs(a) is a function that returns the absolute value of a.

[0034] Int(a) is a function that returns the integer value of a.

[0035] Floor(a) is a function that returns the largest integer less than or equal to a.

[0036] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0037] a / d represents the division of a by d (truncating the decimal part).

[0038] a÷d and a over d represent the division of a by d (without rounding down).

[0039] (Structure of encoded data Te) Before going into a detailed description of the image encoding device 11 and the image decoding device 31 according to this embodiment, the data structure of the encoded data Te generated by the image encoding device 11 and decoded by the image decoding device 31 will be described with reference to Figures 2 and 3.

[0040] The coded data Te is a bitstream consisting of multiple CVS (Coded Video Sequence) and EoB (End of Bitstream) NAL units as shown in FIG. 2. The CVS consists of multiple AU (Access Unit) and EoS (End of Sequence) NAL units. The AU at the beginning of the CVS is called the CVSS (Coded Video Sequence Start) AU. The unit obtained by dividing the CVS into layers is called the CLVS (Coded Layer Video Sequence). The AU consists of one or more layered PUs (Picture Units) with the same output time. If the multilayer coding method is not adopted, the AU consists of one PU. The PU is a unit of coded data for one decoded picture consisting of multiple NAL units. The CLVS consists of PUs of the same layer, and the PU at the beginning of the CLVS is called the CLVSS (Coded Layer Video Sequence Start) PU. The CLVSS PU is limited to PUs consisting of randomly accessible IRAP (Intra Random Access Pictures) and GDR (Gradual Decoder Refresh Picture). A NAL unit consists of a NAL unit header and RBSP (Raw Byte Sequence Payload) data. The NAL unit header consists of 2 bits of 0 data, followed by a 6-bit nuh_layer_id that indicates the layer value, a 5-bit nuh_unit_type that indicates the NAL unit type, and a 3-bit nuh_temporal_id_plus1 that is the Temporal ID value plus 1.

[0041] Fig. 3 is a diagram showing a hierarchical structure of data in the coded data Te in units of PU. The coded data Te illustratively includes a sequence and a plurality of pictures constituting the sequence. Fig. 3 shows a diagram showing a coded video sequence that defines the sequence SEQ, a coded picture that defines the picture PICT, a coded slice that defines the slice S, coded slice data that defines the slice data, a coding tree unit included in the coded slice data, and a coding unit included in the coding tree unit.

[0042] In the coded video sequence, a set of data to be referred to by the image decoding device 31 in order to decode the sequence SEQ to be processed is defined. As shown in Fig. 3, the sequence SEQ includes a video parameter set VPS (Video Parameter Set), a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), an adaptation parameter set (APS), a picture PICT, and supplemental enhancement information SEI (Supplemental Enhancement Information).

[0043] The video parameter set VPS specifies a set of coding parameters common to multiple videos composed of multiple layers, as well as a set of coding parameters related to multiple layers and each individual layer included in the video.

[0044] The sequence parameter set SPS specifies a set of coding parameters that the image decoding device 31 refers to in order to decode the target sequence. For example, the width and height of a picture are specified. Note that there may be multiple SPSs. In that case, one of the multiple SPSs is selected from the PPS.

[0045] Here, the sequence parameter set SPS includes the following syntax elements: pic_width_max_in_luma_samples: This syntax element specifies the width of the image with the largest width among the images in a single sequence, in units of luminance blocks. The value of this syntax element must not be 0 and must be an integer multiple of Max(8, MinCbSizeY). Here, MinCbSizeY is a value determined by the minimum size of a luminance block. pic_height_max_in_luma_samples: This syntax element specifies the height of the image with the maximum height among the images in a single sequence, in units of luminance blocks. The value of this syntax element is required to be a non-zero integer multiple of Max(8, MinCbSizeY).

[0046] The picture parameter set PPS defines a set of coding parameters that the image decoding device 31 refers to in order to decode each picture in the target sequence. Note that there may be multiple PPSs. In that case, one of the multiple PPSs is selected for each picture in the target sequence.

[0047] Here, the picture parameter set PPS includes the following syntax elements: pps_pic_width_in_luma_samples: This syntax element specifies the width of the target picture. The value of this syntax element must be a non-zero integer multiple of Max(8, MinCbSizeY) and equal to or less than sps_pic_width_max_in_luma_samples. InpPicWidthInLumaSamples, described below, is set equal to pps_pic_width_in_luma_samples. pps_pic_height_in_luma_samples: This syntax element specifies the height of the target picture. The value of this syntax element must be a non-zero integer multiple of Max(8, MinCbSizeY) and equal to or less than sps_pic_height_max_in_luma_samples. InpPicHeightInLumaSamples, described below, is set equal to pps_pic_height_in_luma_samples. pps_conformance_window_flag: a flag indicating whether conformance (cropping) window offset parameters are to be signaled subsequently, and where the conformance window is to be displayed. If this flag is 1, the parameters are to be signaled, if it is 0, the conformance window offset parameters are not present. sps_chroma_format_id is a parameter that indicates the chrominance format. ChromaFormatIdc, described below, is set equal to sps_chroma_format_id.

[0048] (Encoded Picture) A coded picture defines a set of data to be referenced by the image decoding device 31 in order to decode a picture PICT to be processed. As shown in Fig. 3, the picture PICT includes a picture header PH and slices 0 to NS-1 (NS is the total number of slices included in the picture PICT).

[0049] (Coded Slice) A coded slice defines a set of data to be referenced by the image decoding device 31 in order to decode a current slice S. As shown in Fig. 3, a slice includes a slice header and slice data.

[0050] The slice header includes a group of coding parameters to be referred to by the image decoding device 31 in order to determine a decoding method for the current slice. Slice type designation information (slice_type) that designates a slice type is an example of a coding parameter included in the slice header.

[0051] Slice types that can be specified by the slice type specification information include (1) an I slice that uses only intra prediction when encoding, (2) a P slice that uses uni-prediction (L0 prediction) or intra prediction when encoding, and (3) a B slice that uses uni-prediction (L0 prediction or L1 prediction), bi-prediction, or intra prediction when encoding. Note that inter prediction is not limited to uni-prediction or bi-prediction, and a predicted image may be generated using more reference pictures. Hereinafter, when referring to P or B slice, it refers to a slice including a block that can use inter prediction.

[0052] (Encoded slice data) The coded slice data specifies a set of data to be referenced by the image decoding device 31 in order to decode the slice data to be processed. The slice data includes a CTU, as shown in the coded slice header in Fig. 3. A CTU is a block of a fixed size (e.g., 64x64) that constitutes a slice, and is also called a Largest Coding Unit (LCU).

[0053] (coding tree unit) 3 specifies a set of data that the image decoding device 31 refers to in order to decode a CTU to be processed. The CTU is divided into coding units CU, which are basic units of encoding processing, by recursive quad tree division (QT (Quad Tree) division), binary tree division (BT (Binary Tree) division), or ternary tree division (TT (Ternary Tree) division). A node of a tree structure obtained by recursive quad tree division is called a coding node. Intermediate nodes of the quad tree, binary tree, and ternary tree are coding nodes, and the CTU itself is specified as the top coding node.

[0054] (Encoding Unit) 3 specifies a set of data to be referenced by the image decoding device 31 in order to decode a coding unit to be processed. Specifically, a CU is composed of a CU header CUH, prediction parameters, transformation parameters, quantization transformation coefficients, etc. The CU header specifies a prediction mode, etc.

[0055] The prediction process may be performed in units of CUs, or in units of sub-CUs obtained by further dividing a CU.

[0056] There are two types of prediction (prediction modes): intra prediction and inter prediction. Intra prediction is a prediction within the same picture, while inter prediction refers to a prediction process performed between different pictures (for example, between display times or between layer images).

[0057] The transform and quantization processes are performed in units of CUs, but the quantized transform coefficients may be entropy coded in units of sub-blocks such as 4x4.

[0058] In this specification, when "a flag indicating whether or not XX" is used, a flag other than 0 (for example, 1) indicates XX and 0 indicates non-XX, and in logical negation, logical product, etc., 1 is treated as true and 0 as false (same below). However, in an actual device or method, other values ​​may be used as true and false values.

[0059] (Configuration of an image decoding device) The configuration of an image decoding device 31 (FIG. 4) according to this embodiment will be described.

[0060] The image decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit (prediction image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a prediction image generating unit (prediction image generating device) 308, an inverse quantization and inverse transform unit 311, an adder 312, and a prediction parameter derivation unit 320. Note that, in accordance with the image encoding device 11 described below, the image decoding device 31 may also be configured not to include the loop filter 305.

[0061] The parameter decoding unit 302 further includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit), and the CU decoding unit 3022 further includes a TU decoding unit 3024. These may be collectively referred to as a decoding module. The header decoding unit 3020 extracts parameter set information such as VPS, SPS, PPS, and APS, slice headers (slice information The CT information decoding unit 3021 decodes the CT from the encoded data. The CU decoding unit 3022 decodes the CU from the encoded data. The TU decoding unit 3024 decodes the QP update information (quantization correction value) and the quantized prediction error (residual_coding) from the encoded data.

[0062] The predicted image generating unit 308 includes an inter predicted image generating unit 309 and an intra predicted image generating unit 310 .

[0063] The entropy decoding unit 301 performs entropy decoding on the encoded data Te input from the outside, and decodes each code (syntax element).

[0064] The entropy decoding unit 301 outputs the decoded code to the parameter decoding unit 302. Control of which code to decode is performed based on an instruction from the parameter decoding unit 302.

[0065] (Basic flow) FIG. 5 is a flowchart illustrating a schematic operation of the image decoding device 31.

[0066] (S1100: Decode Parameter Set Information) The header decoding unit 3020 decodes parameter set information such as the VPS, SPS, and PPS from the encoded data.

[0067] (S1200: Decode slice information) The header decoding unit 3020 decodes the slice header (slice information) from the encoded data.

[0068] Thereafter, the image decoding device 31 repeats the processes from S1300 to S5000 for each CTU included in the target picture to derive a decoded image of each CTU.

[0069] (S1300: Decode CTU Information) The CT information decoding unit 3021 decodes the CTU from the encoded data.

[0070] (S1400: Decode CT Information) The CT information decoding unit 3021 decodes the CT from the encoded data.

[0071] (S1500: CU Decoding) The CU decoding unit 3022 performs S1510 and S1520 to decode the CU from the encoded data.

[0072] (S1510: Decode CU information) The CU decoding unit 3022 decodes the CU information, prediction information, and the like from the encoded data.

[0073] (S1520: Decode TU information) The TU decoding unit 3024 decodes the QP update information and the quantization prediction error, etc. from the encoded data. Note that the QP update information is a difference value from the quantization parameter predicted value qPpred, which is a predicted value of the quantization parameter QP.

[0074] (S2000: Generation of predicted image) The predicted image generation unit 308 generates a predicted image for each block included in the current CU based on the prediction information.

[0075] (S3000: Inverse Quantization and Inverse Transformation) The inverse quantization and inverse transform unit 311 executes inverse quantization and inverse transform processing for each TU included in the target CU.

[0076] (S4000: Generate decoded image) The adder 312 adds the predicted image supplied from the predicted image generation unit 308 and the prediction error supplied from the inverse quantization and inverse transform unit 311 to generate a decoded image of the current CU.

[0077] (S5000: Loop filter) The loop filter 305 applies a deblocking filter to the decoded image. , SAO, ALF, or other loop filters are applied to generate a decoded image.

[0078] (Neural network post-filter characteristic (NNPFC) SEI) Fig. 7 shows a part of the syntax of the neural network post-filter characteristic (NNPFC) SEI message of Non-Patent Document 1. The auxiliary enhancement information decoding device 91 decodes the syntax elements shown in Fig. 7. The NNPFC SEI message specifies a neural network to be applied as a post-filter process. The application of a specific post-filter process to a specific picture is indicated by a neural network post-filter activation SEI message (Fig. 8).

[0079] To apply this SEI message, the following variables must be defined: The width InpPicWidthInLumaSamples and height InpPicHeightInLumaSamples in luminance pixels of the picture decoded by the image decoding device 31. The picture luma pixel array CroppedYPic[idx] and chrominance pixel arrays CroppedCbPic[idx] and CroppedCrPic[idx], with idx ranging from 0 to numInputPics-1, that are used as input for postfiltering. · Pixel bit depth BitDepthY of the luminance pixel array. BitDepthC: the pixel bit length of the chrominance pixel array. Variables SubWidthC and SubHeightC that indicate the chrominance format indicated by ChromaFormatIdc of the picture. When the format is 4:2:0, the variables SubWidthC and SubHeightC are both 2, when the format is 4:2:2, the variables SubWidthC and SubHeightC are both 2 and 1, when the format is 4:4:4, the variables SubWidthC and SubHeightC are both 1.

[0080] nnpfc_auxiliary_inp_idc indicates the presence of auxiliary input data in the input tensor of the neural network postfilter. If the value is equal to 1, the strength control value StrengthControlVal (strength control information) of the input image is a real number ranging from 0 to 1. Note that the real number may be a decimal or floating point number. The strength control value may be a value obtained in the process of generating the decoded image, such as a quantization parameter.

[0081] nnpfc_purpose indicates the purpose of the process. If the value of nnpfc_purpose is 0, it may be used as determined by the application.

[0082] If the value of (nnpfc_purpose & 0x01) is greater than 0, it indicates that the purpose is to improve general visual quality.

[0083] If the value of (nnpfc_purpose & 0x02) is greater than 0, it indicates that chrominance upsampling (from 4:2:0 format to 4:2:2 or 4:4:4 format, or from 4:2:2 format to 4:4:4 format) is occurring.

[0084] If the value of (nnpfc_purpose & 0x04) is greater than 0, it indicates that there is a resolution sampling operation (changing the picture width or height).

[0085] If the value of (nnpfc_purpose & 0x08) is greater than 0, it indicates that picture rate upsampling processing is occurring.

[0086] If the value of (nnpfc_purpose & 0x10) is greater than 0, it indicates that pixel bit depth expansion (increase in luminance pixel bit depth or chrominance pixel bit depth) is being performed.

[0087] If the value of (nnpfc_purpose & 0x20) is greater than 0, it indicates that colorization (from 4:0:0 format to 4:2:0, 4:2:2, or 4:4:4 format) is to be performed.

[0088] Note that if ChromaFormatIdc is 3, the value of (nnpfc_purpose & 0x02) must be 0. Also, if the value of ChromaFormatIdc or (nnpfc_purpose & 0x02) is not 0, (nnpfc_purpose & 0x20) must be 0.

[0089] nnpfc_id indicates an identification number that can be used to identify the post-filter process.

[0090] The value of nnpfc_id must be in the range 0 through 2^32-2. nnpfc_id values ​​from 256 through 511, and from 2^31 through 2^32-2, are reserved for future use.

[0091] If the NNPFC SEI message is the first NNPFC SEI message, in decoding order, with a particular nnpfc_id value within the current CLVS, the following applies: Indicates that this SEI message is a basic post-filter process. This SEI message concerns the current decoded picture and all subsequent decoded pictures of the current layer until the end of the current CLVS.

[0092] nnpfc_base_flag indicates whether this NNPFC SEI message specifies a neural-network post-filter (NNPF) operation or an update to a base NNPF. A value of 1 in nnpfc_base_flag indicates that the SEI message specifies a base NNPF. A value of 0 in nnpfc_base_flag indicates that the SEI message updates a base NNPF.

[0093] The following constraints apply to the value of nnpfc_base_flag: · The NNPFC SEI message is the first NNPFC with a particular nnpfc_id value in the current CLVS in decoding order. If it is an SEI message, the value of nnpfc_base_flag must be 1. · If NNPFC SEI message nnpfcB is not the first NNPFC SEI message in decoding order and has a particular nnpfc_id value in the current CLVS and the value of nnpfc_base_flag is 1, then the NNPFC SEI message is a repeat of the first message. The contents of NNPFC SEI message nnpfcA with the same nnpfc_id value in decoding order shall be identical.

[0094] If the value of nnpfc_base_flag is 0, the following applies: This SEI message defines updates relative to the previous base NNPF in decoding order with the same nnpfc_id value. Updates are not cumulative, each update applies to a base NNPF. A base NNPF is the NNPF specified by the first NNPFC SEI message in decoding order with a particular nnpfc_id value in the current CLVS. The NNPF defined by this SEI message can be obtained by applying the updates defined by this SEI message to a base NNPF with the same nnpfc_id value. This SEI message concerns the current decoded picture and all subsequent decoded pictures of the current layer in output order.

[0095] (Neural Network Post Filter Activation (NNPFA) SEI) 8 shows the syntax of the Neural Network Postfilter Activation (NNPFA) SEI message. The Neural Network Postfilter Activation NNPFA SEI message activates or deactivates the application of a target neural network postfiltering identified by nnpfa_target_id for postfiltering of a sequence of pictures. · If the value of nnpfa_target_base_flag is 1, the target NNPF is the base NNPF with nnpfc_id equal to nnpfa_target_id. Otherwise (if the value of nnpfa_target_base_flag is 0), the target NNPF is the nnpfc_target_id that is equal to the nnpfa_target_id that is earlier in decoding order than the first VCL NAL unit of the current picture. The NNPF specified by the last NNPFC SEI message with that id.

[0096] Note that there may be multiple NNPFA SEI messages for the same picture.

[0097] nnpfa_target_id indicates the neural network post-filtering of the target picture. This identifies one or more NNPFC SEI messages with nnpfc_id equal to nnpfa_target_id for the current picture. nnpfc_id is an identification number that identifies an NNPFC SEI message, and it is possible to identify an NNPFC SEI message by setting this identification number to nnpfa_target_id.

[0098] An NNPFA SEI message with a particular value of nnpfa_target_id should not be present on the current PU unless one or both of the following conditions are true: ·There is an NNPFC SEI message in the current CLVS with nnpfc_id equal to a specific value of nnpfa_target_id present in a PU preceding the current PU in decoding order. · There is an NNPFC SEI message with nnpfc_id equal to a specific value of nnpfa_target_id for the current PU.

[0099] If a PU contains both an NNPFC SEI message with a particular value of nnpfc_id and an NNPFA SEI message with nnpfa_target_id equal to a particular value of nnpfc_id, the NNPFC SEI message shall precede the NNPFA SEI message in decoding order.

[0100] nnpfa_cancel_flag equal to 1 indicates that the continuity of the target neural network postfiltering set by a previous NNPFA SEI message with the same nnpfa_target_id as the current SEI message is cancelled, i.e. the target neural network postfiltering will not be used unless activated by another NNPFA SEI message with the same nnpfa_target_id as the current SEI message and with nnpfa_cancel_flag equal to 0. nnpfa_cancel_flag equal to 0 indicates that nnpfa_persistence_flag will persist.

[0101] nnpfa_persistence_flag indicates the persistence of the target neural network postfilter processing for the current layer.

[0102] If nnpfa_persistence_flag is 0, it indicates that the target neural network postfiltering applies to postfiltering of the current picture only.

[0103] nnpfa_persistence_flag, when set to 1, indicates that the target neural network postfiltering should be applied to the current image in the current layer and all subsequent pictures in output order until one or more of the following conditions become true: - A new CLVS for the current layer is started. Bitstream finished. · Images in the current layer associated with an NNPFA SEI message that has the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag equal to 1 are output after the current image in output order.

[0104] Note that neural network postfiltering is not applied to subsequent pictures of the current layer associated with the NNPFA SEI message if they have the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag is 1.

[0105] nnpfa_num_output_entries specifies the number of nnpfa_output_flag[i] syntax elements present in the NNPFA SEI message. The value of nnpfa_num_output_entries ranges from 0 to the number of pictures output from the network.

[0106] A value of 1 in nnpfa_output_flag[i] indicates that the picture generated corresponding to index i is to be output. A value of 0 in nnpfa_output_flag[i] indicates that the picture generated corresponding to index i is not to be output. If nnpfa_num_output_entries is smaller than the number of pictures to be output from the network, nnpfa_output_flag[i] is assumed to be equal to 1 for the remaining values.

[0107] Furthermore, the auxiliary extension information creating device 71, the auxiliary extension information encoding device 81, and the auxiliary extension information decoding device 91 may commonly hold generic network parameters. The auxiliary extension information creating device 71 uses a framework such as a neural network postfilter characteristic SEI to create network parameters for partially updating the commonly held generic network as auxiliary extension information. The auxiliary extension information may then be encoded by the auxiliary extension information encoding device 81 and decoded by the auxiliary extension information decoding device 91. With this configuration, the amount of code for the auxiliary extension information can be reduced, and auxiliary extension information corresponding to the input image T can be created, encoded, and decoded.

[0108] In addition, in order to support multiple formats as the transmission format of the network parameters, a parameter (identifier) ​​indicating the format may be sent. Furthermore, the actual supplementary extension information following the identifier may be transmitted as a byte string.

[0109] The auxiliary extension information of the network parameters decoded by the auxiliary extension information decoder 91 is input to the postfilter processor 61 .

[0110] The postfilter processor 61 performs post-image processing using a neural network by using the decoded auxiliary enhancement information (neural network postfilter characteristics SEI, neural network postfilter activation SEI) to restore the decoded video Td.

[0111] The auxiliary extension information encoding device 81 encodes the auxiliary extension information based on the syntax tables in Fig. 7 and Fig. 8. The auxiliary extension information is encoded as auxiliary extension information SEI, multiplexed into the encoded data Te output by the image encoding device 11, and output to the network 21.

[0112] The auxiliary extension information decoding device 91 decodes the auxiliary extension information from the encoded data Te based on the syntax tables in Figures 7 and 8, and sends the decoded result to the post-filter processing device 61 and the image recognition device 51 (not shown). The auxiliary extension information decoding device 91 decodes the auxiliary extension information encoded as auxiliary extension information SEI.

[0113] The post-filtering device 61 performs post-image processing on the decoded video image Td1 using the decoded video image Td1 and the auxiliary extension information to generate a post-image processing signal Td2.

[0114] In addition, in the example of this embodiment, the syntax in SEI is shown, but it is not limited to SEI, and syntax such as SPS, PPS, APS, slice header, etc. may be used.

[0115] In this embodiment, in such a configuration, a method is provided that can improve image quality even when the transmission rate is low, and can efficiently code and decode supplemental extension information regardless of the prediction structure.

[0116] (annotated region SEI) 9 and 10 show the syntax of the annotated region SEI disclosed in Non-Patent Document 2. The annotated region SEI will be described below.

[0117] The annotated region SEI message contains parameters that identify an annotated region using a rectangular bounding box that represents the size and location of the identified object in the picture. To use this SEI message, the following variables must be defined: - The width (in pixels) of the picture to be processed by this SEI output from the decoder, CroppedWidth and CroppedHeight ChromaFormatIdc. A value that indicates the chrominance format, where 0 is 4:0:0 (monochrome), 1 is 4:2:0, 2 is 4:2:2, and 3 is 4:4:4. If the value of ChromaFormatIdc is 0, the values ​​of SubWidthC and SubHeightC are both 1, if the value of ChromaFormatIdc is 1, the values ​​of SubWidthC and SubHeightC are both 2, if the value of ChromaFormatIdc is 2, the value of SubWidthC is 2 and the value of SubHeightC is 1, and if the value of ChromaFormatIdc is 3, the values ​​of SubWidthC and SubHeightC are both 1.

[0118] In addition, in the annotated region SEI of non-patent document 2, the left offset of the conformance cropping window ConfWinLeftOffset and the top offset of the conformance cropping window ConfWinTopOffset are defined as inputs, but in this embodiment, the decoded image input to the SEI is defined as having already been cropped, so these are not used as inputs.

[0119] ar_cancel_flag equal to 1 indicates that the SEI message cancels the application of a previous annotated regions SEI message associated with the layer or layers to which the annotated regions SEI message applies. ar_cancel_flag equal to 0 indicates that annotated regions information is present and will be applied. When ar_cancel_flag is equal to 1 or a new CVS for the current layer is started, the variables LabelAssigned[i], ObjectTracked[i], and ObjectBoundingBoxAvail are set to 0 for i in the range of values ​​from 0 to 255.

[0120] ar_not_optimized_for_viewing_flag equal to 1 indicates that the processed picture to which the annotated regions SEI message applies is not optimized for user viewing, but rather optimized for other purposes such as algorithmic object classification performance. ar_not_optimized_for_viewing_flag equal to 0 indicates that it is unknown whether the processed picture to which the annotated regions SEI message applies is optimized for user viewing.

[0121] ar_true_motion_flag equal to 1 indicates that the motion information in the processed picture to which the annotated region SEI message applies is selected for the purpose of accurately representing the object motion of the objects in the annotated region. ar_true_motion_flag equal to 0 indicates that it is unknown whether the motion information in the processed picture to which the annotated region SEI message applies is selected for the purpose of accurately representing the object motion of the objects in the annotated region.

[0122] When ar_occluded_object_flag is equal to 1, it indicates that the ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] represent the size and position of an object or part of an object that may not be displayed or may only be partially displayed in the cropped decoded picture. ar_occluded_object_flag equal to 0 indicates that the ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] syntax elements describe the size and position of an object that is fully displayed in the cropped decoded picture. The value of must be the same for all annotated_regions() syntaxes in the CVS.

[0123] ar_partial_object_flag_present_flag equal to 1 indicates the presence of the ar_partial_object_flag[ar_object_idx[i]] syntax element. ar_partial_object_flag_present_flag equal to 0 indicates the absence of the ar_partial_object_flag[ar_object_idx[i]] syntax element. The value of ar_partial_object_flag_present_flag must be the same for all annotated_regions() syntaxes in the CVS.

[0124] ar_object_label_present_flag equal to 1 indicates that there is label information corresponding to the objects in the annotated region. ar_object_label_present_flag equal to 0 indicates that there is no label information corresponding to the objects in the annotated region.

[0125] ar_object_confidence_info_present_flag equal to 1 indicates the presence of the ar_object_confidence[ar_object_idx[i]] syntax element. ar_object_confidence_info_present_flag equal to 0 indicates the absence of the ar_object_confidence[ar_object_idx[i]] syntax element. It is a bitstream conformance requirement that the value of ar_object_confidence_present_flag be the same for all annotated_regions() syntax structures in the CVS.

[0126] ar_object_confidence_length_minus1 plus 1 specifies the length in bits of the ar_object_confidence[ar_object_idx[i]] syntax element. It is a bitstream conformance requirement that the value of ar_object_confidence_length_minus1 be the same for all annotated_regions() syntax structures in the CVS.

[0127] When ar_object_label_language_present_flag is equal to 1, it indicates that the ar_object_label_language syntax element is present. When ar_object_label_language_present_flag is equal to 0, it indicates that the ar_object_label_language syntax element is not present.

[0128] ar_bit_equal_to_zero shall be equal to zero.

[0129] ar_object_label_language contains a language tag as specified in IETF RFC 5646, followed by a NULL terminating byte equal to 0x00. The length of the ar_object_label_language syntax element MUST NOT exceed 255 bytes, not including the NULL terminating byte. If ar_object_label_language is not present, the language of the label is unspecified.

[0130] ar_num_label_updates indicates the total number of labels associated with the annotated regions being signaled. The value of ar_num_label_updates must be in the range of 0 to 255.

[0131] ar_label_idx[i] indicates the index of the signaled label. The value of ar_label_idx[i] must be in the range of 0 to 255.

[0132] If ar_label_cancel_flag is equal to 1, the application of the ar_label_idx[i]th label is cancelled. If ar_label_cancel_flag is equal to 0, it indicates that the ar_label_idx[i]th label is assigned the signalled value.

[0133] LabelAssigned[ar_label_idx[i]] equal to 1 indicates that the ar_label_idx[i]th label is assigned. LabelAssigned[ar_label_idx[i]] equal to 0 indicates that the ar_label_idx[i]th label is not assigned.

[0134] ar_label[ar_label_idx[i]] specifies the contents of the ar_label_idx[i]th label. The length of the ar_label[ar_label_idx[i]] syntax element must be 255 bytes or less, not including the NULL termination byte.

[0135] ar_num_object_updates indicates the number of object updates to be signaled. ar_num_object_updates must be a value in the range 0 to 255.

[0136] ar_object_idx[i] is the index of the object parameter being signaled. Note that ar_object_idx[i] must be in the range of 0 to 255.

[0137] If ar_object_cancel_flag is equal to 1, the persistence of the ar_object_idx[i]th object is cancelled. If ar_object_cancel_flag is equal to 0, it indicates that the parameters associated with the ar_object_idx[i]th object tracked object are signalled.

[0138] ObjectTracked[ar_object_idx[i]] equal to 1 indicates that the ar_object_idx[i]th object is tracked. ObjectTracked[ar_object_idx[i]] equal to 0 indicates that the ar_object_idx[i]th object is not tracked.

[0139] ar_object_label_update_flag equal to 1 indicates that the object label (label information) is signaled. ar_object_label_update_flag equal to 0 indicates that the object label is not signaled.

[0140] ar_object_label_idx[ar_object_idx[i]] indicates the index of the label corresponding to the ar_object_idx[i]th object. If ar_object_label_idx[ar_object_idx[i]] does not exist, its value is inferred from the value of the previous annotated region SEI message in the same CVS output order, if there is one. Note that the value of ar_object_label_idx[ar_object_idx[i]] must be in the range of 0 to 255.

[0141] ar_bounding_box_update_flag equal to 1 indicates that the bounding box parameters of the object are signaled. ar_bounding_box_update_flag equal to 0 indicates that the bounding box parameters of the object are not signaled.

[0142] When ar_bounding_box_cancel_flag is equal to 1, the duration of ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] are canceled. If ar_bounding_box_cancel_flag is equal to 0, the following syntax elements are signaled: ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], ar_bounding_box_height[ar_object_idx[i]], ar_partial_object_flag[ar_object_idx[i]], and ar_object_confidence[ar_object_idx[i]].

[0143] ObjectBoundingBoxAvail[ar_object_idx[i]] equal to 1 indicates that bounding box information for the ar_object_idx[i]-th object is signaled. ObjectBoundingBoxAvail[ar_object_idx[i]] equal to 0 indicates that bounding box information for the ar_object_idx[i]-th object is not signaled.

[0144] ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_objec ar_object_idx[i] specifies the coordinates of the top-left corner and the width and height, respectively, of the bounding box of the ar_object_idx[i]th object in the cropped decoded picture, relative to the conformance cropping window.

[0145] The value of ar_bounding_box_left[ar_object_idx[i]] must be in the range from 0 to CroppedWidth / SubWidthC-1. The value of ar_bounding_box_top[ar_object_idx[i]] must be in the range from 0 to CroppedHeight / SubHeightC-1. The value of ar_bounding_box_width[ar_object_idx[i]] must be in the range from 0 to CroppedWidth / SubWidthC-ar_bounding_box_left[ar_object_idx[i]]. The value of ar_bounding_box_height[ar_object_idx[i]] must be within the range of 0 to CroppedHeight / SubHeightC-ar_bounding_box_top[ar_object_idx[i]].

[0146] The rectangle of the identified object contains luminance pixels with horizontal picture coordinates from SubWidthCar_bounding_box_left[ar_object_idx[i]] to SubWidthC*ar_bounding_box_left[ar_object_idx[i]]+ar_bounding_box_width[ar_object_idx[i]]-1 and vertical picture coordinates from SubHeightC*ar_bounding_box_top[ar_object_idx[i]] to SubHeightC*ar_bounding_box_top[ar_object_idx[i]]+ar_bounding_box_height[ar_object_idx[i]]-1.

[0147] In addition, in the annotated region SEI of Non-Patent Document 2, the left offset of the conformance cropping window ConfWinLeftOffset and the top offset of the conformance cropping window ConfWinTopOffset are defined as inputs. Therefore, the rectangle of the identified object is defined as including luminance pixels having horizontal picture coordinates from SubWidthC*(ConfWinLeftOffset+ar_bounding_box_left[ar_object_idx[i]] to SubWidthC*(ConfWinLeftOffset+ar_bounding_box_left[ar_object_idx[i]]+ar_bounding_box_width[ar_object_idx[i]])-1 and vertical picture coordinates from SubHeightC*(ConfWinTopOffset+ar_bounding_box_top[ar_object_idx[i]]) to SubHeightC*(ConfWinTopOffset+ar_bounding_box_top[ar_object_idx[i]]+ar_bounding_box_height[ar_object_idx[i]])-1. However, in this embodiment, the decoded image input to SEI is defined as having already been cropped, and therefore the above definition is not used.

[0148] If ChromaFormatIdc is not equal to 0, the corresponding of a given pixel in the two chrominance arrays is the pixel with picture coordinates (x / SubWidthC, y / SubHeightC), where (x, y) are the picture coordinates of the given luminance pixel.

[0149] The values ​​of ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] are preserved for each value of ar_object_idx[i] in output order in CVS. If not present, the values ​​of ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], or ar_bounding_box_height[ar_object_idx[i]] are inferred from the value of a previous annotated region SEI message in output order in CVS, if one exists.

[0150] ar_partial_object_flag[ar_object_idx[i]] equal to 1 indicates that the ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] syntax elements describe the size and position of an object that is only partially displayed in the cropped decoded picture. The ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] syntax elements may or may not represent the size and position of an object that is only partially displayed in the cropped decoded picture. If ar_partial_object_flag[ar_object_idx[i]] is not present, the value of ar_partial_object_flag[ar_object_idx[i]] is inferred from the value of the previous annotated region SEI message in output order in the CVS, if one is present.

[0151] ar_object_confidence[ar_object_idx[i]] indicates the confidence associated with the ar_object_idx[i]th object in units of 2-(ar_object_confidence_length_minus1+1). A larger value of ar_object_confidence[ar_object_idx[i]] indicates a higher confidence. The length of the ar_object_confidence[ar_object_idx[i]] syntax element is ar_object_confidence_length_minus1+1 bits. If not present, the value of ar_object_confidence[ar_object_idx[i]] is inferred from the value of the previous annotated region SEI message in the output order in CVS, if one exists.

[0152] The method disclosed in Non-Patent Document 1 discloses a framework for performing post-filtering using a neural network, but has a problem in that it is not possible to perform filter processing on a part of a picture.

[0153] In Non-Patent Document 1, the Neural Network Post-Filter Characteristics SEI message defines the contents of the filter processing, and the Neural Network Post-Filter Activation SEI message is a framework for determining the application of filter processing to each picture, and only the application of neural network filter processing on a picture-by-picture basis can be controlled.

[0154] In addition, the method disclosed in Non-Patent Document 2 can indicate annotated regions, but has the problem that it does not show how to use the region when applying it to post-filter processing.

[0155] In this embodiment, in order to solve the above problems, a new SEI is proposed.

[0156] Figure 11(a) is an example of a Neural Network Post-Filter Regions (NNPFR) SEI message. This SEI is used together with the annotated regions SEI to define the identification number of which NNPFC SEI is to be combined. By using this together with the NNPFA SEI, which similarly specifies the ID of the NNPFC SEI, neural network post-filter processing for each region defined in the annotated regions SEI message is realized.

[0157] Note that there may be multiple NNPFR SEI messages for the same picture.

[0158] nnpfpr_target_id is an identification number for identifying an NNPFC SEI for performing neural network post-filtering on annotated region SEI of the current picture. By setting nnpfc_id, which is the identification number of an NNPFC SEI message, to nnpfr_target_id, one or more NNPFC SEI messages having the same identification number are identified.

[0159] It also identifies an NNPFA SEI message with nnpfa_target_id that has the same value as nnpfr_target_id, and indicates that neural network post-filtering is to be performed on the region of the bounding box defined by the annotated regions SEI for the current picture.

[0160] An NNPFR SEI message with a particular value of nnpfr_target_id MUST NOT be present on the current PU unless one or both of the following conditions are true: - The feature of nnpfr_target_id in the current CLVS that exists before the current PU in the decoding order There is an NNPFC SEI message with nnpfc_id equal to the specified value. · There is an NNPFC SEI message with nnpfc_id equal to a specific value of nnpfr_target_id for the current PU. · Annotated region SEI exists in the current PU.

[0161] If a PU contains both an NNPFC SEI message with a particular value of nnpfc_id and an NNPFR SEI message with nnpfr_target_id equal to a particular value of nnpfc_id, the NNPFC SEI message shall precede the NNPFR SEI message in decoding order.

[0162] This method allows the timing of activation by the NNPFA SEI to be uniquely synchronized with the timing of the annotated region, and realizes a Region of Interest (ROI) that controls the image quality of a specific region of a picture by filtering. Specifically, the functions of the NNPFC SEI message make it possible to improve the image quality of the ROI, increase the frame rate, perform super-resolution processing, perform color space conversion, and perform color restoration processing. These functions are useful not only for improving human subjective evaluation, but also for video coding methods for machine recognition.

[0163] Fig. 11(b) is another example of a Neural Network Post-Filter Regions (NNPFR) SEI message. In this embodiment, a threshold value of the reliability of annotated region information is added as a syntax element to the NNPFR SEI message shown in Fig. 11(a). If the annotated region SEI message contains reliability information of a region and the reliability is greater than the threshold value, neural network post-filtering is performed on the bounding box region.

[0164] The value of object_confidence_threshold is compared with the value of the object confidence ar_object_confidence in the annotated region SEI, and neural network post-filtering is performed only on objects whose value is greater than object_confidence_threshold.

[0165] If the value of ar_object_confidence_info_present_flag in the annotated region SEI is 0, that is, if there is no object confidence information, the value of object_confidence_threshold is ignored.

[0166] In addition, the value of object_confidence_threshold may be compared with the value of ar_object_confidence, which is the confidence of the object in the annotated region SEI, and neural network post-filtering may be performed only on objects that are equal to or greater than the value of object_confidence_threshold.

[0167] Conversely, the value of object_confidence_threshold may be compared with the value of the object confidence ar_object_confidence in the annotated region SEI, and neural network post-filtering may be performed only on objects whose values ​​are smaller than or equal to the value of object_confidence_threshold.

[0168] Fig. 11(c) is another example of the Neural Network Post-Filter Regions (NNPFR) SEI message. In this embodiment, in addition to Fig. 11(a), the number of labels to be activated and the syntax of the label name are added so that activation can be performed only on the region of a specific object label (label information).

[0169] num_of_activating_label indicates the number of object labels to identify. The value of num_of_activating_label must be greater than or equal to 0 and less than or equal to the value of the syntax element ar_num_label_updates of the annotated region SEI.

[0170] activating_label indicates the character string of the object label to be identified. Neural network filtering is performed on the object with the index whose label name matches the contents of the array ar_label[] of the object labels of the annotated region SEI.

[0171] If the value of ar_object_label_present_flag is 0, that is, if the object label does not exist, num_of_activating_label and activating_label are ignored.

[0172] Multiple NNPFR SEI messages may exist for the same picture, and by specifying the identification number of another NNPFC SEI message in nnpfr_target_id, it is possible to apply separate neural network post-filter processing to areas with object label A and areas with object label B, for example.

[0173] In addition, in the example of FIG. 11(c), the region is specified by the object label, but it may be specified by using the value of the object index ar_object_idx of the annotated region SEI message.

[0174] [Application example] The above-mentioned video encoding device 10 and video decoding device 30 can be mounted and used in various devices that transmit, receive, record, and play back videos. Note that the video may be a natural video captured by a camera or the like, or an artificial video (including CG and GUI) generated by a computer or the like.

[0175] (SEI payload) FIG. 13 is a diagram showing the syntax of the SEI payload, which is a container of the SEI message.

[0176] This is called when nal_unit_type is PREFIX_SEI_NUT. PREFIX_SEI_NUT indicates that the SEI is located before the slice data.

[0177] When payloadType is 202, the annotated region SEI is called.

[0178] When payloadType is 210, the neural network postfilter feature SEI is called.

[0179] When payloadType is 211, the neural network post-filter activation SEI is called.

[0180] When payloadType is 213, the neural network postfilter domain SEI is called.

[0181] (SEI decoding and post-filtering) The header decoder 3020 reads the SEI payload, which is a container of the SEI message, and decodes the neural network post-filter characteristics SEI message. For example, the header decoder 3020 decodes nnpfc_purpose, nnpfc_id, nnpfc_base_flag, and the like.

[0182] 14 is a flowchart showing the processing of the post-filter processor 61. The post-filter processor 61 performs the following processing in accordance with the parameters of the SEI message.

[0183] S6001: Read processing volume and accuracy from SEI.

[0184] S6002: If the amount of post processing exceeds the processable complexity, the process is terminated. Proceed to S6003.

[0185] S6003: End if the accuracy exceeds the processable accuracy of the post-filter processor 61. If not, proceed to S6004.

[0186] S6004: A network model is identified from the SEI, and the topology of the post-filter processing device 61 is set.

[0187] S6005: Derive network model parameters from the updated information of SEI.

[0188] S6006: The derived network model parameters are loaded into the post-filter processor 61.

[0189] S6007: The post-filter processor 61 executes filtering and outputs the result to the outside.

[0190] However, the SEI is not necessarily required to construct luma and chroma samples in the decoding process.

[0191] (Details of the post-filter processing device 61) The NN filter unit performs filtering using a neural network model, using the input image inputTensor and input parameters (e.g., QP, bS, etc.). The input image may be an image for each component, or an image with multiple components as channels. In addition, the input parameters may be assigned to a channel different from the image.

[0192] The NN filter unit may repeatedly apply the following process.

[0193] The NN filter part derives the output image outputTensor by convolving the inputTensor with the kernel k[m][i][j] (conv, convolution) and adding bias. Here, nn=0..n-1, xx=0..width-1, yy=0..height-1, and Σ represents the summation for mm, i, and j, respectively.

[0194] outputTensor[nn][xx][yy]=ΣΣΣ(k[mm][i][j]*inputTensor[mm][xx+i-of][yy+j-of]+bias[nn]) In the case of 1x1 Conv, Σ represents the sum of mm=0..m-1, i=0, j=0. In this case, of=0 is set. In the case of 3x3 Conv, Σ represents the sum of mm=0..m-1, i=0..2, j=0..2. In this case, of=1 is set. n is the number of channels of outSamples, m is the number of channels of inputTensor, width is the width of inputTensor and outputTensor, and height is the height of inputTensor and outputTensor. of is the size of the padding area placed around inputTensor to make the sizes of inputTensor and outputTensor the same. In the following, when the output of the NN filter part is a value (correction value) rather than an image, the output is represented as corrNN instead of outputTensor.

[0195] Note that if you write inputTensor and outputTensor in CHW format instead of CWH format, it is equivalent to the following process.

[0196] outputTensor[nn][yy][xx]=ΣΣΣ(k[mm][i][j]*inputTensor[mm][yy+j-of][xx+i-of]+bias[nn]) In addition, a process called Depth-wise Conv, shown in the following formula, may be performed. Here, nn=0..n-1, xx=0..width-1, yy=0..height-1, and Σ represents the summation for i and j, respectively. n is the number of channels of outputTensor and inputTensor, width is the width of inputTensor and outputTensor, and height is the height of inputTensor and outputTensor.

[0197] outputTensor[nn][xx][yy]=ΣΣ(k[nn][i][j]*inputTensor[nn][xx+i-of][yy+j-of]+bias[nn]) Also, a nonlinear process called Activate, for example, ReLU, may be used. ReLU(x) = x >= 0 ? x : 0 Alternatively, leakyReLU shown in the following formula may be used.

[0198] leakyReLU(x) = x >= 0 ? x : a * x Here, a is a predetermined value, for example, 0.1 or 0.125. In order to perform integer arithmetic, all the values ​​of k, bias, and a above may be integers, and a right shift may be performed after conv.

[0199] With ReLU, 0 is always output for values ​​less than 0, and the input value is output as is for values ​​greater than or equal to 0. On the other hand, with leakyReLU, linear processing is performed for values ​​less than 0 with the gradient set by a. With ReLU, the gradient for values ​​less than 0 disappears, which can make it difficult for learning to progress. With leakyReLU, the gradient for values ​​less than 0 remains, making the above problem less likely to occur. Of the above leakyReLU(x), PReLU, which uses a parameterized value of a, may also be used.

[0200] (NNC) Neural Network Coding (NNC) is an international standard ISO / IEC15938-17 for efficiently compressing neural networks (NNs). Compressing trained NNs makes it possible to store and transmit NNs more efficiently.

[0201] The following provides an overview of the encoding and decoding processes of the NNC.

[0202] FIG. 15 is a diagram showing an encoding device and a decoding device of the NNC.

[0203] The NN coding device 801 has a pre-processing unit 8011, a quantization unit 8012, and an entropy coding unit 8013. The NN coding device 801 receives an uncompressed NN model O, and the quantization unit 8012 quantizes the NN model O to obtain a quantized model Q. The NN coding device 801 may repeatedly apply a parameter reduction method such as pruning or sparsification before quantization in the pre-processing unit 8011. Thereafter, the entropy coding unit 8013 applies entropy coding to the quantized model Q to obtain a bit stream S for storing and transmitting the NN model.

[0204] The NN decoding device 802 includes an entropy decoding unit 8021, a parameter restoration unit 8022, and a post-processing unit 8023. The NN decoding device 802 first inputs the transmitted bit stream S, and the entropy decoding unit 8021 performs entropy decoding of S to obtain an intermediate model RQ. If the operating environment of the NN model supports inference using the quantized representation used in RQ, the RQ may be output and used for inference. If not, the parameter restoration unit 8022 restores the parameters of RQ to their original representation to obtain an intermediate model RP. If the sparse tensor representation used can be processed in the operating environment of the NN model, the RP may be output and used for inference. If not, a reconstructed NN model R that does not include a tensor or structural representation different from the NN model O is obtained and output.

[0205] The NNC standard has decoding methods for certain numeric representations of NN parameters, including integers and floating point.

[0206] The decoding method NNR_PT_INT decodes a model with integer-valued parameters. The decoding method NNR_PT_FLOAT extends NNR_PT_INT by adding a quantization step size delta, which is multiplied by the integer value to produce a scaled integer. Delta is derived from the integer quantization parameter qp and the granularity parameter qp_density of delta as follows:

[0207] mul = 2^(qp_density) + (qp & (2^(qp_density)-1)) delta = mul * 2^((qp >> qp_density)-qp_density) (Format of trained NN) The representation of a trained NN consists of two elements: a topological representation, such as the size of layers and the connections between layers, and a parameter representation, such as weights and biases.

[0208] Topological representation is covered by native formats such as TensorFlow (registered trademark) and PyTorch (registered trademark), but to improve interoperability, exchange formats such as Open Neural Network Exchange Format (ONNX, registered trademark) and Neural Network Exchange Format (NNEF) exist.

[0209] The NNC standard also transports the topology information nnr_topology_unit_payload as part of the NNC bitstream containing the compressed parameter tensors, enabling interoperability with topology information expressed in native formats as well as interchange formats.

[0210] (Configuration of Image Encoding Device) Next, the configuration of the image encoding device 11 according to this embodiment will be described. Fig. 6 is a block diagram showing the configuration of the image encoding device 11 according to this embodiment. The image encoding device 11 includes a prediction image generating unit 101, a subtraction unit 102, a transformation and quantization unit 103, an inverse quantization and inverse transformation unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference image storage unit, frame memory) 109, an encoding parameter determining unit 110, a parameter encoding unit 111, a prediction parameter derivation unit 120, and an entropy encoding unit 104.

[0211] The predicted image generating unit 101 generates a predicted image for each CU.

[0212] The subtraction unit 102 generates a prediction error by subtracting the pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 from the pixel values ​​of the image T. The subtraction unit 102 outputs the prediction error to the transformation and quantization unit 103.

[0213] The transform / quantization unit 103 calculates transform coefficients by frequency transforming the prediction errors input from the subtraction unit 102, and derives quantized transform coefficients by quantizing the prediction errors. The transform / quantization unit 103 outputs the quantized transform coefficients to the parameter coding unit 111 and the inverse quantization / inverse transform unit 105.

[0214] The inverse quantization and inverse transform unit 105 is the same as the inverse quantization and inverse transform unit 311 (FIG. 4) in the image decoding device 31, and a description thereof will be omitted. The calculated prediction error is output to the addition unit .

[0215] The parameter coding unit 111 includes a header coding unit 1110, a CT information coding unit 1111, and a CU coding unit 1112 (prediction mode coding unit). The CU coding unit 1112 further includes a TU coding unit 1114. The following describes an outline of the operation of each module.

[0216] The header encoding unit 1110 performs encoding processing of parameters such as header information, division information, prediction information, and quantized transform coefficients.

[0217] The CT information encoding unit 1111 encodes the QT, MT (BT, TT) division information and the like.

[0218] The CU encoding unit 1112 encodes the CU information, prediction information, division information, and so on.

[0219] When a prediction error is included in a TU, the TU encoding unit 1114 encodes the QP update information and the quantized prediction error.

[0220] The CT information encoding unit 1111 and the CU encoding unit 1112 supply syntax elements such as inter prediction parameters and quantized transform coefficients to the parameter encoding unit 111.

[0221] The entropy coding unit 104 receives the quantized transform coefficients and the coding parameters from the parameter coding unit 111. The entropy coding unit 104 entropy codes these to generate and output coded data Te.

[0222] The prediction parameter derivation unit 120 derives inter prediction parameters and intra prediction parameters from the parameters input from the encoding parameter determination unit 110. The derived inter prediction parameters and intra prediction parameters are output to the parameter encoding unit 111.

[0223] The adder 106 generates a decoded image by adding, for each pixel, the pixel value of the predicted block input from the predicted image generation unit 101 and the prediction error input from the inverse quantization and inverse transform unit 105. The adder 106 stores the generated decoded image in a reference picture memory 109.

[0224] The loop filter 107 performs deblocking filtering, SAO, and ALF on the decoded image generated by the adder 106. Note that the loop filter 107 does not necessarily have to include the above three types of filters, and may be configured, for example, as only a deblocking filter.

[0225] The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in a predetermined location for each current picture and CU.

[0226] The reference picture memory 109 stores the decoded image generated by the loop filter 107 at a predetermined position for each current picture and CU.

[0227] The coding parameter determination unit 110 selects one set from among a plurality of sets of coding parameters. The coding parameters are the above-mentioned QT, BT or TT division information, prediction parameters, or parameters to be coded that are generated in relation to these. The predicted image generation unit 101 generates a predicted image using these coding parameters.

[0228] In addition, a part of the image encoding device 11 and the image decoding device 31 in the above-mentioned embodiment, for example, the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the predicted image generating unit 308, the inverse quantization and inverse transform unit 311, the addition unit 312, the prediction parameter derivation unit 320, the predicted image generating unit 101, the subtraction unit 102, the transform and quantization unit 103, the entropy coding unit 104, the inverse quantization and inverse transform unit 105, the loop filter 107, the coding parameter determination unit 110, the parameter coding unit 111, and the prediction parameter derivation unit 120 may be realized by a computer. In this case, a program for realizing this control function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into and executed by a computer system. In addition, the "computer system" referred to here is a computer system built into either the image encoding device 11 or the image decoding device 31, and includes hardware such as an OS and peripheral devices. Additionally, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" may also include devices that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and devices that hold a program for a certain period of time, such as volatile memory within a computer system that serves as a server or client in such cases. Furthermore, the above-mentioned program may be one that realizes part of the functions described above, and may further realize the functions described above in combination with a program already recorded in the computer system. It may be something like that.

[0229] In addition, a part or the whole of the image encoding device 11 and the image decoding device 31 in the above-mentioned embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image encoding device 11 and the image decoding device 31 may be individually made into a processor, or a part or the whole may be integrated into a processor. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the progress of semiconductor technology, an integrated circuit based on that technology may be used.

[0230] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention.

[0231] This embodiment will be described with reference to FIG. 1. The video decoding device 30 includes an image decoding device 31 that decodes encoded data to generate a decoded image, a postfilter processing device 61 that performs postfilter processing on the decoded image, and an auxiliary extension information decoding device 91 that decodes multiple auxiliary extension information used in the postfilter processing device, and the auxiliary extension information decoding device 91 is characterized in that the auxiliary extension information is decoded by decoding auxiliary extension information having information indicating what type of postfilter processing to perform and an identification number for identifying the type of processing, auxiliary extension information having information on whether or not to apply postfilter processing for each picture and an identification number indicating which postfilter processing to apply, auxiliary extension information indicating a region of interest in a decoded image, and auxiliary extension information having an identification number indicating which postfilter processing to apply to the region of interest in the decoded image.

[0232] In addition, in the auxiliary extension information decoding device 91, the auxiliary extension information indicating the region of interest of the decoded image may include reliability information of the region of interest, and the auxiliary extension information may include an identification number indicating which post-filter processing to apply to the region of interest in the decoded image, and threshold information for the reliability information of the region of interest.

[0233] In addition, in the auxiliary extension information decoding device 91, the auxiliary extension information indicating the region of interest of the decoded image may include label information of the attention information, and the auxiliary extension information may include an identification number indicating which post-filter processing to apply to the region of interest of the decoded image and the label information of the region of interest.

[0234] The video coding device 10 includes an image coding device 11 that codes an input image, an auxiliary extension information generating device 71 that generates multiple auxiliary extension information for postfilter processing, and an auxiliary extension coding device 81 that codes the multiple auxiliary extension information. The auxiliary extension generating device 71 generates auxiliary extension information having information indicating what type of postfilter processing to perform and an identification number for identifying the type of processing, auxiliary extension information having information on whether or not to apply postfilter processing for each picture and an identification number indicating which postfilter processing to apply, auxiliary extension information indicating a region of interest in a decoded image, and auxiliary extension information having an identification number indicating which postfilter processing to apply to the region of interest in the decoded image, and codes these information in the auxiliary extension coding device 81.

[0235] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. In other words, the technical scope of the present invention also includes embodiments obtained by combining technical means that are appropriately modified within the scope of the claims. [Industrial Applicability]

[0236] An embodiment of the present invention relates to a video decoding device that decodes encoded data in which image data is encoded. The present invention can be suitably applied to a video encoding device that generates encoded data in which image data is encoded, and to a data structure of encoded data that is generated by a video encoding device and referenced by a video decoding device. [Explanation of symbols]

[0237] 1. Video transmission system 30 Video Decoding Device 31 Image Decoding Device 301 Entropy Decoding Unit 302 Parameter Decoding Unit 305, 107 Loop Filter 306, 109 Reference Picture Memory 307, 108 Prediction parameter memory 308, 101 Prediction image generation unit 311, 105 Inverse quantization and inverse transformation unit 312, 106 Addition section 320 Prediction Parameter Derivation Unit 10 Video Encoding Device 11 Image encoding device 102 Subtraction section 103 Transformation and Quantization Section 104 Entropy coding unit 110 Encoding parameter determination unit 111 Parameter Encoding Unit 120 Prediction parameter derivation part 41 Image display device 51 Pre-filter processing device 61 Post-filter processing device 71 Auxiliary extension information creation device 81 Supplementary extension information coding device 91 Auxiliary extension information decoding device

Claims

1. an image decoding device that decodes encoded data to generate a decoded image; a postfiltering device that performs postfiltering on the decoded image; a supplemental enhancement information decoding device for decoding a plurality of supplemental enhancement information used in the post-filter processing device; The auxiliary extension information includes information indicating what kind of post-filter processing is to be performed and an identification number for identifying the type of processing, Auxiliary enhancement information having information on whether or not post-filtering is to be applied for each picture and an identification number indicating which post-filtering is to be applied; Supplementary enhancement information indicating a region of interest of a decoded image; A video decoding device comprising: a processor for decoding auxiliary extension information having an identification number indicating which post-filter processing is to be applied to a region of interest of the decoded image;

2. In the video decoding device according to claim 1, The auxiliary extension information indicating the region of interest of the decoded image includes reliability information of the region of interest, A video decoding device characterized by decoding auxiliary extension information including an identification number indicating which post-filter processing is to be applied to the region of interest of the decoded image and threshold information for reliability information of the region of interest.

3. In the video decoding device according to claim 1, The auxiliary extension information indicating the attention area of ​​the decoded image includes label information of the attention information, A video decoding device characterized by decoding auxiliary extension information including an identification number indicating which post-filter processing is to be applied to a region of interest in the decoded image and label information of the region of interest.

4. an image encoding device for encoding an input image; an auxiliary enhancement information generating device for generating a plurality of auxiliary enhancement information for post-filtering; An auxiliary extension coding device for coding the plurality of auxiliary extension information, In the auxiliary extension generating device, information indicating what kind of post-filter processing is to be performed and auxiliary extension information having an identification number for identifying what kind of processing is to be performed; Supplementary enhancement information having information on whether or not post-filtering is to be applied for each picture and an identification number indicating which post-filtering is to be applied; Supplementary enhancement information indicating a region of interest of a decoded image; A video coding device characterized by generating auxiliary extension information having an identification number indicating which post-filter processing to apply to a region of interest of the decoded image, and coding the information using the auxiliary extension coding device.

Images