Moving image encoding device and decoding device

Abstract

A framework in which post-filtering is performed has the problem that the post-filtering cannot be performed for each of partial regions of a picture by using annotated region information. A moving image decoding device according to one aspect of the present invention is characterized by comprising an image decoding device, a post-filtering device, and an auxiliary extension information decoding device, wherein auxiliary extension information that is decoded includes: auxiliary extension information including information indicating what kind of post-filtering is to be performed, and an identification number for identifying the kind of processing; auxiliary extension information including information as to whether or not to apply post-filtering for each picture, and a recognition number indicating which post-filtering is to be applied; auxiliary extension information indicating a region of interest of a decoded image; and auxiliary extension information including a recognition number indicating which post-filtering is to be applied to the region of interest of the decoded image.

Description

Video encoding device and decoding device

[0001] The present invention relates to a video encoding device and a video decoding device.

[0002] In order to efficiently transmit or record moving images, a moving image encoding device is used that encodes moving images to generate coded data, and a moving image decoding device is used that decodes the coded data to generate decoded images.

[0003] Specific examples of video encoding methods include H.264 / AVC and H.265 / HEVC (High-Efficiency Video Coding).

[0004] In such a video coding method, images (pictures) constituting a video are managed in a hierarchical structure consisting of slices obtained by dividing images, coding tree units (CTUs) obtained by dividing slices, coding units (sometimes called coding units (CUs)) obtained by dividing coding tree units, and transform units (TUs) obtained by dividing coding units, and are coded / decoded for each CU.

[0005] In such video coding methods, a predicted image is typically generated based on a locally decoded image obtained by encoding / decoding an input image, and the predicted image is subtracted from the input image (original image) to obtain a prediction error (sometimes called a "difference image" or "residual image"), which is then coded. Methods for generating predicted images include inter-frame prediction (inter-prediction) and intra-frame prediction (intra-prediction).

[0006] Non-Patent Document 1 is another example of a recent video encoding and decoding technology. Non-Patent Document 1 defines a supplemental enhancement information (SEI) message for transmitting image properties, display methods, timing, and the like simultaneously with encoded data. It also discloses an SEI for transmitting the topology and parameters of a neural network filter used as a post-filter in units that allow random access. It also discloses an SEI for transmitting whether post-filtering processing will be performed on a picture-by-picture basis.

[0007] In addition, Non-Patent Document 2 discloses an SEI that transmits annotated region information in which the auxiliary enhancement information is a parameter that identifies the annotated region using a rectangular bounding box that represents the size and position of an identified object in an image (picture).

[0008] Additional SEI messages for VSEI (Draft 4), JVET-AD2006ITU-T H.274 (05 / 22) Versatile supplemental enhancement information messages for coded video bitstreams

[0009] The method disclosed in Non-Patent Document 1 shows a framework for performing post-filtering using a neural network, but has the problem that it is not possible to perform filtering on a part of a picture.

[0010] Furthermore, the method disclosed in Non-Patent Document 2 can indicate annotated regions, but has the problem that it does not show how to use the annotated regions when applied to post-filtering.

[0011] A video decoding device according to one aspect of the present invention comprises an image decoding device that decodes encoded data to generate a decoded image, a post-filter processing device that performs post-filter processing on the decoded image, and an auxiliary extension information decoding device that decodes multiple auxiliary extension information used in the post-filter processing device, wherein the auxiliary extension information is decoded as follows: auxiliary extension information having information indicating what type of post-filter processing to perform and an identification number for identifying the type of processing; auxiliary extension information having information on whether or not to apply post-filter processing for each picture and an identification number indicating which post-filter processing to apply; auxiliary extension information indicating a region of interest in the decoded image; and auxiliary extension information having an identification number indicating which post-filter processing to apply to the region of interest in the decoded image.Furthermore, the auxiliary extension information indicating the region of interest in the decoded image includes reliability information of the region of interest, and the decoding device is characterized in that the auxiliary extension information includes an identification number indicating which post-filter processing to apply to the region of interest in the decoded image and threshold information for the reliability information of the region of interest.

[0012] Furthermore, the auxiliary extension information indicating the region of interest of the decoded image includes label information of the region of interest, and the auxiliary extension information including an identification number indicating which post-filter processing to apply to the region of interest of the decoded image and label information of the region of interest is decoded.

[0013] A video coding device according to one aspect of the present invention comprises an image coding device that codes an input image, an auxiliary extension information generation device that generates multiple auxiliary extension information for post-filter processing, and an auxiliary extension coding device that codes the multiple auxiliary extension information, wherein the auxiliary extension generation device generates auxiliary extension information having information indicating what type of post-filter processing to perform and an identification number for identifying the type of processing, auxiliary extension information having information on whether or not to apply post-filter processing for each picture and an identification number indicating which post-filter processing to apply, auxiliary extension information indicating a region of interest in a decoded image, and auxiliary extension information having an identification number indicating which post-filter processing to apply to the region of interest in the decoded image, and encodes the information using the auxiliary extension coding device.

[0014] This configuration solves the problem of performing post-filtering using a neural network for each partial region of a picture using information on annotated regions.

[0015] FIG. 1 is a schematic diagram showing the configuration of a video transmission system according to this embodiment. FIG. 2 is a diagram showing the hierarchical structure of encoded data. FIG. 3 is a diagram showing the hierarchical structure of encoded data in units of PU. FIG. 3 is a schematic diagram showing the configuration of an image decoding device. FIG. 4 is a flowchart explaining the overall operation of an image decoding device. FIG. 4 is a block diagram showing the configuration of an image encoding device. FIG. 5 is a diagram showing the syntax of a part of a neural network post-filter characteristic (NNPFC) SEI message. FIG. 6 is a diagram showing the syntax of a neural network post-filter activation (NNPFA) SEI message. FIG. 7 is a diagram showing the syntax of an annotated region SEI message. FIG. 8 is a diagram showing the syntax of an annotated region SEI message. FIG. 9 is a diagram showing the syntax of a neural network post-filter region SEI (NNPFR) SEI message. FIG. 10 is a diagram showing the syntax of an SEI payload, which is a container of an SEI message. FIG. 11 is a diagram showing a flowchart of the processing of a post-filter processing device 61. FIG. 12 is a diagram showing an NN encoding device and a decoding device.

[0016] First Embodiment Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0017] FIG. 1 is a schematic diagram showing the configuration of another moving image transmission system according to this embodiment.

[0018] The video transmission system 1 is a system that transmits coded data obtained by coding an image, decodes the transmitted coded data, and displays the data. The video transmission system 1 comprises a video encoding device 10, a network 21, a video decoding device 30, and an image display device 41.

[0019] The video encoding device 10 is composed of an image encoding device (image encoding unit) 11, an auxiliary extension information creation device (auxiliary extension information creation unit) 71, an auxiliary extension information encoding device (auxiliary extension information encoding unit) 81, and a pre-filter processing device (pre-filter processing unit) 51.

[0020] The video encoding device 10 creates a pre-filtered image T2 from an input video T1 using a pre-filter processing device 51, compresses and encodes the image using an image encoding device 11, and analyzes the input video T1 and a locally decoded image T3 from the image encoding device 11 to generate auxiliary extension information to be input to a post-filter processing device 61 using an auxiliary extension information creation device 51, encodes the information using an auxiliary extension information encoding device 81, generates encoded data Te, and sends it to a network 21.

[0021] The video decoding device 30 includes an image decoding device (image decoding unit) 31 , an auxiliary extension information decoding device (auxiliary extension information decoding unit) 91 , and a post-filter processing device (host filter processing unit) 61 .

[0022] The video decoding device 30 decodes the encoded data Te received from the network 21 using the image decoding device 31 and the auxiliary extension information decoding device 91, performs post-filtering on the decoded image Td1 using the auxiliary extension information in the post-filtering processing device 61, and outputs the post-filtered decoded image Td2 to the image display device 41.

[0023] The post-filtering device 61 may output the decoded image Td1 as is without performing post-filtering on the auxiliary extension information.

[0024] The image display device 41 displays all or part of the post-filter image Td2 output from the post-filter processing device 61. The image display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. The display may be in the form of a stationary display, a mobile display, an HMD, or the like. Furthermore, when the image decoding device 31 has high processing power, it displays high-quality images, and when it has only low processing power, it displays images that do not require high processing power or display power.

[0025] The network 21 transmits the encoded supplemental extension information and the encoded data Te to the image decoding device 31. Some or all of the encoded supplemental extension information may be included in the encoded data Te as supplemental extension information SEI. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. The network 21 is not necessarily limited to a bidirectional communication network, and may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. The network 21 may also be replaced by a storage medium on which the encoded data Te is recorded, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blu-ray Disc: registered trademark).

[0026] In one example of a specific embodiment, the pre-filtering process reduces the input image, and the post-filtering process enlarges the decoded image by super-resolution processing based on a neural network. The auxiliary enhancement information encoding device and the auxiliary enhancement information decoding device may encode and decode auxiliary enhancement information for these neural network processes.

[0027] In another example of a specific embodiment, the pre-filtering process does not perform any particular processing, and the post-filtering process performs neural network processing to restore the decoded image to the size of the input video image by image restoration processing based on a neural network. The auxiliary extension information encoding device and the auxiliary extension information decoding device may encode and decode auxiliary extension information for these neural network processes.

[0028] In this configuration, a framework is provided that enables efficient encoding and decoding of supplemental extension information.

[0029] <Operators> The operators used in this specification are listed below.

[0030] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is the OR assignment operator, and || indicates logical sum.

[0031] x? y : z is a ternary operator that takes y when x is true (non-zero) and z when x is false (0).

[0032] Clip3(a, b, c) is a function that clips c to a value between a and b. If c < a, it returns a; if c > b, it returns b; otherwise, it returns c (where a <= b).

[0033] abs(a) is a function that returns the absolute value of a.

[0034] Int(a) is a function that returns the integer value of a.

[0035] Floor(a) is a function that returns the largest integer less than or equal to a.

[0036] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0037] a / d represents the division of a by d (truncating the decimal part).

[0038] a÷d and a over d represent the division of a by d (without rounding).

[0039] (Structure of the encoded data Te) Prior to the detailed description of the image encoding device 11 and the image decoding device 31 according to this embodiment, the data structure of the encoded data Te generated by the image encoding device 11 and decoded by the image decoding device 31 will be described using FIGS. 2 and FIGS. 3.

[0040] The coded data Te is a bitstream consisting of multiple CVS (Coded Video Sequence) and EoB (End of Bitstream) NAL units, as shown in Figure 2. A CVS consists of multiple AUs (Access Units) and EoS (End of Sequence) NAL units. The first AU of a CVS is called a CVSS (Coded Video Sequence Start) AU. A CVS is divided into layers and called a CLVS (Coded Layer Video Sequence). An AU consists of one or more layer picture units (PUs) with the same output time. If a multilayer coding method is not used, an AU consists of one PU. A PU is a unit of coded data for one decoded picture consisting of multiple NAL units. A CLVS consists of PUs of the same layer, and the first PU of a CLVS is called a CLVSS (Coded Layer Video Sequence Start) PU. A CLVSS PU is limited to PUs consisting of randomly accessible Intra Random Access Pictures (IRAPs) or Gradual Decoder Refresh Pictures (GDRs). A NAL unit consists of a NAL unit header and RBSP (Raw Byte Sequence Payload) data. The NAL unit header consists of two 0-bits followed by a 6-bit nuh_layer_id field indicating the layer value, a 5-bit nuh_unit_type field indicating the NAL unit type, and a 3-bit nuh_temporal_id_plus1 field that is the Temporal ID value plus 1.

[0041] 3 is a diagram showing a hierarchical data structure of coded data Te in units of PU. The coded data Te illustratively includes a sequence and a plurality of pictures constituting the sequence. Fig. 3 shows a diagram showing a coded video sequence defining the sequence SEQ, a coded picture defining the picture PICT, a coded slice defining the slice S, coded slice data defining the slice data, a coding tree unit included in the coded slice data, and a coding unit included in the coding tree unit.

[0042] The coded video sequence defines a set of data that the image decoding device 31 refers to in order to decode the sequence SEQ to be processed. As shown in Fig. 3, the sequence SEQ includes a video parameter set VPS (Video Parameter Set), a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), an adaptation parameter set (APS), a picture PICT, and supplemental enhancement information SEI (Supplemental Enhancement Information).

[0043] The video parameter set VPS specifies a set of coding parameters common to multiple videos composed of multiple layers, as well as a set of coding parameters related to multiple layers included in the video and each individual layer.

[0044] The sequence parameter set SPS defines a set of coding parameters that the image decoding device 31 references to decode the target sequence. For example, the width and height of a picture are defined. Note that there may be multiple SPSs. In this case, one of the multiple SPSs is selected from the PPS.

[0045] The sequence parameter set SPS includes the following syntax elements: pic_width_max_in_luma_samples: This syntax element specifies the width of the widest image in a single sequence, in units of luminance blocks. The value of this syntax element must be a non-zero integer multiple of Max(8, MinCbSizeY), where MinCbSizeY is a value determined by the minimum size of a luminance block. pic_height_max_in_luma_samples: This syntax element specifies the height of the tallest image in a single sequence, in units of luminance blocks. The value of this syntax element must be a non-zero integer multiple of Max(8, MinCbSizeY).

[0046] The picture parameter set PPS defines a set of coding parameters that the image decoding device 31 refers to in order to decode each picture in the target sequence. Note that there may be multiple PPSs. In this case, one of the multiple PPSs is selected for each picture in the target sequence.

[0047] Here, the picture parameter set PPS includes the following syntax elements: pps_pic_width_in_luma_samples: This syntax element specifies the width of the target picture. The value of this syntax element is required to be a non-zero, an integer multiple of Max(8, MinCbSizeY), and a value equal to or less than sps_pic_width_max_in_luma_samples. InpPicWidthInLumaSamples, described below, is set equal to pps_pic_width_in_luma_samples. pps_pic_height_in_luma_samples: This syntax element specifies the height of the target picture. The value of this syntax element is required to be a non-zero, an integer multiple of Max(8, MinCbSizeY), and a value equal to or less than sps_pic_height_max_in_luma_samples. InpPicHeightInLumaSamples, described below, is set equal to pps_pic_height_in_luma_samples. pps_conformance_window_flag: A flag indicating whether conformance (cropping) window offset parameters will be subsequently signaled, indicating where the conformance window will be displayed. If this flag is 1, the parameter will be signaled; if it is 0, the conformance window offset parameters are not present. sps_chroma_format_id is a parameter indicating the chroma format. ChromaFormatIdc, described below, is set equal to sps_chroma_format_id.

[0048] (Coded Picture) A coded picture defines a set of data that the image decoding device 31 references in order to decode a picture PICT to be processed. As shown in Fig. 3, the picture PICT includes a picture header PH and slices 0 to NS-1 (NS is the total number of slices included in the picture PICT).

[0049] (Encoded Slice) An encoded slice defines a set of data to be referenced by the image decoding device 31 in order to decode a slice S to be processed. As shown in Fig. 3, a slice includes a slice header and slice data.

[0050] The slice header includes a group of coding parameters that the image decoding device 31 refers to in order to determine a decoding method for the current slice. Slice type designation information (slice_type) that designates the slice type is an example of a coding parameter included in the slice header.

[0051] Slice types that can be specified by the slice type specification information include (1) an I slice that uses only intra prediction during encoding, (2) a P slice that uses uni-prediction (L0 prediction) or intra prediction during encoding, and (3) a B slice that uses uni-prediction (L0 prediction or L1 prediction), bi-prediction, or intra prediction during encoding. Note that inter prediction is not limited to uni-prediction or bi-prediction, and a predicted image may be generated using more reference pictures. Hereinafter, the terms P and B slices refer to slices including blocks that can use inter prediction.

[0052] (Coded Slice Data) The coded slice data defines a set of data that the image decoding device 31 references in order to decode the slice data to be processed. The slice data includes a CTU, as shown in the coded slice header in Fig. 3. A CTU is a block of a fixed size (e.g., 64x64) that constitutes a slice, and is also called a Largest Coding Unit (LCU).

[0053] (Coding Tree Unit) Figure 3 defines a set of data that the image decoding device 31 references to decode the target CTU. The CTU is divided into coding units (CUs), which are basic units of encoding processing, by recursive quad tree (QT) division, binary tree (BT) division, or ternary tree (TT) division. A node in the tree structure obtained by recursive quad tree division is called a coding node. The intermediate nodes of the quad tree, binary tree, and ternary tree are coding nodes, and the CTU itself is defined as the top-level coding node.

[0054] 3 defines a set of data that the image decoding device 31 references in order to decode a coding unit to be processed. Specifically, a CU is composed of a CU header CUH, prediction parameters, transformation parameters, quantization transformation coefficients, etc. The CU header defines a prediction mode, etc.

[0055] The prediction process may be performed in units of CUs, or in units of sub-CUs obtained by further dividing a CU.

[0056] There are two types of prediction (prediction modes): intra prediction and inter prediction. Intra prediction is a prediction within the same picture, while inter prediction refers to a prediction process performed between different pictures (for example, between display times or between layer images).

[0057] The transformation and quantization processes are performed in units of CUs, but the quantized transformation coefficients may be entropy coded in units of sub-blocks such as 4x4.

[0058] In this specification, when referring to a "flag indicating whether or not XX" it means that a flag other than 0 (for example, 1) indicates XX, and 0 indicates that it is not XX, and in logical negation, logical product, etc., 1 is treated as true and 0 as false (the same applies below). However, in actual devices and methods, other values ​​may be used as true and false values.

[0059] (Configuration of Image Decoding Apparatus) The configuration of an image decoding apparatus 31 (FIG. 4) according to this embodiment will be described.

[0060] The image decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit (prediction image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a prediction image generation unit (prediction image generation device) 308, an inverse quantization and inverse transform unit 311, an adder 312, and a prediction parameter derivation unit 320. Note that, in accordance with the image coding device 11 described below, the image decoding device 31 may also be configured without including the loop filter 305.

[0061] The parameter decoding unit 302 further includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit), and the CU decoding unit 3022 further includes a TU decoding unit 3024. These may be collectively referred to as a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, PPS, and APS, and slice headers (slice information) from the coded data. The CT information decoding unit 3021 decodes the CT from the coded data. The CU decoding unit 3022 decodes the CU from the coded data. The TU decoding unit 3024 decodes QP update information (quantization correction value) and quantized prediction error (residual_coding) from the coded data.

[0062] The predicted image generating unit 308 includes an inter predicted image generating unit 309 and an intra predicted image generating unit 310 .

[0063] The entropy decoding unit 301 performs entropy decoding on the coded data Te input from the outside, and decodes each code (syntax element).

[0064] The entropy decoding unit 301 outputs the decoded code to the parameter decoding unit 302. The control of which code to decode is performed based on an instruction from the parameter decoding unit 302.

[0065] (Basic Flow) FIG. 5 is a flowchart illustrating the general operation of the image decoding device 31.

[0066] (S1100: Decode parameter set information) The header decoder 3020 decodes parameter set information such as VPS, SPS, and PPS from the coded data.

[0067] (S1200: Decode slice information) The header decoding unit 3020 decodes the slice header (slice information) from the coded data.

[0068] Thereafter, the image decoding device 31 repeats the processes from S1300 to S5000 for each CTU included in the current picture, thereby deriving a decoded image of each CTU.

[0069] (S1300: Decode CTU information) The CT information decoding unit 3021 decodes the CTU from the encoded data.

[0070] (S1400: Decode CT information) The CT information decoding unit 3021 decodes the CT from the encoded data.

[0071] (S1500: CU Decoding) The CU decoding unit 3022 performs S1510 and S1520 to decode the CU from the encoded data.

[0072] (S1510: Decode CU information) The CU decoding unit 3022 decodes CU information, prediction information, etc. from the coded data.

[0073] (S1520: Decode TU information) The TU decoding unit 3024 decodes QP update information, quantization prediction error, etc. from the coded data. Note that the QP update information is a difference value from the quantization parameter predicted value qPpred, which is a predicted value of the quantization parameter QP.

[0074] (S2000: Generate predicted image) The predicted image generation unit 308 generates a predicted image for each block included in the current CU based on prediction information.

[0075] (S3000: Inverse quantization and inverse transform) The inverse quantization and inverse transform unit 311 executes inverse quantization and inverse transform processing on each TU included in the target CU.

[0076] (S4000: Decoded image generation) The addition unit 312 generates a decoded image of the target CU by adding the predicted image supplied from the predicted image generation unit 308 and the prediction error supplied from the inverse quantization and inverse transform unit 311.

[0077] (S5000: Loop filter) The loop filter 305 applies a loop filter such as a deblocking filter, SAO, or ALF to the decoded image to generate a decoded image.

[0078] (Neural Network Post-Filter Characteristics (NNPFC) SEI) Fig. 7 shows a part of the syntax of the neural network post-filter characteristics (NNPFC) SEI message in Non-Patent Document 1. The supplementary enhancement information decoding device 91 decodes the syntax elements shown in Fig. 7. The NNPFC SEI message specifies a neural network to be applied as post-filter processing. The application of a specific post-filter processing to a specific picture is indicated by a neural network post-filter activation SEI message (Fig. 8).

[0079] To apply this SEI message, the following variables must be defined: the width InpPicWidthInLumaSamples and height InpPicHeightInLumaSamples of the picture decoded by the image decoding device 31, in luma pixel units; the picture luma pixel array CroppedYPic[idx] and chrominance pixel arrays CroppedCbPic[idx] and CroppedCrPic[idx], with idx ranging from 0 to numInputPics-1, used as input for post-filtering; the pixel bit depth BitDepthY of the luma pixel array; the pixel bit length BitDepthC of the chrominance pixel array; and variables SubWidthC and SubHeightC that indicate the chrominance format indicated by the chrominance format ChromaFormatIdc of the picture. When the aspect ratio is 4:2:0, the variables SubWidthC and SubHeightC are both 2; when the aspect ratio is 4:2:2, the variables SubWidthC and SubHeightC are 2 and 1; and when the aspect ratio is 4:4:4, the variables SubWidthC and SubHeightC are both 1.

[0080] nnpfc_auxiliary_inp_idc indicates the presence of auxiliary input data in the input tensor of the neural network postfilter. If the value is equal to 1, the strength control value StrengthControlVal (strength control information) of the input image is a real number ranging from 0 to 1. Note that the real number may be a decimal or floating-point number. The strength control value may also be a value obtained during the process of generating a decoded image, such as a quantization parameter.

[0081] nnpfc_purpose indicates the purpose of the process. If the value of nnpfc_purpose is 0, it may be used as determined by the application.

[0082] If the value of (nnpfc_purpose & 0x01) is greater than 0, it indicates that the purpose is to improve general visual quality.

[0083] If the value of (nnpfc_purpose & 0x02) is greater than 0, it indicates that chrominance upsampling (from 4:2:0 format to 4:2:2 or 4:4:4 format, or from 4:2:2 format to 4:4:4 format) is being performed.

[0084] If the value of (nnpfc_purpose & 0x04) is greater than 0, it indicates that there is a resolution sampling process (changing the picture width or height).

[0085] If the value of (nnpfc_purpose & 0x08) is greater than 0, it indicates that picture rate upsampling processing is being performed.

[0086] If the value of (nnpfc_purpose & 0x10) is greater than 0, it indicates that pixel bit length expansion (increase in pixel bit length of luminance or pixel bit length of chrominance) processing is being performed.

[0087] If the value of (nnpfc_purpose & 0x20) is greater than 0, it indicates that colorization (from 4:0:0 format to 4:2:0, 4:2:2, or 4:4:4 format) is to be performed.

[0088] Note that if ChromaFormatIdc is 3, the value of (nnpfc_purpose & 0x02) must be 0. Also, if the value of ChromaFormatIdc or (nnpfc_purpose & 0x02) is not 0, (nnpfc_purpose & 0x20) must be 0.

[0089] nnpfc_id indicates an identification number that can be used to identify the post-filter processing.

[0090] The value of nnpfc_id must be in the range 0 to 2^32-2. nnpfc_id values ​​from 256 to 511 and from 2^31 to 2^32-2 are reserved for future use.

[0091] If the NNPFC SEI message is the first NNPFC SEI message in decoding order with a particular nnpfc_id value within the current CLVS, the following applies: Indicates that this SEI message is a basic post-filtering operation. This SEI message pertains to the current decoded picture and all subsequent decoded pictures of the current layer until the end of the current CLVS.

[0092] nnpfc_base_flag indicates whether this NNPFC SEI message specifies a neural-network post-filter (NNPF) operation or an update to a base NNPF. A value of 1 in nnpfc_base_flag indicates that the SEI message specifies a base NNPF. A value of 0 in nnpfc_base_flag indicates that the SEI message updates a base NNPF.

[0093] The following constraints apply to the value of nnpfc_base_flag: If the NNPFC SEI message is the first NNPFC SEI message with a particular nnpfc_id value in the current CLVS in decoding order, then the value of nnpfc_base_flag must be 1. If the NNPFC SEI message nnpfcB is not the first NNPFC SEI message in decoding order, has a particular nnpfc_id value in the current CLVS, and the value of nnpfc_base_flag is 1, then the NNPFC SEI message is a repeat of the first message. The contents of the NNPFC SEI message nnpfcA with the same nnpfc_id value in decoding order must be identical.

[0094] If the value of nnpfc_base_flag is 0, the following applies: This SEI message defines updates relative to the previous base NNPF in decoding order with the same nnpfc_id value. Updates are not cumulative; each update applies to the base NNPF. The base NNPF is the NNPF specified by the first NNPFC SEI message in decoding order and with a particular nnpfc_id value in the current CLVS. The NNPF defined by this SEI message can be obtained by applying the updates defined by this SEI message to the base NNPF with the same nnpfc_id value. This SEI message pertains to the current decoded picture and all subsequent decoded pictures of the current layer in output order.

[0095] Neural Network Postfilter Activation (NNPFA) SEI Figure 8 shows the syntax of the Neural Network Postfilter Activation (NNPFA) SEI message. The Neural Network Postfilter Activation NNPFA SEI message activates or deactivates the application of a target neural network postfilter identified by nnpfa_target_id for postfiltering of a series of pictures. If the value of nnpfa_target_base_flag is 1, the target NNPF is the base NNPF with nnpfc_id equal to nnpfa_target_id. Otherwise (if the value of nnpfa_target_base_flag is 0), the target NNPF is the NNPF specified by the last NNPFC SEI message with nnpfc_id equal to nnpfa_target_id that is earlier in decoding order than the first VCL NAL unit of the current picture.

[0096] Note that there may be multiple NNPFA SEI messages for the same picture.

[0097] nnpfa_target_id indicates the neural network post-filtering of the target picture. This identifies one or more NNPFC SEI messages with nnpfc_id equal to nnpfa_target_id for the current picture. nnpfc_id is an identification number that identifies an NNPFC SEI message, and setting this identification number to nnpfa_target_id makes it possible to identify an NNPFC SEI message.

[0098] An NNPFA SEI message with a particular value of nnpfa_target_id must not exist in the current PU unless one or both of the following conditions are true: - There is an NNPFC SEI message in the current CLVS with nnpfc_id equal to a particular value of nnpfa_target_id present in a PU preceding the current PU in decoding order. - There is an NNPFC SEI message with nnpfc_id equal to a particular value of nnpfa_target_id in the current PU.

[0099] If a PU contains both an NNPFC SEI message with a particular value of nnpfc_id and an NNPFA SEI message with nnpfa_target_id equal to a particular value of nnpfc_id, the NNPFC SEI message shall precede the NNPFA SEI message in decoding order.

[0100] If nnpfa_cancel_flag is 1, it indicates that the continuity of the target neural network postfiltering set up by a previous NNPFA SEI message with the same nnpfa_target_id as the current SEI message is canceled. That is, the target neural network postfiltering will not be used unless activated by another NNPFA SEI message with the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag equal to 0. If nnpfa_cancel_flag is 0, it indicates that nnpfa_persistence_flag will persist.

[0101] nnpfa_persistence_flag indicates the continuation of the target neural network post-filtering process for the current layer.

[0102] If nnpfa_persistence_flag is 0, it indicates that the target neural network postfiltering is applied to postfiltering of the current picture only.

[0103] If nnpfa_persistence_flag is 1, it indicates that the targeted neural network postfiltering should be applied to the current picture of the current layer and all subsequent pictures in output order until one or more of the following conditions become true: A new CLVS for the current layer begins. The bitstream ends. Pictures of the current layer associated with an NNPFA SEI message with the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag equal to 1 are output after the current picture in output order.

[0104] Note that neural network post-filtering is not applied to subsequent pictures of the current layer associated with the NNPFA SEI message if they have the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag is 1.

[0105] nnpfa_num_output_entries specifies the number of nnpfa_output_flag[i] syntax elements present in the NNPFA SEI message. The value of nnpfa_num_output_entries ranges from 0 to the number of pictures output from the network.

[0106] If the value of nnpfa_output_flag[i] is 1, it indicates that the picture generated corresponding to index i is to be output. If the value of nnpfa_output_flag[i] is 0, it indicates that the picture generated corresponding to index i is not to be output. If nnpfa_num_output_entries is smaller than the number of pictures to be output from the network, the remaining values ​​of nnpfa_output_flag[i] are assumed to be equal to 1.

[0107] Furthermore, the auxiliary extension information creating device 71, the auxiliary extension information encoding device 81, and the auxiliary extension information decoding device 91 may commonly hold general-purpose network parameters. The auxiliary extension information creating device 71 uses a framework such as a neural network post-filter characteristic SEI to create, as auxiliary extension information, network parameters for partially updating the commonly held general-purpose network. The auxiliary extension information may then be encoded by the auxiliary extension information encoding device 81 and decoded by the auxiliary extension information decoding device 91. With this configuration, the amount of code for the auxiliary extension information can be reduced, and auxiliary extension information according to the input image T can be created, encoded, and decoded.

[0108] In addition, in order to support multiple formats as the transmission format of the network parameters, a parameter (identifier) ​​indicating the format may be sent. Furthermore, the actual supplemental extension information following the identifier may be transmitted as a byte string.

[0109] The auxiliary extension information of the network parameters decoded by the auxiliary extension information decoder 91 is input to the post-filter processor 61 .

[0110] The post-filtering device 61 performs post-image processing using a neural network using the decoded auxiliary enhancement information (neural network post-filter characteristics SEI, neural network post-filter activation SEI) to restore the decoded video Td.

[0111] The auxiliary extension information encoding device 81 encodes the auxiliary extension information based on the syntax tables of Figures 7 and 8. The auxiliary extension information is encoded as auxiliary extension information SEI, multiplexed into the encoded data Te output by the image encoding device 11, and output to the network 21.

[0112] The auxiliary extension information decoding device 91 decodes the auxiliary extension information from the encoded data Te based on the syntax tables in Figures 7 and 8, and sends the decoding result to the post-filter processing device 61 and the image recognition device 51 (not shown). The auxiliary extension information decoding device 91 decodes the auxiliary extension information coded as auxiliary extension information SEI.

[0113] The post-filtering device 61 performs post-image processing on the decoded video Td1 using the decoded video Td1 and the auxiliary extension information, and generates post-image processing Td2.

[0114] In this example of the present embodiment, the syntax is shown as SEI, but is not limited to SEI, and syntax such as SPS, PPS, APS, slice header, etc. may also be used.

[0115] In this embodiment, in such a configuration, a method is provided that can improve image quality even when the transmission rate is low, and can efficiently encode and decode supplemental extension information regardless of the prediction structure.

[0116] (Annotated Region SEI) Figures 9 and 10 show the syntax of the annotated region SEI disclosed in Non-Patent Document 2. The annotated region SEI will be described below.

[0117] The Annotated Region SEI message contains parameters that identify the annotated region using a rectangular bounding box that represents the size and location of the identified object within the picture. To use this SEI message, the following variables must be defined: - CroppedWidth, CroppedHeight, the width and height in pixels of the luminance of the picture output by the decoder that this SEI is intended to process - ChromaFormatIdc, a value that represents the chrominance format, where 0 is 4:0:0 (monochrome), 1 is 4:2:0, 2 is 4:2:2, and 3 is 4:4:4 format. If the value of ChromaFormatIdc is 0, the values ​​of SubWidthC and SubHeightC are both 1; if the value of ChromaFormatIdc is 1, the values ​​of SubWidthC and SubHeightC are both 2; if the value of ChromaFormatIdc is 2, the value of SubWidthC is 2 and the value of SubHeightC is 1; if the value of ChromaFormatIdc is 3, the values ​​of SubWidthC and SubHeightC are both 1.

[0118] In addition, in the annotated region SEI of Non-Patent Document 2, the left offset of the conformance cropping window ConfWinLeftOffset and the top offset of the conformance cropping window ConfWinTopOffset are defined as inputs, but in this embodiment, the decoded image input to the SEI is defined as already cropped, so these are not used as inputs.

[0119] ar_cancel_flag equal to 1 indicates that the SEI message cancels the application of a previous annotated regions SEI message associated with the layer or layers to which the annotated regions SEI message applies. ar_cancel_flag equal to 0 indicates that annotated regions information is present and will be applied. When ar_cancel_flag equals 1 or a new CVS for the current layer is started, the variables LabelAssigned[i], ObjectTracked[i], and ObjectBoundingBoxAvail are set to 0 for i in the range of values ​​from 0 to 255.

[0120] ar_not_optimized_for_viewing_flag equal to 1 indicates that the processed picture to which the annotated regions SEI message applies is not optimized for user viewing, but is optimized for other purposes such as algorithmic object classification performance. ar_not_optimized_for_viewing_flag equal to 0 indicates that it is unknown whether the processed picture to which the annotated regions SEI message applies is optimized for user viewing.

[0121] ar_true_motion_flag equal to 1 indicates that the motion information in the processed picture to which the annotated region SEI message applies has been selected for the purpose of accurately representing the object motion of the objects in the annotated region. ar_true_motion_flag equal to 0 indicates that it is unknown whether the motion information in the processed picture to which the annotated region SEI message applies may be selected for the purpose of accurately representing the object motion of the objects in the annotated region.

[0122] When ar_occluded_object_flag is equal to 1, it indicates that the ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] syntax elements represent the size and position of an object or part of an object that may not be displayed or may only be partially displayed in the cropped decoded picture. If ar_occluded_object_flag is equal to 0, it indicates that the ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] syntax elements describe the size and position of the object that will be fully displayed in the cropped decoded picture. The value of ar_occluded_object_flag must be the same for all annotated_regions() syntaxes in the CVS.

[0123] If ar_partial_object_flag_present_flag is equal to 1, it indicates that the ar_partial_object_flag[ar_object_idx[i]] syntax element is present. If ar_partial_object_flag_present_flag is equal to 0, it indicates that the ar_partial_object_flag[ar_object_idx[i]] syntax element is not present. The value of ar_partial_object_flag_present_flag must be the same for all annotated_regions() syntaxes in the CVS.

[0124] ar_object_label_present_flag equal to 1 indicates that there is label information corresponding to the objects in the annotated region. ar_object_label_present_flag equal to 0 indicates that there is no label information corresponding to the objects in the annotated region.

[0125] ar_object_confidence_info_present_flag equal to 1 indicates the presence of the ar_object_confidence[ar_object_idx[i]] syntax element. ar_object_confidence_info_present_flag equal to 0 indicates the absence of the ar_object_confidence[ar_object_idx[i]] syntax element. It is a bitstream conformance requirement that the value of ar_object_confidence_present_flag be the same for all annotated_regions() syntax structures in CVS.

[0126] ar_object_confidence_length_minus1 plus 1 specifies the length in bits of the ar_object_confidence[ar_object_idx[i]] syntax element. It is a bitstream conformance requirement that the value of ar_object_confidence_length_minus1 be the same for all annotated_regions() syntax structures in CVS.

[0127] If ar_object_label_language_present_flag is equal to 1, it indicates that the ar_object_label_language syntax element is present. If ar_object_label_language_present_flag is equal to 0, it indicates that the ar_object_label_language syntax element is not present.

[0128] ar_bit_equal_to_zero shall be equal to zero.

[0129] ar_object_label_language contains a language tag as specified in IETF RFC 5646, followed by a NULL termination byte equal to 0x00. The length of the ar_object_label_language syntax element MUST be less than or equal to 255 bytes, not including the NULL termination byte. If ar_object_label_language is not present, the label language is unspecified.

[0130] ar_num_label_updates indicates the total number of labels associated with the annotated region being signaled. The value of ar_num_label_updates must be in the range of 0 to 255.

[0131] ar_label_idx[i] indicates the index of the signaled label. The value of ar_label_idx[i] must be in the range of 0 to 255.

[0132] If ar_label_cancel_flag is equal to 1, the application of the ar_label_idx[i]th label is canceled. If ar_label_cancel_flag is equal to 0, it indicates that the ar_label_idx[i]th label is assigned the signaled value.

[0133] LabelAssigned[ar_label_idx[i]] equal to 1 indicates that the ar_label_idx[i]th label is assigned. LabelAssigned[ar_label_idx[i]] equal to 0 indicates that the ar_label_idx[i]th label is not assigned.

[0134] ar_label[ar_label_idx[i]] specifies the content of the ar_label_idx[i]th label. The length of the ar_label[ar_label_idx[i]] syntax element must be 255 bytes or less, not including the NULL termination byte.

[0135] ar_num_object_updates indicates the number of object updates to be signaled. ar_num_object_updates must be a value between 0 and 255.

[0136] ar_object_idx[i] is the index of the object parameter being signaled. Note that ar_object_idx[i] must be in the range of 0 to 255.

[0137] If ar_object_cancel_flag is equal to 1, the persistence of the ar_object_idx[i]th object is cancelled. If ar_object_cancel_flag is equal to 0, it indicates that the parameters associated with the ar_object_idx[i]th object tracked object are signalled.

[0138] ObjectTracked[ar_object_idx[i]] equal to 1 indicates that the ar_object_idx[i]th object is tracked. ObjectTracked[ar_object_idx[i]] equal to 0 indicates that the ar_object_idx[i]th object is not tracked.

[0139] If ar_object_label_update_flag is equal to 1, it indicates that the object label (label information) is signaled. If ar_object_label_update_flag is equal to 0, it indicates that the object label is not signaled.

[0140] ar_object_label_idx[ar_object_idx[i]] indicates the index of the label corresponding to the ar_object_idx[i]th object. If ar_object_label_idx[ar_object_idx[i]] does not exist, its value is inferred from the value of the previous annotated region SEI message in the same CVS output order, if any. Note that the value of ar_object_label_idx[ar_object_idx[i]] must be in the range of 0 to 255.

[0141] ar_bounding_box_update_flag equal to 1 indicates that the object's bounding box parameters are signaled. ar_bounding_box_update_flag equal to 0 indicates that the object's bounding box parameters are not signaled.

[0142] If ar_bounding_box_cancel_flag is equal to 1, the persistence of ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] is cancelled. If ar_bounding_box_cancel_flag is equal to 0, the following syntax elements are signaled: ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], ar_bounding_box_height[ar_object_idx[i]], ar_partial_object_flag[ar_object_idx[i]], and ar_object_confidence[ar_object_idx[i]].

[0143] ObjectBoundingBoxAvail[ar_object_idx[i]] equal to 1 indicates that bounding box information for the ar_object_idx[i]th object is signaled. ObjectBoundingBoxAvail[ar_object_idx[i]] equal to 0 indicates that bounding box information for the ar_object_idx[i]th object is not signaled.

[0144] ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] specify the coordinates of the top left corner, width, and height of the bounding box of the ar_object_idx[i]th object in the cropped decoded picture, relative to the conformance cropping window, respectively.

[0145] The value of ar_bounding_box_left[ar_object_idx[i]] must be in the range of 0 to CroppedWidth / SubWidthC-1. The value of ar_bounding_box_top[ar_object_idx[i]] must be in the range of 0 to CroppedHeight / SubHeightC-1. The value of ar_bounding_box_width[ar_object_idx[i]] must be in the range of 0 to CroppedWidth / SubWidthC-ar_bounding_box_left[ar_object_idx[i]]. The value of ar_bounding_box_height[ar_object_idx[i]] must be in the range of 0 to CroppedHeight / SubHeightC-ar_bounding_box_top[ar_object_idx[i]].

[0146] The rectangle of the identified object contains luminance pixels with horizontal picture coordinates from SubWidthCar_bounding_box_left[ar_object_idx[i]] to SubWidthC*ar_bounding_box_left[ar_object_idx[i]]+ar_bounding_box_width[ar_object_idx[i]]-1 and vertical picture coordinates from SubHeightC*ar_bounding_box_top[ar_object_idx[i]] to SubHeightC*ar_bounding_box_top[ar_object_idx[i]]+ar_bounding_box_height[ar_object_idx[i]]-1.

[0147] In addition, in the annotated region SEI of Non-Patent Document 2, the left offset of the conformance cropping window ConfWinLeftOffset and the top offset of the conformance cropping window ConfWinTopOffset are defined as inputs. Therefore, the rectangle of the identified object is defined as including luminance pixels having horizontal picture coordinates from SubWidthC*(ConfWinLeftOffset+ar_bounding_box_left[ar_object_idx[i]] to SubWidthC*(ConfWinLeftOffset+ar_bounding_box_left[ar_object_idx[i]]+ar_bounding_box_width[ar_object_idx[i]])-1 and vertical picture coordinates from SubHeightC*(ConfWinTopOffset+ar_bounding_box_top[ar_object_idx[i]]) to SubHeightC*(ConfWinTopOffset+ar_bounding_box_top[ar_object_idx[i]]+ar_bounding_box_height[ar_object_idx[i]])-1. However, in this embodiment, the decoded image input to SEI is defined as having already been cropped, and therefore the above definition is not used.

[0148] If ChromaFormatIdc is not equal to 0, the corresponding pixel of the specified pixel in the two chrominance arrays is the pixel with picture coordinates (x / SubWidthC, y / SubHeightC), where (x, y) are the picture coordinates of the pixel with the specified luminance.

[0149] The values ​​of ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] are maintained for each value of ar_object_idx[i] in output order within CVS. If not present, the values ​​of ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], or ar_bounding_box_height[ar_object_idx[i]] are inferred from the value of the previous Annotated Region SEI message in output order within CVS, if present.

[0150] ar_partial_object_flag[ar_object_idx[i]] equal to 1 indicates that the ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] syntax elements describe the size and position of an object that is only partially displayed in the cropped decoded picture. If ar_partial_object_flag[ar_object_idx[i]] is equal to 0, it indicates that the ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] syntax elements may or may not represent the size and position of an object that is only partially displayed in the cropped decoded picture. If ar_partial_object_flag[ar_object_idx[i]] is not present, the value of ar_partial_object_flag[ar_object_idx[i]] is inferred from the value of the previous Annotated Region SEI message in output order in the CVS, if one exists.

[0151] ar_object_confidence[ar_object_idx[i]] indicates the confidence associated with the ar_object_idx[i]th object in units of 2-(ar_object_confidence_length_minus1+1). A larger value of ar_object_confidence[ar_object_idx[i]] indicates a higher confidence. The length of the ar_object_confidence[ar_object_idx[i]] syntax element is ar_object_confidence_length_minus1+1 bits. If not present, the value of ar_object_confidence[ar_object_idx[i]] is inferred from the value of the previous annotated region SEI message in CVS output order, if present.

[0152] The method disclosed in Non-Patent Document 1 shows a framework for performing post-filtering using a neural network, but has the problem that it is not possible to perform filtering on some areas of a picture.

[0153] In Non-Patent Document 1, the Neural Network Post-Filter Characteristics SEI message defines the content of the filter processing, and the Neural Network Post-Filter Activation SEI message is used to determine the application of filter processing to each picture, so that only the application of neural network filter processing can be controlled on a picture-by-picture basis.

[0154] Furthermore, the method disclosed in Non-Patent Document 2 can indicate annotated regions, but has the problem that it does not show how to use the annotated regions when applied to post-filtering.

[0155] In this embodiment, a new SEI is proposed to solve the above problem.

[0156] Figure 11(a) shows an example of a Neural Network Post-Filter Regions (NNPFR) SEI message. This SEI is used together with the annotated region SEI to define the identification number of the NNPFC SEI to be combined with. By using this together with the NNPFA SEI, which also specifies the ID of the NNPFC SEI, neural network post-filtering for each region defined in the annotated region SEI message is realized.

[0157] Note that there may be multiple NNPFR SEI messages for the same picture.

[0158] nnpfpr_target_id is an identification number for identifying the NNPFC SEI for performing neural network post-filtering on the annotated region SEI of the current picture. By setting nnpfc_id, which is the identification number of the NNPFC SEI message, to nnpfr_target_id, one or more NNPFC SEI messages with the same identification number can be identified.

[0159] It also identifies an NNPFA SEI message with nnpfa_target_id that has the same value as nnpfr_target_id, and indicates that neural network post-filtering of the region of the bounding box defined by the annotated region SEI is to be performed on the current picture.

[0160] An NNPFR SEI message with a specific value of nnpfr_target_id must not exist in the current PU unless one or both of the following conditions are true: - There is an NNPFC SEI message in the current CLVS with nnpfc_id equal to a specific value of nnpfr_target_id in a PU that precedes the current PU in decoding order. - There is an NNPFC SEI message with nnpfc_id equal to a specific value of nnpfr_target_id in the current PU. - An annotated region SEI exists in the current PU.

[0161] If a PU contains both an NNPFC SEI message with a particular value of nnpfc_id and an NNPFR SEI message with nnpfr_target_id equal to a particular value of nnpfc_id, the NNPFC SEI message shall precede the NNPFR SEI message in decoding order.

[0162] This method uniquely synchronizes the activation timing of the NNPFC SEI with the timing of the annotated region, enabling the realization of Region of Interest (ROI) that controls the image quality of specific regions of a picture through filtering. Specifically, the NNPFC SEI message functions enable ROI image quality improvement, frame rate increase, super-resolution processing, color space conversion, and color restoration. These functions are useful not only for improving human subjective evaluation but also for video coding methods for machine recognition.

[0163] Figure 11(b) is another example of a Neural Network Post-Filter Regions (NNPFR) SEI message. In this embodiment, a threshold value for the reliability of annotated region information is added as a syntax element to the NNPFR SEI message shown in Figure 11(a). If the annotated region SEI message contains region reliability information and the reliability is greater than the threshold value, neural network post-filtering is performed on the bounding box region.

[0164] The value of object_confidence_threshold is compared with the value of object confidence ar_object_confidence in the annotated region SEI, and neural network post-filtering is performed only on objects whose value is greater than object_confidence_threshold.

[0165] If the value of ar_object_confidence_info_present_flag in the annotated region SEI is 0, that is, if there is no object confidence information, the value of object_confidence_threshold is ignored.

[0166] Note that the value of object_confidence_threshold may be compared with the value of ar_object_confidence, which is the reliability of the object in the annotated region SEI, and neural network post-filtering may be performed only on objects that are equal to or greater than the value of object_confidence_threshold.

[0167] Conversely, the value of object_confidence_threshold may be compared with the value of the object confidence ar_object_confidence in the annotated region SEI, and neural network post-filtering may be performed only on objects whose values ​​are less than or equal to the value of object_confidence_threshold.

[0168] Fig. 11(c) is another example of the Neural Network Post-Filter Regions (NNPFR) SEI message. In this embodiment, in addition to Fig. 11(a), syntax for the number of labels to be activated and the label name is added so that activation can be performed only on regions with specific object labels (label information).

[0169] num_of_activating_label indicates the number of object labels to identify. The value of num_of_activating_label must be greater than or equal to 0 and less than or equal to the value of the ar_num_label_updates syntax element of the annotated region SEI.

[0170] activating_label indicates the character string of the object label to be identified. The content of the object label array ar_label[] of the annotated region SEI is compared, and neural network filtering is performed on the object with the matching label name.

[0171] If the value of ar_object_label_present_flag is 0, that is, if the object label does not exist, num_of_activating_label and activating_label are ignored.

[0172] Multiple NNPFR SEI messages may exist for the same picture, and by specifying the identification number of another NNPFR SEI message in nnpfr_target_id, it is possible to apply different neural network post-filtering processes to regions with object label A and regions with object label B, for example.

[0173] In the example of FIG. 11(c), the region is specified by the object label, but it may be specified using the value of the object index ar_object_idx in the annotated region SEI message.

[0174] [Application Example] The above-described video encoding device 10 and video decoding device 30 can be installed and used in various devices that transmit, receive, record, and play back video. Note that the video may be natural video captured by a camera or the like, or artificial video (including CG and GUI) generated by a computer or the like.

[0175] (SEI Payload) FIG. 13 is a diagram showing the syntax of the SEI payload, which is a container for an SEI message.

[0176] This is called when nal_unit_type is PREFIX_SEI_NUT. PREFIX_SEI_NUT indicates that the SEI is located before the slice data.

[0177] When payloadType is 202, the annotated region SEI is called.

[0178] When payloadType is 210, the neural network post-filter feature SEI is called.

[0179] When payloadType is 211, the neural network post-filter activation SEI is called.

[0180] When payloadType is 213, the neural network postfilter domain SEI is called.

[0181] (SEI Decoding and Post-Filter Processing) The header decoding unit 3020 reads the SEI payload, which is a container of the SEI message, and decodes the neural network post-filter characteristics SEI message. For example, the header decoding unit 3020 decodes nnpfc_purpose, nnpfc_id, nnpfc_base_flag, etc.

[0182] 14 is a flowchart showing the processing of the post-filtering processor 61. The post-filtering processor 61 performs the following processing in accordance with the parameters of the SEI message.

[0183] S6001: Read the processing volume and accuracy from SEI.

[0184] S6002: If the amount of post processing exceeds the processable complexity, the process ends. If not, the process proceeds to S6003.

[0185] S6003: End if the accuracy exceeds the processing capability of the post-filtering processor 61. If not, proceed to S6004.

[0186] S6004: A network model is identified from the SEI, and the topology of the post-filter processing device 61 is set.

[0187] S6005: Derive network model parameters from the updated information of the SEI.

[0188] S6006: The derived network model parameters are read into the post-filter processor 61.

[0189] S6007: The post-filter processor 61 executes filtering and outputs the result to the outside.

[0190] However, the SEI is not necessarily required to construct luma and chroma samples in the decoding process.

[0191] (Details of the post-filtering device 61) The NN filter unit performs filtering using a neural network model, using the input image inputTensor and input parameters (e.g., QP, bS, etc.). The input image may be an image for each component, or an image with multiple components each as a channel. In addition, the input parameters may be assigned to a channel different from the image.

[0192] The NN filter unit may repeatedly apply the following process.

[0193] The NN filter part derives the output image outputTensor by convolving (conv, convolution) the inputTensor with the kernel k[m][i][j] and adding bias, where nn=0..n-1, xx=0..width-1, yy=0..height-1, and Σ represents the summation for mm, i, and j, respectively.

[0194] outputTensor[nn][xx][yy]=ΣΣΣ(k[mm][i][j]*inputTensor[mm][xx+i-of][yy+j-of]+bias[nn]) For a 1x1 Conv, Σ represents the sum of mm=0..m-1, i=0, j=0, respectively. In this case, set of=0. For a 3x3 Conv, Σ represents the sum of mm=0..m-1, i=0..2, j=0..2, respectively. In this case, set of=1. n is the number of channels in outSamples, m is the number of channels in inputTensor, width is the width of inputTensor and outputTensor, and height is the height of inputTensor and outputTensor. of is the size of the padding area around inputTensor to make the sizes of inputTensor and outputTensor the same. In the following, when the output of the NN filter unit is a value (corrected value) rather than an image, the output will be represented as corrNN instead of outputTensor.

[0195] Note that if you write inputTensor and outputTensor in CHW format instead of CWH format, it is equivalent to the following process.

[0196] outputTensor[nn][yy][xx]=ΣΣΣ(k[mm][i][j]*inputTensor[mm][yy+j-of][xx+i-of]+bias[nn]) Alternatively, you can perform a process called Depth-wise Conv, which is shown in the following formula. Here, nn=0..n-1, xx=0..width-1, yy=0..height-1, and Σ represents the sum for i and j, respectively. n is the number of channels of outputTensor and inputTensor, width is the width of inputTensor and outputTensor, and height is the height of inputTensor and outputTensor.

[0197] outputTensor[nn][xx][yy]=ΣΣ(k[nn][i][j]*inputTensor[nn][xx+i-of][yy+j-of]+bias[nn]) You can also use nonlinear processing called Activate, such as ReLU. ReLU(x) = x >= 0 ? x : 0 You can also use leakyReLU, as shown in the following formula.

[0198] leakyReLU(x) = x >= 0 ? x : a * x where a is a predetermined value, such as 0.1 or 0.125. To perform integer arithmetic, all the values ​​of k, bias, and a above can be integers, and a right shift can be performed after conv.

[0199] With ReLU, values ​​less than 0 are always output as 0, and values ​​greater than or equal to 0 output the input value as is. On the other hand, with leakyReLU, linear processing is performed for values ​​less than 0 using the gradient set by a. With ReLU, the gradient for values ​​less than 0 disappears, which can make learning difficult. With leakyReLU, the gradient for values ​​less than 0 remains, making the above problem less likely to occur. Also, of the above leakyReLU(x), PReLU, which uses a parameterized value of a, can be used.

[0200] (NNC) Neural Network Coding (NNC) is an international standard ISO / IEC15938-17 for efficiently compressing neural networks (NNs). Compressing trained NNs makes it possible to store and transmit them more efficiently.

[0201] The following provides an overview of the encoding and decoding processes of the NNC.

[0202] FIG. 15 is a diagram showing an encoding device and a decoding device of the NNC.

[0203] The NN encoding device 801 has a pre-processing unit 8011, a quantization unit 8012, and an entropy encoding unit 8013. The NN encoding device 801 receives an uncompressed NN model O as input, and the quantization unit 8012 quantizes the NN model O to obtain a quantized model Q. Before quantization, the NN encoding device 801 may repeatedly apply parameter reduction techniques such as pruning and sparsification in the pre-processing unit 8011. Then, the entropy encoding unit 8013 applies entropy coding to the quantized model Q to obtain a bitstream S for storing and transmitting the NN model.

[0204] The NN decoding device 802 includes an entropy decoding unit 8021, a parameter restoration unit 8022, and a post-processing unit 8023. The NN decoding device 802 first receives the transmitted bitstream S, and the entropy decoding unit 8021 performs entropy decoding of S to obtain an intermediate model RQ. If the operating environment of the NN model supports inference using the quantized representation used in the RQ, the RQ may be output and used for inference. If not, the parameter restoration unit 8022 restores the parameters of the RQ to their original representation to obtain an intermediate model RP. If the sparse tensor representation used can be processed in the operating environment of the NN model, the RP may be output and used for inference. If not, a reconstructed NN model R that does not include a tensor or structural representation different from that of the NN model O is obtained and output.

[0205] The NNC standard provides decoding methods for specific NN parameter numerical representations, such as integers and floating point numbers.

[0206] The decoding method NNR_PT_INT decodes models with integer-valued parameters. The decoding method NNR_PT_FLOAT extends NNR_PT_INT by adding a quantization step size delta. This delta is multiplied by the integer value to produce a scaled integer. Delta is derived from the integer quantization parameter qp and the delta granularity parameter qp_density as follows:

[0207] mul = 2^(qp_density) + (qp & (2^(qp_density)-1)) delta = mul * 2^((qp >> qp_density)-qp_density) (Format of trained NN) The representation of a trained NN consists of two elements: a topological representation such as the size of layers and connections between layers, and a parameter representation such as weights and biases.

[0208] Topology representation is covered by native formats such as TensorFlow (registered trademark) and PyTorch (registered trademark), but to improve interoperability, exchange formats such as Open Neural Network Exchange Format (ONNX, registered trademark) and Neural Network Exchange Format (NNEF) exist.

[0209] The NNC standard also transports topology information nnr_topology_unit_payload as part of the NNC bitstream containing compressed parameter tensors, enabling interoperability with topology information expressed in native formats as well as interchange formats.

[0210] (Configuration of Image Encoding Device) Next, the configuration of the image encoding device 11 according to this embodiment will be described. Fig. 6 is a block diagram showing the configuration of the image encoding device 11 according to this embodiment. The image encoding device 11 includes a prediction image generation unit 101, a subtraction unit 102, a transformation / quantization unit 103, an inverse quantization / inverse transformation unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference image storage unit, frame memory) 109, a coding parameter determination unit 110, a parameter encoding unit 111, a prediction parameter derivation unit 120, and an entropy encoding unit 104.

[0211] The predicted image generating unit 101 generates a predicted image for each CU.

[0212] The subtraction unit 102 generates a prediction error by subtracting the pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 from the pixel values ​​of the image T. The subtraction unit 102 outputs the prediction error to the transformation / quantization unit 103.

[0213] The transform / quantization unit 103 calculates transform coefficients by frequency transforming the prediction errors input from the subtraction unit 102, and derives quantized transform coefficients by quantizing them. The transform / quantization unit 103 outputs the quantized transform coefficients to the parameter coding unit 111 and the inverse quantization / inverse transform unit 105.

[0214] The inverse quantization and inverse transform unit 105 is the same as the inverse quantization and inverse transform unit 311 (FIG. 4) in the image decoding device 31, and therefore a description thereof will be omitted. The calculated prediction error is output to the adder .

[0215] The parameter coding unit 111 includes a header coding unit 1110, a CT information coding unit 1111, and a CU coding unit 1112 (prediction mode coding unit). The CU coding unit 1112 further includes a TU coding unit 1114. The following describes an outline of the operation of each module.

[0216] The header encoding unit 1110 performs encoding processing of parameters such as header information, division information, prediction information, and quantized transform coefficients.

[0217] The CT information encoding unit 1111 encodes QT, MT (BT, TT) division information and the like.

[0218] The CU encoding unit 1112 encodes CU information, prediction information, division information, and the like.

[0219] When a prediction error is included in a TU, the TU encoding unit 1114 encodes the QP update information and the quantized prediction error.

[0220] The CT information encoding unit 1111 and the CU encoding unit 1112 supply syntax elements such as inter prediction parameters and quantized transform coefficients to the parameter encoding unit 111.

[0221] The entropy coding unit 104 receives the quantized transform coefficients and coding parameters from the parameter coding unit 111. The entropy coding unit 104 entropy codes these to generate and output coded data Te.

[0222] The prediction parameter derivation unit 120 derives inter prediction parameters and intra prediction parameters from the parameters input from the encoding parameter determination unit 110. The derived inter prediction parameters and intra prediction parameters are output to the parameter encoding unit 111.

[0223] The adder 106 generates a decoded image by adding, for each pixel, the pixel values ​​of the predicted block input from the predicted image generation unit 101 and the prediction errors input from the inverse quantization and inverse transform unit 105. The adder 106 stores the generated decoded image in a reference picture memory 109.

[0224] The loop filter 107 performs deblocking filtering, SAO, and ALF on the decoded image generated by the adder 106. Note that the loop filter 107 does not necessarily have to include the above three types of filters, and may be configured with only a deblocking filter, for example.

[0225] The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in a predetermined location for each current picture and CU.

[0226] The reference picture memory 109 stores the decoded image generated by the loop filter 107 at a predetermined location for each current picture and CU.

[0227] The coding parameter determination unit 110 selects one set of coding parameters from among multiple sets of coding parameters. The coding parameters are the above-mentioned QT, BT, or TT division information, prediction parameters, or parameters to be coded that are generated in relation to these. The predicted image generation unit 101 generates a predicted image using these coding parameters.

[0228] Note that a portion of the image encoding device 11 and the image decoding device 31 in the above-described embodiments, such as the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the predicted image generation unit 308, the inverse quantization / inverse transform unit 311, the adder 312, the prediction parameter derivation unit 320, the predicted image generation unit 101, the subtractor 102, the transform / quantization unit 103, the entropy encoding unit 104, the inverse quantization / inverse transform unit 105, the loop filter 107, the encoding parameter determination unit 110, the parameter encoding unit 111, and the prediction parameter derivation unit 120, may be implemented by a computer. In this case, a program for implementing this control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read and executed by a computer system. Note that the "computer system" referred to here refers to a computer system built into either the image encoding device 11 or the image decoding device 31, and includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0229] Furthermore, part or all of the image encoding device 11 and the image decoding device 31 in the above-described embodiments may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image encoding device 11 and the image decoding device 31 may be individually implemented as a processor, or part or all of them may be integrated into a processor. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0230] One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like are possible within the scope that does not deviate from the gist of the present invention.

[0231] This embodiment will be explained based on Figure 1. The video decoding device 30 includes an image decoding device 31 that decodes encoded data to generate a decoded image, a post-filter processing device 61 that performs post-filter processing on the decoded image, and an auxiliary extension information decoding device 91 that decodes multiple auxiliary extension information used in the post-filter processing device, and the auxiliary extension information decoding device 91 is characterized in that the auxiliary extension information is decoded by decoding auxiliary extension information having information indicating what type of post-filter processing to perform and an identification number for identifying the type of processing, auxiliary extension information having information on whether or not to apply post-filter processing for each picture and an identification number indicating which post-filter processing to apply, auxiliary extension information indicating a region of interest in the decoded image, and auxiliary extension information having an identification number indicating which post-filter processing to apply to the region of interest in the decoded image.

[0232] In addition, in the auxiliary extension information decoding device 91, the auxiliary extension information indicating the region of interest of the decoded image may include reliability information of the region of interest, and the auxiliary extension information may include an identification number indicating which post-filter processing to apply to the region of interest of the decoded image and threshold information for the reliability information of the region of interest.

[0233] In addition, in the auxiliary extension information decoding device 91, the auxiliary extension information indicating the region of interest of the decoded image may include label information of the region of interest, and the auxiliary extension information may include an identification number indicating which post-filter processing to apply to the region of interest of the decoded image and label information of the region of interest.

[0234] The video coding device 10 comprises an image coding device 11 that codes an input image, an auxiliary extension information generation device 71 that generates multiple auxiliary extension information for post-filter processing, and an auxiliary extension coding device 81 that codes the multiple auxiliary extension information, and the auxiliary extension generation device 71 generates auxiliary extension information having information indicating what type of post-filter processing to perform and an identification number for identifying the type of processing, auxiliary extension information having information on whether or not to apply post-filter processing for each picture and an identification number indicating which post-filter processing to apply, auxiliary extension information indicating a region of interest in a decoded image, and auxiliary extension information having an identification number indicating which post-filter processing to apply to the region of interest in the decoded image, and encodes the information using the auxiliary extension coding device 81.

[0235] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. In other words, embodiments obtained by combining technical means modified appropriately within the scope of the claims are also included in the technical scope of the present invention.

[0236] The embodiments of the present invention can be suitably applied to a video decoding device that decodes coded data obtained by coding image data, and a video coding device that generates coded data obtained by coding image data, and can also be suitably applied to the data structure of coded data generated by a video coding device and referenced by the video decoding device.

[0237] 1 Video transmission system 30 Video decoding device 31 Image decoding device 301 Entropy decoding unit 302 Parameter decoding unit 305, 107 Loop filter 306, 109 Reference picture memory 307, 108 Prediction parameter memory 308, 101 Prediction image generation unit 311, 105 Inverse quantization / inverse transform unit 312, 106 Addition unit 320 Prediction parameter derivation unit 10 Video coding device 11 Image coding device 102 Subtraction unit 103 Transform / quantization unit 104 Entropy coding unit 110 Coding parameter determination unit 111 Parameter coding unit 120 Prediction parameter derivation unit 41 Image display device 51 Pre-filtering processing device 61 Post-filtering processing device 71 Supplementary enhancement information creation device 81 Supplementary enhancement information coding device 91 Supplementary enhancement information decoding device

Claims

1. A video decoding device comprising: an image decoding device that decodes encoded data to generate a decoded image; a post-filter processing device that performs post-filter processing on the decoded image; and an auxiliary enhancement information decoding device that decodes multiple auxiliary enhancement information used in the post-filter processing device, wherein the auxiliary enhancement information decodes: auxiliary enhancement information having information indicating what type of post-filter processing to perform and an identification number for identifying the type of processing; auxiliary enhancement information having information on whether or not post-filter processing is to be applied for each picture and an identification number indicating which post-filter processing to apply; auxiliary enhancement information indicating a region of interest in a decoded image; and auxiliary enhancement information having an identification number indicating which post-filter processing to apply to the region of interest in the decoded image.

2. In the video decoding device of claim 1, the auxiliary extension information indicating the region of interest of the decoded image includes reliability information of the region of interest, and the video decoding device decodes the auxiliary extension information including an identification number indicating which post-filter processing to apply to the region of interest of the decoded image and threshold information for the reliability information of the region of interest.

3. In the video decoding device of claim 1, the auxiliary extension information indicating the region of interest of the decoded image includes label information of the attention information, and the video decoding device decodes the auxiliary extension information including an identification number indicating which post-filter processing to apply to the region of interest of the decoded image and the label information of the region of interest.

4. A video coding device comprising: an image coding device which codes an input image; an auxiliary extension information generating device which generates multiple auxiliary extension information for post-filter processing; and an auxiliary extension coding device which codes the multiple auxiliary extension information, wherein the auxiliary extension generating device generates auxiliary extension information having information indicating what type of post-filter processing to perform and an identification number for identifying the type of processing, auxiliary extension information having information on whether or not to apply post-filter processing for each picture and an identification number indicating which post-filter processing to apply, auxiliary extension information indicating a region of interest in a decoded image, and auxiliary extension information having an identification number indicating which post-filter processing to apply to the region of interest in the decoded image, and codes the generated information using the auxiliary extension coding device.

Images