Video encoding device and video decoding device

Abstract

To solve a problem where consistency of processing cannot be achieved in a supplementary enhancement information (SEI) processing order SEI message because the sizes of images input and / or output differ depending on the SEI message.SOLUTION: In a video transmission system 1 including a video encoding device 10, a network 21, a video decoding device 30, and an image display device 41, the video decoding device includes an image decoding device for decoding encoded data Te to generate a decoded image, an auxiliary extension processing device for performing auxiliary extension processing on the decoded image, and an auxiliary extension information decoding device for decoding a plurality of pieces of auxiliary extension information for operating the auxiliary extension processing device. The auxiliary extension information indicates a priority processing order of the plurality of auxiliary extension information, and includes information indicating that the parameters of the decoded image to be input to the auxiliary extension processing device are changed according to the priority processing order.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] An embodiment of the present invention relates to a video encoding device and a video decoding device. [Background technology]

[0002] In order to efficiently transmit or record moving images, a moving image encoding device is used that generates encoded data by encoding moving images, and a moving image decoding device is used that generates a decoded image by decoding the encoded data.

[0003] Specific examples of video encoding methods include H.264 / AVC and H.265 / HEVC (High-Efficiency Video Coding).

[0004] In such a video coding method, images (pictures) constituting a video are managed in a hierarchical structure consisting of slices obtained by dividing images, coding tree units (CTUs) obtained by dividing slices, coding units (sometimes called coding units: CUs) obtained by dividing coding tree units, and transform units (TUs) obtained by dividing coding units, and are coded / decoded for each CU.

[0005] In such a video coding method, a predicted image is usually generated based on a locally decoded image obtained by encoding / decoding an input image, and a prediction error (sometimes called a "difference image" or "residual image") obtained by subtracting the predicted image from the input image (original image) is coded. Methods for generating a predicted image include inter-prediction and intra-prediction.

[0006] Furthermore, as a recent technique for video encoding and decoding, a supplemental enhancement information (SEI) message has been defined for transmitting image properties, display methods, timing, etc., simultaneously with encoded data. Non-Patent Document 1 proposes an SEI message that defines information indicating a preferred processing order (priority processing order information) determined by the encoding side when there are multiple SEI messages described in Non-Patent Document 2 and the like. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] SEI processing order SEI message in VVC (draft 5), JVET-AE2027 [Non-Patent Document 2] Consent: H.266 V3 "Versatile video coding" (Rev.), ITU-T SG16(Study Period 2022) Temporary Document 171-PLEN Summary of the Invention [Problem to be solved by the invention]

[0008] In the SEI processing order SEI message disclosed in Non-Patent Document 1, since the image sizes to be input and output differ depending on the SEI message, there is a problem in that the processing cannot be consistent. [Means for solving the problem]

[0009] A video decoding device according to one aspect of the present invention includes: an image decoding device that decodes encoded data to generate a decoded image; an auxiliary extension processing device that performs an auxiliary extension process on the decoded image; an auxiliary extension information decoding device for decoding a plurality of auxiliary extension information for operating the auxiliary extension processing device; The auxiliary extension information has auxiliary extension information indicating a priority processing order of a plurality of auxiliary extension information, The auxiliary extension information indicating a priority processing order of the plurality of auxiliary extension information includes: The information indicates that the parameters of the decoded image to be input to the auxiliary enhancement processor are to be changed according to the priority processing order.

[0010] Furthermore, a video encoding device according to an aspect of the present invention comprises: an image encoding device that encodes a moving image signal to generate encoded data; An auxiliary extension information creating device for creating a plurality of auxiliary extension information; a supplemental extension information encoding device for encoding the plurality of supplemental extension information, The auxiliary extension information has auxiliary extension information indicating a priority processing order of a plurality of auxiliary extension information, The auxiliary extension information indicating a priority processing order of the plurality of auxiliary extension information includes: The present invention is characterized by having information indicating that parameters of a locally decoded image to be input to the auxiliary enhancement processor are changed according to the priority processing order. Effect of the Invention

[0011] This configuration can solve the problem of inconsistent processing due to different input and output image sizes. [Brief description of the drawings]

[0012] [Figure 1] 1 is a schematic diagram showing a configuration of a video transmission system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing a hierarchical structure of encoded data. [Diagram 3] FIG. 13 is a diagram showing a hierarchical structure of encoded data in units of PUs. [Figure 4] FIG. 1 is a schematic diagram showing a configuration of an image decoding device. [Diagram 5] 11 is a flowchart illustrating a schematic operation of an image decoding device. [Figure 6] FIG. 1 is a block diagram showing a configuration of an image encoding device. [Figure 7] A diagram showing the syntax of the SEI processing order SEI message. [Figure 8] FIG. 13 is a diagram showing the SEI payload type of the SEI message. [Figure 9] A diagram showing the syntax of the SEI processing order SEI message of this embodiment. [Figure 10] A figure showing another syntax of the SEI processing order SEI message of this embodiment. [Figure 11] A figure showing another syntax of the SEI processing order SEI message of this embodiment. [Figure 12] A diagram showing the syntax of the SEI payload, which is a container for the SEI message. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] (First embodiment) Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] FIG. 1 is a schematic diagram showing the configuration of another moving image transmission system according to the present embodiment.

[0015] The video transmission system 1 is a system that transmits coded data obtained by coding an image, and decodes and displays the coded data. The video transmission system 1 is made up of a video coding device 10, a network 21, a video decoding device 30, and an image display device 41.

[0016] The video encoding device 10 is composed of an image encoding device (image encoding unit) 11, an auxiliary extension information creation device (auxiliary extension information creation unit) 71, an auxiliary extension information encoding device (auxiliary extension information encoding unit) 81, and a pre-filter processing device (pre-filter processing unit) 51.

[0017] The video encoding device 10 creates a prefiltered image T2 from an input video T1 using a prefilter processing device 51, compresses and encodes the image using an image encoding device 11, and analyzes the input video T1 and a locally decoded image T3 from the image encoding device 11 to generate auxiliary extension information to be input to an auxiliary extension processing device 61 using an auxiliary extension information creation device 51, encodes the information using an auxiliary extension information encoding device 81, generates encoded data Te, and transmits the encoded data Te to a network 21.

[0018] The video decoding device 30 includes an image decoding device (image decoding unit) 31, an auxiliary extension information decoding device (auxiliary extension information decoding unit) 91, and an auxiliary extension processing device (auxiliary extension processing unit) 61.

[0019] The video decoding device 30 decodes the encoded data Te received from the network 21 using the image decoding device 31 and the auxiliary extension information decoding device 91, and performs auxiliary extension processing on the decoded image Td1 using the auxiliary extension information in the auxiliary extension processing device 61, and outputs the auxiliary extended decoded image Td2 to the image display device 41.

[0020] The auxiliary extension processing device 61 may output the decoded image Td1 as it is without performing the auxiliary extension processing using the auxiliary extension information.

[0021] The image display device 41 displays all or a part of the auxiliary extension processed image Td2 output from the auxiliary extension processing device 61. The image display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. The display may be in the form of a stationary, mobile, HMD, or the like. When the image decoding device 31 has high processing capability, it displays a high quality image, and when it has only low processing capability, it displays an image that does not require high processing capability or display capability.

[0022] The network 21 transmits the encoded auxiliary extension information and the encoded data Te to the image decoding device 31. Part or all of the encoded auxiliary extension information may be included in the encoded data Te as the auxiliary extension information SEI. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. The network 21 is not necessarily limited to a bidirectional communication network, and may be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. Also, the network 21 may be replaced by a storage medium that stores the encoded data Te such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blue-ray Disc: registered trademark).

[0023] In such a configuration, a framework is provided that enables efficient encoding and decoding of auxiliary extension information.

[0024] <Operator> The operators used in this specification are described below.

[0025] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is an OR assignment operator, and || represents a logical OR.

[0026] x? y : z is a ternary operator that takes y when x is true (non-zero) and z when x is false (0).

[0027] Clip3(a, b, c) is a function that clips c to a value between a and b (inclusive). If c < a, it returns a; if c > b, it returns b; otherwise, it returns c (where a <= b).

[0028] abs(a) is a function that returns the absolute value of a.

[0029] Int(a) is a function that returns the integer value of a.

[0030] Floor(a) is a function that returns the largest integer less than or equal to a.

[0031] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0032] a / d represents the division of a by d (truncated to an integer).

[0033] a÷d and a over d represent the division of a by d (without rounding down).

[0034] (Structure of encoded data Te) Before going into a detailed description of the image encoding device 11 and the image decoding device 31 according to this embodiment, the data structure of the encoded data Te generated by the image encoding device 11 and decoded by the image decoding device 31 will be described with reference to Figures 2 and 3.

[0035] The coded data Te is a bitstream consisting of multiple CVS (Coded Video Sequence) and EoB (End of Bitstream) NAL units as shown in FIG. 2. The CVS consists of multiple AU (Access Unit) and EoS (End of Sequence) NAL units. The AU at the beginning of the CVS is called the CVSS (Coded Video Sequence Start) AU. The unit obtained by dividing the CVS into layers is called the CLVS (Coded Layer Video Sequence). The AU consists of one or more layered PUs (Picture Units) with the same output time. If the multilayer coding method is not adopted, the AU consists of one PU. The PU is a unit of coded data for one decoded picture consisting of multiple NAL units. The CLVS consists of PUs of the same layer, and the PU at the beginning of the CLVS is called the CLVSS (Coded Layer Video Sequence Start) PU. The CLVSS PU is limited to PUs consisting of randomly accessible IRAP (Intra Random Access Pictures) and GDR (Gradual Decoder Refresh Picture). A NAL unit consists of a NAL unit header and RBSP (Raw Byte Sequence Payload) data. The NAL unit header consists of 2 bits of 0 data, followed by a 6-bit nuh_layer_id that indicates the layer value, a 5-bit nuh_unit_type that indicates the NAL unit type, and a 3-bit nuh_temporal_id_plus1 that is the Temporal ID value plus 1.

[0036] Fig. 3 is a diagram showing a hierarchical structure of data in the coded data Te in units of PU. The coded data Te illustratively includes a sequence and a plurality of pictures constituting the sequence. Fig. 3 shows a diagram showing a coded video sequence that defines the sequence SEQ, a coded picture that defines the picture PICT, a coded slice that defines the slice S, coded slice data that defines the slice data, a coding tree unit included in the coded slice data, and a coding unit included in the coding tree unit.

[0037] In the coded video sequence, a set of data to be referred to by the image decoding device 31 in order to decode the sequence SEQ to be processed is defined. As shown in Fig. 3, the sequence SEQ includes a video parameter set VPS (Video Parameter Set), a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), an adaptation parameter set (APS), a picture PICT, and supplemental enhancement information SEI (Supplemental Enhancement Information).

[0038] The video parameter set VPS specifies a set of coding parameters common to multiple videos composed of multiple layers, as well as a set of coding parameters related to multiple layers and each individual layer included in the video.

[0039] In the sequence parameter set SPS, the image decoding device 3 A set of coding parameters referenced by PPS 1 is specified. For example, the width and height of a picture are specified. Note that there may be multiple SPSs. In that case, one of the multiple SPSs is selected from the PPS.

[0040] Here, the sequence parameter set SPS includes the following syntax elements: pic_width_max_in_luma_samples: A syntax element that specifies the width of the image with the largest width among the images in a single sequence, in units of luminance blocks. The value of this syntax element must not be 0 and must be an integer multiple of Max(8, MinCbSizeY), where MinCbSizeY is a value determined by the minimum size of a luminance block. pic_height_max_in_luma_samples: This syntax element specifies the height of the image with the maximum height among the images in a single sequence, in units of luminance blocks. The value of this syntax element is required to be a non-zero integer multiple of Max(8, MinCbSizeY).

[0041] The picture parameter set PPS defines a set of coding parameters that the image decoding device 31 refers to in order to decode each picture in the target sequence. Note that there may be multiple PPSs. In that case, one of the multiple PPSs is selected for each picture in the target sequence.

[0042] Here, the picture parameter set PPS includes the following syntax elements:

[0043] pps_pic_width_in_luma_samples is a syntax element that specifies the width of the target picture. The value of this syntax element is required to be a non-zero integer multiple of Max(8, MinCbSizeY) and equal to or less than sps_pic_width_max_in_luma_samples.

[0044] pps_pic_height_in_luma_samples is a syntax element that specifies the height of the target picture. The value of this syntax element is required to be a non-zero integer multiple of Max(8, MinCbSizeY) and equal to or less than sps_pic_height_max_in_luma_samples.

[0045] pps_conformance_window_flag is a flag indicating whether conformance (cropping) window offset parameters are to be signaled subsequently and where the conformance window is to be displayed. If this flag is 1, the parameters are to be signaled, if it is 0, the conformance window offset parameters are not present.

[0046] conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset are offset values ​​for specifying the left, right, top, and bottom positions of a picture output by a decoding process with respect to a rectangular area specified by the output picture coordinates. In addition, when the value of conformance_window_flag is 0, the values ​​of conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset are estimated to be 0.

[0047] The output picture will have pixels from SubWidthC * pps_conf_win_left_offset to pps_pic_width_in_luma_samples - (SubWidthC * pps_conf_win_right_offset + 1) horizontally and from SubHeightC * pps_conf_win_top_offset to pps_pic_height_in_luma_samples - (SubHeightC * pps_conf_win_bottom_offset + 1) vertically.

[0048] Here, the chroma format variable ChromaFormatIdc is the value of sps_chroma_format_id, and the variables SubWidthC and SubHightC are values ​​determined by this ChromaFormatIdc. In the case of a monochrome format, SubWidthC and SubHightC are both 1, in the case of a 4:2:0 format, SubWidthC and SubHightC are both 2, and in the case of a 4:2:2 format, SubWidthC and SubHightC are both 3. If idthC is 2 and SubHightC is 1, and the format is 4:4:4, then SubWidthC and SubHightC are both 1.

[0049] (Encoded Picture) A coded picture defines a set of data to be referenced by the image decoding device 31 in order to decode a picture PICT to be processed. As shown in Fig. 3, the picture PICT includes a picture header PH and slices 0 to NS-1 (NS is the total number of slices included in the picture PICT).

[0050] (Coded Slice) A coded slice defines a set of data to be referenced by the image decoding device 31 in order to decode a current slice S. As shown in Fig. 3, a slice includes a slice header and slice data.

[0051] The slice header includes a group of coding parameters to be referred to by the image decoding device 31 in order to determine a decoding method for the current slice. Slice type designation information (slice_type) that designates a slice type is an example of a coding parameter included in the slice header.

[0052] Slice types that can be specified by the slice type specification information include (1) an I slice that uses only intra prediction when encoding, (2) a P slice that uses uni-prediction (L0 prediction) or intra prediction when encoding, and (3) a B slice that uses uni-prediction (L0 prediction or L1 prediction), bi-prediction, or intra prediction when encoding. Note that inter prediction is not limited to uni-prediction or bi-prediction, and a predicted image may be generated using more reference pictures. Hereinafter, when referring to P or B slice, it refers to a slice including a block that can use inter prediction.

[0053] (Encoded slice data) The coded slice data specifies a set of data to be referenced by the image decoding device 31 in order to decode the slice data to be processed. The slice data includes a CTU, as shown in the coded slice header in Fig. 3. A CTU is a block of a fixed size (e.g., 64x64) that constitutes a slice, and is also called a Largest Coding Unit (LCU).

[0054] (coding tree unit) 3 specifies a set of data that the image decoding device 31 refers to in order to decode a CTU to be processed. The CTU is divided into coding units CU, which are basic units of encoding processing, by recursive quad tree division (QT (Quad Tree) division), binary tree division (BT (Binary Tree) division), or ternary tree division (TT (Ternary Tree) division). A node of a tree structure obtained by recursive quad tree division is called a coding node. Intermediate nodes of the quad tree, binary tree, and ternary tree are coding nodes, and the CTU itself is specified as the top coding node.

[0055] (Encoding Unit) 3 specifies a set of data to be referenced by the image decoding device 31 in order to decode a coding unit to be processed. Specifically, a CU is composed of a CU header CUH, prediction parameters, transformation parameters, quantization transformation coefficients, etc. The CU header specifies a prediction mode, etc.

[0056] The prediction process may be performed in units of CUs, or in units of sub-CUs obtained by further dividing a CU.

[0057] There are two types of prediction (prediction modes): intra prediction and inter prediction. Intra prediction is a prediction within the same picture, while inter prediction refers to a prediction process performed between different pictures (for example, between display times or between layer images).

[0058] The transform and quantization processes are performed in units of CUs, but the quantized transform coefficients may be entropy coded in units of sub-blocks such as 4x4.

[0059] In this specification, when "a flag indicating whether or not XX" is used, a flag other than 0 (for example, 1) indicates XX and 0 indicates non-XX, and in logical negation, logical product, etc., 1 is treated as true and 0 as false (same below). However, in an actual device or method, other values ​​may be used as true and false values.

[0060] (Configuration of an image decoding device) The configuration of an image decoding device 31 (FIG. 4) according to this embodiment will be described.

[0061] The image decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit (prediction image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a prediction image generating unit (prediction image generating device) 308, an inverse quantization and inverse transform unit 311, an adder 312, and a prediction parameter derivation unit 320. Note that, in accordance with the image encoding device 11 described below, the image decoding device 31 may also be configured not to include the loop filter 305.

[0062] The parameter decoding unit 302 further includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit), and the CU decoding unit 3022 further includes a TU decoding unit 3024. These may be collectively referred to as a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, PPS, and APS, and slice header (slice information) from the encoded data. The CT information decoding unit 3021 decodes the CT from the encoded data. The CU decoding unit 3022 decodes the CU from the encoded data. The TU decoding unit 3024 decodes QP update information (quantization correction value) and quantization prediction error (residual coding) from the encoded data.

[0063] The predicted image generating unit 308 includes an inter predicted image generating unit 309 and an intra predicted image generating unit 310 .

[0064] The entropy decoding unit 301 performs entropy decoding on the encoded data Te input from the outside, and decodes each code (syntax element).

[0065] The entropy decoding unit 301 outputs the decoded code to the parameter decoding unit 302. Control of which code to decode is performed based on an instruction from the parameter decoding unit 302.

[0066] (Basic flow) FIG. 5 is a flowchart illustrating a schematic operation of the image decoding device 31.

[0067] (S1100: Decode Parameter Set Information) The header decoding unit 3020 decodes parameter set information such as the VPS, SPS, and PPS from the encoded data.

[0068] (S1200: Decode slice information) The header decoding unit 3020 decodes the slice header (slice information) from the encoded data.

[0069] Thereafter, the image decoding device 31 repeats the processes from S1300 to S5000 for each CTU included in the target picture to derive a decoded image of each CTU.

[0070] (S1300: Decode CTU Information) The CT information decoding unit 3021 decodes the CTU from the encoded data.

[0071] (S1400: Decode CT Information) The CT information decoding unit 3021 decodes the CT from the encoded data.

[0072] (S1500: CU Decoding) The CU decoding unit 3022 performs S1510 and S1520 to decode the CU from the encoded data.

[0073] (S1510: Decode CU information) The CU decoding unit 3022 decodes the CU information, prediction information, and the like from the encoded data.

[0074] (S1520: Decode TU information) The TU decoding unit 3024 decodes the QP update information and the quantization prediction error, etc. from the encoded data. Note that the QP update information is a difference value from the quantization parameter predicted value qPpred, which is a predicted value of the quantization parameter QP.

[0075] (S2000: Generation of predicted image) The predicted image generation unit 308 generates a predicted image for each block included in the current CU based on the prediction information.

[0076] (S3000: Inverse Quantization and Inverse Transformation) The inverse quantization and inverse transform unit 311 executes inverse quantization and inverse transform processing for each TU included in the target CU.

[0077] (S4000: Generate decoded image) The adder 312 adds the predicted image supplied from the predicted image generation unit 308 and the prediction error supplied from the inverse quantization and inverse transform unit 311 to generate a decoded image of the current CU.

[0078] (S5000: Loop Filter) The loop filter 305 applies a loop filter such as a deblocking filter, SAO, or ALF to the decoded image to generate a decoded image.

[0079] (SEI processing order SEI message) Figure 7 shows the syntax of the SEI processing order SEI message. The SEI processing order SEI message conveys information indicating a preferred processing order (preferred processing order information) determined by an encoder (i.e., a content producer) for different types of SEI messages that may be present in the CVS.

[0080] When a SEI processing order SEI message is present in any access unit of the CVS, the SEI processing order SEI message MUST be present in the first access unit of the CVS.

[0081] SEI Processing Order SEI messages persist in decoding order from the current access unit until the end of the CVS.

[0082] If multiple SEI processing order SEI messages exist in one CVS, they must have the same content, and in one SEI processing order SEI message, there must be at least two pairs of syntax elements po_sei_payload_type[i] and po_sei_processing_order[i], and there must be at least two values ​​of po_sei_processing_order[i] that are not equal.

[0083] SEI Processing Order An SEI message can carry one or more SEI headers of a particular payloadType. Each SEI header is a byte string that conforms to the SEI payload syntax for that value of payloadType, including some complete syntax elements starting with the first syntax element of the SEI payload, followed possibly by bits that do not represent complete syntax elements of the SEI payload.

[0084] The information at the beginning of these SEIs must be sufficient to distinguish between SEIs with the same payloadType value but in a different order.

[0085] po_num_sei_messages_minus2 plus 2 indicates the number of SEI messages that have a processing order indicated in the SEI processing order SEI message.

[0086] reserved_alignment_6bits shall be equal to 0. The syntax element may be a value other than 0, in which case the value shall be ignored.

[0087] If the value of po_sei_wrapping_flag[i] is 0, the SEI message must exist outside of any SEI processing order SEI message with a payloadType equal to po_sei_payload_type[i]. However, if the value of po_sei_wrapping_flag[i] is 0 and no SEI message with a payloadType equal to po_sei_payload_type[i] exists, the following applies:

[0088] If po_sei_importance_flag[i] is equal to 1, the decoder MUST ignore the entire SEI processing order SEI message.

[0089] Otherwise, the decoder must ignore all data associated with the loop variable value i.

[0090] However, a value of 1 for po_sei_wrapping_flag[i] allows SEI messages to be carried within SEI processing order SEI messages and prevents such SEI messages from being misinterpreted by decoders that do not process SEI processing order SEI messages. Thus, setting po_sei_wrapping_flag[i] to 1 is intended to be used when unintended results may be produced by a decoder if po_sei_wrapping_flag[i] is 0.

[0091] When the value of po_sei_prefix_flag[i] is 1, it means that po_num_prefix_bytes[i] exists, and when the value of po_sei_prefix_flag[i] is 0, it means that po_num_prefix_bytes[i] does not exist.

[0092] po_sei_payload_type[i] specifies the payloadType of the i-th SEI message in the SEI processing order SEI.

[0093] For two distinct non-negative integer values ​​of m and n, the values ​​of po_sei_payload_type[m] and po_sei_payload_type[n] shall not be identical unless po_sei_prefix_flag[m] and po_sei_prefix_flag[n] are both equal to 1.

[0094] The list storing the priority processing order information, SeiProcessingOrderSeiList, is set to consist of the payloadType values ​​specified in Figure 8, excluding the values ​​137, 144, 147, 148, 179, 180, 200, 201, 208, and 213. That is, the following SEI messages are not applicable in the SEI processing order SEI message: ·mastering_display_colour_volume content_light_level_info ·alternative_transfer_characteristic ambient_viewing_environment ·multiview_acquisition_info ·multiview_view_position ·sei_manifest ·sei_prefix_indication ·scalability_dimension_info The value of po_sei_prefix_flag[i] is 0 if po_sei_payload_type[i] is not equal to any value of SeiProcessingOrderSeiList.

[0095] po_num_prefix_bytes[i], if present, specifies the number of bytes associated with the i-th SEI message for which priority processing order information is provided in the SEI processing order SEI message. If not present, the value of po_num_prefix_bytes[i] is inferred to be equal to 0.

[0096] When po_prefix_byte[i][j] is present, it specifies the jth byte value of the ith SEI message.

[0097] po_sei_processing_order[i] indicates the priority processing order of the i-th SEI message. If m and n are two distinct integer values ​​equal to or greater than 0, then if the value of po_sei_processing_order[m] is smaller than po_sei_processing_order[n], it indicates the type of the SEI message whose payloadType is equal to po_sei_payload_type[m].

[0098] The byte-based bit stream of po_prefix_byte[m][p] (p ranges from 0 to po_num_prefix_bytes[m] - 1) in po_sei_processing_order[m] must be processed before the SEI message having the payloadType indicated by po_sei_payload_type[n].

[0099] When po_sei_processing_order[m] is equal to po_sei_processing_order[n], it indicates that there is no preferred processing order among the SEI messages.

[0100] If there are multiple SEI messages with the same values ​​of po_sei_payload_type[i] and po_num_prefix_bytes[i], they shall have the same value of po_sei_processing_order[i].

[0101] When po_sei_processing_order[m] is equal to po_sei_processing_order[n], it indicates that there is no preferred processing order among the SEI messages.

[0102] If there are multiple SEI messages with the same po_sei_payload_type[i], value of po_num_prefix_bytes[i], and byte value of po_prefix_byte[i][j], where j is in the range from 0 to po_num_prefix_bytes[i]-1, they shall have the same value of po_sei_processing_order[i].

[0103] po_sei_processing_order[0] must be 0. If i is greater than 0, po_sei_processing_order[i] is equal to either po_sei_processing_order[i-1] or po_sei_processing_order[i-1] + 1. In addition, the value of po_sei_processing_order[po_num_sei_messages_minus2+1] must not be 0.

[0104] The image size (image area) decoded from the encoded data includes the image size / image area obtained by decoding the parameter set pps_pic_width_in_luma_samples and pps_pic_height_in_luma_samples, and the image size / image area obtained by further cutting out an area called the crop area. The former is called the uncropped image size / image area, and the latter is called the cropped image size / image area.

[0105] The SEI processing order SEI message disclosed in Non-Patent Document 1 had a problem in that the input and output image sizes (range of the screen area treated as the target) differed depending on the SEI message, resulting in inconsistent processing.

[0106] Specifically, according to Annex D of Non-Patent Document 2, ·film grain characteristics SEI message ·decoded picture hash SEI message In this specification, the input image for SEI is defined as an uncropped picture of the same size as pps_pic_width_in_luma_samples and pps_pic_height_in_luma_samples.

[0107] on the other hand, ·annotated regions SEI message ·neural network post-filter characteristics SEI message and neural network post-filter activation SEI message ·phase indication SEI message Here, the input is defined to be a picture with the following cropped image size:

[0108] CropedWidth = pps_pic_width_in_luma_samples - SubWidthC * (conf_win_right_offset + conf_win_left_offset) CropedHeight = pps_pic_height_in_luma_samples - SubHightC * (conf_win_bottom_offset + conf_win_top_offset) As for other SEI messages, basically, it is assumed that an SEI message for inputting a picture will input the image at the cropped size.

[0109] When multiple SEI messages are concatenated and auxiliary extension processing is performed using the SEI processing order SEI message, if the input image sizes are different, for example, when considering implementation, it is necessary to always take into account the larger image size, and the auxiliary extension processing device needs to buffer the picture at the uncropped image size and change the processing range according to the SEI message.

[0110] In this embodiment, in order to solve some of the above problems, ·film grain characteristics SEI message ·decoded picture hash SEI message As the input image for SEI, we will input a picture with the following cropped image size.

[0111] CropedWidth = pps_pic_width_in_luma_samples - SubWidthC * (conf_win_right_offset + conf_win_left_offset) CropedHeight = pps_pic_height_in_luma_samples - SubHightC * (conf_win_bottom_offset + conf_win_top_offset) Also, ·film grain characteristics SEI message ·decoded picture hash SEI message If it is the first (0th priority processing order), the input may be the uncropped image size, otherwise the input may be the cropped image size.

[0112] Furthermore, if there is even one SEI process with a cropped image size from the beginning, the image sizes thereafter, including the following SEI, can be set to the cropped image size. ·film grain characteristics SEI message ·decoded picture hash SEI message By adopting such specifications, it is possible to at least correct the difference in the auxiliary enhancement processing range within a picture due to the SEI message.

[0113] In this embodiment, in order to solve some of the above problems, ·decoded picture hash SEI message may be included in the SEI list excluded by SeiProcessingOrderSeiList and not be applied to the SEI processing order SEI message. That is, SeiProcessingOrderSeiList may include values ​​137, 144, 147, 148, 179, 180, 200, 201, 208, 213, and 132, with 132 representing the decoded picture hash SEI message.

[0114] Alternatively, in this embodiment, in order to solve some of the above problems, ·decoded picture hash SEI message With regard to the decoded picture hash SEI message, a restriction may be imposed that it must always be placed at the top of the processing order. By adopting such a specification, it is possible to eliminate inconsistencies in the definition of the processing order of the SEI for the decoded picture hash SEI message.

[0115] In this embodiment, in order to solve some of the above problems, ·film grain characteristics SEI message Regarding the above, it is acceptable to set a restriction that it should always be at the top of the processing order. By adopting such specifications, SEIs that use the entire uncropped screen can also be processed without any problems.

[0116] In this embodiment, in order to solve some of the above problems, ·film grain characteristics SEI message may be included in the excluded SEI list in SeiProcessingOrderSeiList and not be applied in the SEI processing order SEI message. That is, SeiProcessingOrderSeiList may include values ​​137, 144, 147, 148, 179, 180, 200, 201, 208, 213, and 19, as well as 19, which denotes the film grain characteristics SEI message.

[0117] Another issue is that, for example, in the neural network post-filter characteristics SEI message and the neural network post-filter activation SEI message, the output image size, chrominance format, pixel bit length, and picture rate are changed depending on the content of the processing, so the specifications of the SEI message, which are defined on the premise of images output from an image decoding device, cannot accommodate auxiliary extension processing of the SEI processing order SEI message.

[0118] In this embodiment, in order to solve the above problems, in addition to defining the processing order, input picture parameters, specifically, image size, color difference format, pixel bit length, picture rate, etc., are redefined as necessary.

[0119] FIG. 9 shows a method for redefining the input of an SEI message by adding a syntax element called po_sei_override_flag[i] to the syntax in FIG.

[0120] When the value of po_sei_override_flag[i] is 0, it indicates that the input of the i-th SEI message operates according to the output of the image decoder as defined. When the value of po_sei_override_flag[i] is 1, it indicates that the definition of the input picture of the po_sei_processing_order[i]-th SEI message with the priority order is rewritten to the output contents of the SEI message with the priority order that was last changed before the po_sei_processing_order[i]. Here, the contents to be rewritten are the image size, chrominance format, pixel bit length, picture rate, etc.

[0121] Specifically, in the definition of the conventional SEI message specification, the image size (cropped image size or uncropped image size), chrominance format, pixel bit length, etc. of the decoded image output by the image decoder were inherited as they were, but Replace with the image size, chrominance format, and pixel bit length output by the SEI message of the priority processing order that was last changed before the order[i]th message.

[0122] Here, if the value of po_sei_override_flag[i] is 1, the value of po_sei_processing_order[i] is 0, and the input of the SEI message with the 0th priority processing order indicated by po_sei_payload_type[i] is an uncropped image size, the image is input at the cropped image size. With this configuration, it is possible to execute the auxiliary expansion process even if the parameters of the image to be processed are changed midway.

[0123] FIG. 10 is a diagram showing the syntax when the value of po_sei_processing_order[i] is 0, that is, when the input picture of the SEI message with the 0th priority processing order is not redefined.

[0124] When the value of po_sei_override_flag[i] is 0, it indicates that the input of the i-th SEI message operates according to the output of the image decoder as defined. When the value of po_sei_override_flag[i] is 1, it indicates that the definition of the input picture of the SEI message of the po_sei_processing_order[i]-th priority order is rewritten to the output contents of the SEI message of the priority processing order that was last changed before the po_sei_processing_order[i]-th priority order. Here, the contents to be rewritten are the picture size, chrominance format, pixel bit length, picture rate, etc. When po_sei_override_flag[i] does not exist, the value is estimated to be 0. With this configuration, it is possible to execute the auxiliary extension process even if the parameters of the image to be processed are changed during the process.

[0125] FIG. 11 shows, in addition to the syntax in FIG. 10, a method for explicitly defining a parameter as a syntax when the value of po_sei_override_flag[i] is 1.

[0126] The syntax element po_pic_width_minus1[i] + 1 indicates the number of horizontal luminance pixels of the input image size of the i-th SEI message, the syntax element po_pic_height_minus1[i] + 1 indicates the number of vertical luminance pixels of the input image size of the i-th SEI message, po_bitdepth_minus8 + 8 indicates the pixel bit length of the input image of the i-th SEI message, and po_chroma_format_idc indicates the chrominance format of the input image of the i-th SEI message. With this configuration, it is possible to execute the auxiliary extension process even if the parameters of the image to be processed are changed midway.

[0127] 9, 10, and 11, the redefinition of the input picture is explicitly shown by the syntax, but a method of realizing this by defining only the decoding process without adding the syntax is also conceivable.

[0128] When the value of po_sei_processing_order[i] is 1 or greater, the definition of the input picture of the SEI message of the po_sei_processing_order[i]th priority order is rewritten to the output picture size, chrominance format, pixel bit length, and picture rate of the SEI message of the priority processing order that was last changed before the po_sei_processing_order[i]th priority order. By adopting such a configuration, it is possible to execute the auxiliary expansion process even if the parameters of the image to be processed are changed midway.

[0129] In addition, as another method of this embodiment, ·neural network post-filter characteristics SEI message ·neural network post-filter activation SEI message By applying such a constraint, the image parameters are changed in the final auxiliary expansion process, so that there is no need to redefine them in the middle of the process.

[0130] (SEI payload) FIG. 12 is a diagram showing a portion of the syntax of the SEI payload, which is a container of an SEI message.

[0131] PREFIX_SEI_NUT, which is called when nal_unit_type is PREFIX_SEI_NUT, indicates that the SEI is located before the slice data.

[0132] When payloadType is 19, the film grain characteristics SEI message is called.

[0133] When payloadType is 202, the annotated regions SEI message is called.

[0134] When payloadType is 210, the neural network post-filter characteristics SEI message is called.

[0135] When payloadType is 211, neural network post-filter activation SEI message is called.

[0136] When payloadType is 213, the SEI processing order SEI is called.

[0137] SUFFIX_SEI_NUT, which is called when nal_unit_type is SUFFIX_SEI_NUT, indicates that the SEI is located after the slice data.

[0138] When payloadType is 132, the decoded picture hash SEI message is called.

[0139] (Configuration of Image Encoding Device) Next, the configuration of the image encoding device 11 according to this embodiment will be described. Fig. 6 is a block diagram showing the configuration of the image encoding device 11 according to this embodiment. The image encoding device 11 includes a prediction image generating unit 101, a subtraction unit 102, a transformation and quantization unit 103, an inverse quantization and inverse transformation unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference image storage unit, frame memory) 109, an encoding parameter determining unit 110, a parameter encoding unit 111, a prediction parameter derivation unit 120, and an entropy encoding unit 104.

[0140] The predicted image generating unit 101 generates a predicted image for each CU.

[0141] The subtraction unit 102 generates a prediction error by subtracting the pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 from the pixel values ​​of the image T. The subtraction unit 102 outputs the prediction error to the transformation and quantization unit 103.

[0142] The transform / quantization unit 103 calculates transform coefficients by frequency transforming the prediction errors input from the subtraction unit 102, and derives quantized transform coefficients by quantizing the prediction errors. The transform / quantization unit 103 outputs the quantized transform coefficients to the parameter coding unit 111 and the inverse quantization / inverse transform unit 105.

[0143] The inverse quantization and inverse transform unit 105 is the same as the inverse quantization and inverse transform unit 311 (FIG. 4) in the image decoding device 31, and a description thereof will be omitted. The calculated prediction error is output to the addition unit .

[0144] The parameter coding unit 111 includes a header coding unit 1110, a CT information coding unit 1111, and a CU coding unit 1112 (prediction mode coding unit). The CU coding unit 1112 further includes a TU coding unit 1114. The following describes an outline of the operation of each module.

[0145] The header encoding unit 1110 performs encoding processing of parameters such as header information, division information, prediction information, and quantized transform coefficients.

[0146] The CT information encoding unit 1111 encodes the QT, MT (BT, TT) division information and the like.

[0147] The CU encoding unit 1112 encodes the CU information, prediction information, division information, and so on.

[0148] When a prediction error is included in a TU, the TU encoding unit 1114 encodes the QP update information and the quantized prediction error.

[0149] The CT information encoding unit 1111 and the CU encoding unit 1112 supply syntax elements such as inter prediction parameters and quantized transform coefficients to the parameter encoding unit 111.

[0150] The entropy coding unit 104 receives the quantized transform coefficients and the coding parameters from the parameter coding unit 111. The entropy coding unit 104 entropy codes these to generate and output coded data Te.

[0151] The prediction parameter derivation unit 120 derives inter prediction parameters and intra prediction parameters from the parameters input from the encoding parameter determination unit 110. The derived inter prediction parameters and intra prediction parameters are output to the parameter encoding unit 111.

[0152] The adder 106 generates a decoded image by adding, for each pixel, the pixel value of the predicted block input from the predicted image generation unit 101 and the prediction error input from the inverse quantization and inverse transform unit 105. The adder 106 stores the generated decoded image in a reference picture memory 109.

[0153] The loop filter 107 performs deblocking filtering, SAO, and ALF on the decoded image generated by the adder 106. Note that the loop filter 107 does not necessarily have to include the above three types of filters, and may be configured, for example, as only a deblocking filter.

[0154] The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in a predetermined location for each current picture and CU.

[0155] The reference picture memory 109 stores the decoded image generated by the loop filter 107 at a predetermined position for each current picture and CU.

[0156] The coding parameter determination unit 110 selects one set from among a plurality of sets of coding parameters. The coding parameters are the above-mentioned QT, BT or TT division information, prediction parameters, or parameters to be coded that are generated in relation to these. The predicted image generation unit 101 generates a predicted image using these coding parameters.

[0157] In addition, a part of the image encoding device 11 and the image decoding device 31 in the above-mentioned embodiment, for example, the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the predicted image generating unit 308, the inverse quantization and inverse transform unit 311, the addition unit 312, the prediction parameter derivation unit 320, the predicted image generating unit 101, the subtraction unit 102, the transform and quantization unit 103, the entropy coding unit 104, the inverse quantization and inverse transform unit 105, the loop filter 107, the coding parameter determination unit 110, the parameter coding unit 111, and the prediction parameter derivation unit 120 may be realized by a computer. In this case, a program for realizing this control function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read and executed by a computer system. In addition, the "computer system" referred to here is a computer system built into either the image encoding device 11 or the image decoding device 31, and may include an OS, peripheral devices, etc. The term "computer-readable recording medium" includes hardware such as a portable medium, a flexible disk, an optical magnetic disk, a ROM, a CD-ROM, etc., and a storage device such as a hard disk built into a computer system. Furthermore, the term "computer-readable recording medium" may also include a medium that dynamically holds a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in such a case. The above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0158] In addition, a part or the whole of the image encoding device 11 and the image decoding device 31 in the above-mentioned embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image encoding device 11 and the image decoding device 31 may be individually made into a processor, or a part or the whole may be integrated into a processor. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the progress of semiconductor technology, an integrated circuit based on that technology may be used.

[0159] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention.

[0160] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. In other words, the technical scope of the present invention also includes embodiments obtained by combining technical means that are appropriately modified within the scope of the claims. [Industrial Applicability]

[0161] The embodiments of the present invention can be suitably applied to a video decoding device that decodes coded data in which image data is coded, and a video coding device that generates coded data in which image data is coded, and can also be suitably applied to the data structure of coded data that is generated by a video coding device and referenced by the video decoding device. [Explanation of symbols]

[0162] 1. Video transmission system 30 Video Decoding Device 31 Image Decoding Device 301 Entropy Decoding Unit 302 Parameter Decoding Unit 305, 107 Loop Filter 306, 109 Reference Picture Memory 307, 108 Prediction parameter memory 308, 101 Prediction image generation unit 311, 105 Inverse quantization and inverse transformation unit 312, 106 Addition section 320 Prediction Parameter Derivation Unit 10 Video Encoding Device 11 Image encoding device 102 Subtraction section 103 Transformation and Quantization Section 104 Entropy coding unit 110 Encoding parameter determination unit 111 Parameter Encoding Unit 120 Prediction parameter derivation part 41 Image display device 51 Pre-filter processing device 61 Auxiliary Expansion Processor 71 Auxiliary extension information creation device 81 Supplementary extension information coding device 91 Auxiliary extension information decoding device

Claims

1. an image decoding device that decodes encoded data to generate a decoded image; an auxiliary extension processing device that performs an auxiliary extension process on the decoded image; an auxiliary extension information decoding device that decodes a plurality of auxiliary extension information for operating the auxiliary extension processing device; The auxiliary extension information has auxiliary extension information indicating a priority processing order of a plurality of auxiliary extension information, The auxiliary extension information indicating a priority processing order of the plurality of auxiliary extension information includes: A video decoding device comprising information indicating that parameters of a decoded image to be input to an auxiliary enhancement processor are changed according to a processing priority order.

2. an image encoding device that encodes a moving image signal to generate encoded data; An auxiliary extension information creating device for creating a plurality of auxiliary extension information; a supplemental extension information encoding device for encoding the plurality of supplemental extension information, The auxiliary extension information has auxiliary extension information indicating a priority processing order of a plurality of auxiliary extension information, The auxiliary extension information indicating a priority processing order of the plurality of auxiliary extension information includes: A video encoding device comprising information indicating that parameters of a locally decoded image to be input to an auxiliary enhancement processor are changed according to a processing priority order.

Images