Moving image encoding device and decoding device

Abstract

In an SEI processing sequence SEI message, input and output image sizes are different according to the SEI message, so that processing consistency cannot be obtained. A moving image decoding device according to one embodiment of the present invention is characterized by being provided with: an image decoding device that decodes encoded data and generates a decoded image; an auxiliary enhancement processing device for performing auxiliary enhancement processing on the decoded image; and an auxiliary enhancement information decoding device that decodes a plurality of pieces of auxiliary enhancement information for operating the auxiliary enhancement processing device, the auxiliary enhancement information having auxiliary enhancement information indicating a priority processing order of the plurality of pieces of auxiliary enhancement information. The auxiliary enhancement information includes information indicating that a parameter of a decoded image input to the auxiliary enhancement processing device is changed in accordance with a priority processing order.

Description

Technical Field

[0001] Embodiments of the present invention relate to a moving picture encoding device and a decoding device. Background Art

[0002] In order to efficiently transmit or record moving images, a moving image encoding device that generates encoded data by encoding the moving image and a moving image decoding device that generates a decoded image by decoding the encoded data are used.

[0003] Specific moving picture coding methods include, for example, H.264 / AVC and H.265 / HEVC (High-Efficiency Video Coding).

[0004] In such a motion image coding method, images (pictures) constituting the motion image are managed through a hierarchical structure and encoded / decoded for each CU. The hierarchical structure includes slices obtained by dividing the image, coding tree units (CTU: Coding Tree Unit) obtained by dividing the slices, coding units (sometimes also called coding units (Coding Unit: CU)) obtained by dividing the coding tree units, and transform units (TU: Transform Unit) obtained by dividing the coding units.

[0005] In such a moving picture coding method, a prediction image is usually generated based on a local decoded image obtained by encoding / decoding an input image, and a prediction error (sometimes also referred to as a "difference image" or "residual image") obtained by subtracting the prediction image from the input image (original image) is encoded. Examples of methods for generating a prediction image include inter-picture prediction (inter-frame prediction) and intra-picture prediction (intra-frame prediction).

[0006] In addition, as a technology for motion picture encoding and decoding in recent years, a supplementary enhancement information SEI (Supplemental Enhancement Information) message for transmitting the properties, display method, timing, etc. of an image simultaneously with the encoded data is specified. In non-patent document 1, an SEI message is proposed, which specifies information (priority processing order information) indicating a preferred processing order determined by the encoding side when there are multiple SEI messages described in non-patent document 2, etc. Prior art literature Non-patent literature

[0007] Non-patent document 1: SEI processing order SEI message in VVC (draft 5), JVET-AE2027 Non-patent document 2: Consent: H.266V3 "Versatile video coding" (Rev.), ITU-T SG16 (Study Period 2022) Temporary Document 171-PLEN Summary of the invention Problem that the invention aims to solve

[0008] In the SEI processing order SEI message disclosed in Non-Patent Document 1, the input and output image sizes differ depending on the SEI message, so there is a problem that the consistency of processing cannot be achieved. Technical Solution

[0009] A moving picture decoding device according to one aspect of the present invention is characterized by comprising: An image decoding device decodes the encoded data to generate a decoded image; an auxiliary enhancement processing device, performing auxiliary enhancement processing on the decoded image; and an auxiliary enhancement information decoding device for decoding a plurality of auxiliary enhancement information for operating the auxiliary enhancement processing device; The auxiliary enhancement information includes auxiliary enhancement information indicating a priority processing order of the plurality of auxiliary enhancement information. The auxiliary enhancement information indicating the priority processing order of the plurality of auxiliary enhancement information includes information indicating that a parameter of a decoded image input to the auxiliary enhancement processing device is changed according to the priority processing order.

[0010] Furthermore, a moving picture encoding device according to one aspect of the present invention is characterized by comprising: An image coding device encodes a moving image signal to generate coded data; An auxiliary enhancement information generating device generates a plurality of auxiliary enhancement information; and an auxiliary enhancement information encoding device, encoding the plurality of auxiliary enhancement information, The auxiliary enhancement information includes auxiliary enhancement information indicating a priority processing order of the plurality of auxiliary enhancement information. The auxiliary enhancement information indicating the priority processing order of the plurality of auxiliary enhancement information includes information indicating that a parameter of a local decoded image input to the auxiliary enhancement processing device is changed according to the priority processing order. Beneficial Effects

[0011] By adopting such a configuration, it is possible to solve the problem that the processing cannot be made consistent due to the difference in the input and output image sizes. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a schematic diagram showing the configuration of a moving image transmission system according to the present embodiment. Figure 2 It is a diagram showing the hierarchical structure of encoded data. Figure 3 This is a diagram showing the hierarchical structure of coded data in units of PUs. Figure 4 This is a schematic diagram showing the structure of an image decoding device. Figure 5 This is a flowchart for explaining the schematic operation of the image decoding device. Figure 6 This is a block diagram showing the structure of an image encoding device. Figure 7 2 is a diagram showing the syntax of SEI message in SEI processing order. Figure 8 is a diagram representing the SEI payload type of the SEI message. Fig. 9 This is a diagram showing the syntax of the SEI message in the SEI processing order according to this embodiment. Fig.10 This is a diagram showing another syntax of the SEI processing order SEI message according to this embodiment. Fig.11 This is a diagram showing another syntax of the SEI processing order SEI message according to this embodiment. Fig.12 This is a diagram showing the syntax of the SEI payload which is a container of the SEI message. DETAILED DESCRIPTION (First Embodiment)

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0014] Figure 1 This is a schematic diagram showing the configuration of a moving image transmission system according to the present embodiment.

[0015] The moving picture transmission system 1 is a system that transmits coded data obtained by encoding an image, decodes the transmitted coded data, and displays the coded data. The moving picture transmission system 1 is composed of a moving picture encoding device 10 , a network 21 , a moving picture decoding device 30 , and an image display device 41 .

[0016] The moving image encoding device 10 is composed of an image encoding device (image encoding unit) 11, an auxiliary enhancement information generating device (auxiliary enhancement information generating unit) 71, an auxiliary enhancement information encoding device (auxiliary enhancement information encoding unit) 81, and a pre-filter processing device (pre-filter processing unit) 51.

[0017] The motion image encoding device 10 generates a pre-filtered processed image T2 from the input motion image T1 through the pre-filtering processing device 51, compresses and encodes the image through the image encoding device 11, and analyzes the input motion image T1 and the local decoded image T3 of the image encoding device 11, generates auxiliary enhancement information for input to the auxiliary enhancement processing device 61 through the auxiliary enhancement information generating device 71, encodes through the auxiliary enhancement information encoding device 81, generates encoded data Te, and sends it to the network 21.

[0018] The moving picture decoding device 30 is composed of an image decoding device (image decoding unit) 31 , an auxiliary enhancement information decoding device (auxiliary enhancement information decoding unit) 91 , and an auxiliary enhancement processing device (auxiliary enhancement processing unit) 61 .

[0019] The motion image decoding device 30 decodes the encoded data Te received from the network 21 through the image decoding device 31 and the auxiliary enhancement information decoding device 91, performs auxiliary enhancement processing on the decoded image Td1 using the auxiliary enhancement information through the auxiliary enhancement processing device 61, and outputs the auxiliary enhanced decoded image Td2 to the image representation device 41.

[0020] It should be noted that the auxiliary enhancement processing device 61 may not perform the auxiliary enhancement processing using the auxiliary enhancement information, but may output the decoded image Td1 as it is.

[0021] The image display device 41 displays all or part of the auxiliary enhancement processing image Td2 output from the auxiliary enhancement processing device 61. The image display device 41 includes, for example, a display device such as a liquid crystal display, an organic EL (Electroluminescence) display, etc. As the form of the display, fixed type, mobile type, HMD (Head Mounted Display), etc. can be listed. In addition, when the image decoding device 31 has high processing power, a high-quality image is displayed, and when it has only low processing power, an image that does not require high processing power and display power is displayed.

[0022] The network 21 transmits the encoded auxiliary enhancement information and the encoded data Te to the image decoding device 31. Part or all of the encoded auxiliary enhancement information may also be included in the encoded data Te as the auxiliary enhancement information SEI. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. The network 21 is not necessarily limited to a two-way communication network, and may also be a one-way communication network that transmits broadcast waves such as terrestrial digital broadcasts and satellite broadcasts. In addition, the network 21 may be replaced by a storage medium such as a DVD (Digital Versatile Disc, registered trademark) or a BD (Blue-ray Disc, registered trademark) on which the encoded data Te is recorded.

[0023] In such a configuration, an architecture that can efficiently encode and decode auxiliary enhancement information is provided. <Operator>

[0024] The following describes the operators used in this specification.

[0025] >> is a right shift, << is a left shift, & is a bitwise AND, | is a bitwise OR, |= is an OR assignment operator, and || represents a logical OR.

[0026] x? y:z is a ternary operator that takes y when x is true (non-zero) and takes z when x is false (0).

[0027] Clip3(a, b, c) is a function that limits c to a value between a and b (inclusive), returning a when c < a, b when c > b, and c otherwise (where a <= b).

[0028] abs(a) is a function that returns the absolute value of a.

[0029] Int(a) is a function that returns the integer value of a.

[0030] Floor(a) is a function that returns the largest integer less than or equal to a.

[0031] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0032] a / d means a divided by d (truncating the decimal part).

[0033] a÷d and a over d mean a divided by d (without rounding). (Structure of the encoded data Te)

[0034] Before describing in detail the image encoding device 11 and the image decoding device 31 of this embodiment, Figure 2 , Figure 3 The data structure of the encoded data Te generated by the image encoding device 11 and decoded by the image decoding device 31 will be described.

[0035] The coded data Te is composed of Figure 2 The bitstream shown is composed of multiple CVS (Coded Video Sequence) and EoB (End of Bitstream) NAL units. CVS is composed of multiple AU (Access Unit) and EoS (End of Sequence) NAL units. The AU at the beginning of CVS is called CVSS (Coded Video Sequence Start) AU. The unit obtained by dividing CVS per layer is called CLVS (Coded Layer Video Sequence). AU is composed of PU (Picture Unit) of one or more layers at the same output time. If the multi-layer encoding method is not adopted, AU is composed of one PU. PU is a unit of encoded data of a decoded picture composed of multiple NAL units. CLVS is composed of PUs of the same layer, and the PU at the beginning of CLVS is called CLVSS (Coded Layer Video Sequence Start) PU. CLVSS PU is limited to PUs composed of IRAP (Intra Random Access Pictures) and GDR (Gradual Decoder Refresh Picture) that can be randomly accessed. NAL unit consists of NAL unit header and RBSP (Raw Byte Sequence Payload) data. NAL unit header consists of 6-bit nuh_layer_id indicating layer value followed by 2 bits of 0 data, 5-bit nuh_unit_type indicating NAL unit type, and 3-bit nuh_temporal_id_plus1 which is the value obtained by adding 1 to Temporal ID value.

[0036] Figure 3 : is a diagram showing the hierarchical structure of data in coded data Te in units of PUs. The coded data Te exemplarily includes a sequence and a plurality of pictures constituting the sequence. Figure 3 The figure shows a coded video sequence of a specified sequence SEQ, a coded picture of a specified picture PICT, a coded slice of a specified slice S, coded slice data of the specified slice data, a coding tree unit included in the coded slice data, and a coding unit included in the coding tree unit.

[0037] In the coded video sequence, a set of data is specified for the image decoding device 31 to refer to in order to decode the sequence SEQ to be processed. Figure 3 As shown, the sequence SEQ includes a video parameter set (Video Parameter Set), a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), an adaptation parameter set (Adaptation Parameter Set: APS), a picture PICT and supplemental enhancement information SEI (Supplemental Enhancement Information).

[0038] In the video parameter set VPS, in a moving picture composed of multiple layers, a set of coding parameters common to a plurality of moving pictures, and a set of multiple layers included in the moving picture and coding parameters associated with each layer are defined.

[0039] In the sequence parameter set SPS, a set of coding parameters is specified for the image decoding device 31 to refer to in order to decode the object sequence. For example, the width and height of the picture are specified. It should be noted that there can be multiple SPSs. In this case, any one of the multiple SPSs is selected from the PPS.

[0040] Here, the sequence parameter set SPS includes the following syntax elements. pic_width_max_in_luma_samples: This is a syntax element that specifies the width of the image with the maximum width in units of luma blocks among the images in a single sequence. In addition, the value of this syntax element is required to be non-zero and an integer multiple of Max(8, MinCbSizeY). Here, MinCbSizeY is a value determined based on the minimum size of a luma block. pic_height_max_in_luma_samples: This is a syntax element that specifies the height of the image with the maximum height in units of luma blocks among images in a single sequence. In addition, the value of this syntax element is required to be non-zero and an integer multiple of Max(8, MinCbSizeY).

[0041] In the picture parameter set PPS, a set of coding parameters is defined for reference by the image decoding device 31 to decode each picture in the target sequence. It should be noted that there may be multiple PPSs. In this case, any one of the multiple PPSs is selected from each picture in the target sequence.

[0042] Here, the picture parameter set PPS includes the following syntax elements.

[0043] pps_pic_width_in_luma_samples is a syntax element for specifying the width of the target picture. The value of this syntax element is required to be non-zero, an integer multiple of Max(8, MinCbSizeY), and a value less than or equal to sps_pic_width_max_in_luma_samples.

[0044] pps_pic_height_in_luma_samples is a syntax element for specifying the height of the target picture. The value of this syntax element is required to be non-zero, an integer multiple of Max(8, MinCbSizeY), and a value less than or equal to sps_pic_height_max_in_luma_samples.

[0045] pps_conformance_window_flag is a flag indicating whether to continuously notify the conformance (cropping) window offset parameter, and is a flag indicating the location where the conformance window is displayed. When this flag is 1, the parameter is notified, and when this flag is 0, it indicates that there is no conformance window offset parameter.

[0046] conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset are offset values ​​for specifying the left, right, top, and bottom positions of the picture output by the decoding process with respect to the rectangular area specified by the picture coordinates for output. In addition, when the value of conformance_window_flag is 0, the values ​​of conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset are estimated to be 0.

[0047] The output image is horizontally from SubWidthC*pps_conf_win_left_offset to pps_pic_width_in_luma_samples- The vertical direction is the pixels from SubHeightC*pps_conf_win_top_offset to pps_pic_height_in_luma_samples-(SubHeightC*pps_conf_win_bottom_offset+1).

[0048] Here, the variable ChromaFormatIdc of the color difference format is the value of sps_chroma_format_id, and the variables SubWidthC and SubHightC are values ​​determined by the ChromaFormatIdc. In the case of monochrome format, SubWidthC and SubHightC are both 1, in the case of 4:2:0 format, SubWidthC and SubHightC are both 2, in the case of 4:2:2 format, SubWidthC is 2, SubHightC is 1, and in the case of 4:4:4 format, SubWidthC and SubHightC are both 1. (Encoded image)

[0049] In the coded picture, a set of data is defined for reference by the image decoding device 31 in order to decode the picture PICT to be processed. Figure 3 As shown, the picture PICT includes a picture header PH, and slices 0 to NS-1 (NS is the total number of slices included in the picture PICT). (Encoding slice)

[0050] In the coded slice, a set of data is defined for reference by the image decoding device 31 in order to decode the slice S to be processed. Figure 3 The shown includes a slice header and slice data.

[0051] The slice header includes a coding parameter group that the image decoding device 31 refers to in order to determine a decoding method of a target slice. Slice type designation information (slice_type) that designates a slice type is one example of a coding parameter included in the slice header.

[0052] Examples of slice types that can be specified by the slice type specification information include: (1) I slices that use only intra-frame prediction when encoding, (2) P slices that use unidirectional prediction (L0 prediction) or intra-frame prediction when encoding, and (3) B slices that use unidirectional prediction (L0 prediction or L1 prediction), bidirectional prediction, or intra-frame prediction when encoding. It should be noted that inter-frame prediction is not limited to unidirectional prediction and bidirectional prediction, and more reference pictures can be used to generate predicted images. Hereinafter, the cases referred to as P and B slices refer to slices that include blocks that can use inter-frame prediction. (Encoded slice data)

[0053] The coded slice data specifies a set of data that the image decoding device 31 refers to in order to decode the slice data to be processed. Figure 3 As shown in the coded slice header of , the slice data includes CTU. CTU is a fixed-size (for example, 64×64) block constituting a slice, and is also called the Largest Coding Unit (LCU). (Coding Tree Unit)

[0054] exist Figure 3 In the , a set of data is specified for the image decoding device 31 to refer to in order to decode the CTU of the processing object. The CTU is divided into coding units CU as the basic unit of the encoding process by recursive quadtree division (QT (Quad Tree) division), binary tree division (BT (Binary Tree) division) or ternary tree division (TT (Ternary Tree) division). The nodes of the tree structure obtained by recursive quadtree division are called coding nodes. The middle nodes of the quadtree, binary tree and ternary tree are coding nodes, and the CTU itself is also specified as the top coding node. (Coding unit)

[0055] exist Figure 3 In the CU header, a set of data is specified for the image decoding device 31 to refer to in order to decode the coding unit to be processed. Specifically, the CU is composed of a CU header CUH, prediction parameters, conversion parameters, quantized conversion coefficients, etc. The prediction mode, etc. are specified in the CU header.

[0056] The prediction process may be performed in units of CUs or in units of sub-CUs obtained by further dividing the CU.

[0057] There are two types of prediction (prediction modes): intra prediction and inter prediction. Intra prediction is prediction within the same picture, while inter prediction refers to prediction processing between different pictures (for example, between display times, between layer images).

[0058] The conversion / quantization process is performed in units of CUs, but the quantized conversion coefficients may be entropy-coded in units of 4×4 or other sub-blocks.

[0059] It should be noted that in the present specification, when a "flag indicating whether it is ××" is written, a flag other than 0 (e.g., 1) is regarded as ××, and a flag 0 is regarded as not ××, and 1 is regarded as true and 0 is regarded as false in logical negation, logical product, etc. (the same applies hereinafter). However, other values ​​may be used as true values ​​and false values ​​in actual devices and methods. (Configuration of Image Decoding Device)

[0060] The image decoding device 31 ( Figure 4 ) is described below.

[0061] The image decoding device 31 is configured to include an entropy decoding unit 301, a parameter decoding unit (prediction image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a prediction image generation unit (prediction image generation device) 308, an inverse quantization / inverse conversion unit 311 and an addition unit 312, and a prediction parameter derivation unit 320. It should be noted that, according to the image encoding device 11 described later, there is also a configuration in which the loop filter 305 is not included in the image decoding device 31.

[0062] The parameter decoding unit 302 also includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit). The CU decoding unit 3022 also includes a TU decoding unit 3024. They can also be collectively referred to as decoding modules. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, PPS, APS, and slice header (slice information) from the encoded data. The CT information decoding unit 3021 decodes CT from the encoded data. The CU decoding unit 3022 decodes CU from the encoded data. The TU decoding unit 3024 decodes QP (Quantization Parameter) update information (quantization correction value) and quantization prediction error (residual coding) from the encoded data.

[0063] The predicted image generation unit 308 is configured to include an inter-predicted image generation unit 309 and an intra-predicted image generation unit 310 .

[0064] The entropy decoding unit 301 performs entropy decoding on the coded data Te input from the outside, and decodes each code (syntax element).

[0065] The entropy decoding unit 301 outputs the decoded code to the parameter decoding unit 302. Based on the instruction of the parameter decoding unit 302, control is performed as to which code is to be decoded. (Basic process)

[0066] Figure 5 This is a flowchart for explaining the schematic operation of the image decoding device 31 .

[0067] (S1100: Parameter set information decoding) The header decoding unit 3020 decodes parameter set information such as VPS, SPS, and PPS from the encoded data.

[0068] (S1200: Slice Information Decoding) The header decoding unit 3020 decodes a slice header (slice information) from the encoded data.

[0069] Thereafter, the image decoding device 31 derives a decoded image of each CTU by repeatedly performing the processes from S1300 to S5000 for each CTU included in the target picture.

[0070] (S1300: CTU Information Decoding) The CT information decoding unit 3021 decodes CTU from the encoded data.

[0071] (S1400: CT Information Decoding) The CT information decoding unit 3021 decodes CT from the encoded data.

[0072] (S1500: CU Decoding) The CU decoding unit 3022 implements S1510 and S1520 to decode the CU from the encoded data.

[0073] (S1510: CU Information Decoding) The CU decoding unit 3022 decodes CU information, prediction information, and the like from the encoded data.

[0074] (S1520: TU Information Decoding) The TU decoding unit 3024 decodes QP update information, quantization prediction error, etc. from the coded data. It should be noted that the QP update information is a difference value from a quantization parameter prediction value qPpred which is a prediction value of the quantization parameter QP.

[0075] (S2000: Prediction Image Generation) The prediction image generation unit 308 generates a prediction image for each block included in the target CU based on the prediction information.

[0076] (S3000: Inverse Quantization / Inverse Transformation) The inverse quantization / inverse transformation unit 311 performs an inverse quantization / inverse transformation process on each TU included in the target CU.

[0077] (S4000: Decoded Image Generation) The addition unit 312 generates a decoded image of the target CU by adding the predicted image supplied from the predicted image generation unit 308 and the prediction error supplied from the inverse quantization / inverse conversion unit 311 .

[0078] (S5000: Loop Filtering) The loop filter 305 applies loop filtering such as deblocking filtering, SAO (Sample Adaptive Offset), and ALF (Adaptive Loop Filter) to the decoded image to generate a decoded image. (SEI processing order SEI message)

[0079] Figure 7 The syntax of the SEI processing order SEI message is shown. The SEI processing order SEI message notifies information indicating a preferred processing order (priority processing order information) determined by the encoder (ie, content producer) for different types of SEI messages that may exist in the CVS.

[0080] When the SEI processing order SEI message exists in any access unit of the CVS, the SEI processing order SEI message must exist in the first access unit of the CVS.

[0081] SEI processing order SEI messages continue in decoding order, from the current access unit until the end of the CVS.

[0082] In the case where there are multiple SEI processing order SEI messages in one CVS, they must have the same content. In addition, within one SEI processing order SEI message, there need to be at least two pairs of syntax elements po_sei_payload_type[i] and po_sei_processing_order[i], and there are at least two values ​​of po_sei_processing_order[i] that are not equal.

[0083] SEI Processing Order SEI messages can notify the information of the beginning of one or more SEIs of a specific payloadType. The information of the beginning of each SEI is a byte string of the SEI payload syntax according to the value of payloadType, including several complete syntax elements starting from the first syntax element of the SEI payload. In addition, there are sometimes bits that do not represent the complete syntax elements of the SEI payload.

[0084] The information at the beginning of these SEIs must be sufficient to distinguish when there are SEIs with the same payloadType value but in a different order.

[0085] po_num_sei_messages_minus2 plus 2 indicates the number of SEI messages having the processing order indicated by the SEI Processing Order SEI message.

[0086] It is assumed that reserved_alignment_6bits is set equal to 0. The syntax element may also have a value other than 0, but in this case the value is ignored.

[0087] In case the value of po_sei_wrapping_flag[i] is 0, SEI messages need to exist outside SEI processing order SEI messages with payloadType equal to po_sei_payload_type[i]. However, in case the value of po_sei_wrapping_flag[i] is 0 and there is no SEI message with payloadType equal to po_sei_payload_type[i], the following applies.

[0088] In case po_sei_importance_flag[i] is equal to 1, the decoder needs to ignore the entire SEI processing order SEI message.

[0089] In all other cases, the decoder needs to ignore all data associated with the loop variable value i.

[0090] However, when the value of po_sei_wrapping_flag[i] is 1, SEI messages can be carried within SEI processing order SEI messages, which can prevent decoders that do not process SEI processing order SEI messages from erroneously interpreting such SEI messages. Therefore, po_sei_wrapping_flag[i] equal to 1 is intended to be used when po_sei_wrapping_flag[i] is 0 and unexpected results are produced by such decoding devices.

[0091] When the value of po_sei_prefix_flag[i] is 1, it means that po_num_prefix_bytes[i] exists, and when the value of po_sei_prefix_flag[i] is 0, it means that po_num_prefix_bytes[i] does not exist.

[0092] po_sei_payload_type[i] specifies the payloadType of the i-th SEI message in the SEI processing order SEI.

[0093] For two different non-negative integer values ​​of m and n, the values ​​of po_sei_payload_type[m] and po_sei_payload_type[n] must be different unless po_sei_prefix_flag[m] and po_sei_prefix_flag[n] are both equal to 1.

[0094] The list SeiProcessingOrderSeiList storing the priority processing order information is set to be composed of the values ​​other than 137, 144, 147, 148, 179, 180, 200, 201, 208, and 213. Figure 8 That is, the following SEI message is not applied in the SEI processing order SEI message. ·mastering_display_colour_volume content_light_level_info ·alternative_transfer_characteristic ·ambient_viewing_environment ·multiview_acquisition_info multiview_view_position ·sei_manifest sei_prefix_indication scalability_dimension_info When po_sei_payload_type[i] is not equal to any value in SeiProcessingOrderSeiList, the value of po_sei_prefix_flag[i] is 0.

[0095] In case po_num_prefix_bytes[i] is present, the priority processing order information specifies the number of bytes associated with the i-th SEI message provided in the SEI processing order SEI message. In case it is not present, the value of po_num_prefix_bytes[i] is estimated to be equal to 0.

[0096] If po_prefix_byte[i][j] exists, it specifies the j-th byte value of the i-th SEI message.

[0097] po_sei_processing_order[i] indicates the priority processing order of the i-th SEI message. For m and n being two different integer values ​​greater than 0, po_sei_processing_order[m] being less than po_sei_processing_order[n] indicates the type of the SEI message whose payloadType is equal to po_sei_payload_type[m].

[0098] The byte-based bitstream of po_prefix_byte[m][p] (the range of p is greater than or equal to 0 and less than or equal to po_num_prefix_bytes[m]-1) of po_sei_processing_order[m] needs to be processed before any SEI message whose payloadType is equal to po_sei_payload_type[n].

[0099] The case where po_sei_processing_order[m] is equal to po_sei_processing_order[n] indicates that there is no priority processing order between SEI messages.

[0100] When there are multiple SEI messages with the same value of po_sei_payload_type[i] and the same value of po_num_prefix_bytes[i], it is assumed that they must have the same value of po_sei_processing_order[i].

[0101] The case where po_sei_processing_order[m] is equal to po_sei_processing_order[n] indicates that there is no priority processing order between SEI messages.

[0102] In the case where there are multiple SEI messages having values ​​of po_sei_payload_type[i] and po_num_prefix_bytes[i] and the byte value of po_prefix_byte[i][j] having the same value with j ranging from 0 to po_num_prefix_bytes[i]-1, it is assumed that they have the same value of po_sei_processing_order[i].

[0103] po_sei_processing_order[0] must be 0. If i is greater than 0, po_sei_processing_order[i] is equal to either po_sei_processing_order[i-1] or po_sei_processing_order[i-1] + 1. Furthermore, it is assumed that the value of po_sei_processing_order[po_num_sei_messages_minus2+1] is not 0.

[0104] As the image size (image area) decoded from the encoded data, there are an image size / image area obtained by decoding pps_pic_width_in_luma_samples and pps_pic_height_in_luma_samples of the parameter set and an image size / image area obtained by further cropping an area called a cropping area. The former is called an uncropped image size / image area, and the latter is called a cropped image size / image area.

[0105] In the SEI processing order SEI message disclosed in Non-Patent Document 1, input and output image sizes (ranges of screen areas to be processed) differ according to the SEI message, so there is a problem that processing consistency cannot be achieved.

[0106] Specifically, according to Annex D of Non-Patent Document 2, Film grain characteristics SEI message Decoded picture hash SEI message In, it is defined as inputting an uncropped picture of the same size as pps_pic_width_in_luma_samples and pps_pic_height_in_luma_samples as the input image for SEI.

[0107] On the other hand, Annotated regions SEI message Neural network post-filter characteristics SEI message and neural network post-filter activation SEI message Phase indication SEI message In , the input is defined as the following cropped image of size (CropedWidth, CropedHight).

[0108] CropedWidth=pps_pic_width_in_luma_samples-SubWidthC* (conf_win_right_offset+conf_win_left_offset) CropedHight=pps_pic_height_in_luma_samples-SubHightC* (conf_win_bottom_offset+conf_win_top_offset) For other SEI messages, basically the SEI message related picture is assumed to be a picture with a cropped image size.

[0109] When linking multiple SEI messages and performing auxiliary enhancement processing according to the SEI processing order SEI messages, if the input image sizes are different, a larger image size needs to be considered for execution, and the auxiliary enhancement processing device needs to perform measures such as pre-buffering the image at an uncropped image size and changing the processing range according to the SEI message.

[0110] In this embodiment, in order to solve part of the above problems, it is assumed that Film grain characteristics SEI information Decode the picture hash SEI message In , set the input to the following cropped image size (CropedWidth, CropedHight).

[0111] CropedWidth=pps_pic_width_in_luma_samples-SubWidthC* (conf_win_right_offset+conf_win_left_offset) CropedHight=pps_pic_height_in_luma_samples-SubHightC* (conf_win_bottom_offset+conf_win_top_offset) In addition, for Film grain characteristics SEI information Decode the picture hash SEI message In the case of the beginning (0th priority processing order), the input can be set to the uncropped image size, and in other cases, the input can be set to the cropped image size.

[0112] Furthermore, in the case where there is SEI processing using a cropped image size, the subsequent SEI must input the picture set to the cropped image size. Film grain characteristics SEI information Decode the picture hash SEI message By setting such a specification, at least the difference in the auxiliary enhancement processing range within the picture caused by the SEI message can be corrected.

[0113] In this embodiment, in order to solve part of the above problems, it can be set to Decode the picture hash SEI message Put it in the excluded SEI list in SeiProcessingOrderSeiList, so this SEI is not applied in the SEI processing order SEI message. That is, SeiProcessingOrderSeiList can include values ​​137, 144, 147, 148, 179, 180, 200, 201, 208, 213 and 132, as well as 132 representing the decoded picture hash SEI message.

[0114] Alternatively, in this embodiment, in order to solve part of the above problems, it is also possible to Decode the picture hash SEI message A restriction is set to always start with the processing order. By setting such a specification, it is possible to eliminate contradictions in the SEI processing order definition for decoding the picture hash SEI message.

[0115] In this embodiment, in order to solve part of the above problems, it is also possible to Film grain characteristics SEI information Set a restriction that always starts with the processing order. By setting this specification, SEI that uses the entire uncropped screen can be processed without any problems.

[0116] In this embodiment, in order to solve part of the above problems, it can be set to Film grain characteristics SEI information Put into the excluded SEI list in SeiProcessingOrderSeiList, so this SEI is not applied in the SEI processing order SEI message. That is, SeiProcessingOrderSeiList can include values ​​137, 144, 147, 148, 179, 180, 200, 201, 208, 213 and 19. 19 represents the film grain characteristics SEI message.

[0117] As another problem, for example, in the neural network post-filter characteristic SEI message and the neural network post-filter activation SEI message, the output image size, color difference format, pixel bit length, and picture rate are changed according to the processing content. Therefore, in the specification of the SEI message defined based on the image output from the image decoding device, it is impossible to cope with the auxiliary enhancement processing of the SEI processing order SEI message.

[0118] In this embodiment, in order to solve the above problem, in addition to the definition of the processing order, the parameters of the input image are redefined as needed, specifically, the image size, color difference format, pixel bit length, and picture rate are redefined.

[0119] Fig. 9 Indicates that through Figure 7 The syntax element po_sei_override_flag[i] is added to the syntax to redefine the method of inputting the SEI message.

[0120] When the value of po_sei_override_flag[i] is 0, it means that the input of the i-th SEI message is operated according to the output of the image decoder as defined. When the value of po_sei_override_flag[i] is 1, it means that the definition of the input picture of the SEI message of the po_sei_processing_order[i]th priority order is rewritten to the output content of the SEI message that was last changed in the priority order before po_sei_processing_order[i]. Here, the rewritten content is the picture size, color difference format, pixel bit length, and picture rate.

[0121] Specifically, in the definition of the existing SEI message specifications, the image size (cropped image size or uncropped image size), chrominance format, pixel bit length, etc. of the decoded image output by the image decoder are inherited as is, but are replaced by the image size, chrominance format, and pixel bit length of the output of the SEI message that was last changed in the priority processing order before po_sei_processing_order[i].

[0122] Here, it is assumed that when the value of po_sei_override_flag[i] is 1, the value of po_sei_processing_order[i] is 0, and the input of the SEI message of the 0th priority processing order indicated by po_sei_payload_type[i] is the uncropped image size, the cropped image size is input. By adopting such a configuration, even if the parameters of the image to be processed are changed in the middle of the bitstream, auxiliary enhancement processing can be performed.

[0123] Fig.10 This is a diagram showing the syntax when the value of po_sei_processing_order[i] is 0, that is, when the input picture of the SEI message of the 0th priority processing order is not redefined.

[0124] When the value of po_sei_override_flag[i] is 0, it means that the input of the i-th SEI message acts according to the output of the image decoder as defined. When the value of po_sei_override_flag[i] is 1, it means that the definition of the input picture of the SEI message of the po_sei_processing_order[i]th priority order is rewritten to the output content of the SEI message that was last changed in the priority processing order before po_sei_processing_order[i]. Here, the rewritten content is the picture size, color difference format, pixel bit length, and image rate. When po_sei_override_flag[i] does not exist, the value is estimated to be 0. By adopting such a configuration, auxiliary enhancement processing can be performed even if the parameters of the image to be processed are changed in the middle of the bit stream.

[0125] Fig.11 It shows that when the value of po_sei_override_flag[i] is 1, except Fig.10 grammar, explicitly define the parameters as methods of the grammar.

[0126] The syntax element po_pic_width_minus1[i]+1 indicates the number of pixels in the horizontal direction of the brightness of the input image size of the i-th SEI message, the syntax element po_pic_height_minus1[i]+1 indicates the number of pixels in the vertical direction of the brightness of the input image size of the i-th SEI message, po_bitdepth_minus8+8 indicates the pixel bit depth of the input image of the i-th SEI message, and po_chroma_format_idc indicates the color difference format of the input image of the i-th SEI message. By adopting such a configuration, auxiliary enhancement processing can be performed even if the parameters of the image to be processed are changed in the middle of the bit stream.

[0127] exist Fig. 9 , Fig.10 and Fig.11 In , the redefinition of the input picture is clearly indicated by syntax, but a method of implementing it only by defining the decoding process without adding syntax can also be considered.

[0128] When the value of po_sei_processing_order[i] is greater than or equal to 1, the definition of the input image of the SEI message of the priority order po_sei_processing_order[i] is rewritten to the picture size, color difference format, pixel bit depth, and image rate output by the SEI message that was last changed in the priority order before po_sei_processing_order[i]. By adopting such a configuration, auxiliary enhancement processing can be performed even if the parameters of the image to be processed are changed in the middle of the bit stream.

[0129] In addition, as another method of this embodiment, it is also possible to set Neural network post-filtering characteristics SEI message Neural network post-filter activation SEI message Such a restriction must be set at the end of the priority processing order. By setting such a restriction, the image parameters are changed in the final auxiliary enhancement process, so there is no need to redefine the bitstream in the middle. (SEI Payload)

[0130] Fig.12 This is a diagram showing a part of the syntax of the SEI payload which is a container of the SEI message.

[0131] PREFIX_SEI_NUT called when nal_unit_type is PREFIX_SEI_NUT indicates that it is the SEI located before the slice data.

[0132] When payloadType is 19, the film grain characteristics SEI message is called.

[0133] When payloadType is 202, the comment area SEI message is called.

[0134] When payloadType is 210, the neural network post-filtering characteristic SEI message is called.

[0135] When ayloadType is 211, the neural network post-filter is called to activate the SEI message.

[0136] When payloadType is 213, SEI processing sequence SEI is called.

[0137] SUFFIX_SEI_NUT called when nal_unit_type is SUFFIX_SEI_NUT indicates that it is the SEI located after the slice data.

[0138] When payloadType is 132, call the decode picture hash SEI message. (Configuration of Image Coding Device)

[0139] Next, the configuration of the image encoding device 11 according to this embodiment will be described. Figure 6 1 is a block diagram showing the structure of the image encoding device 11 of the present embodiment. The image encoding device 11 is configured to include a prediction image generation unit 101, a subtraction unit 102, a conversion / quantization unit 103, an inverse quantization / inverse conversion unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference picture storage unit, frame memory) 109, a coding parameter determination unit 110, a parameter coding unit 111, a prediction parameter derivation unit 120, and an entropy coding unit 104.

[0140] The predicted image generation unit 101 generates a predicted image for each CU.

[0141] The subtraction unit 102 generates a prediction error by subtracting the pixel value of the predicted image of the block input from the predicted image generation unit 101 from the pixel value of the image T. The subtraction unit 102 outputs the prediction error to the conversion / quantization unit 103 .

[0142] The conversion / quantization unit 103 calculates a conversion coefficient by frequency conversion of the prediction error input from the subtraction unit 102 , and derives a quantized conversion coefficient by quantization. The conversion / quantization unit 103 outputs the quantized conversion coefficient to the parameter encoding unit 111 and the inverse quantization / inverse conversion unit 105 .

[0143] The inverse quantization / inverse conversion unit 105 and the inverse quantization / inverse conversion unit 311 ( Figure 4 ) is the same as that of FIG. 106, and its description is omitted. The calculated prediction error is output to the addition unit 106.

[0144] The parameter coding unit 111 includes a header coding unit 1110, a CT information coding unit 1111, and a CU coding unit 1112 (prediction mode coding unit). The CU coding unit 1112 further includes a TU coding unit 1114. The following describes the schematic operation of each module.

[0145] The header coding unit 1110 performs coding processing on parameters such as header information, segmentation information, prediction information, and quantized transform coefficients.

[0146] The CT information encoding unit 1111 encodes QT, MT (BT, TT) division information and the like.

[0147] The CU encoding unit 1112 encodes CU information, prediction information, partition information, and the like.

[0148] When a prediction error is included in a TU, the TU encoding unit 1114 encodes the QP update information and the quantized prediction error.

[0149] The CT information encoding unit 1111 and the CU encoding unit 1112 provide syntax elements such as inter prediction parameters and quantized transform coefficients to the parameter encoding unit 111 .

[0150] The entropy coding unit 104 receives the quantized transform coefficient and the coding parameter from the parameter coding unit 111. The entropy coding unit 104 performs entropy coding on these, generates and outputs coded data Te.

[0151] The prediction parameter derivation unit 120 derives inter prediction parameters and intra prediction parameters from the parameters input from the encoding parameter determination unit 110. The derived inter prediction parameters and intra prediction parameters are output to the parameter encoding unit 111.

[0152] The adder 106 generates a decoded image by adding the pixel value of the prediction block input from the prediction image generator 101 and the prediction error input from the inverse quantization / inverse conversion unit 105 for each pixel. The adder 106 stores the generated decoded image in the reference picture memory 109.

[0153] The loop filter 107 applies a deblocking filter, SAO, and ALF to the decoded image generated by the adding unit 106. It should be noted that the loop filter 107 does not necessarily include the above three filters, and may include only a deblocking filter, for example.

[0154] The prediction parameter memory 108 stores the prediction parameter generated by the encoding parameter determination unit 110 in a predetermined position for each target picture and CU.

[0155] The reference picture memory 109 stores the decoded image generated by the loop filter 107 at a predetermined position for each target picture and CU.

[0156] The coding parameter determination unit 110 selects one of a plurality of sets of coding parameters. The coding parameters refer to the QT, BT or TT partition information, prediction parameters or parameters generated in association with them as the coding target. The predicted image generation unit 101 generates a predicted image using these coding parameters.

[0157] It should be noted that the image encoding device 11 and part of the image decoding device 31 in the above-mentioned embodiment, for example, the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the prediction image generation unit 308, the inverse quantization / inverse conversion unit 311, the addition unit 312, the prediction parameter derivation unit 320, the prediction image generation unit 101, the subtraction unit 102, the conversion / quantization unit 103, the entropy encoding unit 104, the inverse quantization / inverse conversion unit 105, the loop filter 107, the encoding parameter determination unit 110, the parameter encoding unit 111 and the prediction parameter derivation unit 120 can be implemented by a computer. In this case, the program for realizing the control function can be recorded in a computer-readable recording medium, and the computer system can read the program recorded in the recording medium and execute it. It should be noted that the "computer system" mentioned here refers to a computer system built into any one of the image encoding device 11 and the image decoding device 31, and a computer system including hardware such as an OS and peripheral devices is used. In addition, "computer-readable recording medium" refers to removable media such as floppy disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into computer systems. Moreover, "computer-readable recording medium" may also include: recording media that dynamically store programs for a short period of time, such as communication lines when sending programs via networks such as the Internet or communication lines such as telephone lines; and recording media that store programs for a fixed period of time, such as volatile memories inside computer systems that serve as servers or clients in this case. In addition, the above-mentioned program may be a program for realizing a part of the above-mentioned functions, or a program that can realize the above-mentioned functions by combining with programs already recorded in the computer system.

[0158] In addition, part or all of the image encoding device 11 and the image decoding device 31 in the above-mentioned embodiment may be implemented as an integrated circuit such as LSI (Large Scale Integration). Each functional block of the image encoding device 11 and the image decoding device 31 may be individually processor-based, or part or all may be integrated to form a processor. In addition, the method of integrated circuitization is not limited to LSI, and it may be implemented by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears with the advancement of semiconductor technology, an integrated circuit based on the technology may also be used.

[0159] As mentioned above, one embodiment of the present invention has been described in detail with reference to the drawings, but the specific configuration is not limited to the above embodiment, and various design changes and the like can be made within the scope not departing from the gist of the present invention.

[0160] The embodiments of the present invention are not limited to the above-mentioned embodiments, and various modifications can be made within the scope of the claims. That is, embodiments obtained by combining technical solutions appropriately modified within the scope of the claims are also included in the technical scope of the present invention. Industrial Applicability

[0161] The embodiments of the present invention can be preferably applied to a motion picture decoding device that decodes coded data obtained by encoding image data, and a motion picture encoding device that generates coded data obtained by encoding image data. In addition, it can be appropriately applied to the data structure of the coded data generated by the motion picture encoding device and referred to by the motion picture decoding device. Description of Reference Numerals 1: Motion picture transmission system 30: Motion picture decoding device 31: Image decoding device 301: Entropy decoding unit 302: Parameter decoding unit 305, 107: Loop filter 306, 109: Reference image memory 307, 108: Prediction parameter memory 308, 101: Prediction image generation unit 311, 105: Inverse quantization / inverse conversion unit 312, 106: Addition Department 320: Prediction parameter derivation department 10: Motion picture encoding device 11: Image coding device 102: Subtraction Department 103: Conversion / Quantization Department 104: Entropy coding unit 110: Coding parameter determination unit 111: Parameter encoding unit 120: Prediction parameter derivation department 41: Image display device 51: Pre-filtering device 61: Auxiliary Enhancement Processing Device 71: Auxiliary enhancement information generation device 81: Auxiliary enhancement information encoding device 91: Auxiliary enhanced information decoding device

Claims

1. A moving picture decoding device, characterized in that: have: An image decoding device decodes the encoded data to generate a decoded image; an auxiliary enhancement processing device for performing auxiliary enhancement processing on the decoded image; as well as an auxiliary enhancement information decoding device for decoding a plurality of auxiliary enhancement information for operating the auxiliary enhancement processing device; The auxiliary enhancement information includes auxiliary enhancement information indicating a priority processing order of the plurality of auxiliary enhancement information. The auxiliary enhancement information indicating the priority processing order of the plurality of auxiliary enhancement information includes information indicating that a parameter of a decoded image input to the auxiliary enhancement processing device is changed according to the priority processing order.

2. A motion picture encoding device, characterized in that: have: An image coding device encodes a moving image signal to generate coded data; An auxiliary enhancement information generating device generates a plurality of auxiliary enhancement information; as well as an auxiliary enhancement information encoding device, encoding the plurality of auxiliary enhancement information, The auxiliary enhancement information includes auxiliary enhancement information indicating a priority processing order of the plurality of auxiliary enhancement information. The auxiliary enhancement information indicating the priority processing order of the plurality of auxiliary enhancement information includes information indicating that a parameter of a local decoded image input to the auxiliary enhancement processing device is changed according to the priority processing order.

Images