Signaling processing priority of metadata messaging in video coding

Abstract

To provide methods for determining processing order of metadata messaging, such as supplemental enhancement information (SEI) messaging in MPEG video coding.SOLUTION: A processor receives input SEI messages defining supplemental processing to be applied to decoded video pictures of an input video bitstream, parses the input SEI messages to identify and detect a processing order SEI (PO SEI) message defining a preferred processing order of selected input SEI messages, extracts information determining a total number of the selected input SEI messages with a priority processing order, and extracts identification information and priority information for each of the selected input SEI messages. The priority information includes an integer value determining a process in the preferred processing order. When the priority information of a SEI message m is smaller than the priority information of a SEI message n, the SEI message m is processed before the SEI message n.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] [Related Applications] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 216,318, filed June 29, 2021, which is incorporated herein by reference.

[0002] [Technical field] This document relates generally to images and videos. More particularly, embodiments of the present invention relate to signaling processing priority of metadata messaging (e.g., supplemental enhancement information (SEI) messaging) in image and video sequences. [Background technology]

[0003] As described in Annex D of the AVC and HEVC standards, or H.274 (also known as VSEI) (References [1-4]), Supplemental Enhancement Information (SEI) messages within a coded video bitstream assist in processing related to decoding, display, or other purposes within a video processing pipeline. Despite its widespread use, at least until now, compliant decoders (e.g., AVC, HEVC, or VVC decoders) have not been required to process SEI messaging in order to comply with any of the MPEG video coding standards (e.g., AVC, HEVC, VVC, etc.).

[0004] In a typical video bitstream, multiple SEI messages may coexist. However, existing video coding standards do not define the processing order of such messages. For some SEI messages, there is no specific processing defined by the standard other than the information embedded in the syntax elements. Examples of such messages include SEI messages that define mastering display color volume or content light level information. In addition to syntax elements, some other SEI messages, such as those describing film grain characteristics (FGC) or color remapping information, may also define additional post-processing. For the latter, the final video output may differ depending on the processing order of these SEI messages. As recognized by the inventors, an improved technique for signaling the priority of SEI messages is described herein.

[0005] The term "metadata," as used herein, relates to any supplemental information transmitted as part of a coded bitstream that assists a decoder in rendering a decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and film grain modeling parameters, as described herein. While the examples presented herein may refer to SEI messaging associated with MPEG-based video coding standards, those skilled in the art will understand that the techniques discussed herein are applicable to any such metadata messaging and any audio or video coding format (e.g., AV1, etc.).

[0006] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise noted, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, problems identified with one or more approaches should not be assumed to have been recognized in any prior art under this section unless specifically indicated. [Brief explanation of the drawings]

[0007] Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the accompanying figures, in which like reference symbols represent similar elements and in which:

[0008] [Figure 1A] 1 illustrates an example of a video processing pipeline where metadata includes multiple SEI messages, the order in which they are post-processed can affect the video output. [Figure 1B] 1 illustrates an example of a video processing pipeline where metadata includes multiple SEI messages, the order in which they are post-processed can affect the video output.

[0009] [Figure 2] 1 illustrates an exemplary processing pipeline at a decoder when a Priority Order of Messaging (POM) message is available in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Exemplary embodiments relating to prioritized processing of metadata are described herein. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the present invention. It will be apparent, however, that various embodiments of the present invention may be practiced without some of these specific details. In other instances, well-known structures and devices have not been described in exhaustive detail to avoid unnecessarily occluding, obscuring, or obscuring embodiments of the present invention.

[0011] <Summary> Example embodiments described herein relate to signaling priority of processing of metadata (such as SEI messaging) in video coding. In embodiments, a processor includes: receiving an input video bitstream and a corresponding input metadata message defining supplemental processing to be applied to decoded video pictures of the input video bitstream; parsing the selected input metadata messages to identify a metadata processing order messaging (POM) message that defines a desired processing order for the selected input metadata messages; When the POM message is detected, extracting information to determine a total number of selected input metadata messages having a processing priority; extracting identification and priority information of the selected input metadata message having a processing priority; Apply two or more of the selected input metadata messages to a decoded video picture of the input video bitstream in their determined processing priority.

[0012] In another embodiment, the processor: generating a video bitstream and corresponding metadata messages defining supplemental processing to be applied to decoded video pictures of the video bitstream; Generate a Metadata Processing Order (POM) message that defines the desired processing order of the selected metadata messages, the POM message comprising: information determining the total number of selected input metadata messages having a processing priority; and identification and priority information of the selected input metadata messages having the processing priority; and generate an output stream comprising the video bitstream, the metadata message, and the processing order of the metadata message.

[0013] In another embodiment, a tangible computer medium includes a bitstream, the bitstream comprising: an encoded video picture; an input metadata message defining supplementary processing to be applied to the decoded video picture of the input bitstream; a Metadata Processing Order Messaging (POM) message defining the desired processing order of the selected input metadata messages; and the POM message comprises: information determining the total number of the selected input metadata messages having a processing priority; and identification and priority information of the selected input metadata messages having the processing priority.

[0014] <Example of SEI Messaging Flow Processing> ​For existing video coding standards such as AVC, HEVC, and VVC (References [1-4]) (collectively referred to as MPEG or MPEG Video Standards), Figures 1A and 1B show an exemplary processing pipeline where the SEI messaging includes messaging related to both film grain characteristics (FGC) and color remapping information (CRI) or color transform information (CTI).

[0015] 1A and 1B, given an input video 102, the encoder can generate two sets of SEI messages: a) CTI / CRI SEI metadata (112) related to desired display parameters (e.g., as generated by block 110), and b) film grain characteristics (117) to be added to the decoder to emulate the film grain removed by the encoder (e.g., by block 105) and modeled by the film grain modeling block (115) to improve coding efficiency. Accordingly, a video encoder (120) compresses the output of the film grain remover (107) to generate a compressed bitstream 122. The compressed bitstream 122 and the metadata (112 and 117) are transmitted to the decoder.

[0016] 1A, after decoding (125), the decoded output (127) is passed to a film grain insertion unit 130, which generates and adds film grain noise using information embedded in the FGC metadata (117). The output of the film grain insertion unit (130) is then passed to a display mapping unit (135), which optimally maps the dynamic range and color gamut of the input stream to the target display using information embedded in the CRI / CTI metadata (112). The output is signal 137.

[0017] As shown in FIG. 1B, after decoding (125), compared with FIG. 1A, the processing order of the two sets of metadata is reversed. Here, the decoded output (127) first passes to a display mapping unit (135) that maps the dynamic range and color gamut of the decoded stream to the target display using the information embedded in the CRI / CTI metadata (112). Next, the output of the display mapping unit is passed to a film grain insertion unit 130, which generates and adds film grain noise using the information embedded in the FGC metadata (117) to generate the output video 132.

[0018] Since the order of display mapping and film grain addition is different, it is expected that the video outputs 132 and 137 will be different, especially when the target display is significantly different from the intended display. Therefore, it is considered important to specify the appropriate processing order of the metadata within the bitstream so that the decoder output is as close as possible to the creative intent of the content creator.

[0019] <Example of Signaling Priorities in SEI Messaging> The exemplary embodiments presented here relate to specifying the processing order between multiple metadata sets when multiple SEI messages coexist in video coding standards such as AVC, HEVC, and VVC. In particular, high level syntax (HLS) is proposed to specify such an order. Such high level syntax can be inserted at various levels of the coded bitstream, for example, without limitation, in the video parameter set (VPS), picture parameter set (PPS), sequence parameter set (SPS), auxiliary parameter set (APS), picture header (PH), as individual SEI messaging, etc.

[0020] In an embodiment, such high-level syntax shall include the following information: 1) the SEI messages that may be present in a coded video stream (CVS), 2) whether a processing order is relevant for a particular SEI message, and 3) if so, what is the processing order for a particular SEI message.

[0021] The processing (or priority) order of SEI messages can be specified in a variety of ways, including: 1) absolute order: For example, in one embodiment, messages with lower processing / priority should be processed sooner than messages with higher priority. 2) relative order: in one embodiment, processing order can be specified as a dependency on the processing order of another SEI message. If a coding standard defines that the decoding process requires the processing of certain metadata, then the processing order of such SEI messages must always be specified. If a compliance point is defined for a particular SEI message, then the processing order of such SEI messages must also be specified.

[0022] Table 1 shows an example of specifying processing order among SEI messages. In this example, a proposed new SEI message, the "SEI messaging processing order," allows specifying which SEI messages are part of a CVS where processing order is important, and their processing priority. The SEI messages of interest are identified through their unique payload type (see references [1-4]). [Table 1] SEI processing order SEI messages [Table 1] po_num_sei_msg_types specifies the number of types of SEI messages that provide information to the SEI processing priority SEI message. po_sei_payload_type[i] indicates the payload type value of the i-th type of SEI message whose information is provided in the SEI processing order SEI message. If m is not equal to n, the values ​​of po_sei_payload_type[m] and po_sei_payload_type[n] are not the same. po_sei_order[i] provides the absolute order of SEI messages with payloadType equal to po_sei_payload_type[i]. If m is not equal to n, the values ​​of po_sei_order[m] and po_sei_order[n] are not identical. If the value of po_sei_order[m] is less than the value of po_sei_order[n], SEI messages with po_sei_payload_type[m] are processed before SEI messages with po_sei_payload_type[n].

[0023] In Table 1, instead of the absolute po_sei_order[i] parameter, embodiments may also describe a relative order, such as po_sei_rel_order[i][j], where the order of SEI message i is defined relative to the order of SEI message j. For example, po_sei_rel_order[3][1]=2 may define that the priority of SEI message "3" is the priority of SEI message "1" plus (or alternatively, minus) 2. In such a scenario, it may be assumed that all SEI messages with processing priority start with a default value (e.g., 1) and are adjusted accordingly.

[0024] In embodiments, Table 1 can also be simplified by removing the po_sei_order[i] parameter. In such a scenario, the encoder already lists the po_sei_payload_type[i] values ​​in an implicit, predetermined order of priority (e.g., lowest to highest or highest to lowest). Such an approach limits the possible parallelization of SEI messaging-related tasks, since priority is enforced even when there is no such requirement (e.g., SEI messages "2" and "4" can have the same priority but are listed in order). The implicit ordering is done at the start of a new CVS, where all existing SEI messages are listed.

[0025] The SEI manifest SEI message for HEVC and VVC conveys information about SEI messages that are expected (i.e., likely) to be present or absent in the bitstream, but does not indicate any priority among these SEI messages. Table 2 shows another example of extending the SEI manifest SEI to include priority information. Additional syntax to the existing SEI manifest message is shown in italics. [Table 2] Example 1 of SEI Manifest SEI Message Extension [Table 2] manifest_sei_po_flag[i] equal to 1 specifies that SEI messages with payloadType equal to manifest_sei_payload_type[i] request processing order information. manifest_sei_po_flag[i] equal to 0 specifies that SEI messages with payloadType equal to manifest_sei_payload_type[i] do not request processing order information. manifest_sei_order[i] provides the absolute order of SEI messages with payloadType equal to manifest_sei_payload_type[i]. If m is not equal to n, the values ​​of manifest_sei_order[m] and manifest_sei_order[n] are not identical. If the value of manifest_sei_order[m] is less than the value of manifest_sei_order[n], SEI messages with manifest_sei_payload_type[m] are processed before SEI messages with manifest_sei_payload_type[n]. NOTE: As mentioned above, if the processing order is implicitly defined (e.g., the order in which SEI messages are listed in manifest_sei_po_flag[i]), the syntax element manifest_sei_order[i] is not required and can be removed.

[0026] Table 3 shows another approach to extending the SEI manifest SEI that supports backward compatibility with legacy decoders that already support older versions of the SEI manifest message. [Table 3] Example 2 of SEI Manifest SEI message extension [Table 3]

[0027] From HEVC, more_data_in_payload() is specified as follows: If byte_aligned() is equal to TRUE and the current position of the sei_payload() syntax structure is 8*payloadSize bits from the beginning of the sei_payload() syntax structure, the return value of more_data_in_payload() is equal to FALSE. - Otherwise, the return value of more_data_in_payload() is TRUE.

[0028] From HEVC, payload_extension_present() is specified as follows: - If the current position of the sei_payload() syntax structure is not the position of the last (lowest, rightmost) bit equal to 1, but is less than 8*payloadSize bits from the beginning of the syntax structure (i.e., the position of the payload_bit_equal_to_one syntax element), the return value of payload_extension_present() is equal to TRUE. - Otherwise, the return value of payload_extension_present() is equal to FALSE.

[0029] In Table 3, existing decoders that implement SEI manifest messaging can continue to do so as before by ignoring the SEI processing order syntax, while decoders implemented after the adoption of the new syntax can take advantage of the updated syntax and associated processing order information.

[0030] As discussed in relation to Table 1, the processing order of the SEI messages in Tables 2 and 3 may also be implied or defined relative to the processing order of other SEI messages.

[0031] In an embodiment, Table 1 and its semantics may be modified as shown in the description of Table 4. In this embodiment, the semantics rely on the syntax element MaxNumPayloadTypes to address the issue that the number of possible payload types varies by coding standard (e.g., as of this draft, AVC has 77 SEI messages, HEVC has 67 SEI messages, and VVC has 39 SEI messages), and furthermore, the actual number may increase in future versions of the standard, to discourage signaling of more payload types than may exist.

[0032] Alternatively (or additionally), the actual valid payloadTypes for each standard can also be specified in a specific list (e.g., the list SeiAssociatedSeiList). For example, the following paragraph can be added to the VVC or VSEI standard (references [3-4]): "SEI Message Processing Order SEI Message Syntax Usage SEI Message Processing Order For purposes of interpreting SEI messages, the list SeiAssociatedSeiList is set to consist of payloadType values ​​0, 1, 3, 4, 5, 19, 45, 47, 129, 130, 132, 133, 137, 142, 144, 145, 147 through 150, 153 through 156, 165, 168, 177, 179, 180, 200 through 208, inclusive." In another embodiment, this list can be part of the SEI Processing order SEI message or other header information, and is modified by the encoder as needed. [Table 4] SEI processing order SEI messages [Table 4] po_num_sei_payload_types specifies the maximum number of payloadType values ​​for which the syntax elements po_sei_payload_type[i] and po_sei_processing_order[i] exist. The value of po_num_sei_payload_types must be less than or equal to MaxNumPayloadTypes, which defines the maximum number of payloadType values. Note: If the syntax element SeiAssociateSeiList is added, the above definition can be replaced by the following: po_num_sei_payload_types specifies the maximum number of payloadType values ​​for which the syntax elements po_sei_payload_type[i] and po_sei_processing_order[i] exist, as specified by the encoder (eg, via the list SeiAssociateSeiList). po_sei_payload_type[i] specifies the value of payloadType of the i-th SEI message. If m is not equal to n, the value of po_sei_payload_type[m] shall not be equal to the value of po_sei_payload_type[n]. Note: If the syntax element SeiAssociateSeiList is added, add the following: The value of po_sei_payload_type[i] shall be equal to the payloadType value in SeiAssociateSeiList. po_sei_order[i] indicates the desired order of processing SEI messages whose payloadType is equal to po_sei_payload_type[i]. If m is not equal to n, the value of po_sei_order[m] shall not be equal to the value of po_sei_order[n]. If po_sei_order[m] is greater than 0 and less than po_sei_order[n], it indicates that SEI messages with payloadType equal to po_sei_payload_type[m] should be processed before SEI messages with payloadType equal to po_sei_payload_type[n]. If po_sei_order[i] is equal to 0, it specifies that the desired processing order of SEI messages with payloadType equal to po_sei_payload_type[i] is determined by external means not specified in this specification.

[0033] FIG. 2 illustrates an exemplary processing pipeline according to an embodiment when SEI processing order of messaging (POM) messages are available. Starting at step 210, the decoder can search the SEI payloadType of the incoming SEI message to detect whether an SEI POM message is present. If no such message is present, processing of the SEI messages (215) continues without considering their order. If an SEI POM message is present, then in step 220, the decoder reads the number of selected SEI messages for which a processing priority has been defined. Next, in step 225, the decoder reads, for each such message, its identity (e.g., payloadType) and its priority, typically an integer value with a predetermined ranked importance (e.g., lower or higher numbers have higher priority). In an embodiment, the priority may be implied by the order in which the selected SEI messages are listed. Finally, in step 230, the decoder processes these selected SEI messages in the appropriate order of priority. References Each of the references listed herein is incorporated herein by reference in its entirety. [1] AdvancedV Video Coding, Rec. ITU-T H.264, May 2019, ITU [2] High Efficiency Video Coding, Rec. ITU-T H.265, November 2019, ITU [3] Versatile Video Coding, Rec. ITU-T H.266, August 2020, ITU [4] Versatile supplemental enhancement information messages for coded video bitstreams, Rec. ITU-T H.274, Aug. 2020, ITU

[0034] Exemplary Computer System Implementation Embodiments of the present invention may be implemented by a computer system, a system configured in electronic circuits and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA) or other configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or perform instructions related to signaling metadata processing priority messaging in video coding as described herein. The computer and / or IC may calculate any of various parameters or values ​​related to signaling metadata processing priority messaging in video coding as described herein. Image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0035] Particular implementations of the present invention include computer processors executing software instructions that cause the processor to perform the methods of the present invention. For example, one or more processors in a display, encoder, set-top box, transcoder, etc. may implement the methods related to signaling metadata processing priority messaging in video coding described above by executing software instructions in a program memory accessible to the processor. Embodiments of the present invention may be provided in the form of a program product. The program product may include any non-transitory tangible medium that carries a set of computer-readable signals that include instructions that, when executed by a data processor, cause the data processor to perform the methods of the present invention. A program product according to the present invention may be in any of a variety of non-transitory tangible forms. The program product may include physical media such as, for example, magnetic data storage media including floppy disks, hard disk drives, optical data storage media including CD-ROMs, DVDs, ROMs, electronic data storage media including flash RAM, etc. The computer-readable signals on the program product may be optically compressed or encrypted. Although components (e.g., software modules, processors, components, devices, circuits, etc.) have been referred to above, unless otherwise indicated, references to those components (including references to "means") should be interpreted to include equivalents of those components, any components that perform the functions of the described components (e.g., are functionally equivalent), and components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated exemplary embodiments of the present invention.

[0036] <Equivalents, Extensions, Alternatives and Miscellaneous> Exemplary embodiments relating to signaling processing priority of metadata messaging in video coding have been described. In the foregoing specification, embodiments of the present invention have been described with reference to numerous specific details that may vary from implementation to implementation. Accordingly, the sole and exclusive indication of what the invention is, and what Applicant intends to be the invention, is set forth in the claims as issued in particular form hereby, including any subsequent amendments. Any definitions expressly set forth herein for terms contained in such claims shall control the meaning of such terms as used in the claims. Accordingly, any limitation, element, feature, advantage, or attribute not expressly recited in the claims should not in any way limit the scope of the claims. The specification and drawings are, therefore, to be considered illustrative, rather than restrictive.

Claims

1. 1. A method for determining priority handling among a plurality of supplemental enhancement information (SEI) messages, the method comprising: receiving an input video bitstream and a corresponding input SEI message defining supplemental processing to be applied to decoded video pictures of the input video bitstream; parsing the selected incoming SEI message to identify a processing order (PO) SEI (PO SEI) message that defines a desired processing order for the selected incoming SEI message; Upon detecting the PO SEI message, extracting information that determines a total number of selected input SEI messages that have a processing priority; extracting, for each of the selected input SEI messages, identification information and order information, the order information including an integer value determining processing in the desired processing order, and wherein if the order information of SEI message m is less than the order information of SEI message n, then SEI message m is processed before SEI message n; A method comprising:

2. The step of extracting identification information and ranking information of the selected input SEI message having a processing priority includes:

2. The method of claim 1, comprising reading the list of total numbers of the selected input SEI messages having an implicit descending processing priority or an implicit ascending processing priority.

3. The method of claim 1 , wherein the identification information includes SEI payload type information.

4. 1. A method for signaling priority handling among multiple SEI messages, comprising: specifying an order in which the plurality of SEI messages are to be processed; transmitting the coded bitstream; Including, The step of specifying the processing order includes: generating a video bitstream and a corresponding SEI message defining supplemental processing to be applied to decoded video pictures of the video bitstream; generating a processing order (PO) SEI (PO SEI) message defining a desired processing order of the selected SEI message, the PO SEI message comprising: Information determining a total number of selected input SEI messages having a desired processing priority; For each of the selected input SEI messages, identification information and ranking information; the order information includes an integer value that determines processing in the desired processing order, and if the order information of SEI message m is less than the order information of SEI message n, then the SEI message m is processed before the SEI message n; generating the coded bitstream including the video bitstream, the SEI message, and the PO SEI message; A method comprising:

5. The method of claim 4 , wherein the identification information includes SEI payload type information.

6. 1. A method for specifying priority handling among a plurality of SEI messages, the method comprising: generating a video bitstream and a corresponding SEI message defining supplemental processing to be applied to decoded video pictures of the video bitstream; generating a processing order (PO) SEI (PO SEI) message defining a desired processing order of the selected SEI message, the PO SEI message comprising: information for determining a total number of selected input SEI messages having a processing priority; For each of the selected input SEI messages, identification information and ranking information; the order information includes an integer value that determines processing in the desired processing order, and if the order information of SEI message m is less than the order information of SEI message n, then the SEI message m is processed before the SEI message n; generating an output stream including the video bitstream, the SEI message, and the PO SEI message; A method comprising:

7. The method of claim 6 , wherein the identification information includes SEI payload type information.

Images