Signaling of metadata messaging processing priority in video coding

Abstract

Signals the processing priority of Supplemental Extended Information (SEI) messaging. [Solution] The method receives an input video bitstream and corresponding input SEI messages that define supplemental processing to be applied to the decoded video picture of the input video bitstream. The input SEI messages are parsed to identify processing priority (PO) SEI messages that define the desired processing order of selected input SEI messages. Upon detecting a PO SEI message, information is extracted to determine the total number of selected input SEI messages that have processing priority. For each selected input SEI message, identification information and ranking information are extracted. The ranking information includes an integer value that determines the processing in the desired processing order. If the ranking information of SEI message m is smaller than the ranking information of SEI message n, SEI message m is processed before SEI message n.

Description

Technical Field

[0001] [Related Applications] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 216,318, filed Jun. 29, 2021, which is incorporated herein by reference.

[0002] [Technical Field] This specification generally relates to images and videos. More specifically, embodiments of the present invention relate to signaling of processing priorities for metadata messaging (e.g., supplemental enhancement information (SEI) messaging) in image and video sequences.

Background Art

[0003] As described in Annex D of the AVC and HEVC standards, or H.264 (also called VSEI) (References [1-4]), supplemental enhancement information (SEI) messages within a coded video bitstream assist in decoding, display, or other processing related to purposes within the video processing pipeline. Despite their widespread use, at least until now, compliant decoders (e.g., AVC, HEVC, or VVC decoders) have not been required to process SEI messaging to comply with any of the MPEG video coding standards (such as AVC, HEVC, VVC).

[0004] In a typical video bitstream, multiple SEI messages may coexist. However, existing video coding standards do not define the processing order of such messaging. Some SEI messages have no specific processing defined by the standard other than the information embedded in the syntax element. Examples of such messages include SEI messaging that defines mastering display color volume or content light level information. In addition to the syntax element, some other SEI messages, such as those describing film grain characteristics (FGC) or color remapping information, may also have additional post-processing defined. For the latter, the final video output may differ depending on the processing order of these SEI messages. An improved technique for signaling the priority of SEI messaging, as recognized by the inventors, is described here.

[0005] The term “metadata” in this specification refers to any supplemental information transmitted as part of a coded bitstream to assist the decoder in rendering the decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and film grain modeling parameters, as described in this specification. While the examples presented herein may refer to SEI messaging related to MPEG-based video coding standards, those skilled in the art will understand that the techniques discussed herein are applicable to any such metadata messaging and any audio or video coding format (e.g., AV1, etc.).

[0006] The approaches described in this chapter are potential approaches to be pursued, but not necessarily approaches that have been previously devised or pursued. Therefore, unless otherwise specified, none of the approaches described in this chapter should be considered prior art simply because they are included in this chapter. Similarly, any problems identified with respect to one or more approaches should not be assumed to have been recognized in any prior art based on this chapter, unless otherwise specified. [Brief explanation of the drawing]

[0007] Embodiments of the present invention are described by example, not limiting them, and similar reference numerals in the accompanying figures represent similar elements.

[0008] [Figure 1A] This example shows a video processing pipeline where metadata contains multiple SEI messages, and the order in which they are post-processed may affect the video output. [Figure 1B] This example shows a video processing pipeline where metadata contains multiple SEI messages, and the order in which they are post-processed may affect the video output.

[0009] [Figure 2] This describes an exemplary processing pipeline in a decoder when a Priority Order of Messaging (POM) message is available according to an embodiment of the present invention. [Modes for carrying out the invention]

[0010] Exemplary embodiments relating to metadata priority processing are described herein. In the following detailed description, numerous specific details are described for illustrative purposes to provide a complete understanding of the various embodiments of the invention. However, it will be apparent that various embodiments of the invention may be performed without some of these specific details. In other examples, well-known structures and apparatus are not described in thorough detail to avoid unnecessarily occluding, obscuring, or obscuring embodiments of the invention.

[0011] <Summary> The exemplary embodiments described herein relate to signaling the prioritization of metadata processing (such as SEI messaging) in video coding. In the embodiments, the processor, The system receives an input video bitstream and a corresponding input metadata message that defines supplementary processing to be applied to the decoded video picture of the input video bitstream. In order to identify metadata processing priority messaging (POM) messages that define the desired processing order of selected input metadata messages, the input metadata messages are parsed, When the aforementioned POM message is detected, Extract information to determine the total number of selected input metadata messages that have a processing priority. Identify the selected input metadata messages that have a processing priority and extract priority information. Two or more of the selected input metadata messages are applied to the decoded video picture of the input video bitstream in the order of their determined processing priority.

[0012] In another embodiment, the processor is A video bitstream and a corresponding metadata message that defines supplementary processing to be applied to the decoded video picture of the video bitstream are generated. Generate a metadata processing order (POM) message that defines the desired processing order of the selected metadata messages, and the POM message includes information for determining the total number of selected input metadata messages having a processing priority, and identification and priority information of the selected input metadata messages having the processing priority, and including Generate an output stream including the video bitstream, the metadata message, and the processing order of the metadata message.

[0013] In another embodiment, the tangible computer medium includes a bitstream, and the bitstream includes encoded video pictures, and an input metadata message that defines supplementary processing applied to the decoded video pictures of the input bitstream, and a metadata processing order messaging (POM) message that defines the desired processing order of the selected input metadata messages, and including The POM message includes information for determining the total number of the selected input metadata messages having a processing priority, and identification and priority information of the selected input metadata messages having the processing priority, and including.

[0014] <Example of SEI messaging flow processing> With respect to existing video coding standards such as AVC, HEVC, and VVC (references [1-4]) (collectively referred to as MPEG or MPEG video standards), Figures 1A and 1B show an exemplary processing pipeline when SEI messaging includes messaging related to both film grain characteristics (FGC) and color remapping information (CRI) or color transform information (CTI).

[0015] As shown in Figures 1A and 1B, given an input video 102, the encoder can generate two sets of SEI messages: a) CTI / CRI SEI metadata (112) related to desired display parameters (e.g., generated by block 110), and b) film grain characteristics (117) to be added to the decoder to emulate film grain that has been removed by the encoder (e.g., by block 105) and modeled by the film grain modeling block (115) in order to improve encoding efficiency. Thus, the video encoder (120) compresses the output of the film grain remover (107) to generate a compressed bitstream 122. The compressed bitstream 122 and metadata (112 and 117) are sent to the decoder.

[0016] As shown in Figure 1A, after decoding (125), the decoded output (127) is passed to a film grain insertion unit 130, which generates and adds film grain noise using the information embedded in the FGC metadata (117). Next, the output of the film grain insertion unit (130) is passed to a display mapping unit (135), which uses the information embedded in the CRI / CTI metadata (112) to optimally map the dynamic range and color gamut of the input stream to the target display. The output is signal 137.

[0017] As shown in FIG. 1B, after decoding (125), compared with FIG. 1A, the processing order of the two sets of metadata is reversed. Here, the decoded output (127) first passes to a display mapping unit (135) that maps the dynamic range and color gamut of the decoded stream to the target display using the information embedded in the CRI / CTI metadata (112). Next, the output of the display mapping unit is passed to a film grain insertion unit 130, which generates and adds film grain noise using the information embedded in the FGC metadata (117) to generate an output video 132.

[0018] Since the order of display mapping and film grain addition is different, the video outputs 132 and 137 are expected to be different, especially when the target display is significantly different from the intended display. Therefore, it is considered important to specify the appropriate processing order of the metadata within the bitstream so that the decoder output is as close as possible to the creative intent of the content creator.

[0019] <Example of Signaling Priority in SEI Messaging> The exemplary embodiments presented herein relate to specifying the processing order between multiple metadata sets when multiple SEI messages coexist in video coding standards such as AVC, HEVC, and VVC. In particular, high level syntax (HLS) is proposed to specify such an order. Such high level syntax can be inserted at various levels of the coded bitstream, for example, without limitation, in the video parameter set (VPS), picture parameter set (PPS), sequence parameter set (SPS), auxiliary parameter set (APS), picture header (PH), as individual SEI messaging, etc.

[0020] In the embodiment, such high-level syntax shall include the following information: 1) SEI messages that may be present in the coded video stream (CVS), 2) whether the processing order is related to a particular SEI message, and 3) if so, what the processing order of a particular SEI message is.

[0021] The processing (or priority) order of SEI messages can be specified in various ways, including: 1) Absolute order: For example, in one embodiment, messages with lower processing / priority must be processed earlier than messages with higher priority. 2) Relative order: In one embodiment, the processing order can be specified as a dependency on the processing order of another SEI message. If the coding standard defines that processing of certain metadata is mandatory in the decryption process, the processing order of such SEI messages must always be specified. If a compliance point is defined for a particular SEI message, the processing order of such SEI messages must also be specified.

[0022] Table 1 shows an example of specifying the processing order between SEI messages. In this example, the proposed new SEI message, the "SEI messaging processing order" message, can specify which SEI messages are part of the CVS where processing order is important, and their processing priority. The target SEI messages are identified through their unique payload types (see references [1-4]). [Table 1] SEI Processing Order SEI Message [Table 1] po_num_sei_msg_types specifies the number of SEI message types that provide information to the SEI processing order SEI message. po_sei_payload_type[i] indicates the payload type (payloadType) value of the i-th type of SEI message for which information is provided in the SEI processing order SEI message. If m is not equal to n, the values ​​of po_sei_payload_type[m] and po_sei_payload_type[n] are not identical. po_sei_order[i] provides the absolute order of SEI messages with a payloadType equal to po_sei_payload_type[i]. If m is not equal to n, the values ​​of po_sei_order[m] and po_sei_order[n] are not identical. If the value of po_sei_order[m] is less than the value of po_sei_order[n], SEI messages with po_sei_payload_type[m] are processed before SEI messages with po_sei_payload_type[n].

[0023] In Table 1, instead of the absolute po_sei_order[i] parameter, in embodiments, a relative order can also be described, such as po_sei_rel_order[i][j], where the order of SEI message i is defined relative to the order of SEI message j. For example, po_sei_rel_order[3][1]=2 can define that the priority of SEI message "3" is the priority of SEI message "1" + (or alternatively, "-") 2. In such a scenario, it can be assumed that all SEI messages with processing priorities start with a default value (e.g., 1) and are adjusted accordingly.

[0024] In some embodiments, Table 1 can also be simplified by removing the po_sei_order[i] parameter. In such scenarios, the encoder has already listed the po_sei_payload_type[i] values ​​with an implicit and predetermined priority (e.g., lowest to highest or highest to lowest). This approach limits the possible parallelization of SEI messaging-related tasks because priority is enforced even when there is no such requirement (e.g., SEI messages "2" and "4" can have the same priority but are listed in order). The implicit order is determined at the start of a new CVS where all existing SEI messages are listed.

[0025] The SEI manifest message in HEVC and VVC conveys information about SEI messages that are present or not expected (i.e., likely) in the bitstream. However, it does not indicate the priority between these SEI messages. Table 2 shows another example of extending the SEI manifest to include priority information. Additional syntax for existing SEI manifest messages is shown in italics. [Table 2] Example 1 of SEI Manifest SEI Message Extension [Table 2] If manifest_sei_po_flag[i] is equal to 1, it indicates that an SEI message with a payloadType equal to manifest_sei_payload_type[i] requests processing priority information. If manifest_sei_po_flag[i] is equal to 0, it indicates that an SEI message with a payloadType equal to manifest_sei_payload_type[i] does not request processing priority information. manifest_sei_order[i] provides the absolute order of SEI messages with a payloadType equal to manifest_sei_payload_type[i]. If m is not equal to n, the values ​​of manifest_sei_order[m] and manifest_sei_order[n] are not identical. If the value of manifest_sei_order[m] is less than the value of manifest_sei_order[n], SEI messages with manifest_sei_payload_type[m] are processed before SEI messages with manifest_sei_payload_type[n]. Note: As mentioned above, if the processing order is implicitly defined (for example, if the SEI messages are listed in order within manifest_sei_po_flag[i]), the syntax element manifest_sei_order[i] is unnecessary and can be deleted.

[0026] Table 3 shows another approach to extending SEI Manifest SEI to support backward compatibility with legacy decoders that already support older versions of SEI Manifest messages. [Table 3] Example 2 of extending SEI manifest SEI messages [Table 3]

[0027] From HEVC, more_data_in_payload() is specified as follows: If -byte_aligned() is equal to TRUE and the current position of the sei_payload() syntax structure is 8*payloadSize bits from the beginning of the sei_payload() syntax structure, then the return value of more_data_in_payload() is equal to FALSE. - Otherwise, the return value of more_data_in_payload() will be TRUE.

[0028] From HEVC, payload_extension_present() is specified as follows: -If the current position of the sei_payload() syntax structure is less than 8*payloadSize bits from the beginning of the syntax structure (i.e., the position of the payload_bit_equal_to_one syntax element), rather than the position of the last (least significant, rightmost) bit equal to 1, then the return value of payload_extension_present() is equal to TRUE. - Otherwise, the return value of payload_extension_present() will be equal to FALSE.

[0029] Table 3 shows that existing decoders implementing SEI manifest messaging can continue their implementation as before by ignoring the SEI processing order syntax. On the other hand, decoders implemented after the adoption of the new syntax can utilize the updated syntax and associated processing order information.

[0030] As discussed in relation to Table 1, the processing order of SEI messages in Tables 2 and 3 can also be implicit or defined in relation to the processing order of other SEI messages.

[0031] In this embodiment, Table 1 and its semantics can be modified as shown in the description of Table 4. In this embodiment, the semantics rely on the syntax element MaxNumPayloadTypes to address the issue that the number of possible payload types differs depending on the coding standard (for example, as of this draft, AVC has 77 SEI messages, HEVC has 67 SEI messages, and VVC has 39 SEI messages), and furthermore, the actual number may increase in future versions of the standard, suppressing the signaling of more payload types than may exist.

[0032] Alternatively (or additionally), the actual valid payloadType for each standard can also be specified in a specific list (e.g., the list SeiAssociatedSeiList). For example, the following paragraph can be added to the VVC or VSEI standard (references [3-4]): SEI message processing order, using SEI message syntax SEI Message Processing Order: For the purpose of interpreting SEI messages, the list SeiAssociatedSeiList is set to consist of payloadType values ​​0, 1, 3, 4, 5, 19, 45, 47, 129, 130, 132, 133, 137, 142, 144, 145, 147 to 150, 153 to 156, 165, 168, 177, 179, 180, 200 to 208, including both ends. In another embodiment, this list may be part of an SEI Processing order SEI message or other header information, and may be modified by the encoder as needed. [Table 4] SEI Processing Order SEI Message [Table 4] po_num_sei_payload_types specifies the maximum payloadType value for which the syntax elements po_sei_payload_type[i] and po_sei_processing_order[i] exist. The value of po_num_sei_payload_types must be less than or equal to MaxNumPayloadTypes, which defines the maximum payloadType value. Note: If the syntax element SeiAssociateSeiList is added, the above definition can be replaced as follows: po_num_sei_payload_types specifies the maximum payloadType value for which the syntax elements po_sei_payload_type[i] and po_sei_processing_order[i] exist, as specified by the encoder (for example, via the list SeiAssociateSeiList). po_sei_payload_type[i] specifies the payloadType value of the i-th SEI message. If m is not equal to n, the value of po_sei_payload_type[m] is not equal to the value of po_sei_payload_type[n]. Note: If the syntax element SeiAssociateSeiList is added, add the following: The value of po_sei_payload_type[i] should be equal to the payloadType value in SeiAssociateSeiList. po_sei_order[i] indicates the preferred order for processing SEI messages whose payloadType is equal to po_sei_payload_type[i]. If m is not equal to n, the value of po_sei_order[m] is not equal to the value of po_sei_order[n]. If po_sei_order[m] is greater than 0 and less than po_sei_order[n], it indicates that SEI messages with a payloadType equal to po_sei_payload_type[m] should be processed before SEI messages with a payloadType equal to po_sei_payload_type[n]. If po_sei_order[i] is equal to 0, it indicates that the preferred processing order for SEI messages with a payloadType equal to po_sei_payload_type[i] is determined by an external means not specified in this specification.

[0033] Figure 2 shows an exemplary processing pipeline according to an embodiment when SEI processing order of messaging (POM) messages are available. Starting from step 210, the decoder can find out if SEI POM messages exist by looking up the SEIpayloadType of the incoming SEI messages. If no such messages exist, the processing of the SEI messaging (215) continues without considering their order. If SEI POM messages exist, in step 220 the decoder reads the number of selected SEI messages for which processing priority is defined. Next, in step 225 the decoder reads for each such message its identity (e.g., payloadType) and its priority, typically an integer value having a predetermined ranked importance (e.g., smaller numbers or larger numbers have higher priority). In an embodiment, the priority may be implied by the order in which the selected SEI messages are listed. Finally, in step 230 the decoder processes these selected SEI messages in the appropriate priority order. References Each of the references listed here is incorporated here in its entirety by reference. [1] AdvancedV Video Coding, Rec. ITU-T H.264, May 2019, ITU [2] High Efficiency Video Coding, Rec. ITU-T H.265, November 2019, ITU [3] Versatile Video Coding, Rec. ITU-T H.266, August 2020, ITU [4] Versatile supplemental enhancement information messages for coded video bitstreams, Rec. ITU-T H.274, Aug. 2020, ITU

[0034] <Example of a computer system implementation> Embodiments of the present invention may be implemented by computer systems, systems configured within electronic circuits and components, integrated circuit (IC) devices such as microcontrollers, FPGAs (field programmable gate arrays), or other configurable or programmable logic devices (PLDs), individual time or digital signal processors (DSPs), application-specific ICs (ASICs), and / or devices including one or more such systems, devices, or components. Computers and / or ICs may execute, control, or perform instructions related to signaling metadata processing priority messaging in video coding as described in this specification. Computers and / or ICs may calculate any of various parameters or values ​​related to signaling metadata processing priority messaging in video coding as described in this specification. Embodiments of images and videos may be implemented in hardware, software, firmware, and various combinations thereof.

[0035] A particular implementation of the present invention includes a computer processor that executes software instructions causing the processor to perform the method of the present invention. For example, one or more processors in a display, encoder, set-top box, transcoder, etc., may perform a method related to signaling metadata processing priority messaging in the video coding described above by executing software instructions in the processor's accessible program memory. Embodiments of the present invention may be provided in the form of a program product. The program product may include any non-temporary tangible medium that carries a set of computer-readable signals containing instructions that, when executed by a data processor, cause the data processor to perform the method of the present invention. The program product according to the present invention may be any of the various non-temporary tangible forms. The program product may include a physical medium such as, for example, a magnetic data storage medium including a floppy disk, an optical data storage medium including a hard disk drive, a CD-ROM, a DVD, an electronic data storage medium including ROM, flash RAM, etc. The computer-readable signals on the program product may be optically compressed or encrypted. Although components (e.g., software modules, processors, parts, devices, circuits, etc.) have been mentioned above, unless otherwise specified, any mention of such components (including any mention of “means”) should be interpreted as including equivalents of such components, any components that perform the function of the described components (e.g., functionally equivalent), and components that are not structurally equivalent to the structures of the disclosures that perform the function in the illustrated exemplary embodiments of the present invention.

[0036] <Equivalents, Extensions, Alternatives, and Miscellaneous> Exemplary embodiments relating to signaling the processing priority of metadata messaging in video coding have been described. In the above specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Accordingly, the sole and exclusive indication of what the invention is, and what the applicant intends to be the invention, is stated in the claims, which are issued in a specific form, including any subsequent amendments. Any definitions of terms included in such claims expressly stated in this specification should govern the meaning of such terms as used in the claims. Accordingly, any limitations, elements, features, advantages, or attributes not expressly stated in the claims should not in any way limit the scope of the claims. The specification and drawings should therefore be considered descriptive, not restrictive.

Claims

1. A method for determining priority processing among multiple Supplemental Extended Information (SEI) messages, wherein the method is: The steps include receiving an input video bitstream and a corresponding input SEI message that defines supplementary processing to be applied to the decoded video picture of the input video bitstream, The steps include: parsing the input SEI messages to identify priority-of-process (PO) SEI messages (PO SEI) that define the desired processing order of the selected input SEI messages; When the aforementioned PO SEI message is detected, A step of extracting information to determine the total number of selected input SEI messages that have a processing priority, A step of extracting identification information and ranking information for each of the selected input SEI messages, wherein the ranking information includes an integer value that determines the processing in the desired processing order, and if the ranking information of SEI message m is smaller than the ranking information of SEI message n, then SEI message m is processed before SEI message n. Includes, A method in which the integer value represents the priority of a selected input message, either as an absolute processing order among the selected input SEI messages, or as a relative processing order with respect to a second selected input SEI message among the selected input SEI messages.

2. The method according to claim 1, further comprising the step of applying two or more of the selected input SEI messages to the decoded video picture of the input video bitstream in the order of their determined processing priority.

3. The step of extracting the identification information and ranking information of the selected input SEI messages having a processing priority is: The method according to claim 1, comprising the step of reading a list of the total number of selected input SEI messages having an implicit descending or implicit ascending processing priority.

4. The method according to claim 1, wherein the identification information includes SEI payload type information.

5. A method for signaling priority processing between multiple SEI messages, The steps include specifying the order in which to process the multiple SEI messages, The steps include sending the coded bitstream, Includes, The step of specifying the order of processing is, The steps include generating a video bitstream and a corresponding SEI message that defines supplementary processing to be applied to the decoded video picture of the video bitstream, A step of generating a processing priority (PO) SEI (PO SEI) message that defines the desired processing order of selected SEI messages, wherein the PO SEI message is: Information that determines the total number of selected input SEI messages having a desirable processing order, For each of the selected input SEI messages, identification information and ranking information are provided. The steps include, wherein the ranking information includes an integer value that determines the processing in the desired processing order, and if the ranking information of SEI message m is smaller than the ranking information of SEI message n, then SEI message m is processed before SEI message n, and the integer value represents the priority of the selected input messages, either in absolute processing order among the selected input SEI messages or in relative processing order with respect to a second selected input SEI message among the selected input SEI messages, The steps of generating the coded bitstream which includes the video bitstream, the SEI message, and the PO SEI message, Methods that include...

6. The method according to claim 5, wherein the identification information includes SEI payload type information.

7. A method for specifying priority processing among multiple SEI messages, wherein the method is: The steps include generating a video bitstream and a corresponding SEI message that defines supplementary processing to be applied to the decoded video picture of the video bitstream, A step of generating a processing priority (PO) SEI (PO SEI) message that defines the desired processing order of selected SEI messages, wherein the PO SEI message is: Information that determines the total number of selected input SEI messages with processing priority, For each of the selected input SEI messages, identification information and ranking information are provided. The steps include, wherein the ranking information includes an integer value that determines the processing in the desired processing order, and if the ranking information of SEI message m is smaller than the ranking information of SEI message n, then SEI message m is processed before SEI message n, and the integer value represents the priority of the selected input messages, either in absolute processing order among the selected input SEI messages or in relative processing order with respect to a second selected input SEI message among the selected input SEI messages, A step of generating an output stream including the video bitstream, the SEI message, and the PO SEI message, A method that includes this.

8. The method according to claim 7, wherein the identification information includes SEI payload type information.

Images