Signaling of priority processing order for metadata messaging in video coding

Abstract

To provide methods for determining processing order of metadata messaging, such as supplemental enhancement information (SEI) messaging in MPEG video coding.SOLUTION: A processor receives input SEI messages defining supplemental processing to be applied to decoded video pictures of an input video bitstream, parses the input SEI messages to identify and detect a processing order SEI (PO SEI) message defining a preferred processing order of selected input SEI messages, extracts information determining a total number of the selected input SEI messages with a priority processing order, and extracts identification information and priority information for each of the selected input SEI messages. The priority information includes an integer value determining a process in the preferred processing order. When the priority information of a SEI message m is smaller than the priority information of a SEI message n, the SEI message m is processed before the SEI message n.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] [Related Applications] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 216,318, filed Jun. 29, 2021, which is incorporated herein by reference.

[0002] [Technical Field] This specification generally relates to images and videos. More particularly, embodiments of the invention relate to signaling of processing priorities for metadata messaging (e.g., supplemental enhancement information (SEI) messaging) in image and video sequences.

Background Art

[0003] As described in Annex D of the AVC and HEVC standards, or H.274 (also called VSEI) (References [1-4]), supplemental enhancement information (SEI) messages within a coded video bitstream assist in decoding, display, or other processing related to purposes within the video processing pipeline. Despite their widespread use, at least until now, compliant decoders (e.g., AVC, HEVC, or VVC decoders) have not needed to process SEI messaging in order to comply with any of the MPEG video coding standards (such as AVC, HEVC, VVC).

[0004] In a typical video bitstream, multiple SEI messages may coexist. However, existing video coding standards do not define the processing order of such messaging. For some SEI messages, there is no specific processing defined by the standard other than the information embedded in the syntax elements. Examples of such messages include SEI messaging that defines the mastering display color volume or content light level information. In addition to the syntax elements, for some other SEI messages, such as those describing film grain characteristics (FGC) or color remapping information, additional post-processing can also be defined. For the latter, the final video output may vary depending on the processing order of these SEI messages. As recognized by the inventors, an improved technique for signaling the priority order of SEI messaging is described herein.

[0005] The term "metadata" as used herein relates to any supplementary information transmitted as part of a coded bitstream and aids the decoder in rendering the decoded image. Such metadata may include, but is not limited to, color space or full gamut information, reference display parameters, and film grain modeling parameters, as described herein. The examples presented herein may refer to SEI messaging related to MPEG-based video coding standards, but those skilled in the art will understand that the techniques discussed herein are applicable to any such metadata messaging and any audio or video coding format (e.g., AV1, etc.).

[0006] The approaches described in this chapter are possible approaches to be pursued, but they are not necessarily approaches that have been previously devised or pursued. Therefore, unless otherwise specified, none of the approaches described in this chapter should be considered as being recognized as prior art merely by virtue of their inclusion in this chapter. Similarly, the problems identified with respect to one or more approaches should not be assumed to be recognized in any prior art based on this chapter unless specifically indicated.

Brief Description of the Drawings

[0007] Embodiments of the present invention are described by way of example and not limitation, and like reference numerals in the accompanying drawings represent like elements.

[0008]

Figure 1A

Figure 1B

[0009]

Figure 2

Modes for Carrying Out the Invention

[0010] Exemplary embodiments related to the prioritization of metadata are described herein. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments of the invention. It will be apparent, however, that the various embodiments of the invention may be practiced without some of these specific details. In other instances, well-known structures and devices are not described in detail to avoid unnecessarily obscuring, obfuscating, or rendering unclear the embodiments of the invention.

[0011] <Abstract> The exemplary embodiments described herein relate to the signaling of the prioritization of the processing of metadata (such as SEI messaging) in video coding. In an embodiment, a processor receives an input video bitstream and a corresponding input metadata message that defines supplementary processing to be applied to the decoded video pictures of the input video bitstream, analyzes the input metadata message to identify a metadata processing priority messaging (POM) message that defines a desired processing order for the selected input metadata message, when the POM message is detected, extracts information for determining a total number of selected input metadata messages having a processing priority, extracts the identification and priority information of the selected input metadata messages having a processing priority, and applies two or more of the selected input metadata messages to the decoded video pictures of the input video bitstream in their determined processing priorities.

[0012] In another embodiment, a processor generates a video bitstream and a corresponding metadata message that defines supplementary processing to be applied to the decoded video pictures of the video bitstream, Generate a metadata processing order (POM) message that defines the desired processing order of the selected metadata messages, said POM message comprising: information determining the total number of selected input metadata messages having a processing priority; and identification and priority information of said selected input metadata messages having said processing priority; and generate an output stream including said video bitstream, said metadata message, and the processing order of said metadata message.

[0013] In another embodiment, a tangible computer medium includes a bitstream, said bitstream comprising: an encoded video picture; an input metadata message defining supplementary processing to be applied to the decoded video picture of said input bitstream; a metadata processing order messaging (POM) message defining the desired processing order of selected input metadata messages; and said POM message comprising: information determining the total number of said selected input metadata messages having a processing priority; and identification and priority information of said selected input metadata messages having a processing priority. and

[0014] <Example of SEI messaging flow processing> Regarding existing video coding standards such as AVC, HEVC, and VVC (collectively referred to as MPEG or the MPEG video standard in References [1-4]), FIGS. 1A and 1B show an exemplary processing pipeline when SEI messaging includes messaging related to both film grain characteristics (FGC) and color remapping information (CRI) or color transform information (CTI).

[0015] As shown in FIGS. 1A and 1B, when an input video 102 is provided, the encoder can generate two sets of SEI messages: a) CTI / CRI SEI metadata (112) related to desired display parameters (e.g., as generated by block 110), and b) film grain characteristics (117) added to the decoder to emulate film grain modeled by the film grain modeling block (115) and removed by the encoder (e.g., by block 105) to improve encoding efficiency. Thus, the video encoder (120) compresses the output of the film grain remover (107) to generate a compressed bitstream 122. The compressed bitstream 122 and the metadata (112 and 117) are transmitted to the decoder.

[0016] As shown in FIG. 1A, after decoding (125), the decoded output (127) is passed to a film grain insertion unit 130 that generates and adds film grain noise using the information embedded in the FGC metadata (117). Next, the output of the film grain insertion unit (130) is passed to a display mapping unit (135) that optimally maps the dynamic range and color gamut of the input stream to the target display using the information embedded in the CRI / CTI metadata (112). The output is signal 137.

[0017] As shown in FIG. 1B, after decoding (125), as compared with FIG. 1A, the processing order of the two sets of metadata is reversed. Here, the decoded output (127) is first passed to a display mapping unit (135) that maps the dynamic range and color gamut of the decoded stream to the target display using the information embedded in the CRI / CTI metadata (112). Next, the output of the display mapping unit is passed to a film grain insertion unit 130, which generates and adds film grain noise using the information embedded in the FGC metadata (117) to generate the output video 132.

[0018] Since the order of display mapping and film grain addition is different, the video outputs 132 and 137 are expected to be different, especially when the target display is significantly different from the intended display. Therefore, it is considered important to specify the appropriate processing order of the metadata within the bitstream so that the decoder output is as close as possible to the creative intent of the content creator.

[0019] <Example of Signaling Priority in SEI Messaging> The exemplary embodiments presented herein relate to specifying the processing order between multiple metadata sets when multiple SEI messages coexist in video coding standards such as AVC, HEVC, and VVC. In particular, high level syntax (HLS) is proposed to specify such an order. Such high level syntax can be inserted at various levels of the coded bitstream, for example, but not limited to, video parameter set (VPS), picture parameter set (PPS), sequence parameter set (SPS), auxiliary parameter set (APS), picture header (PH), as individual SEI messaging, etc.

[0020] In an embodiment, such high-level syntax shall include the following information: 1) SEI messages that may exist in a coded video stream (CVS), 2) whether the processing order is related to a specific SEI message, and 3) if related, what the processing order of the specific SEI message is.

[0021] The processing (or priority) order of SEI messages can be specified in various ways as follows. 1) Absolute order: For example, in one embodiment, a message with a smaller processing / priority order needs to be processed earlier than a message with a larger priority order. 2) Relative order: In one embodiment, the processing order can be specified as a dependency on the processing order of another SEI message. When the coding standard defines that the processing of specific metadata is essential in the decoding process, the processing order of such SEI messages must always be specified. If the compliance point of a specific SEI message is defined, the processing order of such SEI messages must also be specified.

[0022] Table 1 shows an example of specifying the processing order between SEI messages. In this example, in the "SEI messaging processing order" SEI message, which is a newly proposed SEI message, the SEI messages that are part of the CVS where the processing order is important and their processing priorities can be specified. The target SEI messages are identified through their unique payload types (see References [1-4]). [Table 1] SEI Processing Order SEI Message [Table 1] po_num_sei_msg_types specifies the number of types of SEI messages that provide information to the SEI processing order SEI message. po_sei_payload_type[i] indicates the payloadType value of the i-th type of SEI message for which information is provided in the SEI processing order SEI message. If m is not equal to n, the values of po_sei_payload_type[m] and po_sei_payload_type[n] are not the same. po_sei_order[i] provides the absolute order of the SEI message having the payloadType equal to po_sei_payload_type[i]. If m is not equal to n, the values of po_sei_order[m] and po_sei_order[n] are not the same. If the value of po_sei_order[m] is less than the value of po_sei_order[n], the SEI message having po_sei_payload_type[m] is processed before the SEI message having po_sei_payload_type[n].

[0023] In Table 1, instead of the absolute po_sei_order[i] parameter, in an embodiment, a relative order can also be described where the order of SEI message i is defined relative to the order of SEI message j, such as po_sei_rel_order[i][j]. For example, po_sei_rel_order[3][1]=2 can define that the priority of SEI message "3" is the priority of SEI message "1" + (or alternatively "-") 2. In such a scenario, it can be assumed that all SEI messages with processing priorities start with a default value (e.g., 1) and are adjusted accordingly.

[0024] In an embodiment, Table 1 can also be simplified by removing the po_sei_order[i] parameter. In such a scenario, the encoder already lists the po_sei_payload_type[i] values in an implicit and pre-determined priority (e.g., lowest to highest or highest to lowest). In such an approach, the priority is enforced even when there are no such requirements (e.g., SEI messages "2" and "4" can have the same priority but are listed in order), which limits the possible parallelization of SEI messaging-related tasks. The implicit ranking is done at the start of a new CVS where all existing SEI messages are listed.

[0025] The HEVC and VVC SEI manifest SEI messages convey information about SEI messages that are present or expected to be present (i.e., likely) in the bitstream. However, the priority among these SEI messages is not indicated. Table 2 shows another example of extending the SEI manifest SEI to include priority information. The additional syntax for the existing SEI manifest messages is shown in italics. [Table 2] Example 1 of the extension of the SEI manifest SEI message

Table 2

[0026] Table 3 shows another approach to extending SEI manifest SEIs that support backward compatibility with legacy decoders that already support older versions of SEI manifest messages. [Table 3] Example 2 of the extension of SEI manifest SEI messages

Table 3

[0027] From HEVC, more_data_in_payload() is specified as follows. - If byte_aligned() is equal to TRUE and the current position of the sei_payload() syntax structure is 8 * payloadSize bits from the start of the sei_payload() syntax structure, the return value of more_data_in_payload() is equal to FALSE. - Otherwise, the return value of more_data_in_payload() is TRUE.

[0028] From HEVC, payload_extension_present() is specified as follows. - If the current position of the -sei_payload() syntax structure is not the position of the last (least significant, rightmost) bit equal to 1, but less than 8*payloadSize bits from the start of the syntax structure (i.e., the position of the payload_bit_equal_to_one syntax element), the return value of payload_extension_present() is equal to TRUE. - Otherwise, the return value of payload_extension_present() is equal to FALSE.

[0029] In Table 3, existing decoders implementing SEI manifest messaging can continue their implementation as before by ignoring the SEI processing order syntax. On the other hand, decoders implemented after the adoption of the new syntax can utilize the processing order information associated with the updated syntax.

[0030] As discussed in relation to Table 1, the processing order of the SEI messages in Tables 2 and 3 is also either implied or can be defined in relation to the processing order of other SEI messages.

[0031] In an embodiment, Table 1 and its semantics can be changed as shown in the description of Table 4. In this embodiment, the semantics depend on the syntax element MaxNumPayloadTypes to address the problem that the number of possible payload types varies according to the coding standard (e.g., at the time of this draft, there are 77 SEI messages in AVC, 67 SEI messages in HEVC, and 39 SEI messages in VVC), and furthermore, the actual number may increase in future versions of the standard, suppressing the signaling of more payload types than may exist.

[0032] As an alternative (or in addition), the actual valid payloadType of each specification can also be specified in a specific list (e.g., the list SeiAssociatedSeiList). For example, the following paragraphs can be added to the VVC or VSEI specification (References [3-4]). "SEI message processing order Use of SEI message syntax For the purpose of interpreting SEI messages in the SEI message processing order, the list SeiAssociatedSeiList is set to be composed of payloadType values from 0, 1, 3, 4, 5, 19, 45, 47, 129, 130, 132, 133, 137, 142, 144, 145, 147 to 150, 153 to 156, 165, 168, 177, 179, 180, 200 to 208, inclusive at both ends." In another embodiment, this list can be part of the SEI processing order SEI message or other header information and can be changed by the encoder as needed. [Table 4] SEI processing order SEI message

Table 4

[0033] Figure 2 shows an exemplary processing pipeline according to an embodiment when a SEI processing order of messaging (POM) message is available. Starting from step 210, the decoder can search the SEI payloadType of the incoming SEI message to detect whether an SEI POM message exists. If there is no such message, the processing of SEI messaging (215) continues without considering their order. If an SEI POM message exists, at step 220, the decoder reads the number of selected SEI messages for which the processing priority is defined. Next, at step 225, for each such message, the decoder reads its identity (e.g., payloadType) and its priority, typically an integer value having a predetermined ranked importance (e.g., a small number or a large number has a higher priority). In an embodiment, the priority may be implied by the order in which the selected SEI messages are listed. Finally, at step 230, the decoder processes these selected SEI messages in the appropriate priority order. References Each of the references listed herein is hereby incorporated by reference in its entirety. [1] AdvancedV Video Coding, Rec. ITU-T H.264, May 2019, ITU [2] High Efficiency Video Coding, Rec. ITU-T H.265, November 2019, ITU [3] Versatile Video Coding, Rec. ITU-T H.266, August 2020, ITU [4] Versatile supplemental enhancement information messages for coded video bitstreams, Rec. ITU-T H.274, Aug. 2020, ITU

[0034] <Implementation of an Exemplary Computer System> Embodiments of the present invention may be implemented by a computer system, a system configured within an electronic circuit and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD), an individual time or digital signal processor (DSP), an application specific IC (ASIC), and / or a device including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or perform instructions related to signaling metadata processing priority messaging in video coding as described herein. The computer and / or IC may calculate any of the various parameters or values related to signaling metadata processing priority messaging in video coding as described herein. Image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0035] Certain implementations of the present invention include a computer processor that executes software instructions that cause the processor to execute the methods of the present invention. For example, one or more processors in a display, encoder, set-top box, transcoder, etc. may implement a method related to signaling metadata processing priority messaging in video coding as described above by executing software instructions in a program memory accessible to the processor. Embodiments of the present invention may be provided in the form of a program product. The program product may include any non-transitory tangible medium carrying a set of computer-readable signals that, when executed by a data processor, cause the data processor to execute the methods of the present invention. The program product according to the present invention may be in any of various non-transitory tangible forms. The program product may include, for example, physical media such as magnetic data storage media including floppy disks, hard disk drives, optical data storage media including CD-ROMs, DVDs, ROMs, electronic data storage media including flash RAM, etc. The computer-readable signals on the program product may be optically compressed or encrypted. Components (e.g., software modules, processors, parts, devices, circuits, etc.) have been mentioned above, but unless otherwise specified, references to those components (including references to "means") shall be construed to include equivalents of those components, any component that performs the functions of the described components (e.g., functionally equivalent), and components that are not structurally equivalent to the disclosed structure that performs the functions in the illustrated exemplary embodiments of the present invention.

[0036] <Equivalents, Extensions, Alternatives and Miscellaneous> Exemplary embodiments related to signaling the processing priority of metadata messaging in video coding have been described. In the above specification, embodiments of the present invention have been described with reference to a number of specific details that may vary from implementation to implementation. Accordingly, the sole and exclusive indication of what the present invention is, and what the applicant intends to be the present invention, is set forth in the claims, which are issued in a specific form, including any later amendments, by this application. Any definitions explicitly set forth in this specification for terms included in such claims should govern the meaning of such terms as used in the claims. Accordingly, no limitation, element, feature, advantage, or attribute not explicitly recited in the claims should limit the scope of the claims in any way. The specification and drawings are, therefore, to be regarded in an illustrative rather than a limiting sense.

Claims

1. A method for determining priority processing among a plurality of supplementary enhancement information (SEI) messages, the method comprising: receiving an input video bitstream and a corresponding input SEI message defining supplementary processing applied to a decoded video picture of the input video bitstream; analyzing the input SEI message to identify a processing order (PO) SEI (PO SEI) message that defines a desired processing order for the selected input SEI message; when the PO SEI message is detected, extracting information for determining a total number of selected input SEI messages having a processing priority; for each of the selected input SEI messages, extracting identification information and ranking information, the ranking information including an integer value for determining processing in the desired processing order, and when the ranking information of SEI message m is less than the ranking information of SEI message n, the SEI message m is processed before the SEI message n; A method comprising the above steps.

2. The step of extracting identification information and the ranking information for the selected input SEI messages having a processing priority includes: The method according to claim 1, including the step of reading a list of the total number of the selected input SEI messages having an implicit descending processing priority or an implicit ascending processing priority.

3. The method according to claim 1, wherein the identification information includes SEI payload type information.

4. A method for signaling priority processing among a plurality of SEI messages, comprising: specifying an order for processing the plurality of SEI messages; transmitting a coded bitstream; including: The step of specifying the order for processing includes: generating a video bitstream and a corresponding SEI message defining supplementary processing applied to a decoded video picture of the video bitstream; generating a processing order (PO) SEI (PO SEI) message that defines a desired processing order for the selected SEI message, the PO SEI message including: information for determining a total number of selected input SEI messages having a desired processing order; for each of the selected input SEI messages, identification information and ranking information; including, the ranking information includes an integer value for determining a process in the desired processing order, and when the ranking information of the SEI message m is smaller than the ranking information of the SEI message n, the SEI message m is processed before the SEI message n, step; generating the coded bitstream including the video bitstream, the SEI message, and the PO SEI message; A method including.

5. The method according to claim 4, wherein the identification information includes SEI payload type information.

6. A method for specifying preferential processing among a plurality of SEI messages, the method comprising: generating a video bitstream and a corresponding SEI message defining supplementary processing applied to the decoded video picture of the video bitstream; generating a processing order (PO) SEI (PO SEI) message defining a desired processing order of the selected SEI message, the PO SEI message including: information for determining the total number of selected input SEI messages having a processing priority; for each of the selected input SEI messages, identification information and ranking information; including, the ranking information includes an integer value for determining a process in the desired processing order, and when the ranking information of the SEI message m is smaller than the ranking information of the SEI message n, the SEI message m is processed before the SEI message n, step; generating an output stream including the video bitstream, the SEI message, and the PO SEI message; A method including.

7. The method according to claim 6, wherein the identification information includes SEI payload type information.

Images