Signaling processing priority of metadata messaging in video coding - Patents.com

Abstract

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A method, system, and bitstream syntax for determining metadata processing priority messaging, such as supplemental enhancement information (SEI) messaging in MPEG video coding, are described.

Description

[Technical field]

[0001] [Related Applications] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 216,318, filed June 29, 2021, which is incorporated herein by reference.

[0002] [Technical field] FIELD OF THE DISCLOSURE This document relates generally to images and videos. More particularly, embodiments of the present invention relate to signaling processing priority of metadata messaging (e.g., supplemental enhancement information (SEI) messaging) in image and video sequences. [Background technology]

[0003] As described in Annex D of the AVC and HEVC standards, or H.274 (also referred to as VSEI) (References [1-4]), Supplemental Enhancement Information (SEI) messages in a coded video bitstream aid in processing related to decoding, display, or other purposes within a video processing pipeline. Despite its widespread use, at least until now, a compliant decoder (e.g., an AVC, HEVC, or VVC decoder) has not been required to process SEI messaging in order to be compliant with any of the MPEG video coding standards (AVC, HEVC, VVC, etc.).

[0004] In a typical video bitstream, multiple SEI messages may coexist. However, existing video coding standards do not define the processing order of such messaging. For some SEI messages, there is no specific processing defined in the standard other than the information embedded in the syntax elements. Examples of such messages include SEI messaging that defines mastering display color volume or content light level information. In addition to syntax elements, for some other SEI messages, such as those that describe film grain characteristics (FGC) or color remapping information, additional post-processing can also be defined. For the latter, the final video output may differ depending on the processing order of these SEI messages. As recognized by the inventors, improved techniques for signaling priority of SEI messaging are described herein.

[0005] The term "metadata" as used herein relates to any supplemental information transmitted as part of a coded bitstream to assist a decoder in rendering a decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and film grain modeling parameters, as described herein. Although the examples presented herein may refer to SEI messaging associated with MPEG-based video coding standards, those skilled in the art will understand that the techniques discussed herein are applicable to any such metadata messaging and any audio or video coding format (e.g., AV1, etc.).

[0006] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise noted, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, problems identified with one or more approaches should not be assumed to have been recognized in any prior art under this section unless specifically indicated. [Brief description of the drawings]

[0007] Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the accompanying drawings, in which like reference symbols represent similar elements and in which:

[0008] [Figure 1A] 1 illustrates an example of a video processing pipeline where metadata includes multiple SEI messages, whose post-processing order can affect the video output. [Figure 1B] 1 illustrates an example of a video processing pipeline where metadata includes multiple SEI messages, whose post-processing order can affect the video output.

[0009] [Diagram 2] 1 illustrates an exemplary processing pipeline at a decoder when a Priority Order of Messaging (POM) message is available in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Exemplary embodiments relating to prioritized processing of metadata are described herein. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the present invention. It will be apparent, however, that various embodiments of the present invention may be practiced without some of these specific details. In other instances, well-known structures and devices have not been described in exhaustive detail to avoid unnecessarily occluding, obscuring, or obscuring embodiments of the present invention.

[0011] <Summary> Example embodiments described herein relate to signaling priority of processing of metadata (such as SEI messaging) in video coding. In an embodiment, a processor includes: receiving an input video bitstream and a corresponding input metadata message defining supplemental processing to be applied to decoded video pictures of the input video bitstream; parsing selected incoming metadata messages to identify a metadata processing order messaging (POM) message that defines a desired processing order for the selected incoming metadata messages; When the POM message is detected, Extracting information to determine a total number of selected input metadata messages having a processing priority; Extracting identification and priority information of the selected input metadata messages having processing priorities; Applying two or more of the selected input metadata messages in their determined processing priority to a decoded video picture of the input video bitstream.

[0012] In another embodiment, the processor: generating a video bitstream and a corresponding metadata message defining supplemental processing to be applied to decoded video pictures of said video bitstream; Generate a metadata processing order (POM) message that defines the desired processing order of the selected metadata messages, where the POM message information for determining the total number of selected input metadata messages having a processing priority, and identification and priority information of the selected input metadata messages having the processing priority, and including Generate an output stream including the video bitstream, the metadata message, and the processing order of the metadata message.

[0013] In another embodiment, the tangible computer medium includes a bitstream, where the bitstream an encoded video picture, and an input metadata message that defines supplementary processing applied to the decoded video picture of the input bitstream, and a metadata processing order messaging (POM) message that defines the desired processing order of the selected input metadata messages, and including where the POM message information for determining the total number of the selected input metadata messages having a processing priority, and identification and priority information of the selected input metadata messages having the processing priority, and including.

[0014] <Example of SEI messaging flow processing> For existing video coding standards such as AVC, HEVC and VVC (References [1-4]) (collectively referred to as MPEG or MPEG Video Standards), Figures 1A and 1B show an example processing pipeline where the SEI messaging includes messaging related to both film grain characteristics (FGC) and color remapping information (CRI) or color transform information (CTI).

[0015] 1A and 1B, given an input video 102, the encoder can generate two sets of SEI messages: a) CTI / CRI SEI metadata (112) related to desired display parameters (e.g., as generated by block 110), and b) film grain characteristics (117) to be added to the decoder to emulate the film grain removed by the encoder (e.g., by block 105) and modeled by the film grain modeling block (115) to improve coding efficiency. Thus, the video encoder (120) compresses the output of the film grain remover (107) to generate a compressed bitstream 122. The compressed bitstream 122 and the metadata (112 and 117) are transmitted to the decoder.

[0016] As shown in Figure 1A, after decoding (125), the decoded output (127) is passed to a film grain insertion unit 130 which generates and adds film grain noise using information embedded in the FGC metadata (117). The output of the film grain insertion unit (130) is then passed to a display mapping unit (135) which optimally maps the dynamic range and color gamut of the input stream to the target display using information embedded in the CRI / CTI metadata (112). The output is signal 137.

[0017] As shown in FIG. 1B, after decoding (125), compared with FIG. 1A, the processing order of the two sets of metadata is reversed. Here, the decoded output (127) first passes to a display mapping unit (135) that maps the dynamic range and color gamut of the decoded stream to the target display using the information embedded in the CRI / CTI metadata (112). Next, the output of the display mapping unit is passed to a film grain insertion unit 130, which generates and adds film grain noise using the information embedded in the FGC metadata (117) to generate the output video 132.

[0018] Since the order of display mapping and film grain addition is different, the video outputs 132 and 137 are expected to be different, especially when the target display is significantly different from the intended display. Therefore, it is considered important to specify the appropriate processing order of the metadata within the bitstream so that the decoder output is as close as possible to the creative intent of the content creator.

[0019] <Example of Signaling Priorities in SEI Messaging> The exemplary embodiments presented herein relate to specifying the processing order between multiple metadata sets when multiple SEI messages coexist in video coding standards such as AVC, HEVC, and VVC. In particular, high level syntax (HLS) is proposed to specify such an order. Such high level syntax can be inserted at various levels of the coded bitstream, for example, but not limited to, video parameter set (VPS), picture parameter set (PPS), sequence parameter set (SPS), auxiliary parameter set (APS), picture header (PH), as individual SEI messaging, etc.

[0020] In an embodiment, such high level syntax shall include the following information: 1) what SEI messages may be present in a coded video stream (CVS); 2) whether a processing order is relevant for a particular SEI message; and 3) if so, what is the processing order for a particular SEI message.

[0021] The processing (or priority) order of SEI messages can be specified in a variety of ways: 1) Absolute order: For example, in one embodiment, messages with lower processing / priority need to be processed sooner than messages with higher priority. 2) Relative order: In one embodiment, the processing order can be specified as a dependency on the processing order of another SEI message. If a coding standard defines that the processing of certain metadata is mandatory for the decoding process, then the processing order of such SEI messages must always be specified. If a compliance point is defined for a particular SEI message, then the processing order of such SEI messages must also be specified.

[0022] Table 1 shows Here is an example of specifying the processing priority between SEI messages: 。 In this example, a proposed new SEI message, the "SEI messaging processing order" SEI message, allows specification of SEI messages that are part of a CVS where order matters and their processing priority. The SEI messages are identified through their unique payload type (see References [1-4]). [Table 1] SEI processing order SEI messages [Table 1] po_num_sei_msg_types specifies the number of types of SEI messages that provide information to the SEI processing priority SEI message. po_sei_payload_type[i] indicates the payload type value of the i-th type of SEI message whose information is provided in the SEI processing order SEI message. If m is not equal to n, the values ​​of po_sei_payload_type[m] and po_sei_payload_type[n] are not the same. po_sei_order[i] provides the absolute ranking of SEI messages with payloadType equal to po_sei_payload_type[i]. If m is not equal to n, the values ​​of po_sei_order[m] and po_sei_order[n] are not identical. If the value of po_sei_order[m] is less than the value of po_sei_order[n], SEI messages with po_sei_payload_type[m] are processed before SEI messages with po_sei_payload_type[n].

[0023] In Table 1, instead of the absolute po_sei_order[i] parameter, embodiments may also describe a relative order, such as po_sei_rel_order[i][j], where the order of SEI message i is defined relative to the order of SEI message j. For example, po_sei_rel_order[3][1]=2 may define that the priority of SEI message "3" is the priority of SEI message "1" + (or alternatively "-") 2. In such a scenario, it may be assumed that all SEI messages with processing priority start out with a default value (e.g., 1) and are adjusted accordingly.

[0024] In an embodiment, Table 1 can also be simplified by removing the po_sei_order[i] parameter. In such a scenario, the encoder already lists the po_sei_payload_type[i] values ​​in an implicit and pre-determined order of priority (e.g., lowest to highest or highest to lowest). Such an approach limits possible parallelization of SEI messaging related tasks, since priority is enforced even in the absence of such a requirement (e.g., SEI messages "2" and "4" can have the same priority, but are listed in order). The implicit ordering is done at the start of a new CVS, where all existing SEI messages are listed.

[0025] The HEVC and VVC SEI manifest SEI message conveys information about SEI messages that are expected (i.e., likely) to be present or absent in the bitstream, but does not indicate any priority among these SEI messages. Table 2 shows another example of extending the SEI manifest SEI to include priority information. The additional syntax to the existing SEI manifest message is shown in italics. [Table 2] Example 1 of SEI Manifest SEI Message Extension [Table 2] manifest_sei_po_flag[i] equal to 1 specifies that an SEI message with a payloadType equal to manifest_sei_payload_type[i] requests processing order information. manifest_sei_po_flag[i] equal to 0 specifies that an SEI message with a payloadType equal to manifest_sei_payload_type[i] does not request processing order information. manifest_sei_order[i] provides the absolute ordering of SEI messages with payloadType equal to manifest_sei_payload_type[i]. If m is not equal to n, the values ​​of manifest_sei_order[m] and manifest_sei_order[n] are not identical. If the value of manifest_sei_order[m] is less than the value of manifest_sei_order[n], SEI messages with manifest_sei_payload_type[m] are processed before SEI messages with manifest_sei_payload_type[n]. NOTE: As mentioned above, if the processing order is defined implicitly (e.g., the order in which SEI messages are listed in manifest_sei_po_flag[i]), the syntax element manifest_sei_order[i] is not necessary and can be removed.

[0026] Table 3 shows an alternative approach to extending the SEI manifest SEI that supports backward compatibility with legacy decoders that already support older versions of the SEI manifest message. [Table 3] Example 2 of SEI Manifest SEI message extension [Table 3]

[0027] From HEVC, more_data_in_payload() is specified as follows: - If byte_aligned() is equal to TRUE and the current position of the sei_payload() syntax structure is 8*payloadSize bits from the beginning of the sei_payload() syntax structure, the return value of more_data_in_payload() is equal to FALSE. - Otherwise, the return value of more_data_in_payload() is TRUE.

[0028] From HEVC, payload_extension_present() is specified as follows: - If the current position of the sei_payload() syntax structure is not the position of the last (lowest, rightmost) bit equal to 1, and is less than 8*payloadSize bits from the beginning of the syntax structure (i.e., the position of the payload_bit_equal_to_one syntax element), the return value of payload_extension_present() is equal to TRUE. - Otherwise, the return value of payload_extension_present() is equal to FALSE.

[0029] In Table 3, existing decoders that implement SEI manifest messaging can continue to do so as before by ignoring the SEI processing order syntax, while decoders implemented after the adoption of the new syntax can take advantage of the updated syntax and associated processing order information.

[0030] As discussed in relation to Table 1, the processing order of the SEI messages in Tables 2 and 3 may also be implied or defined relative to the processing order of other SEI messages.

[0031] In an embodiment, Table 1 and its semantics may be modified as shown in the description of Table 4. In this embodiment, the semantics rely on the syntax element MaxNumPayloadTypes to address the issue that the number of possible payload types varies by coding standard (e.g., as of this draft, AVC has 77 SEI messages, HEVC has 67 SEI messages, and VVC has 39 SEI messages), and furthermore, the actual number may increase in future versions of the standard, to discourage signaling of more payload types than may exist.

[0032] Alternatively (or additionally), the actual valid payloadTypes for each standard can also be specified in a specific list (e.g., the list SeiAssociatedSeiList). For example, the following paragraph can be added to the VVC or VSEI standard (References [3-4]): "SEI Message Processing Order SEI Message Syntax Usage SEI Message Processing Order For purposes of interpreting SEI messages, the list SeiAssociatedSeiList is set to consist of payloadType values ​​0, 1, 3, 4, 5, 19, 45, 47, 129, 130, 132, 133, 137, 142, 144, 145, 147 through 150, 153 through 156, 165, 168, 177, 179, 180, 200 through 208, inclusive." In another embodiment, this list may be part of the SEI Processing order SEI message or other header information, and is modified by the encoder as needed. [Table 4] SEI processing order SEI messages [Table 4] po_num_sei_payload_types specifies the maximum value of the payloadType value for which the syntax elements po_sei_payload_type[i] and po_sei_processing_order[i] exist. The value of po_num_sei_payload_types must be less than or equal to MaxNumPayloadTypes, which defines the maximum value of the payloadType value. NOTE: If the syntax element SeiAssociateSeiList is added, the above definition can be replaced by the following: po_num_sei_payload_types specifies the maximum number of payloadType values ​​for which there are syntax elements po_sei_payload_type[i] and po_sei_processing_order[i], as specified by the encoder (eg, via the list SeiAssociateSeiList). po_sei_payload_type[i] specifies the value of payloadType of the i-th SEI message. If m is not equal to n, the value of po_sei_payload_type[m] shall not be equal to the value of po_sei_payload_type[n]. NOTE: If the syntax element SeiAssociateSeiList is added, add the following: The value of po_sei_payload_type[i] shall be equal to the payloadType value in SeiAssociateSeiList. po_sei_order[i] indicates the desired order of processing SEI messages with payloadType equal to po_sei_payload_type[i]. If m is not equal to n, the value of po_sei_order[m] shall not be equal to the value of po_sei_order[n]. If po_sei_order[m] is greater than 0 and less than po_sei_order[n], it indicates that SEI messages with payloadType equal to po_sei_payload_type[m] should be processed before SEI messages with payloadType equal to po_sei_payload_type[n]. If po_sei_order[i] is equal to 0, it specifies that the desired order of processing SEI messages with payloadType equal to po_sei_payload_type[i] is determined by external means not specified in this specification.

[0033] FIG. 2 illustrates an exemplary processing pipeline according to an embodiment when SEI processing order of messaging (POM) messages are available. Starting at step 210, the decoder may search the SEI payloadType of the incoming SEI message to detect whether a SEI POM message is present. If there is no such message, the processing of the SEI messaging (215) continues without considering their order. If there is a SEI POM message, then in step 220 the decoder reads the number of selected SEI messages for which a processing priority has been defined. Then, in step 225, the decoder reads for each such message its identity (e.g., payloadType) and its priority, typically an integer value with a predefined ordered importance (e.g., lower or higher numbers have higher priority). In an embodiment, the priority may be implied by the order of how the selected SEI messages are listed. Finally, in step 230, the decoder processes these selected SEI messages with the appropriate priority. References Each of the references cited herein is hereby incorporated by reference in its entirety. [1] AdvancedV Video Coding, Rec. ITU-T H.264, May 2019, ITU [2] High Efficiency Video Coding, Rec. ITU-T H.265, November 2019, ITU [3] Versatile Video Coding, Rec. ITU-T H.266, August 2020, ITU [4] Versatile supplemental enhancement information messages for coded video bitstreams, Rec. ITU-T H.274, Aug. 2020, ITU

[0034] Exemplary Computer System Implementation Embodiments of the invention may be implemented by a computer system, a system configured in electronic circuits and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA) or other configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or perform instructions related to signaling metadata processing priority messaging in video coding as described herein. The computer and / or IC may calculate any of various parameters or values ​​related to signaling metadata processing priority messaging in video coding as described herein. Image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0035] A particular implementation of the present invention includes a computer processor executing software instructions that cause the processor to perform the method of the present invention. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc. may perform the method related to signaling metadata processing priority messaging in video coding described above by executing software instructions in a program memory accessible to the processor. An embodiment of the present invention may be provided in the form of a program product. The program product may include any non-transitory tangible medium that carries a set of computer-readable signals that include instructions that, when executed by a data processor, cause the data processor to perform the method of the present invention. A program product according to the present invention may be in any of a variety of non-transitory tangible forms. The program product may include a physical medium, such as, for example, a magnetic data storage medium including a floppy disk, a hard disk drive, an optical data storage medium including a CD-ROM, a DVD, an electronic data storage medium including a ROM, a flash RAM, etc. The computer-readable signals on the program product may be optically compressed or encrypted. Although components (e.g., software modules, processors, components, devices, circuits, etc.) have been referred to above, unless otherwise indicated, references to those components (including references to "means") should be interpreted to include equivalents of those components, any components that perform the functions of the described components (e.g., are functionally equivalent), and components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated exemplary embodiments of the present invention.

[0036] <Equivalents, Extensions, Alternatives and Miscellaneous> Exemplary embodiments related to signaling processing priority of metadata messaging in video coding have been described. In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indication of what the invention is, and what applicant intends to be the invention, is set forth in the claims issued in a particular form hereby, including any subsequent amendments. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Thus, any limitations, elements, features, advantages, or attributes not expressly set forth in the claims should not limit the scope of the claims in any manner. The specification and drawings should therefore be considered illustrative, and not restrictive.

Claims

1. 1. A method for determining a prioritized handling among a plurality of metadata messages, the method comprising: receiving an input video bitstream and a corresponding input metadata message defining supplemental processing to be applied to decoded video pictures of the input video bitstream; parsing selected incoming metadata messages to identify a metadata processing order messaging (POM) message that defines a desired processing order for the incoming metadata messages; When the POM message is detected, extracting information determining a total number of selected input metadata messages having a processing priority; extracting, for each of the selected input metadata messages defining the desired processing order, identification information and corresponding priority information, the priority information comprising integer values ​​with smaller or larger numerical values ​​resulting in earlier processing in the desired processing order, the integer values ​​representing the priority of a selected input message in terms of absolute processing priority among the selected input metadata messages or in terms of processing priority relative to a second selected input metadata message among the selected input metadata messages; applying two or more of the selected input metadata messages with their determined processing priority to a decoded video picture of the input video bitstream; The method includes:

2. 2. The method of claim 1, wherein the POM message is part of at least one of a slice header, a picture header, a sequence parameter set, a picture parameter set, a network abstraction layer unit header, and a supplemental enhancement information (SEI) message.

3. The method of claim 2 , wherein the POM message is part of an SEI manifest message.

4. The step of extracting identification information and priority information of the selected input metadata message having a processing priority comprises: The method of claim 1 , comprising reading the list of total numbers of the selected input metadata messages having an implicit descending processing priority or an implicit ascending processing priority.

5. The method of claim 1 , wherein the identification information includes payloadType information in SEI messaging.

6. A method for signaling priority handling among a plurality of metadata messages, comprising: specifying the priority process; transmitting the coded bitstream; Including, The step of designating the priority process includes: generating a video bitstream and a corresponding metadata message defining supplemental processing to be applied to decoded video pictures of said video bitstream; generating a metadata processing order (POM) message defining a desired processing order of the selected metadata messages, said POM message comprising: information for determining a total number of selected input metadata messages having a processing priority; for each of the selected input metadata messages defining the processing priority, identification information and corresponding priority information; wherein the priority information comprises integer values ​​with smaller or larger numerical values ​​resulting in earlier processing in the desired processing order, the integer values ​​representing the priority of a selected input metadata message in terms of absolute processing priority among selected input metadata messages or in terms of processing priority relative to a second selected input metadata message among the selected input metadata messages; generating the coded bitstream including the video bitstream, the metadata messages, and a processing order for the metadata messages; A method comprising:

7. The method of claim 6 , wherein the POM message is part of an SEI manifest message or an SEI metadata processing order message.

8. The method of claim 7 , wherein the identification information includes payloadType information in SEI messaging.

9. 1. A method for specifying priority handling among a plurality of metadata messages, the method comprising: generating a video bitstream and a corresponding metadata message defining supplemental processing to be applied to decoded video pictures of said video bitstream; generating a metadata processing order (POM) message defining a desired processing order of the selected metadata messages, said POM message comprising: information for determining a total number of selected input metadata messages having a processing priority; for each of the selected input metadata messages defining the processing priority, identification information and corresponding priority information; wherein the priority information comprises integer values ​​with smaller or larger numerical values ​​resulting in earlier processing in the desired processing order, the integer values ​​representing the priority of a selected input metadata message in terms of absolute processing priority among selected input metadata messages or in terms of processing priority relative to a second selected input metadata message among the selected input metadata messages; generating an output stream comprising said video bitstream, said metadata messages and a processing order for said metadata messages; The method includes:

10. A non-transitory computer readable storage medium storing computer executable instructions for carrying out the method of any one of claims 1 to 5 or 9 by one or more processors.

11. An apparatus including a processor and configured to perform any one of the methods according to claims 1 to 5 or 9.

Images