Signaling payload length in a bitstream

Abstract

A method for obtaining from a bitstream a first payload of a first message, wherein the first payload of the first message has a size. The method includes obtaining from the bitstream information regarding the first payload of the first message and / or obtaining from the bitstream information regarding the first message. The method also includes determining, based on the obtained information, whether the first message includes at least a first size syntax element for use in specifying the size of the first payload of the first message. The method also includes, if it is determined that the first message includes at least the first size syntax element, then using at least the first size syntax element to obtain a first payload size value specifying the size of the first payload of the first message, otherwise using other information to obtain the first payload size value.

Description

SIGNALING PAYLOAD LENGTH IN A BITSTREAMTECHNICAL FIELD

[0001] Disclosed are embodiments related to payload length signaling.BACKGROUND

[0002] Versatile Video Coding (VVC)

[0003] VVC and its predecessors Advanced Video Coding (AVC) and High Efficiency Video Coding (HEVC) are block-based video codecs standardized and developed jointly by ITU-T and MPEG. The codecs utilize both temporal and spatial prediction to generate predicted pixel data. VVC, HEVC and AVC are similar in many aspects. Spatial prediction is achieved using intra (I) prediction from within a current picture. Temporal prediction is achieved using uni-directional (P) or bi-directional inter (B) prediction on the block level from previously decoded reference pictures.

[0004] In the encoder, the difference between the original pixel data and the predicted pixel data, referred to as the residual, is transformed into the frequency domain, quantized and then entropy coded (e.g., coded using arithmetic coding) before transmitted together with necessary prediction parameters such as prediction mode and motion vectors, also entropy coded (e.g., also arithmetically coded). To reconstruct a picture the decoder obtains the residual for the picture (i. e. , the decoder performs decoding (e.g., arithmetic decoding), inverse quantization and inverse transformation to obtain the residual) and then adds the residual for the picture to an intra or inter prediction.

[0005] The VVC version 1 specification was published as Rec. ITU-T H.266 | ISO / IEC 23090-3, “Versatile Video Coding”, in 2020 followed by v2 and v3 which is the latest published version at the time of writing. MPEG and ITU-T are working together within the Joint Video Exploratory Team (JVET) on updated versions of HEVC and VVC as well as the successor to VVC, i.e. the next generation video codec.

[0006] Video, Picture, and Components

[0007] A video (a.k.a., video sequence) consists of a series of pictures where each picture consists of one or more components. A picture in a video sequence is sometimes denoted ‘image’ or ‘frame’. Each component in a picture can be described as a two-dimensional rectangular array of sample values. It is common that a picture in a video sequence consists of three components; one luma component Y where the sample values are luma values and two chroma components Cb and Cr, where the sample values are chroma values. Other common representations include ICtCb, IPT, constant-luminance YCbCr, YCoCg and others. It is also common that the dimensions of the chroma components are smaller than the luma components by a factor of two in each dimension. For example, the size of the luma component of an HD picture would be 1920x1080 and the chroma components would each have the dimension of 960x540. Components are sometimes referred to as ‘color components’, and other times as ‘channels’.

[0008] Blocks and Units

[0009] In many video coding standards, such as HEVC and VVC, each component is split into blocks and a bitstream, which is a sequence of bits, produced by the encoder consists of a series of coded blocks. A block is a two-dimensional array of sample values (or “samples” for short). It is common in video coding that the picture is split into units that cover a specific area of the picture. Each unit consists of all blocks from all components that make up that specific area and each block belongs fully to one unit. The macroblock in H.264 and the Coding unit (CU) in HEVC are examples of units.

[0010] A block may be a “transform block” or a “prediction block.” A transform block is a block to which a transform used in coding is applied, and a “prediction block” is a block to which a single prediction mode is applied.

[0011] NAL units

[0012] Both VVC and HEVC define a Network Abstraction Layer (NAL). All the data, i.e., both Video Coding Layer (VCL) data and non-VCL data in HEVC and VVC is encapsulated in NAL units. A VCL NAL unit contains data that represents picture sample values, such as coded slices. A non-VCL NAL unit contains additional associated data (a.k.a., metadata) such as parameter sets and supplemental enhancement information (SEI) messages. The NAL unit in VVC and HEVC begins with a header called the NAL unit header. The syntax for the NAL unit header for HEVC starts with a forbidden zero bit that shall always be equal to 0 to prevent start code emulations. Without it, some MPEG systems might confuse the HEVC video bitstream with other data, but the 0 bit in the NAL unit header makes all possible HEVC bitstreams uniquely identifiable as HEVC bitstreams. The NAL unit header in VVC is very similar to the one in HEVC but uses 1 bit less for thenal unit type and instead reserves this bit for future use. The nal unit type, nuh layer id and nuh_temporal_id_plusl code words specify the NAL unit type of the NAL unit that identifies what type of data is carried in the NAL unit, the scalability layer ID and the temporal layer ID for which the NAL unit belongs to. The NAL unit type indicates and specifies how the NAL unit should be parsed and decoded. The rest of the bytes of the NAL unit is payload of the type indicated by the NAL unit type. A bitstream consists of a series of concatenated NAL units (i. e. , an ordered set of NAL units).

[0013] The syntax for the NAL unit header for VVC is shown in Table 1 below:TABLE- 1

[0014] A decoder can determine how the NAL unit should be handled, e.g., parsed and decoded, after looking at the NAL unit header. All VVC or HEVC bitstreams consist of a series of concatenated NAL units. The decoding order is the order in which NAL units shall be decoded, which is the same as the order of the NAL units within the bitstream. The decoding order may be different from the output order, which is the order in which decoded pictures are to be output, such as for display, by the decoder.

[0015] Parameter Sets

[0016] HEVC and VVC specifies three types of parameter sets, the picture parameter set (PPS), the sequence parameter set (SPS) and the video parameter set (VPS). The PPS contains data that is common for a whole picture, the SPS contains data that is common for a coded video sequence (CVS) and the VPS contains data that is common for multiple CVSs, e.g. data for multiple scalability layers in the bitstream.

[0017] VVC also specifies one additional parameter set, the adaptation parameter set (APS). The APS carries parameters needed for the adaptive loop filter (ALF) tool, the luma mapping and chroma scaling (LMCS) tool and the scaling list tool.

[0018] Both HEVC and VVC allow certain information (e.g. parameter sets) to be provided by external means. “By external means” should be interpreted as the information is not provided in the coded video bitstream but by some other means not specified in the video codec specification, e.g. via metadata possibly provided in a different data channel, as a constant in the decoder, or provided through an API to the decoder.

[0019] RBSP and Syntax Functions and Descriptors

[0020] A raw byte sequence payload (RBSP) as defined in VVC is a syntax structure containing an integer number of bytes that is encapsulated in aNAL unit and is either empty or has the form of a string of data bits containing syntax elements followed by an RBSP stop bit and zero or more subsequent bits equal to 0.

[0021] VVC further defines syntax functions and descriptors, shown in Table 2 below, which may be useful for the understanding of some syntax in in this disclosure. These functions are expressed in terms of the value of a bitstream pointer that indicates the position of the next bit to be read by the decoding process from the bitstream. The value or position of the bitstream pointer is here called the current position.TABLE-2

[0022] SEI Messages

[0023] Supplementary Enhancement Information (SEI) messages are syntax structures in the coded bitstream that are conveyed in SEI NAL units and do not influence the decoding process of coded pictures from VCL NAL units. SEI messages usually address issues of representation and / or rendering of the decoded bitstream. The overall concept of SEI messages and many of the messages themselves have been inherited from the H.264and HEVC specifications into VVC specification. In VVC, an SEI RBSP contains one or more SEI messages.

[0024] SEI messages assist in processes related to decoding, display or other purposes. However, SEI messages are not required for constructing the luma or chroma sample values by the decoding process. Some SEI messages are required for checking bitstream conformance and for output timing decoder conformance. Other SEI messages are not required for checking bitstream conformance. The HEVC and VVC specifications do not require decoders to implement SEI messages. Some SEI messages are specified as mandatory to implement in certain application specifications. Usually, if a decoder encounters an unsupported SEI message, it is discarded.

[0025] ITU-T H.SEI | ISO / IEC 23002-7, also referred to as VSEI, specifies the syntax and semantics of SEI messages and is particularly intended for use with VVC bitstreams, although it is written in a manner intended to be sufficiently generic that it may also be used with other types of coded video bitstreams. The first version of ITU-T H.SEI | ISO / IEC 23002-7 was finalized in July 2020.

[0026] The persistence of an SEI message indicates the pictures to which the values signalled in the instance of the SEI message may apply. The part of the bitstream that the values of the SEI message may apply to are referred to as the persistence scope of the SEI message.

[0027] The SEI RBSP and the SEI message syntax table describing the general structure of an SEI message in VVC is shown in Table 3 and Table 4. The type of each SEI message is identified by its payload type.TABLE-3TABLE-4

[0028] payload_type_byte is a byte of the payload type of an SEI message. payloadType is calculated from the one or more payload type byte syntax elements.

[0029] payload_size_byte is a byte of the payload size of an SEI message. payloadSize is calculaded from the one or more payload size byte syntax elemetns.

[0030] Annex D in the VVC v3 specification, specifies syntax and semantics for SEI message payloads for some SEI messages, and specifies the use of the SEI messages and VUI parameters for which the syntax and semantics are specified in ITU-T H.274 | ISO / IEC 23002-7, ISO / IEC 23001-11 and ISO / IEC 23090-13. Table 5 below specifies the general SEI payload syntax.TABLE-5

[0031] Large Supplemental Enhancement Information Message (Isei)

[0032] To overcome an issue of inefficient signalling of large SEI payload sizes, a new design for SEI data structure so called lsei_message () has been included in the draft of technology under consideration, TuC, for future extensions of VSEI (JVET-AF2032). The syntax table and semantics for the lsei_message () are shown below in Table 6.TABLE-6

[0033] Each large SEI message consists of the variables specifying the type payloadType and size payloadSize of the large SEI message payload. Large SEI message payloads are specified in Annex D. The derived large SEI message payload size payloadSize is specified in bytes and shall be equal to the number of RBSP bytes in the large SEI message payload.

[0034] The NAL unit byte sequence containing the large SEI message might include one or more emulation prevention bytes (represented by emulation_prevention_three_byte syntax elements). Because the payload size of a large SEI message is specified in RBSPbytes, the quantity of emulation prevention bytes is not included in the size payloadSize of a large SEI payload.

[0035] Table 7 below provides the semantics of the syntax elments of the lsei_message.TABLE-7

[0036] Versatile SEI RBSP

[0037] Another relevant part of the published technology is the Versatile SEI RBSP proposed in the JVET-AG0051 contribution and included in the following version of Technology under Consideration (TuC) document for future extensions of VSEI (JVET- AG2032).

[0038] The elements from JVET-AG0051 shown in Table 8 below were agreed to be included into the TuC:TABLE-8

[0039] The following text implementations were proposed by JVET-AG0051 and included in the JVET-AG2032 TuC.

[0040] Versatile SEI RBSP syntax is shown in Table 9:TABLE-9

[0041] The versatile SEI RBSP semantics is shown in Table 10:TABLE- 10

[0042] The versatile SEI message syntax is shown in Table 11 :TABLE- 11

[0043] The versatile SEI message semantics is shown in Table 12:TABLE- 12SUMMARY

[0044] Certain challenges presently exist. For instance, in the current specification of SEI RBSP in VVC, the RBSP payload size needs to be signalled prior to the payload. One reason for this is to be able to bundle several SEI payloads in one RBSP and correctly parsethem. For SEIs with large payload size, a solution for signaling the size of the SEI RBSP payload using a particular fixed number of bits has been proposed to JVET (with 16 bits for now mentioned in the VSEI technology under consideration (TuC) for the large supplemental enhancement information message, Isei). Putting a limit on the maximum size of an SEI RBSP payload size, however, could be problematic. For example, the size of an SEI message can vary significantly depending on the specific content being encoded, the metadata being conveyed, and the implementation choices made by the encoder. Some SEI messages carry more parameters and have naturally larger payloads. Considering new applications and technologies it could be that even larger SEI messages need to be added in the future. So, the codewords regarding the length of the SEI RBSP can potentially become larger than the limit specified in the current design. Even with a non-fixed size codeword, signalling the length of the RBSP might not be an efficient use of bits.

[0045] Another problem is that the current design requires all the SEI messages to use at least 8 bits for the SEI payload size, which, in some cases, is not an efficient use of bits.

[0046] Another problem with the current implementation of the SEI mechanism is that all SEIs have the same status in the sense that different SEI messages are treated the same in the bitstream, including: there is one NAL unit type for all SEI messages, different SEI messages can be signalled in the same NAL unit, All SEI messages have the same importance level in the sense that all are treated similarly and can be discarded by the systems layer, etc. This makes it difficult to differentiate them for instance based on their length or importance or giving them any special treatment based on their essentiality.

[0047] Some published technologies propose solutions which use more than one NAL unit type for carrying different SEI messages, for instance a first NAL unit type being used for a first group of SEI messages that are larger or more important and a second NAL unit type for the rest of the SEI messages. But this is not an optimal design choice because the NAL unit types are limited and could be put to better use.

[0048] Additionally, some published technologies include a flag in the beginning of the SEI RBSP or the SEI message structure to indicate the importance of the SEI message. This potentially provides a means for distinguishing important SEIs and potentially treat them separately for instance not removing those by functionalities in the systems layer. This solution, however, may not be backwards compatible and the concept of the importance for an SEI is not clearly defined.

[0049] Accordingly, in one aspect there is provided a method for obtaining from a bitstream a first pay load of a first message, wherein the first payload of the first message has a size. The method includes obtaining from the bitstream information regarding the first payload of the first message and / or obtaining from the bitstream information regarding the first message. The method also includes determining, based on the obtained information, whether the first message includes at least a first size syntax element for use in specifying the size of the first payload of the first message. The method also includes, if it is determined that the first message includes at least the first size syntax element, then using at least the first size syntax element to obtain a first payload size value specifying the size of the first payload of the first message, otherwise using other information to obtain the first payload size value.

[0050] In another aspect there is provided a method for processing a bitstream comprising a sequence of network abstraction layer, NAL, units. The method includes obtaining type information for a NAL unit included in the bitstream, wherein the type information specifies that the NAL unit is a supplemental enhancement information, SEI, NAL unit, wherein the SEI NAL unit comprises a first SEI message having a payload, wherein the payload consists of an ordered sequence of N bits, where N > 0. The method also includes determining the last bit in the ordered sequence of N bits. Determining the last bit in the ordered sequence of N bits comprises locating a code pattern in the SEI NAL unit, wherein the last bit in the ordered sequence of bits immediately precedes the code pattern in the SEI NAL unit.

[0051] In another aspect there is provided a method performed by an encoder. The method includes obtaining a first message comprising a first payload having a size and a type. The method also includes determining whether to include in a bitstream one or more size syntax elements specifying the size of the first pay load of the first message. The determination is based on at least one of the following: an encoder profile; an encoder configuration; an encoder setting; the type of the payload; the size of the payload; or whether the first message is the only message that will be included in a network abstraction layer, NAL, unit containing the first message.

[0052] In another aspect there is provided an apparatus that is configured to perform the methods disclosed herein. The apparatus may include memory and processing circuitry coupled to the memory.

[0053] In another aspect there is provided a computer program comprising instructions which when executed by processing circuitry of an apparatus causes the apparatus to perform any of the methods disclosed herein. In one embodiment, there is provided a carrier containing the computer program wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[0054] The embodiments disclosed herein provide several advantages, including: 1) not requiring the use of extra NAL unit types; 2) improving coding efficiency by removing the need to signal the size of large SEI messages; 3) improving coding efficiency by removing the need for signalling the length of SEI messages especially when they are signalled frequently, such as, per picture; 4) providing a better balancing of the length of the SEI NAL units by avoiding grouping of large SEIs together or bundling large and small SEIs in the same NAL unit; 5) being able to categorize or group SEI messages; 5) avoiding having to hard code the maximum SEI payload size, which makes the solution extendable to a future codec that allows larger payload sizes; 6) removing the need for signalling the payload size under some conditions, hence making it possible to incorporate larger SEIs as well. Further, some embodiments provide backwards compatibility with the current VVC and VSEI specifications.BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0056] FIG. 1 illustrates a system according to an embodiment.

[0057] FIG. 2 is a schematic block diagram of an encoder according to an embodiment.

[0058] FIG. 3 is a schematic block diagram of a decoder according to an embodiment.

[0059] FIG. 4 is a flowchart illustrating a process according to an embodiment.

[0060] FIG. 5 is a flowchart illustrating a process according to an embodiment.

[0061] FIG. 6 is a flowchart illustrating a process according to an embodiment.

[0062] FIG. 7 is a block diagram of an encoding apparatus according to an embodiment.DETAILED DESCRIPTION

[0063] FIG. 1 illustrates a system 100 according to an embodiment. System 100 includes an encoder 102 and a decoder 104, wherein encoder 102 is in communication with decoder 104 via a network 110 (e.g., the Internet or other network). Encoder 102 encodes a source video sequence 101 into a bitstream comprising an encoded video sequence and transmits the bitstream to decoder 104 via network 110. In some embodiments, encoder 102 is not in communication with decoder 104, and, in such an embodiment, rather than transmitting bitstream to decoder 104, the bitstream is stored in a data storage unit. Decoder 104 decodes the pictures included in the encoded video sequence to produce video data for display and / or further image processing (e.g. a machine vision task). Accordingly, decoder 104 may be part of a device 103 having an image processor 105 and / or a display 106. The image processor 105 may perform machine vision tasks on the decoded pictures. One such machine vision task may be identifying objects in the picture. The device 103 may be a mobile device, a set-top device, a head-mounted display, or any other device.

[0064] FIG. 2 illustrates functional components of encoder 102 according to some embodiments. It should be noted that encoders may be implemented differently so implementation other than this specific example can be used. Encoder 102 employs a subtractor 241 to produce a residual block which is the difference in sample values between an input block and a prediction block (i.e., the output of a selector 251, which is either an inter prediction block output by an inter predictor 250 (a.k.a., motion compensator) or an intra prediction block output by an intra predictor 249). Then a forward transform 242 is performed on the residual block to produce a transformed block comprising transform coefficients. A quantization unit 243 quantizes the transform coefficients based on a quantization parameter (QP) value (e.g., a QP value obtained based on a picture QP value for the picture in which the input block is a part and a block specific QP offset value for the input block), thereby producing quantized transform coefficients which are then encoded into the bitstream by encoder 244 (e.g., an arithmetic encoder) and the bitstream with the encoded transform coefficients is output from encoder 102. Next, encoder 102 uses the quantized transform coefficients to produce a reconstructed block. This is done by first applying inverse quantization 245 and inverse transform 246 to the transform coefficients to produce a reconstructed residual block and using an adder 247 to add the prediction block to the reconstructed residual block, thereby producing the reconstructed block, which is stored in the reconstruction picture buffer (RPB) 266. Loop filtering by a loop filter (LF) stage 267 is applied and the final decoded picture is stored in a decoded picture buffer (DPB) 268, whereit can then be used by the inter predictor 250 to produce an inter prediction block for the next picture to be processed. LF stage 267 may include any combination of the following three sub-stages: i) a deblocking filter, ii) a sample adaptive offset (SAO) filter, and iii) an Adaptive Loop Filter (ALF).

[0065] FIG. 3 illustrates functional components of decoder 104 according to some embodiments. It should be noted that decoder 104 may be implemented differently so implementations other than this specific example can be used. Decoder 104 includes a bitstream parser 301 that receives the bitstream (e.g., retrieves the bitstream form a file or receives the bitstream via network 110) and parses the bitstream to obtain each NAL unit in the bitstream. When bitstream parser 301 obtains a NAL unit comprising encoded picture sample values (e.g., the encoded quantized transform coefficient values) bitstream parser 301 provides the encoded sample values to a decoder module 361 (e.g., a function that performs arithmetic decoding) that decodes the sample values to obtain decoded sample values (e.g., the quantized transform coefficient values). Decoder 104 also includes a reconstruction stage 398 in which the quantized transform coefficient values are subject to an inverse quantization process 362 and inverse transform process 363 to produce a residual block. This residual block is input to adder 364 that adds the residual block and a prediction block output from selector 390 to form a reconstructed block. Selector 390 either selects to output an inter prediction block or an intra prediction block. The reconstructed block is stored in a RPB 365. The inter prediction block is generated by the inter prediction module 350 and the intra prediction block is generated by the intra prediction module 369. Following the reconstruction stage 398, a loop filter stage 367 applies loop filtering and the final decoded picture may be stored in a decoded picture buffer (DPB) 368 and output to image processor 105. Pictures are stored in the DPB for two primary reasons: 1) to wait for picture output and 2) to be used for reference when decoding future pictures.

[0066] As described above, a challenge presently exists because, in the current specification of SEI RBSP in VVC, the payload size of a message, such as the RBSP payload size of an SEI message, needs to be signalled prior to the message’s payload.

[0067] Accordingly, this disclosure provides for, in one aspect, that the payload size, such as length, is signalled conditionally. That is, the payload size is not necessarily always signalled. In one embodiment, if the pay load size is not signalled, the payload size is derived.

[0068] In one embodiment, a syntax element is used to signal whether or not the payload size is signalled. For instance, when the syntax element has a first value, then this specifies that the payload size is signalled, and when the syntax element has a second value, then this specifies that the payload size is not signalled.

[0069] In one embodiment, the conditional signalling of the pay load size of a message, such as an SEI message, is based on the pay load type of the message. For instance, in one embodiment, the payload size for a first message is signalled if the payload type of the first message is equal to a first payload type, whereas the payload size for a second message is not signalled if the payload type of the second message is equal to a second payload type.

[0070] In one embodiment, the conditional signalling of the pay load size of a message is based on whether or not the message has a fixed payload size. For instance, in one embodiment, the payload size is not signalled for messages that have a fixed payload size.

[0071] In one embodiment, in the case that a NAL unit includes more than one message, then, instead of signalling the payload size for each message, one or more codewords are included in the NAL unit to separate the messages.

[0072] Implementation Options

[0073] 1. Payload Size Flag

[0074] As noted above, in one embodiment, for some messages, the payload size of the message is signalled, but, for some other messages, the payload size of the message is not signalled. For example, for a NAL unit having a first type of SEI message, information for specifying the size of the payload of the SEI message of the first type is included in the NAL unit, whereas, for a NAL unit having a second type of SEI message, no information for specifying the size of the payload of the SEI message of the second type is included in the NAL unit.

[0075] In one embodiment, the RBSP payload size of an SEI message is signalled depending on the group to which the SEI message belongs. Examples of such grouping may be essential SEIs and non-essential SEIs with one or more SEI types in a group. Essential SEIs may be defined as SEIs which are essential for a particular use case, and non-essential SEIs may be defined as SEIs which are not essential for a particular use case. Grouping of SEIs may be specified in a technical specification, may depend on the application, or may besubject to an encoder decision. The grouping of the SEIs into those that will have the SEI payload size signalled and those that SEI payload size is not signalled for may be static, for instance being specified in a technical specification, or being dynamic, for instance being signalled in the APS or another parameter set.

[0076] In one embodiment, the RBSP payload size for an SEI message (or “SEI” for short) is not signalled if the SEI is a large SEI. A large SEI may be defined for instance as an SEI having an RBSP payload size larger than a certain number of bytes, such as, for example, 256 bytes.

[0077] In one embodiment, when the RBSP payload size of an SEI is not signalled in the NAL unit containing the SEI, the decoder can assume that there is only one SEI message in the NAL unit. In contrast, in one embodiment, when the RBSP payload size of an SEI is signalled in the NAL unit containing the SEI, there may be more than one SEI in the NAL unit. In one version of this embodiment, it is an encoder decision to bundle the SEI messages together in one NAL unit or not, e.g. by doing a bit cost optimization. In one embodiment there are more than one SEI message in the NAL unit and a single syntax element in the NAL unit is present where one value specifies that payload sizes are signalled for each SEI message in the NAL unit and another value specifies that there are no payload sizes signalled for any SEI message in the NAL unit.

[0078] Table 13 below illustrates an SEI according to one embodiment.TABLE- 13

[0079] In this embodiment, payload_size_flag equal to 0 specifies that the payload size is not signalled and there is only one SEI payload in this NAL unit. payload_size_flag equal to 1 specifies that the payload size is signalled and there may be more than one SEI payload in this NAL unit.

[0080] In one version, the payload size flag having a first value, e.g. 0, specifies that no payload size byte syntax element is present in the SEI message. In one version the payload_size_flag having a second value, e.g. 1, specifies that one or more payload_size_byte syntax elements are present in the SEI message.

[0081] In one version when the payload size flag has a certain value, e.g. 0, the SEI message shall be the only SEI message in the NAL unit.

[0082] Table 14 below illustrates an SEI according to another embodiment.TABLE- 14

[0083] In this example, payload_size_flag equal to 0 specifies that no payload size byte syntax element is present in the SEI message, payload size flag equal to 1 specifies that one or more payload size byte syntax elements are present in the SEI message. When payload_size_flag is equal to 0, the SEI message shall be the only SEI message in the NAL unit.

[0084] In this example, when payload size flag is equal to 0, payloadSize is derived as NumBytesInNalUnit - 3, where NumBytesInNalUnit is the number of bytes for the current NAL unit (here the NAL unit containing the SEI) and -3 specifies that the two bytes for the NAL unit header and the one byte for the payload type byte should not be counted.

[0085] In another version, a bit of the first byte of the sei message () is used for the payload size flag, and the other bits may be reserved or used for other purposes. This version of the proposed solution has the advantage of keeping the sei_message () byte aligned.

[0086] In the following example shown in Table 15 built on top of the syntax from the JVET-AG0051 proposal where anewNAL unit type is used for SEIs, one of the reserved bits in the first byte of the vsei_rbsp () is used to signal the payload size flag.TABLE- 15

[0087] In this example, vsei_payload_size_flag equal to 0 specifies that no payload_size_byte syntax element is present in the SEI message or NAL unit. payload size flag equal to 1 specifies that one or more payload size byte syntax elements are present in the SEI message(s) or NAL unit. When vsei_payload_size_flag is equal to 0, the SEI message shall be the only SEI message in the NAL unit containing the SEI message.TABLE- 16

[0088] Note that in this example there is a -4 instead of -3 because there is one byte specified in the vsei_rbsp() syntax structure. Alterantively, the payloadSize could be derived as payloadSize = NumBytesInNalUnit - 3 - payloadType / 256.

[0089] In another version of the embodiment the first byte in the sei message () is shared between the payload size flag and the first payload type part, which would then only be 7 bits instead of 8 bits. This version of the proposed solution also has the advantage of keeping the sei_message () byte aligned. This is illustrated in the following example shown in Table 17.TABLE- 17

[0090] In this example, payload_size_flag equal to 0 specifies that no payload size byte syntax element is present in the SEI message, payload size flag equal to 1 specifies that one or more payload size byte syntax elements are present in the SEI message. When payload_size_flag is equal to 0, the SEI message shall be the only SEI message in the NAL unit.

[0091] payload_type_7_bits is a value of the payload type of an SEI message.

[0092] In the example, when payload_type_7_bits is not 0x7F the payloadType is between 0 and 126 and signaled with one byte. In another version of the example, the meaning of the payload_type_7_bits is switched such that when payload_type_7_bits is not 0x7F the payloadType is between 128 and 254 instead of between 0 and 126. Then only one byte would be needed when the payloadType is between 128 and 254. This may be advantageuos when most SEI messages has a payload format between 128 and 254, such as for VVC.

[0093] In one embodiment, a decoder may perform all or a subset of the following steps:

[0094] Step 1 : obtain information from a NAL unit, wherein the information specifies that the NAL unit is an SEI NAL unit, i.e., that the NAL unit comprises a first SEI message;

[0095] Step 2: determine whether the first SEI message includes at least a first size syntax element for use in specifying the size of the payload of the first SEI message;

[0096] Step 3: as a result of determining that the first SEI message includes the first size syntax element, go to Step 4, otherwise skip Steps 4 and 5 and go to Step 6;

[0097] Step 4: decode the first size syntax element of the first SEI message to produce a first decoded size value;

[0098] Step 5: use the first decoded size value to determine the size of the payload of the first SEI message and then go to Step 7 (skip Step 6);

[0099] Step 6: determine the size of the payload of the first SEI message based on a value specifying the size of the SEI NAL unit; and

[0100] Step 7 : decode the rest of the SEI NAL unit. For example, in one embodiment this step is performed using the determined size of the of the pay load to decode the rest of the SEI NAL unit.

[0101] In another embodiment, the decoder may perform all or a subset of the following steps:

[0102] Step 1 : obtain information from a NAL unit, wherein the information specifies that the NAL unit is an SEI NAL unit, i.e., that the NAL unit includes at least a first SEI message;

[0103] Step 2: decode a payload size flag syntax element included in the SEI NAL unit to produce a payload size flag value;

[0104] Step 3: determine whether the payload size flag value is equal to a certain value (e.g., 1).

[0105] Step 4: as a result of determining that the payload size flag value is equal to the certain value, go to Step 5, otherwise go to Step 7;

[0106] Step 5: decode a first size syntax element from the SEI NAL unit to produce a first decoded size value;

[0107] Step 6: use the first decoded size value to determine the size of the payload of the first SEI message and then go to Step 8 (skip Step 7);

[0108] Step 7: determine the size of the pay load of the first SEI message based on a value specifying the size of the SEI NAL unit; and

[0109] Step 8: decode the rest of the SEI NAL unit. In one variant, using the determined size of the of the payload to decode the rest of the SEI NAL unit.

[0110] In one embodiment, an SEI NAL unit comprises NAL unit header syntax elements, followed by SEI payload format syntax elements, followed by SEI message syntax elements. The SEI message syntax element may be the SEI payload as described above.

[0111] 2. Conditional Signalling of Payload Size Based on Payload Type

[0112] In one embodiment, the conditional signalling of the pay load size of a message, such as an SEI message is based on the payload type of the SEI message. In one version of this embodiment, the pay load size is signalled only if the pay load type of the SEI message is a payload type included in a particular set of one or more payload types. Forexample, if the payload type of the SEI message is one of the legacy payload types, then the payload size for this SEI message will be signalled in the SEI message.

[0113] One example of this embodiment is illustrated in Table 18 below. In this example, the condition for signalling the payload size is based on the type of the SEI message in a way that the payload size of the SEI message is signalled if the pay load type of the SEI message is on a list (or in a set or a look up table), and no payload size syntax elements are present if the pay load type is not on the list. In one specific example the list is named LegecyPayloadTypes and it contains the types 0, 1, 3, 4, 5, 9, 45, 129, 130, 133, 137, 144, 145, 147, 148, 149, 150, 153, 154, 155, 156, 168, 203, 204, 205, 206, 207, 208, 209, 211 and 212. These are the payload types of the SEI messages in VVC v3. In this embodiment, the SEI message includes a payload size syntax element if the value of the payload type byte is included in the list. For instance, if the list includes payload types of n SEI messages, the syntax table will not have any payload size syntax element for any of those n SEI messages and hence the design will be backwards compatible with existing design which does not contain such a condition. That is, in this example, if the pay load type is one of the existing SEI messages, then the payload size is signalled, and the payload size is not signalled for SEI messages with payload types not on the list. This example has the benefit of being backwards compatible with the current VSEI version.TABLE- 18

[0114] payload_type_byte is a byte of the payload type of an SEI message. The variable payloadType is set equal to the value of payload type byte.

[0115] LegecyPayloadTypes = {0 1 3 4 5 945 129 130 133 137 144 145 147 148 149 150 153 154 155 156 168 203 204205 206207 208 209211 212}

[0116] payload_size_byte is a byte of the payload size of an SEI message.

[0117] In another version of this embodiment, the condition on the payload types is a function of the payload type byte syntax value. In one variant of this version of the embodiment, the condition is defined as a bitwise operation on the payload byte type value. In one example, bit zero (bO) to bit seven (b7) is the 8 bit representation of the payload type byte syntax element, and the condition for signalling the payload size is true when !( b5 || ( bO && bl && b4 )) has a value equal to 1, where ! is the negation operation, || is the logic or and && is the logic and operation. A syntax table for this version is shown below in Table 19.TABLE- 19

[0118] 3. Duplicated Payload Types for Backwards Compatibility

[0119] In one embodiment, an SEI message having a particular type (e.g., buffering_period) can be associated with either a first pay load type value (e.g., 0) specifying the particular type or a second payload type value (e.g., 70) also specifying the particular type. In this embodiment, the payload size for the SEI message of the particular type is signalled if the payload type value associated with the SEI message is equal to the first payload type value, and the pay load size for the SEI message is not signalled if the pay load type value associated with the SEI message is equal to the second payload type value.

[0120] In one version of this embodiment, the second payload type value is defined as a function of first payload type value. In one example, the second payload type value is equal to 255 minus the first pay load type value, so for instance if the first payload type value is 0, or 0000 0000 in binary format, the second payload type value is 255 - 0 = 255, or 1111 1111 in binary format. In another example, the second payload value is defined such that it does not overlap with any payload types of VVC v3. For example, the second payload type value is equal to 240 minus the first payload type value.

[0121] In one embodiment, a function can be used to determine whether the payload type value signalled for an SEI message is included in a first set of payload type values (e.g., the set of pay load type values defined in VVC v3) or a second set of pay load type values (e.g., a set of payload type values defined in VVC v4), wherein the two sets of payload type values are disjoint. The function may be defined as: value = func(payloadType), where payloadType is the payload type value signalled for the SEI message. The function may be used to determine whether the SEI message has the RBSP payload size signalled or not. For example, if the function returns a value of x (e.g., 0), this indicates that the payload size is signalled, but if the function returns a value of y (e.g., 1), this indicates that the payload size is not signalled.

[0122] In one version the function is implemented using a look-up table. In one version, for each defined SEI message type there is a first payload type value for the SEI message where the RBSP payload size is signalled and a second payload type value for the SEI message where the RBSP payload size is not signalled

[0123] For a new version v4 of VVC this could be expressed with the Table 20 shown below, where “payloadType of SEI w / payload size” indicate the payloadTypes that are also present in v3 of VVC for which the payload size is signalled and “payloadType of SEI w / o payload size” indicate the payloadTypes that are new for v4 and have the same payload type as the corresponding SEI message in v3 of VVC but where the payload size is not signalled. Table 20 shows the pairing of these payload type values for each SEI message. For instance, SEI messages with payloadType 137 and 83 carries the same SEI payload type and SEI messages with payloadType 205 and 105 carries the same SEI payload type. VVC v4 could potentially also define new SEI messages for which two payloadTypes could be used, one where the payload size is signalled and one where the payload size is not signalled.TABLE-20

[0124] The example could be expressed by the following syntax tables shown in Table 21, Table 22, and Table 23.TABLE-21TABLE-22TABLE-23

[0125] Tables 21 and 22 may alternatively be replaced by the following syntax table.TABLE-24

[0126] In one embodiment, the function may return a vector of two values, e.g., the function may be defined as: (value 1, value2) = func(payloadType), where value 1 specifieswhether or not the value payloadType is included in the first set of payload type values, and, if the value payloadType is not included in the first set of payload type values, then value2 is the payload type value from the first set of payload type values to which the value payloadType is mapped. As a specific example, if the value of payloadType is 73, then func(73) returns a first value (e.g., 0) indicating that 73 is not included in the first set of payload type values and returns a second value (i. e. , 4) which is the value from the first set of payload type values to which, as shown in Table 20, the value 73 is mapped. As another example, func(4) returns a first value (e.g., 1) indicating that 4 is included in the first set of payload type value and the second value returned can be ignored.

[0127] In one version, when no payload size syntax element is present in the SEI message, the SEI message shall be the only SEI message in the NAL unit. For instance, when a nesting SEI message is used that groups multiple SEI messages in a NAL unit, the RBSP payload size for each SEI message in the nested SEI message needs to be signalled in order to parse the SEI messages correctly.

[0128] In one embodiment, the function is defined as one or more bit operators of the payloadType. For example, in one embodiment, the function is defined as: value = func(payloadType) = payloadType & X, where & is a bitwise AND operator and X is an integer value. In another embodiment, the function is defined as: value = func(payloadType) = (payloadType & X) && (payloadType & Y), where & is a bitwise AND operator, && is a logical AND operator and X and Y are integers. The inverse may be expressed as: inv_value = ! ((payloadType & X) && (payloadType & Y)) or as: inv_value = ! ((payloadType & X)) || ! ((payloadType & Y)), where || is a logical OR operator.

[0129] If X = 32 and Y = 64, then when func(payloadType) returns a value of 0 or an inv value of 1 this will indicate that the payloadType is defined in v3 of VVC. This payloadType would require the RBSP payload size to be read from the SEI NAL unit. In one version a payloadType for which the value is 1 (or the inv value is 0), is used for a following version of VVC, e.g., VVC v4, for which the RBSP payload size of the SEI message is not read from the SEI message NAL unit.

[0130] The table below shows the defined SEI payloadTypes in VVC v3, the corresponding binary representation and the values of value and inv value. The bottom SEI payloadType 243 is an example of anew payloadType added for a following VVC version. The SEI with payloadType 243 may for instance correspond to the SEI message ofpayloadType 211, the NN postfiler characteristics SEI message. Note that there is not possible to have a new unique equivalent payloadType for each v3 SEI message payloadType using the ( X & 64 ) && ( X & 32 ) or !((x & 64)) || !((x & 32)) expressions of the table.TABLE-25

[0131] How to derive whether to decode the RBSP payload size from the SEI message NAL unit is exemplified with the following syntax shown in Table 26.TABLE-26

[0132] payload_size_byte is a byte of the payload size of an SEI message. When no payload_size_byte syntax element is present in the SEI message, the SEI message shall be the only SEI message in the NAL unit.

[0133] The expression “!((payloadType & 64)) || !(( payloadType & 32))” may alternatively be written "(payloadType & 96) != 96”, which is an equivalent rewriting of the expression.

[0134] In yet another version of this embodiment, the first payload type byte, when set to 255 indicate that the payload size is not signalled for the SEI message. In other words, SEI messages with payloadTypes from 0 to 254 is signalled with payload size and SEI messages with payloadTypes above 254 is signalled without payload size. This is illustrated with the syntax below.TABLE-27

[0135] 3.5 One flag for multiple SEI messages

[0136] In one embodiment, there is one syntax element per SEI NAL unit that specifies whether there is an SEI payload size syntax element for each SEI message in the NAL unit or not. As an example, the one syntax element may be a 1 -bit flag, a flag value of 0 means that there are no SEI payload size syntax elements in the SEI NAL unit, and a flag value of 1 means that there are SEI payload size syntax elements in the SEI NAL unit.

[0137] In one variant, when the one syntax element specifies that there is not any SEI payload size syntax elements in the SEI NAL unit (the exemplary flag value is equal to 0), there is only one SEI message in the SEI NAL unit. A decoder may in this variant decode the one syntax element and based on the one syntax element value determine that there is only one SEI message in the SEI NAL unit.

[0138] Exemplary syntax and semantics for this embodiment is shown below in Table 28, table 29, and table 30TABLE-28TABLE-29TABLE-30

[0139] In the exemplary syntax above, the sei_rbsp() syntax follows a NAL unit header that specifies that the NAL unit is a versatile SEI message NAL unit.

[0140] A decoder first decodes the vsei_payload_size_flag syntax element.

[0141] If the value of the decoded vsei_payload_size_flag syntax element is equal to 0, there is only one vsei_message() in the SEI NAL unit, and when the decoder decodes the vsei_message() there is no vsei_payload_size_16bits syntax element present in the NAL unit. The decoder will therefore not decode the payload size of the single SEI message, but may infer the payload size of the single SEI message as equal to the size of the NAL unit minus the number of bytes preceding the vsei_payload() syntax structure in the NAL unit.

[0142] If the value of the decoded vsei_payload_size_flag syntax element is equal to 1, a decoder may determine that there may be one or more SEI messages in the SEI NAL unit. When the decoder decodes the one or more vsei_message() syntax structures, there is one vsei_payload_size syntax element present for each SEI message and the decoder decodes the vsei_payload_size syntax elements for all the one or more SEI messages in the SEI NAL unit.

[0143] 4. Static Payload Size

[0144] In one embodiment, the payload size is not signalled for SEI messages that have a fixed payload size. In this embodiment, a flag (e.g., static_payload_size_flag) can be used to indicate whether or not the payload size is static for the SEI message. For instance, if static_payload_size_flag is equal to 1, then the size can be determined from the SEI payload type and the size is not signalled but derived from an SEI size table in the specification, e.g. a DRAP SEI message should always be empty and have the payload size 0. If the static_payload_size_flag is equal to 0, the payload size is signalled. The static_payload_size_flag may be signalled in the bitstream, e.g. in the SEI message, or be derived based on the value of the SEI payload type.

[0145] 5. Dividers to Separate Multiple SEI messages in a NAL Unit

[0146] In this embodiment, instead of signalling the payload size of the individual SEI messages for a NAL unit with multiple SEI messages, a codeword is signalled between the SEI messages, where the codeword is used as a separator between the SEI messages. The codeword is a unique bit pattern that cannot be mixed with other signalling of the SEI messages. The codeword may be a start code or a separator code.

[0147] FIG. 4 is a flowchart illustrating a process 400 for obtaining from a bitstream a first pay load of a first message, wherein the first pay load of the first message has a size. Process 400 may begin in step s402.

[0148] Step s402 comprises obtaining from the bitstream information regarding the first pay load of the first message and / or obtaining from the bitstream information regarding the first message.

[0149] Step s404 comprises determining, based on the obtained information, whether the first message includes at least a first size syntax element for use in specifying the size of the first payload of the first message.

[0150] If it is determined that the first message includes at least the first size syntax element, then process 400 further includes using at least the first size syntax element to obtain a first payload size value specifying the size of the first payload of the first message (step s406), otherwise process 400 includes using other information to obtain the first payload size value (step s408). For example, if the S is a variable for representing the size of the first payload and SEI is the first syntax element, such as a byte of the bitstream, then using SEI to obtain the first payload size value may comprise setting S equal to SEI. If the message includes a second size syntax element (SE2), such as another byte of the bitstream, then obtaining the payload size value comprises setting S = S + SE2.

[0151] In some embodiments, the method comprises obtaining from the bitstream information regarding the first message, and the information regarding the first message is a second syntax element (e.g., a one bit flag or a sequence of bits) specifying whether the first message includes the first size syntax element.

[0152] In some embodiments, the obtained information is a second syntax element (e.g., a one bit flag or a sequence of bits) specifying whether the first message includes the first size syntax element.

[0153] In some embodiments, the second syntax element has a length of 1 bit.

[0154] In some embodiments, the second syntax element is included in the first message.

[0155] In some embodiments, the bitstream is or comprises a network abstraction layer, NAL, unit, and the first message is a raw byte sequence payload, RBSP, of the NAL unit.

[0156] In some embodiments, the first payload of the first message has a pay load type, the method comprises obtaining from the bitstream information regarding the first message, the information regarding the first message comprises a first type value specifying the pay load type of the first pay load.

[0157] In some embodiments, the first payload of the first message has a pay load type, the method comprises obtaining from the bitstream information regarding the first message, the information regarding the first message comprises one or more type syntax elements, and the method further comprises using the one or more type syntax elements to obtain a first type value specifying the payload type of the first payload.

[0158] In some embodiments, determining, based on the obtained information, whether the first message includes at least a first size syntax element comprises: determining whether the first type value is included in a particular set of values.

[0159] In some embodiments, determining whether the first type value is included in a particular set of values comprises using a function or a look up table taking the first type value as an input.

[0160] In some embodiments, the bitstream further comprises a second message comprising a pay load, the pay load of the second message has the same type as the first payload of the first message, the method further comprises obtaining, using one or more syntax elements in the bitstream, a second type value specifying the payload type of the payload of the second message, the first type value specifying the first message’s payload’s type is different than the second type value specifying the second message’s payload’s type even though the payload of the second message has the same type as the first payload of the first message.

[0161] In some embodiments, it is determined that the first message does not include the first size syntax element, and the method further comprises determining the size of the first payload of the first message based on the first type value.

[0162] In some embodiments, the bitstream comprises a sequence of network abstraction layer, NAL, units, the sequence of NAL units comprises a first NAL unit, wherein the first NAL unit is not necessarily the first NAL unit in the sequence, the first NAL unit comprises the first message, and obtaining the information from the bitstream comprises obtaining the information from the first NAL unit.

[0163] In some embodiments, it is determined that the first message does not include the first size syntax element, and the method further comprises using information obtained from the first NAL unit to obtain the first pay load size value specifying the size of the first payload of the first message.

[0164] In some embodiments, the information obtained from the first NAL unit is information specifying the size of the first NAL unit.

[0165] In some embodiments, it is determined that the first message does not include the first size syntax element, and the other information used to obtain the first payload size value comprises information indicating that the first message is the only message in the first NAL unit.

[0166] In some embodiments, it is determined that the first message does not include the first size syntax element, and the other information used to obtain the first payload size value comprises information about the size of the first NAL unit. In some embodiments, the other information used to obtain the first payload size value further comprises information indicating that the first message is the only message in the first NAL unit.

[0167] In some embodiments, the method further comprises obtaining NAL unit type information for the first NAL unit, the NAL unit type information specifies that the first NAL unit is a supplemental enhancement information, SEI, NAL unit, and the first message is an SEI message.

[0168] In some embodiments, the bitstream comprises a sequence of network abstraction layer, NAL, units, the sequence of NAL units comprises a first NAL unit (e.g., a SPS NAL unit) and a second NAL unit (e.g., SEI NAL unit) following the first NAL unit in the sequence of NAL units, wherein the first NAL unit is not necessarily the first NAL unit in the sequence, the second NAL unit comprises the first message, and obtaining the information from the bitstream comprises obtaining the information from the first NAL unit.

[0169] In some embodiments, the process also includes obtaining the first payload of the first message using the first payload size value.

[0170] FIG. 5 is a flowchart illustrating a process 500 for obtaining from a bitstream a first pay load of a first message, wherein the first pay load of the first message has a size.Process 500 may begin in step s502.

[0171] Step s502 comprises obtaining type information for a NAL unit included in the bitstream, wherein the type information specifies that the NAL unit is a supplemental enhancement information, SEI, NAL unit, wherein the SEI NAL unit comprises a first SEI message having a payload, wherein the payload consists of an ordered sequence of N bits, where N > 0.

[0172] Step s504 comprises determining the last bit in the ordered sequence of N bits, wherein determining the last bit in the ordered sequence of N bits comprises locating a code pattern in the SEI NAL unit, wherein the last bit in the ordered sequence of bits immediately precedes the code pattern in the SEI NAL unit. In some embodiments, the code pattern is start code or separator code.

[0173] FIG. 6 is a flowchart illustrating a process 600 for obtaining from a bitstream a first pay load of a first message, wherein the first pay load of the first message has a size.Process 600 may begin in step s602.

[0174] Step s602 comprises obtaining a first message comprising a first payload having a size and a type.

[0175] Step s604 comprises determining whether to include in a bitstream one or more size syntax elements specifying the size of the first pay load of the first message. The determination is based on at least one of the following: an encoder profile; an encoder configuration; an encoder setting; the type of the payload; the size of the payload; or whether the first message is the only message that will be included in a network abstraction layer, NAL, unit containing the first message.

[0176] FIG. 7 is a block diagram of an apparatus 700 for implementing encoder 102, decoder 104, and / or network node 1104, according to some embodiments. When apparatus 700 implements encoder 102, apparatus 700 may be referred to as an encoder apparatus, when apparatus 700 implements decoder 104, apparatus 700 may be referred to as a decoder apparatus, and when apparatus 700 implements network node 1104, apparatus 700may be referred to as a network node. As shown in FIG. 7, apparatus 700 may comprise: processing circuitry (PC) 702, which may include one or more processors (P) 755 (e.g., one or more general purpose microprocessors and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (i.e., encoder apparatus 700 may be a distributed computing apparatus); at least one network interface 748 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 745 and a receiver (Rx) 747 for enabling apparatus 700 to transmit data to and receive data from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network) to which network interface 748 is connected (physically or wirelessly) (e.g., network interface 748 may be coupled to an antenna arrangement comprising one or more antennas for enabling encoder apparatus 700 to wirelessly transmit / receive data); and a storage unit (a.k.a., “data storage system”) 708, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 702 includes a programmable processor, a computer readable storage medium (CRSM) 742 may be provided. CRSM 742 may store a computer program (CP) 743 comprising computer readable instructions (CRI) 744. CRSM 742 may be a non- transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 744 of computer program 743 is configured such that when executed by PC 702, the CRI causes encoder apparatus 700 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, encoder apparatus 700 may be configured to perform steps described herein without the need for code. That is, for example, PC 702 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0177] Summary of Various Embodiments

[0178] Al. A method for obtaining from a bitstream a first payload of a first message, wherein the first pay load of the first message has a size, the method comprising: obtaining from the bitstream information regarding the first payload of the first message and / or obtaining from the bitstream information regarding the first message; determining, based on the obtained information, whether the first message includes at least a first size syntaxelement for use in specifying the size of the first payload of the first message; and if it is determined that the first message includes at least the first size syntax element, then using at least the first size syntax element to obtain a first pay load size value specifying the size of the first payload of the first message, otherwise using other information to obtain the first payload size value.

[0179] A2a. The method of embodiment Al, wherein the method comprises obtaining from the bitstream information regarding the first message, and the information regarding the first message is a second syntax element (e.g., a one bit flag or a sequence of bits) specifying whether the first message includes the first size syntax element.

[0180] A2b. The method of embodiment Al, wherein the obtained information is a second syntax element (e.g., a one bit flag or a sequence of bits) specifying whether the first message includes the first size syntax element.

[0181] A3. The method of embodiment A2a or A2b, wherein the second syntax element has a length of 1 bit.

[0182] A4. The method of embodiment A2a, A2b, or A3, wherein the second syntax element is included in the first message.

[0183] A5. The method of embodiment A4, wherein the bitstream is or comprises a network abstraction layer, NAL, unit, and the first message is a raw byte sequence payload, RBSP, of the NAL unit.

[0184] A6. The method of embodiment Al, wherein the first payload of the first message has a payload type, the method comprises obtaining from the bitstream information regarding the first message, the information regarding the first message comprises a first type value specifying the payload type of the first payload.

[0185] A7. The method of embodiment Al, wherein the first payload of the first message has a payload type, the method comprises obtaining from the bitstream information regarding the first message, the information regarding the first message comprises one or more type syntax elements, and the method further comprises using the one or more type syntax elements to obtain a first type value specifying the payload type of the first payload.

[0186] A8. The method of embodiment A6 or A7, wherein determining, based on the obtained information, whether the first message includes at least a first size syntax element comprises: determining whether the first type value is included in a particular set of values.

[0187] A9. The method of embodiment A8, wherein determining whether the first type value is included in a particular set of values comprises using a function or a look up table taking the first type value as an input.

[0188] A10. The method of any one of embodiment A6-A9, wherein the bitstream further comprises a second message comprising a payload, the pay load of the second message has the same type as the first payload of the first message, the method further comprises obtaining, using one or more syntax elements in the bitstream, a second type value specifying the payload type of the payload of the second message, the first type value specifying the first message’s payload’s type is different than the second type value specifying the second message’s payload’s type even though the payload of the second message has the same type as the first pay load of the first message.

[0189] All. The method of any one of embodiment A6-A10, wherein it is determined that the first message does not include the first size syntax element, and the method further comprises determining the size of the first payload of the first message based on the first type value.

[0190] A12. The method of any one of embodiments Al-Al 1, wherein the bitstream comprises a sequence of network abstraction layer, NAL, units, the sequence of NAL units comprises a first NAL unit, wherein the first NAL unit is not necessarily the first NAL unit in the sequence, the first NAL unit comprises the first message, and obtaining the information from the bitstream comprises obtaining the information from the first NAL unit.

[0191] Al 3. The method of embodiment Al 2, wherein it is determined that the first message does not include the first size syntax element, and the method further comprises using information obtained from the first NAL unit to obtain the first payload size value specifying the size of the first payload of the first message.

[0192] A14. The method of embodiment Al 3, wherein the information obtained from the first NAL unit is information specifying the size of the first NAL unit.

[0193] A15a. The method of embodiment A12, wherein it is determined that the first message does not include the first size syntax element, and the other information used to obtain the first payload size value comprises information indicating that the first message is the only message in the first NAL unit.

[0194] Al 5b. The method of embodiment Al 2, wherein it is determined that the first message does not include the first size syntax element, and the other information used to obtain the first payload size value comprises information about the size of the first NAL unit.

[0195] A15c. The method of embodiment A15b, wherein the other information used to obtain the first payload size value further comprises information indicating that the first message is the only message in the first NAL unit.

[0196] A16. The method of any one of embodiments A12-A15c, wherein the method further comprises obtaining NAL unit type information for the first NAL unit, the NAL unit type information specifies that the first NAL unit is a supplemental enhancement information, SEI, NAL unit, and the first message is an SEI message.

[0197] A17. The method of any one of embodiments Al-Al 1, wherein the bitstream comprises a sequence of network abstraction layer, NAL, units, the sequence of NAL units comprises a first NAL unit (e.g., a SPS NAL unit) and a second NAL unit (e.g., SEI NAL unit) following the first NAL unit in the sequence of NAL units, wherein the first NAL unit is not necessarily the first NAL unit in the sequence, the second NAL unit comprises the first message, and obtaining the information from the bitstream comprises obtaining the information from the first NAL unit.

[0198] A18. The method of any one of embodiments A1-A17, further comprising obtaining the first pay load of the first message using the first pay load size value.

[0199] Al 9. The method of embodiment A6 wherein, if the value of the second syntax element is equal to a first value, then there is at most one message in the NAL unit and the first size syntax element is not present in the NAL unit.

[0200] A20. The method of embodiment Al 9, wherein, if the value of the second syntax element is equal to the first value, then the size of the first pay load is set to the size of the NAL unit minus the number of bytes preceding the first payload of the first message in the NAL unit.

[0201] A21. The method of embodiment A6 wherein, if the value of the second syntax element is equal to a second value, then there is at least one message in the NAL unit, and the NAL unit includes a payload size syntax element for each message in the NAL unit.

[0202] Bl . A method for processing a bitstream comprising a sequence of network abstraction layer, NAL, units, the method comprising: obtaining type information for a NALunit included in the bitstream, wherein the type information specifies that the NAL unit is a supplemental enhancement information, SEI, NAL unit, wherein the SEI NAL unit comprises a first SEI message having a payload, wherein the payload consists of an ordered sequence of N bits, where N > 0; and determining the last bit in the ordered sequence of N bits, wherein determining the last bit in the ordered sequence of N bits comprises locating a code pattern in the SEI NAL unit, wherein the last bit in the ordered sequence of bits immediately precedes the code pattern in the SEI NAL unit.

[0203] B2. The method of embodiment Bl, wherein the code pattern is start code or separator code.

[0204] CL A method performed by an encoder, the method comprising: obtaining a first message comprising a first payload having a size and a type; and determining whether to include in a bitstream one or more size syntax elements specifying the size of the first payload of the first message, wherein the determination is based on at least one of the following: an encoder profile; an encoder configuration; an encoder setting; the type of the payload; the size of the payload; or whether the first message is the only message that will be included in a network abstraction layer, NAL, unit containing the first message.

[0205] DI. A computer program (743) comprising instructions (744) which when executed by processing circuitry (702) of an apparatus (700) causes the apparatus to perform the method of any one of the above embodiments.

[0206] D2. A carrier containing the computer program of embodiment DI, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (742).

[0207] El . A decoder apparatus (700) configured to perform a method for obtaining from a bitstream a first pay load of a first message, wherein the first pay load of the first message has a size, and the method comprises: obtaining from the bitstream information regarding the first payload of the first message and / or obtaining from the bitstream information regarding the first message; determining, based on the obtained information, whether the first message includes at least a first size syntax element for use in specifying the size of the first payload of the first message; and if it is determined that the first message includes at least the first size syntax element, then using at least the first size syntax element to obtain a first payload size value specifying the size of the first payload of the first message, otherwise using other information to obtain the first payload size value.

[0208] E2. The decoder apparatus of embodiment El, wherein the decoder apparatus is further configured to perform the method of any one of embodiments A1-A18.

[0209] Fl . A decoder apparatus (700) configured to perform a method for processing a bitstream comprising a sequence of network abstraction layer, NAL, units, wherein the method comprises: obtaining type information for a NAL unit included in the bitstream, wherein the type information specifies that the NAL unit is a supplemental enhancement information, SEI, NAL unit, wherein the SEI NAL unit comprises a first SEI message having a payload, wherein the payload consists of an ordered sequence of N bits, where N > 0; and determining the last bit in the ordered sequence of N bits, wherein determining the last bit in the ordered sequence of N bits comprises locating a code pattern (e.g., a start code or a separator code) in the SEI NAL unit, wherein the last bit in the ordered sequence of bits immediately precedes the code pattern in the SEI NAL unit.

[0210] Gl. An encoder apparatus (700) configured to perform a method comprising: obtaining a first message comprising a first payload having a size and a type; and determining whether to include in a bitstream one or more size syntax elements specifying the size of the first payload of the first message, wherein the determination is based on at least one of the following: an encoder profile; an encoder configuration; an encoder setting; the type of the payload; the size of the payload; or whether the first message is the only message that will be included in a network abstraction layer, NAL, unit containing the first message.

[0211] Additional Disclosure

[0212] The TuC for future extensions of VSEI (JVET-AG2032) includes two solutions for reducing the payload size signalling for SEI messages, the large SEI message proposed in JVET-AF0148, and the versatile SEI RBSP / versatile SEI message solution proposed in JVET-AG0051.

[0213] The large SEI message uses 16 bits for the payload size to be used for large SEI messages instead of the mechanism used in the sei_message() that uses one byte for signalling the payload size for each 255 bytes of payload. The large SEI message in the TuC has the following syntax:TABLE-31

[0214] The versatile SEI RBSP solution omits the signalling of the pay load size in the RBSP, with the motivation that all other RBSPs of VVC does not signal the payload size prior to the payload. The versatile SEI RBSP may be carried in a NAL unit. For existing codecs, a new NAL unit type may be added for each of prefix and suffix SEI messages. The versatile SEI RBSP solution in the TuC has the following syntax:TABLE-32TABLE-33

[0215] The large SEI message in the TuC targets reducing the payload size signalling for large SEI messages. However, the overall compression of the bitstream would also benefit from not having to signal the payload size for any SEI message when not needed.

[0216] In VVC the payload size is needed for SEI messages carried in the scalable nesting SEI, where a call is made to the sei_message() struct that includes the payload size bytes. The scalable nesting SEI message is not only useful for reducing the bitrate by not having to send the NAL unit header for each nested SEI message, but is also intended for grouping SEI messages and provides a mechanism to associate SEI messages with specific OLSs, specific layers, or specific sets of subpictures. Versatile SEI messages in the versatile SEI RBSP solution in the TuC would not be supported by the scalable nesting SEI message since the payload size is not signaled.

[0217] Proposal

[0218] This proposal contains four options to conditionally signal the payload size for SEI messages. The three first options are claimed to be backwards compatible with earlier versions of VVC and the fourth option targets the next generation video codec. It is proposed to include the options to the TuC for future extensions of VSEI and to include one of the three first options to a working draft for VVC v4.

[0219] The first option proposes to add a vsei_payload_size_flag to the Versatile SEI RBSP solution in the TuC for future extensions of VSEI. The vsei_payload_size_flag specifies if the pay load size is signalled in the RBSP or not. By allowing the pay load to be signalled in the RBSP, the SEI message may be carried in a scalable nesting SEI message.

[0220] The second option harmonizes the large SEI message and the Versatile RBSP SEI solutions from the TuC. The payload size is signalled with a 2-byte syntax element vsei_payload_size_16bits for all SEI messages in the NAL unit when vsei_payload_size_flag is equal to 1. When vsei_payload_size_flag is equal to 0, the payload size is not signalled and the SEI message shall be the only SEI message in the NAL unit.

[0221] The third option proposes to for each SEI message assign an additional payload type for which the payload size of the SEI message is not signalled in the SEI RBSP. Compared to the versatile SEI RBSP solution in the TuC for future extensions of VSEI no new NAL unit type is needed. In the proposed syntax, an additional payload type has been added for all existing SEI messages, but it can be discussed for which SEI messages an additional payload type should be added, e.g. only for long SEI messages or only for SEI messages that are new to VVC v4.

[0222] The fourth option proposes to add a payload size flag syntax element in the sei_message() that specifies whether the payload size of the SEI message is signalled in the RBSP or not. In this option, the first syntax element for the payload type is signalled with 7 bits.

[0223] Syntax and semantics for the 4 options are shown below with additions on top of existing VVC text or TuC text show in italics and removed text shown between double brackets: [[]].

[0224] Option 1 : Backwards compatible - Payload size flag in versatile SEI RBSPTABLE-34vsei_payload_size_flag equal to 0 specifies that no payload_size_byte syntax element is present in the SEI message. payload_size_flag equal to 1 specifies that one or more payload_size_byte syntax elements are present in the SEI message. When vsei_payload_size_flag is equal to 0, the SEI message shall be the only SEI message in the NAL unit.TABLE-35

[0225] Option 2: Harmonization of large SEI message and versatile SEI RBSPTABLE-36vsei_position indicates if the SEI message corresponds to the PREFIX_SEI_NUT and SUFFIX_SEI_NUT. vsei_position equal 0 indicates that the SEI message is treated as PREFIX_SEI_NUT. vsei_position equal 1 indicates that the SEI message is treated as SUFFIX_SEI_NUT. Values 3 and 4 of vsei_position are reserved for future use and shall be ignored. vsei_relevance indicates the relevance of the SEI message for the target application. vsei_relevance ranges from 0 to 3, with 0 being the least relevant and 3 being the most relevant. vsei_relevance equal to 3 indicates that the versatile SEI message may be a required versatile SEI message. vsei_payload_size_flag equal to 0 specifies that no vsei_payload_size_16bits syntax element is present in the versatile SEI message. vsei_payload_size_flag equal to 1 specifies that one or more vsei_payload_size_16bits syntax elements are present in the SEI message. When vsei_payload_size_flag is equal to 0, the SEI message shall be the only SEI message in the NAL unit.TABLE-37i / se / _payload_size_16bits specifies the payload size in bytes of a versatile SEI message.When vsei_payload_size_16bits is not present it is inferred to be equal to NumBytesInNalUnit - 4.

[0226] Option 3: Backwards compatible - New payload types for not signalling payload sizeTABLE-38payload_size_byte is a byte of the payload size of an SEI message. When no payload_size_byte syntax element is present in the SEI message, the SEI message shall be the only SEI message in the NAL unit.TABLE-39

[0227] Option 4: For next generation codecTABLE-40payload_size_flag equal to 0 specifies that no payload_size_byte syntax element is present in the SEI message. payload_size_flag equal to 1 specifies that one or more payload_size_byte syntax elements are present in the SEI message. When payload_size_flag is equal to 0, the SEI message shall be the only SEI message in the NAL unit. payload_type_7_bits is a value of the payload type of an SEI message.

[0228] While the terminology in this disclosure is described in terms of VVC, the embodiments of this disclosure also apply to any existing or future codec, which may use a different, but equivalent terminology.

[0229] Also, while various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context. As used herein “a” means “at least one” or “one or more”. Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

Claims

CLAIMS1. A method (400) for obtaining from a bitstream a first payload of a first message, wherein the first payload of the first message has a size, the method comprising: obtaining (s402) from the bitstream information regarding the first pay load of the first message and / or obtaining from the bitstream information regarding the first message; determining (s404), based on the obtained information, whether the first message includes at least a first size syntax element for use in specifying the size of the first pay load of the first message; and if it is determined that the first message includes at least the first size syntax element, then using at least the first size syntax element to obtain a first payload size value specifying the size of the first payload of the first message, otherwise using other information to obtain the first payload size value (s406).

2. The method of claim 1 , wherein the method comprises obtaining from the bitstream information regarding the first message, and the information regarding the first message is a second syntax element having a value specifying whether the first message includes the first size syntax element.

3. The method of claim 1, wherein the obtained information is a second syntax element having a value specifying whether the first message includes the first size syntax element.

4. The method of claim 2 or 3, wherein the second syntax element has a length of 1 bit.

5. The method of claim 2, 3, or 4, wherein the second syntax element is included in the first message.

6. The method of any one of claims 2-5, wherein the bitstream is or comprises a network abstraction layer, NAL, unit, andthe first message is a raw byte sequence payload, RBSP, of the NAL unit.

7. The method of claim 6 wherein, if the value of the second syntax element is equal to a first value, then there is at most one message in the NAL unit and the first size syntax element is not present in the NAL unit.

8. The method of claim 7, wherein, if the value of the second syntax element is equal to the first value, then the size of the first payload is set to the size of the NAL unit minus the number of bytes preceding the first payload of the first message in the NAL unit.

9. The method of claim 6 wherein, if the value of the second syntax element is equal to a second value, then there is at least one message in the NAL unit, and the NAL unit includes a payload size syntax element for each message in the NAL unit.

10. The method of claim 1, wherein the first payload of the first message has a payload type, the method comprises obtaining from the bitstream information regarding the first message, the information regarding the first message comprises a first type value specifying the pay load type of the first pay load.

11. The method of claim 1 , wherein the first payload of the first message has a payload type, the method comprises obtaining from the bitstream information regarding the first message, the information regarding the first message comprises one or more type syntax elements, and the method further comprises using the one or more type syntax elements to obtain a first type value specifying the payload type of the first payload.

12. The method of claim 10 or 11, wherein determining, based on the obtained information, whether the first message includes at least a first size syntax element comprises: determining whether the first type value is included in a particular set of values.

13. The method of claim 12, wherein determining whether the first type value is included in a particular set of values comprises using a function or a look up table taking the first type value as an input.

14. The method of any one of claim 10-13, wherein the bitstream further comprises a second message comprising a payload, the payload of the second message has the same type as the first payload of the first message, the method further comprises obtaining, using one or more syntax elements in the bitstream, a second type value specifying the payload type of the payload of the second message, the first type value specifying the first message’s payload’s type is different than the second type value specifying the second message’s payload’s type even though the payload of the second message has the same type as the first payload of the first message.

15. The method of any one of claim 10-14, wherein it is determined that the first message does not include the first size syntax element, and the method further comprises determining the size of the first pay load of the first message based on the first type value.

16. The method of any one of claims 1-15, wherein the bitstream comprises a sequence of network abstraction layer, NAL, units, the sequence of NAL units comprises a first NAL unit, wherein the first NAL unit is not necessarily the first NAL unit in the sequence, the first NAL unit comprises the first message, and obtaining the information from the bitstream comprises obtaining the information from the first NAL unit.

17. The method of claim 16, wherein it is determined that the first message does not include the first size syntax element, andthe method further comprises using information obtained from the first NAL unit to obtain the first pay load size value specifying the size of the first pay load of the first message.

18. The method of claim 17, wherein the information obtained from the first NAL unit is information specifying the size of the first NAL unit.

19. The method of claim 16, wherein it is determined that the first message does not include the first size syntax element, and the other information used to obtain the first payload size value comprises information indicating that the first message is the only message in the first NAL unit.

20. The method of claim 16, wherein it is determined that the first message does not include the first size syntax element, and the other information used to obtain the first payload size value comprises information about the size of the first NAL unit.

21. The method of claim 20, wherein the other information used to obtain the first payload size value further comprises information indicating that the first message is the only message in the first NAL unit.

22. The method of any one of claims 16-21, wherein the method further comprises obtaining NAL unit type information for the first NAL unit, the NAL unit type information specifies that the first NAL unit is a supplemental enhancement information, SEI, NAL unit, and the first message is an SEI message.

23. The method of any one of claims 1-15, wherein the bitstream comprises a sequence of network abstraction layer, NAL, units,the sequence of NAL units comprises a first NAL unit and a second NAL unit following the first NAL unit in the sequence of NAL units, wherein the first NAL unit is not necessarily the first NAL unit in the sequence, the second NAL unit comprises the first message, and obtaining the information from the bitstream comprises obtaining the information from the first NAL unit.

24. The method of any one of claims 1-23, further comprising obtaining the first payload of the first message using the first payload size value.

25. A method (500) for processing a bitstream comprising a sequence of network abstraction layer, NAL, units, the method comprising: obtaining (s502) type information for a NAL unit included in the bitstream, wherein the type information specifies that the NAL unit is a supplemental enhancement information, SEI, NAL unit, wherein the SEI NAL unit comprises a first SEI message having a payload, wherein the payload consists of an ordered sequence of N bits, where N > 0; and determining (s504) the last bit in the ordered sequence of N bits, wherein determining the last bit in the ordered sequence of N bits comprises locating a code pattern in the SEI NAL unit, wherein the last bit in the ordered sequence of bits immediately precedes the code pattern in the SEI NAL unit.

26. The method of claim 25, wherein the code pattern is start code or separator code.

27. A method (600) performed by an encoder (102), the method comprising: obtaining (s602) a first message comprising a first payload having a size and a type; and determining (s604) whether to include in a bitstream one or more size syntax elements specifying the size of the first payload of the first message, wherein the determination is based on at least one of the following: an encoder profile; an encoder configuration; an encoder setting; the type of the payload;the size of the payload; or whether the first message is the only message that will be included in a network abstraction layer, NAL, unit containing the first message.

28. A computer program (743) comprising instructions (744) which when executed by processing circuitry (702) of an apparatus (700) causes the apparatus to perform the method of any one of the above claims.

29. A carrier containing the computer program of claim 28, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (742).

30. A decoder apparatus (700) configured to perform a method for obtaining from a bitstream a first pay load of a first message, wherein the first payload of the first message has a size, and the method comprises: obtaining from the bitstream information regarding the first pay load of the first message and / or obtaining from the bitstream information regarding the first message; determining, based on the obtained information, whether the first message includes at least a first size syntax element for use in specifying the size of the first payload of the first message; and if it is determined that the first message includes at least the first size syntax element, then using at least the first size syntax element to obtain a first payload size value specifying the size of the first payload of the first message, otherwise using other information to obtain the first payload size value.

31. The decoder apparatus of claim 30, wherein the decoder apparatus is further configured to perform the method of any one of claims 1-24.

32. A decoder apparatus (700) configured to perform a method for processing a bitstream comprising a sequence of network abstraction layer, NAL, units, wherein the method comprises: obtaining type information for a NAL unit included in the bitstream, wherein the type information specifies that the NAL unit is a supplemental enhancement information, SEI,NAL unit, wherein the SEI NAL unit comprises a first SEI message having a payload, wherein the payload consists of an ordered sequence of N bits, where N > 0; and determining the last bit in the ordered sequence of N bits, wherein determining the last bit in the ordered sequence of N bits comprises locating a code pattern in the SEI NAL unit, wherein the last bit in the ordered sequence of bits immediately precedes the code pattern in the SEI NAL unit.

33. The decoder apparatus of claim 32, wherein the code pattern is start code or separator code..

34. An encoder apparatus (700) configured to perform a method comprising: obtaining a first message comprising a first payload having a size and a type; and determining whether to include in a bitstream one or more size syntax elements specifying the size of the first payload of the first message, wherein the determination is based on at least one of the following: an encoder profile; an encoder configuration; an encoder setting; the type of the payload; the size of the payload; or whether the first message is the only message that will be included in a network abstraction layer, NAL, unit containing the first message.

35. An apparatus (700), the apparatus comprising: memory; and processing circuitry, wherein the apparatus is configured to perform the method of any one of claims 1-27.

Images