Constraints on supplemental enhancement information in video coding

Abstract

Several techniques for video encoding and video decoding are described. An example method of processing video data includes performing a conversion between a video and a bitstream of the video, wherein the conversion is performed in accordance with a format rule that specifies, in response to a supplemental enhancement information network abstraction layer unit including a scalable nested supplemental enhancement information message, the supplemental enhancement information network abstraction layer unit including a network abstraction layer unit type that is equal to a prefix supplemental enhancement information network abstraction layer unit type, wherein the scalable nested supplemental enhancement information message includes a supplemental enhancement information message that is not associated with a particular payload type.

Description

[0001] Cross-references to related applications

[0002] This application is based on International Patent Application No. PCT / US2021 / 036353, filed June 8, 2021, which claims priority and interest in U.S. Provisional Patent Application No. 63 / 036,743, filed June 9, 2020. All of the above-mentioned patent applications are incorporated herein by reference in their entirety. Technical Field

[0003] The patent document relates to image and video encoding and decoding. Background Technology

[0004] Digital video accounts for the largest share of bandwidth usage on the Internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for digital video is expected to continue to grow. Summary of the Invention

[0005] This document discloses techniques that can be used by video encoders and decoders to process encoded or decoded representations of video or images.

[0006] In one example aspect, a video processing method is disclosed. The method includes performing a conversion between a video comprising one or more sub-images and a bitstream of the video, wherein, during the conversion, one or more supplementary enhancement information messages with filler payloads are processed according to format rules, and wherein the format rules disallow one or more supplementary enhancement information messages with filler payloads within scalable nesting supplementary enhancement information messages.

[0007] In another example, a video processing method is disclosed. The method includes performing a conversion between a video and a video bitstream, wherein, during the conversion, one or more syntax elements are processed according to format rules, and wherein the format rules specify that the one or more syntax elements are used to indicate sub-picture information of a video layer having pictures having multiple sub-pictures.

[0008] In another example, a video processing method is disclosed. This method includes performing a conversion between a video comprising multiple sub-images and a bitstream of the video, wherein, during the conversion, a scalable-nested supplemental enhancement information message is processed according to format rules, and wherein the format rules specify that one or more sub-image indices are used to associate one or more sub-images with the scalable-nested supplemental enhancement information message.

[0009] In another example, a video processing method is disclosed. The method includes performing a conversion between a video and a video bitstream comprising one or more sub-pictures according to a format rule, wherein the format rule specifies that, in response to a scalable nested supplemental enhancement information message comprising one or more sub-picture level supplemental enhancement information messages, a first syntax element in the scalable nested supplemental enhancement information message in the bitstream is set to a specific value, and wherein the specific value of the first syntax element indicates that the scalable nested supplemental enhancement information message comprises one or more scalable nested supplemental enhancement information messages applied to a specific output video layer set.

[0010] In another example, a video processing method is disclosed. This method includes performing a conversion between a video comprising multiple sub-images and a video bitstream, wherein the conversion is based on format rules that prohibit scalable nested supplemental enhancement messages from including a first supplemental enhancement message of a first payload type and a second supplemental enhancement message of a second payload type.

[0011] In another example, a video processing method is disclosed. The method includes performing a conversion between a video and a video bitstream, wherein the conversion is performed according to a format rule specifying that, in response to a supplemental enhancement information network abstraction layer unit (NEL unit), the NLB unit includes a scalable nested NLB message, the NLB unit including a NLB type equal to a prefixed NLB unit type, wherein the scalable nested NLB message includes an NLB message unrelated to a specific payload type.

[0012] In another example, a video processing method is disclosed. The method includes performing a conversion between a video and a video bitstream, wherein the conversion is performed according to a format rule specifying that, in response to a supplemental enhancement information network abstraction layer unit (NEL unit), a scalable nested LDL message is included, the LDL unit including a NLB type equal to the suffix LDL unit type, wherein the scalable nested LDL message includes a LDL message associated with a specific payload type.

[0013] In another example, a video processing method is disclosed. This method includes performing a conversion between a video comprising one or more sub-pictures or a sequence of one or more sub-pictures and a codec representation of the video, wherein the codec representation conforms to a format rule specifying whether and how scalable nested supplementary enhancement information (SEI) is included in the codec representation.

[0014] In yet another example, a video encoder apparatus is disclosed. The video encoder includes a processor configured to implement the methods described above.

[0015] In yet another example, a video decoder apparatus is disclosed. The video decoder includes a processor configured to implement the methods described above.

[0016] In yet another example, a computer-readable medium on which code is stored is disclosed. This code embodies one of the methods described herein in the form of processor-executable code.

[0017] These and other features are described in this document. Attached Figure Description

[0018] Figure 1 An example of raster scan strip segmentation of an image is shown, where the image is divided into 12 slices and 3 raster scan strips.

[0019] Figure 2 An example of rectangular strip segmentation of an image is shown, where the image is divided into 24 slices (6 slice columns and 4 slice rows) and 9 rectangular strips.

[0020] Figure 3 An example of an image divided into slices and rectangular strips is shown, where the image is divided into 4 slices (2 slice columns and 2 slice rows) and 4 rectangular strips.

[0021] Figure 4 The image is shown as being divided into 15 slices, 24 strips, and 24 sub-images.

[0022] Figure 5 This is a block diagram of an example video processing system.

[0023] Figure 6 This is a block diagram of a video processing device.

[0024] Figure 7 A flowchart of an example method for video processing.

[0025] Figure 8 This is a block diagram illustrating a video encoding / decoding system according to some embodiments of the present disclosure.

[0026] Figure 9 This is a block diagram illustrating an encoder according to some embodiments of the present disclosure.

[0027] Figure 10 This is a block diagram illustrating a decoder according to some embodiments of the present disclosure.

[0028] Figure 11 An example of a typical sub-picture-based viewport-dependent 360° video encoding / decoding scheme is shown.

[0029] Figure 12A viewport-dependent 360° video encoding and decoding scheme based on sub-images and spatial scalability is presented.

[0030] Figures 13 to 19 This is a flowchart of an example method for processing video data. Detailed Implementation

[0031] The use of chapter headings in this document is for ease of understanding and does not limit the applicability of the techniques and embodiments disclosed in each chapter to that chapter only. Furthermore, the use of H.266 terminology in some descriptions is merely for ease of understanding and not to limit the scope of the disclosed techniques. Therefore, the techniques described herein are also applicable to other video codec protocols and designs. In this document, edit changes to text relative to the current draft of the VVC specification are indicated by strikethrough to indicate canceled text and highlighting to indicate added text (including bold and italics).

[0032] 1. Introduction

[0033] This document relates to video codec technology. Specifically, it concerns specifying and signaling notification level information for sub-picture sequences. It can be applied to any video codec standard or non-standard video codec that supports single-layer and multi-layer video codecs, such as the Multi-Functional Video Codec (VVC) currently under development.

[0034] 2. Abbreviation

[0035] APS (Adaptation Parameter Set)

[0036] AU (Access Unit)

[0037] AUD (Access Unit Delimiter)

[0038] AVC (Advanced Video Coding)

[0039] BP (Buffering Period)

[0040] CLVS (Coded Layer Video Sequence)

[0041] CPB (Coded Picture Buffer) is a codec image buffer.

[0042] CRA (Clean Random Access)

[0043] CTU (Coding Tree Unit)

[0044] CVS (Coded Video Sequence) is a video sequence encoding / decoding mechanism.

[0045] DPB (Decoded Picture Buffer)

[0046] DPS (Decoding Parameter Set)

[0047] DUI (Decoding Unit Information)

[0048] EOB (End Of Bitstream) - End of Bitstream

[0049] EOS (End Of Sequence)

[0050] GCI (General Constraints Information)

[0051] GDR (Gradual Decoding Refresh) gradually decodes and refreshes.

[0052] HEVC (High Efficiency Video Coding)

[0053] HRD (Hypothetical Reference Decoder)

[0054] IDR (Instantaneous Decoding Refresh)

[0055] JEM (Joint Exploration Model)

[0056] MCTS (Motion-Constrained Tile Sets)

[0057] NAL (Network Abstraction Layer)

[0058] OLS (Output Layer Set)

[0059] PH (Picture Header)

[0060] PPS (Picture Parameter Set)

[0061] PT (Picture Timing)

[0062] PTL (Profile, Tier, and Level) refers to the level, tier, and grade of a product or service.

[0063] PU (Picture Unit)

[0064] RRP (Reference Picture Resampling)

[0065] RBSP (Raw Byte Sequence Payload)

[0066] SEI (Supplemental Enhancement Information)

[0067] SH (Slice Header)

[0068] SLI (Subpicture Level Information)

[0069] SPS (Sequence Parameter Set)

[0070] SVC (Scalable Video Coding)

[0071] VCL (Video Coding Layer)

[0072] VPS (Video Parameter Set)

[0073] VTM (VVC Test Model)

[0074] VUI (Video Usability Information)

[0075] VVC (Versatile Video Coding) is a multi-functional video codec.

[0076] 3. Preliminary Discussion

[0077] Video codec standards have primarily evolved through the development of well-known ITU-T and ISO / IEC standards. ITU-T developed the H.261 and H.263 standards, while ISO / IEC developed the MPEG-1 and MPEG-4 Visual standards. The two organizations jointly developed the H.262 / MPEG-2 video standard, the H.264 / MPEG-4 Advanced Video Codec (AVC) standard, and the H.265 / HEVC standard. Starting with H.262, video codec standards are based on a hybrid video codec architecture, utilizing temporal prediction plus transform coding. To explore future video codec technologies beyond HEVC, the Joint Video Exploration Team (JVET) was jointly established by VCEG and MPEG in 2015. Since then, JVET has adopted many new methods and incorporated them into reference software called the Joint Exploration Model (JEM). JVET meetings are held quarterly, and the goal of new codec standards is to reduce the bitrate by 50% compared to HEVC. The new video codec standard was officially named Multifunctional Video Coding (VVC) at the JVET meeting in April 2018, and the first version of the VVC Test Model (VTM) was also released at that time. With ongoing efforts to standardize VVC, new codec technologies have been adopted into the VVC standard at each JVET meeting. The VVC working draft and test model VTM are updated after each meeting. The current goal of the VVC project is to achieve Technical Completion (FDIS) at the meeting in July 2020.

[0078] 3.1. Image Segmentation Schemes in HEVC

[0079] HEVC includes four different image segmentation schemes: regular striping, non-independent striping, slice, and wavefront parallel processing (WPP). These can be applied to maximum transfer unit (MTU) size matching, parallel processing, and reduction of end-to-end latency.

[0080] Regular slices are similar to those in H.264 / AVC. Each regular slice is encapsulated in its own NAL unit, and intra-frame prediction (intra-sample prediction, motion information prediction, and encoding / decoding mode prediction) and entropy encoding / decoding dependencies across slice boundaries are disabled. Therefore, regular slices can be reconstructed independently of other regular slices within the same image (although interdependencies may still exist due to loop filtering operations).

[0081] Regular striping is the only tool available for parallelization, and it is available in almost the same form in H.264 / AVC. Parallelization based on regular striping requires minimal inter-processor or inter-core communication (except for inter-processor or inter-core data sharing for motion compensation during decoding prediction of encode-decode images, which is typically much heavier than inter-processor or inter-core data sharing due to intra-frame image prediction). However, for the same reason, using regular striping can incur significant encoding / decoding overhead due to the bit cost of the stripe header and the lack of prediction across stripe boundaries. Furthermore, due to the intra-image independence of regular striping and the fact that each regular stripe is encapsulated in its own NAL, regular striping (compared to other tools mentioned below) can also serve as a key mechanism for bitstream segmentation to match MTU size requirements. In many cases, the goals of parallelization and MTU size matching are contradictory in terms of stripe layout requirements within the image. The implementation of such situations led to the development of the parallelization tools mentioned below.

[0082] Non-independent striping has a short stripe header and allows the bitstream to be split at tree block boundaries without disrupting any intra-picture predictions. Essentially, non-independent striping divides a regular stripe into multiple NAL units, reducing end-to-end latency by allowing a portion of the regular stripe to be sent before the entire regular stripe's encoding is complete.

[0083] In WPP, images are segmented into single-row coding tree blocks (CTBs). Entropy decoding and prediction are allowed to utilize data from CTBs in other segments. Parallel processing is possible through parallel decoding of CTB rows, where decoding of a CTB row begins with a delay of two CTBs to ensure that data associated with CTBs above and to the right of the main CTB is available before the main CTB being decoded. Using this staggered start (which looks like a wavefront when represented graphically), as many processors / cores as images containing CTB rows can be parallelized. Because intra-image prediction is allowed between adjacent tree block rows within an image, the inter-processor / inter-core communication required to achieve intra-image prediction can be substantial. WPP segmentation does not result in additional NAL units compared to segmentation without WPP application, therefore WPP is not a tool for MTU size matching. However, if MTU size matching is required, regular striping can be used with WPP, but with some encoding / decoding overhead.

[0084] A slice is defined by the horizontal and vertical boundaries that divide an image into slice columns and slice rows. Slice columns extend from the top to the bottom of the image. Similarly, slice rows extend from the left to the right of the image. The number of slices in an image can be simply derived by multiplying the number of slice columns by the number of slice rows.

[0085] Before decoding the top left CTB of the next slice in the order of the slice raster scans of the image, the scan order of the CTBs is changed to the local scan order within the slice (according to the order of the slice's CTB raster scans). Similar to regular stripes, slices break the intra-image prediction dependency and the entropy decoding dependency. However, they do not need to be contained in separate NAL units (the same as WPP in this respect); therefore, slices cannot be used for MTU size matching. Each slice can be processed by one processor / core, and in the case of a stripe spanning multiple slices, the inter-processor / inter-core communication required for decoding intra-image predictions between processing units of adjacent slices is limited to transmitting the shared stripe header and loop filtering associated with the sharing of reconstructed samples and metadata. When a stripe contains more than one slice or WPP segment, the entry point byte offset of each slice or WPP segment in the stripe, except for the first one, is signaled in the stripe header.

[0086] For simplicity, restrictions are specified in HEVC for the application of four different image segmentation schemes. For most levels specified in HEVC, a given codec video sequence cannot contain both slices and wavefronts simultaneously. For each strip and slice, one or both of the following conditions must be met: 1) All codec tree blocks in a strip belong to the same slice; 2) All codec tree blocks in a slice belong to the same strip. Finally, a wavefront segment contains exactly one CTB line, and when using WPP, if a strip begins within a CTB line, then that strip must end within the same CTB line.

[0087] The most recent revision of HEVC is specified in the JCT-VC output document JCTVC-AC1005, by J. Boyce, A. Ramasubramonian, R. Skupin, G. J. Sullivan, A. Tourapis, and Y.-K. Wang (editors). "HEVC Additional Supplemental Enhancement Information (Draft 4)," published on October 24, 2017, is available here: http: / / phenix.intevry.fr / jct / doc_end_user / documents- / 29_Macau / wg11 / JCTVC-AC1005-v2.zip. Including this revision, HEVC specifies three SEI messages related to MCT: the i.e., the domain MCTS SEI message, the MCTS extracted information set SEI message, and the MCTS extracted information nested SEI message.

[0088] The temporal MCTS SEI message indicates the presence of an MCTS in the bitstream and signals this to the MCTS. For each MCTS, motion vectors are restricted to pointing to full-sample locations within the MCTS and fractional-sample locations that require interpolation only from full-sample locations within the MCTS. Motion vector candidates derived from temporal motion vector predictions from blocks outside the MCTS are not allowed. This allows each MCTS to be decoded independently even if there are no slices not included in the MCTS.

[0089] The MCTS Extraction Information Set (SEI) message provides supplementary information that can be used for MCTS sub-bitstream extraction (specified as part of the semantics of the SEI message) to generate a bitstream conforming to the MCTS group. This information consists of multiple extraction information sets, each defining multiple MCTS groups and containing RBSP bytes for replacing the VPS, SPS, and PPS to be used during the MCTS sub-bitstream extraction process. When extracting the sub-bitstream according to the MCTS sub-bitstream extraction process, the parameter sets (VPS, SPS, and PPS) need to be rewritten or replaced, and the slice header needs to be slightly updated because one or all syntax elements related to the slice address (including first_slice_segment_in_pic_flag and slice_segment_address) typically need to have different values.

[0090] 3.2. Image Segmentation in VVC

[0091] In VVC, an image is divided into one or more slice rows and one or more slice columns. A slice is a sequence of CTUs covering a rectangular area of ​​the image. The CTUs within a slice are scanned in raster scan order within that slice.

[0092] A strip consists of an integer number of complete slices or an integer number of consecutive complete CTU lines within a slice of an image.

[0093] Two stripe modes are supported: raster scan stripe mode and rectangular stripe mode. In raster scan stripe mode, a stripe contains a complete stripe sequence within a sheet raster scan of the image. In rectangular stripe mode, a stripe contains multiple complete sheets that together form a rectangular area of ​​the image, or multiple consecutive complete CTU rows of a sheet that together forms a rectangular area of ​​the image. Sheets within a rectangular stripe are scanned in sheet raster scan order within the rectangular area corresponding to that stripe.

[0094] A sub-image contains one or more stripes that collectively cover a rectangular area of ​​the image.

[0095] Figure 1 An example of raster scan strip segmentation of an image is shown, where the image is divided into 12 slices and 3 raster scan strips.

[0096] Figure 2 An example of rectangular strip segmentation of an image is shown, where the image is divided into 24 slices (6 slice columns and 4 slice rows) and 9 rectangular slices.

[0097] Figure 3 An example of an image divided into slices and rectangular strips is shown, where the image is divided into 4 slices (2 slice columns and 2 slice rows) and 4 rectangular strips.

[0098] Figure 4 An example of sub-image segmentation of an image is shown, where the image is segmented into 18 pieces: 12 pieces on the left (each covering a strip with 4x4 CTUs) and 6 pieces on the right (each covering two vertically stacked strips with 2x2 CTUs), resulting in a total of 24 strips and 24 sub-images of different sizes (each strip being a sub-image).

[0099] 3.3. Image resolution variations within a sequence

[0100] In AVC and HEVC, the spatial resolution of an image cannot be changed unless a new sequence with a new SPS begins with an IRAP image. VVC allows changing the image resolution within a sequence at locations where IRAP images are not encoded; IRAP images are always intra-frame encoded and decoded. This feature is sometimes called Reference Image Resampling (RPR) because it requires resampling the reference image used for inter-frame prediction when the reference image has a different resolution than the current image being decoded.

[0101] The scaling ratio is limited to greater than or equal to 1 / 2 (2x downsampling from the reference image to the current image) and less than or equal to 8 (8x upsampling). Three sets of resampling filters with different frequency cutoffs are specified to handle various scaling ratios between the reference and current images. The three sets of resampling filters are applied to scaling ratios ranging from 1 / 2 to 1 / 1.75, from 1 / 1.75 to 1 / 1.25, and from 1 / 1.25 to 8, respectively. Each set of resampling filters has 16 phases for luminance and 32 phases for chrominance, which is the same as in motion-compensated interpolation filters. In fact, the normal MC interpolation process is a special case of the resampling process, where the scaling ratio ranges from 1 / 1.25 to 8. The horizontal and vertical scaling ratios are derived based on the image width and height, as well as the left, right, top, and bottom scaling offsets specified for the reference and current images.

[0102] Other aspects of VVC designs that support this feature, unlike HEVC, include: i) signaling the image resolution and corresponding consistency window in the PPS instead of the SPS, while in the SPS the signaling indicates the maximum image resolution. ii) for a single-layer bitstream, each image storage (a slot storing one decoded image in the DPB) occupies the buffer size required to store the decoded image with the maximum image resolution.

[0103] 3.4. Scalable Video Codec (SVC) in General Purpose and VVC

[0104] Scalable video codec (SVC, sometimes also called scalability in video codec) refers to video codec using a base layer (BL) (sometimes called a reference layer (RL)) and one or more scalable enhancement layers (EL). In SVC, the base layer can carry video data with a base quality level. One or more enhancement layers can carry additional video data to support, for example, higher spatial, temporal, and / or signal-to-noise (SNR) levels. Enhancement layers can be defined relative to previously encoded layers. For example, the bottom layer can be used as a BL, while the top layer can be used as an EL. Intermediate layers can be used as ELs or RLs, or both. For example, an intermediate layer (e.g., a layer that is neither the lowest nor the highest layer) can be an EL of a layer below the intermediate layer (e.g., a base layer or any intermediate enhancement layer) and simultaneously serve as an RL of one or more enhancement layers above the intermediate layer. Similarly, in the HEVC standard's multiview or 3D extension, there may be multiple views, and information from one view can be used to encode or decode (e.g., encode or decode) information from another view (e.g., motion estimation, motion vector prediction, and / or other redundancy).

[0105] In SVC, the parameters used by the encoder or decoder are grouped into parameter sets based on the codec level in which they can be used (e.g., video level, sequence level, picture level, stripe level, etc.). For example, parameters that can be used by one or more codec video sequences at different layers in a bitstream can be included in the Video Parameter Set (VPS), and parameters that can be used by one or more pictures in a codec video sequence can be included in the Sequence Parameter Set (SPS). Similarly, parameters used by one or more stripes in a picture can be included in the Picture Parameter Set (PPS), and other parameters specific to a single strip can be included in the stripe header. Likewise, indications of which parameter set(s) a particular layer uses at a given time can be provided at various codec levels.

[0106] Because of VVC's support for Reference Picture Resampling (RPR), support for multi-layered bitstreams can be designed without requiring any additional signaling notification processing level codec tools. For example, two layers in VVC with SD and HD resolutions can be supported because the upsampling required for spatial scalability can be achieved using only RPR upsampling filters. However, supporting scalability requires a higher level of syntax changes (compared to not supporting scalability at all). Scalability support was specified in VVC version 1. Unlike scalability support in any earlier video codec standards, including extensions to AVC and HEVC, VVC scalability was designed to be as friendly as possible to single-layer decoder designs. The decoding capability of multi-layered bitstreams is specified as if there were only a single layer in the bitstream. For example, decoding capabilities such as DPB size are specified in a way that is independent of the number of layers in the bitstream to be decoded. Essentially, decoders designed for single-layered bitstreams do not require many changes to decode multi-layered bitstreams. Compared to the multi-layered extensions of AVC and HEVC, the HLS aspect is significantly simplified at the expense of some flexibility. For example, IRAP AU requires a picture of every layer present in CVS.

[0107] 3.5. Viewport-dependent 360° video stream based on sub-images

[0108] In 360° video streaming (also known as omnidirectional video), at any given moment, only a subset of the entire omnidirectional video sphere (i.e., the current viewport) is presented to the user, who can rotate their head at any time to change their viewing orientation, thus changing the current viewport. While it's desirable to have at least some lower-quality representations of areas not covered by the current viewport on the client side, ready to be presented to the user in case they suddenly change their viewing orientation to any location on the sphere, the high-quality representation of the omnidirectional video is only needed for the current viewport being presented to the user. This optimization can be achieved by dividing the high-quality representation of the entire omnidirectional video into sub-pictures with appropriate granularity. Using VVC, these two representations can be encoded as two independent layers.

[0109] A typical sub-image-based viewport-dependent 360° video transmission scheme is as follows: Figure 11 As shown, a higher resolution representation of the complete video consists of sub-pictures, while a lower resolution representation of the complete video does not use sub-pictures and can be encoded and decoded using lower-frequency random access points in the higher resolution representation. The client receives the lower resolution complete video, while for the higher resolution video, it only receives and decodes the sub-pictures covering the current viewport.

[0110] The latest VVC draft specification also supports such Figure 12 The improved 360° video encoding / decoding scheme is shown. (Compared to...) Figure 11The only difference between the methods shown is that inter-layer prediction (ILP) is applied to the methods described. Figure 12 The method shown.

[0111] 3.6. Parameter Set

[0112] AVC, HEVC, and VVC specify parameter sets. Parameter set types include SPS, PPS, APS, and VPS. All AVC, HEVC, and VVC versions support SPS and PPS. VPS was introduced with HEVC and is included in both HEVC and VVC. APS is not included in AVC or HEVC, but is included in the latest VVC draft text.

[0113] SPS is designed to carry sequence-level header information, and PPS is designed to carry infrequently changing image-level header information. Using SPS and PPS eliminates the need to repeat infrequently changing information for each sequence or image, thus avoiding redundant signaling notifications. Furthermore, using SPS and PPS enables out-of-band transmission of critical header information, thereby not only avoiding redundant transmission but also improving error recovery capabilities.

[0114] The VPS was introduced to carry sequence-level header information common to all layers in a multi-layer bitstream.

[0115] The purpose of APS is to carry such image-level or stripe-level information, which requires a considerable number of bits for encoding and decoding, can be shared by multiple images, and can have many different variations in the sequence.

[0116] 3.7. Specifying and Signaling Notifications for Nested SEI Messages in Sub-Picture Sequences in VVC

[0117] In the latest VVC draft text, the specification and signaling notification of subpicture sequences nested in VVC via scalable nested SEI messages are performed. Subpicture sequences are defined within the semantics of Subpicture Level Information (SLI) SEI messages. Subpicture sequences can be extracted from the bitstream by applying the subpicture sub-bitstream extraction procedure specified in VVC Clause C.7.

[0118] The syntax and semantics of scalable nested SEI messages in the latest VVC draft text are as follows.

[0119] D.6.1 Scalable Nested SEI Message Syntax

[0120]

[0121] D.6.2 Scalable Nested SEI Message Semantics

[0122] Scalable nested SEI messages provide a mechanism to associate SEI messages with a specific OLS or layer, and with a specific sub-group of images.

[0123] A scalable nested SEI message contains one or more SEI messages. The SEI message contained within a scalable nested SEI message is also referred to as a scalable nested SEI message.

[0124] The bitstream consistency requirement applies to SEI messages contained within scalable nested SEI messages, with the following constraints:

[0125] - SEI messages with payloadType equal to 132 (decoded image hash) can only be included in scalable nested SEI messages in which sn_subpic_flag equals 1.

[0126] SEI messages with payloadType equal to 133 (scalable nested) should not be included in scalable nested SEI messages.

[0127] – When a scalable nested SEI message contains a BP, PT, or DUI SEI message, the scalable nested SEI message should not contain any other SEI message in which the payloadType is not equal to 0 (BP), 1 (PT), or 130 (DUI).

[0128] The requirement for bitstream consistency is that the following constraints apply to the value of nal_unit_type of SEI NAL units containing scalable nested SEI messages:

[0129] – When a scalable nested SEI message contains an SEI message in which payloadType is equal to 0 (BP), 1 (PT), 130 (DUI), 145 (DRAP indication), or 168 (frame field information), the SEI NAL unit containing the scalable nested SEI message should have nal_unit_type equal to PREFIX_SEI_NUT.

[0130] A sn_ols_flag value of 1 specifies that scalable nested SEI messages are applicable to a particular OLS.

[0131] A sn_ols_flag value of 0 specifies that scalable nested SEI messages apply to a particular layer.

[0132] The requirements for bitstream consistency are as follows, and the following constraints apply to the value of sn_ols_flag:

[0133] – When a scalable nested SEI message contains an SEI message with a payloadType equal to 0 (BP), 1 (PT), or 130 (DUI), the value of sn_ols_flag should be equal to 1.

[0134] – When a scalable nested SEI message contains an SEI message with a payloadType equal to a value in VclAssociatedSeiList, the value of sn_ols_flag should be equal to 0.

[0135] A sn_subpic_flag value of 1 indicates that a scalable nested SEI message applicable to a specified OLS or layer applies only to a specific subpicture of the specified OLS or layer. A sn_subpic_flag value of 0 indicates that a scalable nested SEI message applicable to a specific OLS or layer applies to all subpictures of the specified OLS or layer.

[0136] Increasing 1 to sn_num_olss_minus1 specifies the number of OLS applicable to scalable nested SEI messages. The value of sn_num_olss_minus1 should be in the range of 0 to Total NumOlss-1 (inclusive).

[0137] sn_ols_idx_delta_minus1[i] is used to derive the variable NestingOlsIdx[i], which specifies the OLS index applicable to the i-th OLS of the scalable nested SEI message when sn_ols_flag equals 1. The value of sn_ols_idx_delta_minus1[i] should be in the range of 0 to TotalNumOlss-2 (inclusive).

[0138] The derivation of the variable NestingOlsIdx[i] is as follows:

[0139]

[0140] `sn_all_layers_flag` equal to 1 indicates that scalable nested SEI messages apply to all layers, and all layers have a `nuh_layer_id` greater than or equal to the `nuh_layer_id` of the current SEI NAL unit. `sn_all_layers_flag` equal to 0 indicates that scalable nested SEI messages may or may not apply to all layers, and all layers have a `numh_layer_id` greater than or equal to the `numh_layer_id` of the current SEI NAL unit.

[0141] Increasing sn_num_layers_minus1 by 1 specifies the number of layers applicable to scalable nested SEI messages. The value of sn_num_layers_minus1 should be in the range of 0 to vps_max_layers_minus1-GeneralLayerIdx[nuh_layer_id] (inclusive), where nuh_layer_id is the nuh_layer_id of the current SEI NAL unit.

[0142] sn_layer_id[i] specifies the nuh_layer_id value applicable to the i-th layer of the scalable nested SEI message when sn_all_layers_flag equals 0. The value of sn_layer_id[i] should be greater than nuh_layer_id, where nuh_layer_id is the nuh_layer_id of the current SEI NAL unit.

[0143] When sn_ols_flag equals 0, the variable nestingNumLayers specifies the number of layers applicable to the scalable nested SEI message, and nestingLayerId[i] is a list of nuh_layer_id values ​​applicable to the layer in the scalable nested SEI message for i in the range of 0 to nestingNumLayers-1 (inclusive), as deduced below, where nuh_layer_id is the nuh_layer_id of the current SEI NAL unit:

[0144]

[0145]

[0146] Incrementing sn_num_subpics_minus1 by 1 specifies the number of subpicks applicable to scalable nested SEI messages. The value of sn_num_subpics_minus1 should be less than or equal to the value of sps_num_subpics_minus1 in the SPS of the picture reference in CLVS.

[0147] Increasing 1 in sn_subpic_id_len_minus1 specifies the number of bits used to represent the syntax element sn_subpic_id[i]. The value of sn_subpic_id_len_minus1 should be in the range of 0 to 15 (inclusive).

[0148] The requirement for bitstream consistency is that the value of sn_subpic_id_len_minus1 should be the same for all scalable nested SEI messages present in CLVS.

[0149] sn_subpic_id[i] indicates the ID of the i-th subpick associated with the scalable nested SEI message. The length of the sn_subpic_id[i] syntax element is sn_subpic_id_len_minus1+1 bits.

[0150] Incrementing sn_num_seis_minus1 by 1 specifies the number of scalable nested SEI messages. The value of sn_num_seis_minus1 should be in the range of 0 to 63 (inclusive).

[0151] sn_zero_bit should be equal to 0.

[0152] 4. The technical problem solved by the disclosed technical solution

[0153] Existing VVC designs that specify and signal nested SEI messages for sub-pictures and sub-picture sequences using scalable nested SEI messages have the following problems:

[0154] 1) To associate scalable nested SEI messages with one or more sub-images, the scalable nested SEI messages use the sub-image ID. However, the duration of a scalable nested SEI message can be multiple consecutive AUs, and the sub-image ID of a sub-image with a specific sub-image index within a layer can change within CLVS. Therefore, the sub-image index should be used in scalable nested SEI messages, rather than the sub-image ID.

[0155] 2) When a relevant sub-image is removed, the padding payload SEI message (if present) needs to be removed from the output bitstream during sub-image sub-bitstream extraction. However, when the padding payload SEI message can be included in the scalable nested SEI message, removing the padding payload SEI message during sub-image sub-bitstream extraction sometimes requires extracting some scalable nested SEI messages from the scalable nested SEI message.

[0156] 3) Since SLI SEI messages are applicable to OLS, like the other three HRD-related SEI messages (i.e., BP / PT / DUISEI messages), the value of sn_ols_flag needs to be equal to 1 when the SLI SEI message is scalable and nested. Furthermore, because the SLI SEI message specifies information about all subpicks of the image in the OLS applicable to the SLI SEI message, it is meaningless for the value of sn_subpic_flag to be equal to 1 for scalable and nested SEI messages containing the SLI SEI message.

[0157] 4) Lack of constraints requiring that when a scalable nested SEI message contains BP, PT, DUI, or SLI SEI messages, the scalable nested SEI message should not contain any other SEI message whose payloadType is not equal to 0 (BP), 1 (PT), 130 (DUI), or 203 (SLI).

[0158] 5) It is specified that when a scalable nested SEI message contains an SEI message whose payloadType is equal to 0 (BP), 1 (PT), 130 (DUI), 145 (DRAP indication), or 168 (frame field information), the SEI NAL unit containing the scalable nested SEI message should have a nal_unit_type equal to PREFIX_SEI_NUT. However, when many other SEI messages are nested, the value of the scalable nested SEI message should also have a nal_unit_type equal to PREFIX_SEI_NUT.

[0159] 6) Missing constraint: When a scalable nested SEI message contains an SEI message in which payloadType equals 132 (decoded image hash), the SEI NAL unit containing the scalable nested SEI message should have nal_unit_type equal to SUFFIX_SEI_NUT.

[0160] 7) The semantics of sn_num_subpics_minus1 and sn_subpic_idx[i] need to be specified in such a way that the syntax elements are subpicks of layers with multiple subpicks per image, so as to support the case that OLS has some layers with multiple subpicks per image and some other layers with a single subpick per image.

[0161] 5. List of solutions and implementation examples

[0162] To address the aforementioned and other issues, the following summarized methodology is disclosed. Solution items should be considered as examples illustrating general concepts, rather than interpreted in a narrow sense. Furthermore, these items can be used individually or in combination in any way.

[0163] 1) To solve the first problem, use the sub-image index (instead of using the sub-image ID) to associate the sub-image with the scalable nested SEI message in the scalable nested SEI message.

[0164] a. In one example, the syntax element sn_subpic_id[i] is changed to sn_subpic_idx[i], and thus the syntax element sn_subpic_id_len_minus1 is removed.

[0165] 2) To address the second issue, the prohibited padding payload SEI message is scalable nested, meaning it is contained within a scalable nested SEI message.

[0166] 3) To address the third issue, a constraint is added such that the value of sn_ols_flag should be equal to 1 when a scalable nested SEI message contains one or more SLISEI messages.

[0167] a. In one example, additionally, or alternatively, a constraint is added such that the value of sn_subpic_flag should be equal to 0 when a scalable nested SEI message contains one or more SLI SEI messages.

[0168] 4) To address the fourth issue, it is required that when a scalable nested SEI message contains BP, PT, DUI, or SLI SEI messages, the scalable nested SEI message must not contain any other SEI message whose payloadType is not equal to 0 (BP), 1 (PT), 130 (DUI), or 203 (SLI).

[0169] 5) To address the fifth issue, it is stipulated that when a scalable nested SEI message contains an SEI message in which payloadType is not equal to 3 (padding payload) or 132 (decoded image hash), the SEI NAL unit containing the scalable nested SEI message should have nal_unit_type equal to PREFIX_SEI_NUT.

[0170] 6) To address the sixth issue, a constraint is added such that when a scalable nested SEI message contains an SEI message in which payloadType equals 132 (decoded image hash), the SEI NAL unit containing the scalable nested SEI message should have nal_unit_type equal to SUFFIX_SEI_NUT.

[0171] 7) To address the 7th problem, the semantics of sn_num_subpics_minus1 and sn_subpic_idx[i] are defined in such a way that the syntax elements specify information about the subpicks of a layer with multiple subpicks for each picture.

[0172] 6. Example

[0173] Below are some example embodiments of aspects of the invention summarized above in this chapter, which are applicable to the VVC specification. The modified text is based on the latest VVC text in JVET-S0152-v5. Most of the relevant parts that have been added or modified are indicated in bold italic text, and some deleted parts are marked with double brackets (e.g., [[]]), where the deleted text is enclosed in double brackets.

[0174] 6.1. Example 1

[0175] This example applies to projects 1 to 5 and some of their sub-projects.

[0176] D.6.1 Scalable Nested SEI Message Syntax

[0177]

[0178]

[0179] D.6.2 Scalable Nested SEI Message Semantics

[0180] Scalable nested SEI messages provide a mechanism to associate SEI messages with a specific OLS or layer, as well as with a specific sub-group of images.

[0181] A scalable nested SEI message contains one or more SEI messages. The SEI message contained within a scalable nested SEI message is also referred to as a scalable nested SEI message.

[0182] The bitstream consistency requirement applies to SEI messages contained within scalable nested SEI messages, with the following constraints:

[0183] SEI messages with payloadType equal to 132 (decoded image hash) can only be included within scalable nested SEI messages where sn_subpic_flag equals 1.

[0184] SEI messages with payloadType equal to 3 (padding payload) or 133 (scalable nested) should not be included in scalable nested SEI messages.

[0185] – When a scalable nested SEI message contains a BP, PT, [[or]]DUI or SLI SEI message, the scalable nested SEI message should not contain any other SEI message in which the payloadType is not equal to 0 (BP), 1 (PT), [[or]]130 (DUI) or 203 (SLI).

[0186] The requirement for bitstream consistency is that the following constraints apply to the value of nal_unit_type of SEI NAL units containing scalable nested SEI messages:

[0187] – When a scalable nested SEI message contains an SEI message in which payloadType is not equal to 3 (fill payload) or 132 (decoded image hash) [[equal to 0 (BP), 1 (PT), 130 (DUI), 145 (DRAP indication) or 168 (frame field information)]], the SEI NAL unit containing the scalable nested SEI message should have nal_unit_type equal to PREFIX_SEI_NUT.

[0188] – When a scalable nested SEI message contains an SEI message in which payloadType equals 132 (decoded image hash), the SEI NAL unit containing the scalable nested SEI message should have nal_unit_type equal to SUFFIX_SEI_NUT.

[0189] A sn_ols_flag value of 1 specifies that scalable nested SEI messages are applicable to a particular OLS.

[0190] A sn_ols_flag value of 0 specifies that scalable nested SEI messages apply to a particular layer.

[0191] The requirements for bitstream consistency are as follows, and the following constraints apply to the value of sn_ols_flag:

[0192] – When a scalable nested SEI message contains an SEI message with a payloadType equal to 0 (BP), 1 (PT), [[or]]130 (DUI) or 203 (SLI), the value of sn_ols_flag should be equal to 1.

[0193] – When a scalable nested SEI message contains an SEI message with a payloadType equal to but not equal to 203 (SLI) in VclAssociatedSeiList, the value of sn_ols_flag should be equal to 0.

[0194] A sn_subpic_flag value of 1 indicates that a scalable nested SEI message applicable to a specified OLS or layer applies only to a specific subpicture of the specified OLS or layer. A sn_subpic_flag value of 0 indicates that a scalable nested SEI message applicable to a specific OLS or layer applies to all subpictures of the specified OLS or layer.

[0195] The requirement for bitstream consistency applies the following restrictions to the value of sn_subpic_flag:

[0196] – When a scalable nested SEI message contains an SEI message in which payloadType equals 132 (decoded image hash), the value of sn_subpic_flag should be equal to 1.

[0197] – When a scalable nested SEI message contains an SEI message in which payloadType equals 203 (SLI), the value of sn_subpic_flag should be equal to 0.

[0198] Increasing 1 to sn_num_olss_minus1 specifies the number of OLS applicable to scalable nested SEI messages. The value of sn_num_olss_minus1 should be in the range of 0 to Total NumOlss-1 (inclusive).

[0199] sn_ols_idx_delta_minus1[i] is used to derive the variable NestingOlsIdx[i], which specifies the OLS index applicable to the i-th OLS of the scalable nested SEI message when sn_ols_flag equals 1. The value of sn_ols_idx_delta_minus1[i] should be in the range of 0 to TotalNumOlss-2 (inclusive).

[0200] The derivation of the variable NestingOlsIdx[i] is as follows:

[0201]

[0202] `sn_all_layers_flag` equal to 1 indicates that scalable nested SEI messages apply to all layers, and all layers have a `nuh_layer_id` greater than or equal to the `nuh_layer_id` of the current SEI NAL unit. `sn_all_layers_flag` equal to 0 indicates that scalable nested SEI messages may or may not apply to all layers, and all layers have a `numh_layer_id` greater than or equal to the `numh_layer_id` of the current SEI NAL unit.

[0203] Increasing sn_num_layers_minus1 by 1 specifies the number of layers applicable to scalable nested SEI messages. The value of sn_num_layers_minus1 should be in the range of 0 to vps_max_layers_minus1-GeneralLayerIdx[nuh_layer_id] (inclusive), where nuh_layer_id is the nuh_layer_id of the current SEI NAL unit.

[0204] sn_layer_id[i] specifies the nuh_layer_id value applicable to the i-th layer of the scalable nested SEI message when sn_all_layers_flag equals 0. The value of sn_layer_id[i] should be greater than nuh_layer_id, where nuh_layer_id is the nuh_layer_id of the current SEI NAL unit.

[0205] When sn_ols_flag equals 0, the variable nestingNumLayers specifies the number of layers applicable to scalable nested SEI messages, and nestingLayerId[i] is a list of nuh_layer_id values ​​for layers applicable to scalable nested SEI messages for i in the range of 0 to nestingNumLayers-1 (inclusive), as deduced below, where nuh_layer_id is the nuh_layer_id of the current SEI NAL unit:

[0206]

[0207]

[0208] In layers of an OLS applicable to scalable nested SEI messages (when sn_ols_flag equals 1), or in layers of an OLS applicable to scalable nested SEI messages (when sn_ols_flag equals 0), the layers of the referenced SPS with sps_num_subpics_minus1 greater than 0 are called multiSubpicLayers.

[0209] Incrementing sn_num_subpics_minus1 by 1 specifies the number of subpicks in each picture within the multiSubpicLayers [[applicable to scalable nested SEI messages]]. The value of sn_num_subpics_minus1 should be less than or equal to the value of sps_num_subpics_minus1 in the SPS referenced by the pictures in the multiSubpicLayers[[CLVS]].

[0210] The increment of 1 in `sn_subpic_id_len_minus1` specifies the number of bits used to represent the syntax element `sn_subpic_id[i]`. The value of `sn_subpic_id_len_minus1` should be in the range of 0 to 15 (inclusive).

[0211] The requirement for bitstream consistency is that the value of sn_subpic_id_len_minus1 should be the same for all scalable nested SEI messages present in CLVS.

[0212] `sn_subpic_idx[i][[indicator]]` specifies the subpick index of the i-th subpick [[ID]] in each picture within `multiSubpicLayers` [[associated with the scalable nested SEI message]]. The value of `sn_subpic_idx[i]` should be less than or equal to the value of `sps_num_subpics_minus1` in the SPS referenced by the picture in `multiSubpicLayers`. The length of the `sn_subpic_id[i]` syntax element is `sn_subpic_id_len_minus1 + 1 bits`. Scalable nested SEI messages also apply to individual subpicks in each picture that are not in `multiSubpicLayers` but are in an OLS applicable to scalable nested SEI messages (when `sn_ols_flag` equals 1) or in a layer within a layer applicable to scalable nested SEI messages (when `sn_ols_flag` equals 0).

[0213] Incrementing sn_num_seis_minus1 by 1 specifies the number of scalable nested SEI messages. The value of sn_num_seis_minus1 should be in the range of 0 to 63 (inclusive).

[0214] sn_zero_bit should be equal to 0.

[0215] Figure 5 This is a block diagram of an example video processing system 1900 that can implement the various techniques disclosed herein. Various implementations may include some or all of the components in system 1900. System 1900 may include an input 1902 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8 or 10-bit multi-component pixel values), or in a compressed or encoded format. Input 1902 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as Ethernet, passive optical network (PON), etc.) and wireless interfaces (such as Wi-Fi or cellular interfaces).

[0216] System 1900 may include a codec component 1904 capable of implementing the various codec or encoding methods described in this document. Codec component 1904 can reduce the average bit rate of the video from input 1902 to the output of codec component 1904 to produce a codec representation of the video. Therefore, codec techniques are sometimes referred to as video compression or video transcoding techniques. The output of codec component 1904 can be stored or transmitted via connected communication, as represented by component 1906. The stored or communicated bitstream (or codec) representation of the video received at input 1902 can be used by component 1908 to generate pixel values ​​or displayable video that are sent to display interface 1910. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. Furthermore, although some video processing operations are referred to as "codec" operations or tools, it should be understood that the codec tool or operation is used at the encoder, and the corresponding decoding tool or operation will be inverted by the decoder to retrieve the result of the codec.

[0217] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or DisplayPort. Examples of storage interfaces include SATA (Serial Advanced Technology Accessory), PCI, IDE, etc. The technologies described in this document can be implemented in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of digital data processing and / or video display.

[0218] Figure 6 This is a block diagram of a video processing apparatus 3600. Apparatus 3600 can be used to implement one or more of the methods described herein. Apparatus 3600 can be implemented in smartphones, tablets, computers, Internet of Things (IoT) receivers, etc. Apparatus 3600 may include one or more processors 3602, one or more memories 3604, and video processing hardware 3606. The processors(multiple) 3602 can be configured to implement one or more methods described herein. The memories(multiple) 3604 can be used to store data and code used to implement the methods and techniques described herein. The video processing hardware 3606 can be used to implement some of the techniques described herein in hardware circuitry.

[0219] Figure 8 This is a block diagram illustrating an example video codec system 100 that can utilize the techniques disclosed herein.

[0220] like Figure 8As shown, the video encoding / decoding system 100 may include a source device 110 and a target device 120. The source device 110 generates encoded video data and may be referred to as a video encoding device. The target device 120 can decode the encoded video data generated by the source device 110 and may be referred to as a video decoding device.

[0221] The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0222] Video source 112 may include sources such as video capture devices, interfaces for receiving video data from video content providers, and / or computer graphics systems that generate video data, or combinations of these sources. Video data may include one or more images. Video encoder 114 encodes the video data from video source 112 to generate a bitstream. The bitstream may include a sequence of bits forming a codec representation of the video data. The bitstream may include codec images and associated data. A codec image is a codec representation of an image. Associated data may include sequence parameter sets, image parameter sets, and other syntax elements. I / O interface 116 includes a modulator / demodulator (modem) and / or a transmitter. Encoded video data may be transmitted directly to target device 120 via network 130a through I / O interface 116. Encoded video data may also be stored on storage medium / server 130b for access by target device 120.

[0223] The target device 120 may include an I / O interface 126, a video decoder 124, and a display device 122.

[0224] I / O interface 126 may include a receiver and / or a modem. I / O interface 126 may acquire encoded video data from source device 110 or storage medium / server 130b. Video decoder 124 may decode the encoded video data. Display device 122 may display the decoded video data to a user. Display device 122 may be integrated with target device 120 or may be external to target device 120 configured to connect to an external display device.

[0225] The video encoder 114 and the video decoder 124 can operate according to video compression standards such as High Efficiency Video Codec (HEVC), Multi-Functional Video Codec (VVC), and other current and / or other standards.

[0226] Figure 9 This is a block diagram illustrating an example of a video encoder 200, which may be... Figure 8 The video encoder 114 in the system 100 shown in the figure.

[0227] The video encoder 200 can be configured to perform any or all of the techniques disclosed herein. Figure 9 In the example, the video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video encoder 200. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0228] The functional components of the video encoder 200 may include a segmentation unit 201, a prediction unit 202 (which may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205, and an intra-frame prediction unit 206), a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213, and an entropy coding unit 214.

[0229] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in IBC mode, where at least one reference picture is the picture in which the current video block is located.

[0230] Furthermore, some components, such as the motion estimation unit 204 and the motion compensation unit 205, can be highly integrated, but for interpretive purposes... Figure 5 The examples are shown separately.

[0231] The segmentation unit 201 can segment an image into one or more video blocks. The video encoder 200 and the video decoder 300 can support various video block sizes.

[0232] The mode selection unit 203 can, for example, select one of the intra-frame or inter-frame codec modes based on the error result, and provide the obtained intra-frame or inter-frame codec blocks to the residual generation unit 207 to generate residual block data and to the reconstruction unit 212 to reconstruct the codec blocks for use as reference images. In some examples, the mode selection unit 203 can select a combined intra-frame and inter-frame prediction (CIIP) mode, where the prediction is based on the inter-frame prediction signal and the intra-frame prediction signal. The mode selection unit 203 can also select the resolution of the motion vector (e.g., sub-pixel or integer pixel precision) for the blocks in the inter-frame prediction case.

[0233] To perform inter-frame prediction for the current video block, motion estimation unit 204 can generate motion information for the current video block by comparing one or more reference frames from buffer 213 with the current video block. Motion compensation unit 205 can determine the predicted video block for the current video block based on the motion information of the image from buffer 213 (rather than the image associated with the current video block) and decoded samples.

[0234] The motion estimation unit 204 and the motion compensation unit 205 can perform different operations on the current video block, for example, the different operations performed depend on whether the current video block is in an I-strip, a P-strip, or a B-strip.

[0235] In some examples, motion estimation unit 204 can perform unidirectional prediction of the current video block, and can search for a reference video block for the current video block in the reference images of list 0 or list 1. Motion estimation unit 204 can then generate a reference index indicating that the reference image in list 0 or list 1 contains the reference video block, and a motion vector indicating the spatial displacement between the current video block and the reference video block. Motion estimation unit 204 can output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 205 can generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0236] In other examples, motion estimation unit 204 can perform bidirectional prediction of the current video block. Motion estimation unit 204 can search for a reference video block for the current video block in the reference images of list 0 and can also search for another reference video block for the current video block in the reference images of list 1. Motion estimation unit 204 can then generate a reference index indicating that the reference images in list 0 or list 1 contain the reference video block, and a motion vector indicating the spatial displacement between the reference video block and the current video block. Motion estimation unit 204 can output the reference index and the motion vector of the current video block as the motion information of the current video block. Motion compensation unit 205 can generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0237] In some examples, the motion estimation unit 204 can output the complete set of motion information for the decoder's decoding process.

[0238] In some examples, motion estimation unit 204 may not output the complete set of motion information for the current video. Instead, motion estimation unit 204 may signal the motion information of the current video block by referencing the motion information of another video block. For example, motion estimation unit 204 may determine that the motion information of the current video block is sufficiently similar to the motion information of adjacent video blocks.

[0239] In one example, the motion estimation unit 204 may indicate in the syntax structure associated with the current video block that the current video block has the same motion information value as another video block.

[0240] In another example, motion estimation unit 204 can identify another video block and motion vector difference (MVD) in the syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicating video block. Video decoder 300 can use the motion vector of the indicating video block and the motion vector difference to determine the motion vector of the current video block.

[0241] As discussed above, the video encoder 200 can predictively signal motion vectors. Two examples of predictive signaling notification techniques that can be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and merge pattern signaling notification.

[0242] Intra-prediction unit 206 can perform intra-prediction on the current video block. When intra-prediction unit 206 performs intra-prediction on the current video block, it can generate prediction data for the current video block based on decoded samples from other video blocks in the same frame. The prediction data for the current video block can include the predicted video block and various syntax elements.

[0243] The residual generation unit 207 can generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) multiple predicted video blocks from the current video block. The residual data for the current video block can include residual video blocks corresponding to different sample components of the samples in the current video block.

[0244] In other examples, such as in skip mode, residual data for the current video block may not exist, and the residual generation unit 207 may not perform a subtraction operation.

[0245] The transform processing unit 208 can generate one or more transform coefficient video blocks of the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0246] After the transform processing unit 208 generates a transform coefficient video block associated with the current video block, the quantization unit 209 can quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0247] The inverse quantization unit 210 and the inverse transform unit 211 can apply inverse quantization and inverse transform to the transform coefficient video block respectively to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 212 can add the reconstructed residual video block to the corresponding samples of one or more predicted video blocks generated by the prediction unit 202 to produce a reconstructed video block associated with the current block for storage in the buffer 213.

[0248] After the video block is reconstructed in reconstruction unit 212, a loop filtering operation can be performed to reduce video block artifacts in the video block.

[0249] Entropy encoding unit 214 can receive data from other functional components of video encoder 200. When entropy encoding unit 214 receives data, it can perform one or more entropy encoding operations to generate entropy encoded data and output a bitstream including the entropy encoded data.

[0250] Figure 10 This is a block diagram illustrating an example of a video decoder 300, which may be... Figure 8 The video decoder 114 in the system 100 shown in the figure.

[0251] The video decoder 300 can be configured to perform any or all of the techniques disclosed herein. Figure 10 In the example, the video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video decoder 300. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0252] exist Figure 10 In the example, the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra-frame prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, a reconstruction unit 306, and a buffer 307. In some examples, the video decoder 300 can perform operations related to the video encoder 200 ( Figure 9 The decoding process is the overall inversion of the encoding process described.

[0253] Entropy decoding unit 301 can retrieve the encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded blocks of video data). Entropy decoding unit 301 can decode the entropy-encoded video, and based on the entropy-encoded video data, motion compensation unit 302 can determine motion information including motion vectors, motion vector precision, reference image list index, and other motion information. Motion compensation unit 302 can determine such information, for example, by performing AMVP and merge modes.

[0254] The motion compensation unit 302 can generate motion compensation blocks, possibly based on interpolation filters. The identifier of the interpolation filter to be used at sub-pixel precision can be included in the syntax element.

[0255] The motion compensation unit 302 can use the interpolation filter used by the video encoder 200 during the encoding of the video block to calculate the interpolation values ​​of a sub-integer number of pixels of the reference block. The motion compensation unit 302 can determine the interpolation filter used by the video encoder 200 based on the received syntax information and use the interpolation filter to generate the prediction block.

[0256] The motion compensation unit 302 can use some syntactic information to determine: the size of the blocks used to encode (multiple) frames and / or (multiple) stripes of the encoded video sequence, segmentation information describing how each macroblock of the image of the encoded video sequence is segmented, a mode indicating how each segment is encoded, one or more reference frames (and a list of reference frames) for each inter-frame coded block, and other information for decoding the encoded video sequence.

[0257] Intra-prediction unit 303 can use, for example, an intra-prediction mode received in the bitstream to form prediction blocks from spatially adjacent blocks. Inverse quantization unit 303 inverse quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 301. Inverse transform unit 303 applies an inverse transform.

[0258] The reconstruction unit 306 can sum the residual blocks using the corresponding prediction blocks generated by the motion compensation unit 202 or the intra-frame prediction unit 303 to form a decoded block. As desired, a deblocking filter can also be applied to filter the decoded block to remove block artifacts. The decoded video block is then stored in a buffer 307, which provides a reference block for subsequent motion compensation / intra-frame prediction and also produces the decoded video for presentation on the display device.

[0259] The following provides a list of preferred solutions for some embodiments.

[0260] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., all projects).

[0261] 1. A video processing method comprising: performing a conversion between a video comprising one or more sub-pictures or a sequence of one or more sub-pictures and a video codec representation, wherein the codec representation conforms to a format rule specifying whether or how scalable nested supplementary enhancement information (SEI) is included in the codec representation.

[0262] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 1).

[0263] 2. According to the method described in Solution 1, wherein the format rules specify that the encoding and decoding representation uses sub-image indexes to associate sub-images with corresponding scalable nested SEI information.

[0264] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 2).

[0265] 3. The method according to any one of solutions 1-2, wherein the format rules do not allow the use of the filter payload SEI message in a scalable nested manner.

[0266] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 3).

[0267] 4. The method according to any one of solutions 1-3, wherein the format rules specify that for scalable nested SEI messages that include one or more sub-picture level information SEI messages, a flag is included in the codec representation to indicate its presence.

[0268] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 4).

[0269] 5. The method according to any one of solutions 1-4, wherein the formatting rules prohibit nested SEI messages containing payloads of a specific type within a scalable SEI message containing a specific type of message.

[0270] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 5).

[0271] 6. The method according to any one of Schemes 1-5, wherein the format rules specify that SEI messages that are not padding payload types or decoded image hash types need to have a specific Network Abstraction Layer Unit type.

[0272] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 6).

[0273] 7. The method according to any one of Schemes 1-6, wherein the format rules specify that decoding an SEI message of image hash type requires having a specific network abstraction layer unit type.

[0274] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 7).

[0275] 8. The method according to any one of solutions 1-7, wherein the formatting rules specify that the syntax elements specify information about the sub-pictures of a layer having multiple sub-pictures for each picture.

[0276] 9. The method according to any one of solutions 1 to 8, wherein the conversion includes encoding the video into a codec representation.

[0277] 10. The method according to any one of solutions 1 to 8, wherein the conversion includes decoding the encoding / decoding representation to generate pixel values ​​of the video.

[0278] 11. A video decoding apparatus, comprising a processor configured to implement one or more of the methods described in solutions 1 to 10.

[0279] 12. A video encoding apparatus, comprising a processor configured to implement one or more of the methods described in solutions 1 to 10.

[0280] 13. A computer program product having computer code stored thereon, which, when executed by a processor, causes the processor to implement the method described in any one of solutions 1 to 10.

[0281] 14. The methods, apparatus or systems described in this document.

[0282] In the solution described in this paper, the encoder conforms to the format rules by generating a codec representation based on those rules. In the solution described in this paper, the decoder uses the format rules to parse the syntax elements in the codec representation, determining the presence or absence of syntax elements to generate the decoded video.

[0283] Figure 13 This is a flowchart of an example method 1300 for processing video data. Operation 1302 includes performing a conversion between a video comprising one or more sub-pictures and a video bitstream, wherein, during the conversion, one or more supplementary enhancement information messages with padding payloads are processed according to format rules, and wherein the format rules do not allow one or more supplementary enhancement information messages with padding payloads to be in scalable nested supplementary enhancement information messages.

[0284] In some embodiments of method 1300, one or more supplemental enhancement messages with padding payloads include a payload type with a value equal to 3. In some embodiments of method 1300, the formatting rules do not allow one or more second supplemental enhancement messages with scalable nesting to be within a scalable nested supplemental enhancement message.

[0285] Figure 14This is a flowchart of an example method 1400 for processing video data. Operation 1402 includes performing a conversion between video and a video bitstream, wherein, during the conversion, one or more syntax elements are processed according to format rules, and wherein the format rules specify one or more syntax elements for indicating sub-picture information of a layer of video having pictures having multiple sub-pictures.

[0286] In some embodiments of method 1400, one or more syntax elements include a first syntax element, wherein the value of the first syntax element plus 1 specifies the number of subpicks in an image having multiple subpicks. In some embodiments of method 1400, the value of the first syntax element is less than or equal to the value of a second syntax element in a sequence parameter set referenced by the images in the multiple subpick layers. In some embodiments of method 1400, one or more syntax elements include a third syntax element, wherein the third syntax element indicates the subpick index of the i-th subpick in each of the images having multiple subpicks. In some embodiments of method 1400, the first syntax element is labeled sn_num_subpics_minus1. In some embodiments of method 1400, the third syntax element is labeled sn_subpic_idx[i].

[0287] Figure 15 This is a flowchart of an example method 1500 for processing video data. Operation 1502 includes performing a conversion between a video comprising multiple sub-pictures and a video bitstream, wherein, during the conversion, a scalable nested supplemental enhancement information message is processed according to format rules, and wherein the format rules specify the use of one or more sub-picture indices to associate one or more sub-pictures with the scalable nested supplemental enhancement information message.

[0288] In some embodiments of method 1500, the formatting rule disallows the use of one or more subpicture identifiers to associate one or more subpictures with a scalable nested supplemental enhancement information message. In some embodiments of method 1500, the formatting rule replaces the first syntax element with a second syntax element in the scalable nested supplemental enhancement information message, and removes a third syntax element from the scalable nested supplemental information message. The first syntax element indicates a subpicture identifier for the i-th subpicture in each of one or more video layers, the second syntax element indicates a subpicture index for the i-th subpicture in each of one or more video layers, and the third syntax element plus 1 specifies the number of bits used to represent the first syntax element.

[0289] Figure 16This is a flowchart of an example method 1600 for processing video data. Operation 1602 includes performing a conversion between a video and a video bitstream comprising one or more sub-pictures according to a format rule, wherein the format rule specifies that, in response to a scalable nested supplemental enhancement information message comprising one or more sub-picture level information supplemental enhancement information messages, a first syntax element in the scalable nested supplemental enhancement information message in the bitstream is set to a specific value, and wherein the specific value of the first syntax element indicates that the scalable nested supplemental enhancement information message comprises one or more scalable nested supplemental enhancement information messages applicable to a particular set of output video layers.

[0290] In some embodiments of method 1600, the format rule specifies that, in response to a scalable nested supplemental enhancement message including one or more sub-picture level supplemental enhancement messages, the value of a second syntax element in the bitstream is equal to 0, and the value of the second syntax element being equal to 0 indicates that the scalable nested supplemental enhancement message includes one or more scalable nested supplemental enhancement messages applicable to one or more output video layer sets or one or more video layers, and the one or more output video layer sets or one or more video layers are applicable to all subpictures of the one or more output video layer sets or one or more video layers. In some embodiments of method 1600, the format rule specifies that the payload type of one or more scalable nested supplemental enhancement messages in the scalable nested supplemental enhancement message is 203, and the payload type 203 of the supplemental enhancement message indicates that the supplemental enhancement message is a sub-picture level supplemental enhancement message. In some embodiments of method 1600, a specific value of the first syntax element is equal to 1.

[0291] Figure 17 This is a flowchart of an example method 1700 for processing video data. Operation 1702 includes performing a conversion between a video comprising multiple sub-pictures and a bitstream of the video, wherein the conversion is based on a format rule that disallows scalable nested supplemental enhancement messages including a first supplemental enhancement message of a first payload type and a second supplemental enhancement message of a second payload type.

[0292] In some embodiments of method 1700, the first payload type includes a payload type of a buffer period supplemental enhancement information message. In some embodiments of method 1700, the first payload type includes a payload type of a picture timing supplemental enhancement information message. In some embodiments of method 1700, the first payload type includes a payload type of a decoding unit information supplemental enhancement information message. In some embodiments of method 1700, the first payload type includes a payload type of a sub-picture level information supplemental enhancement information message. In some embodiments of method 1700, the second payload type includes a payload type that is not one of the following: (i) a buffer period supplemental enhancement information message payload type, (ii) a picture timing supplemental enhancement information message payload type, (iii) a decoding unit information supplemental enhancement information message payload type, and (iv) a sub-picture level information supplemental enhancement information message payload type.

[0293] Figure 18 This is a flowchart of an example method 1800 for processing video data. Operation 1802 includes performing a conversion between video and a video bitstream, wherein the conversion is performed according to format rules that specify that, in response to the supplemental enhancement information network abstraction layer unit (NEL unit), a scalable nested LEL message is included. The LEL unit includes a NLB type equal to the prefixed LEL unit type, wherein the scalable nested LEL message includes a LEL message unrelated to a specific payload type.

[0294] In some embodiments of method 1800, the network abstraction layer unit type is equal to PREFIX_SEI_NUT.

[0295] Figure 19 This is a flowchart of an example method 1900 for processing video data. Operation 1902 includes performing a conversion between video and a video bitstream, wherein the conversion is performed according to format rules that specify that, in response to a supplemental enhancement information network abstraction layer unit (NEL unit), a scalable nested LEL message is included. The LEL unit includes a NLB type equal to the suffix LEL unit type, wherein the scalable nested LEL message includes a LEL message associated with a specific payload type.

[0296] In some embodiments of method 1900, the network abstraction layer unit type is equal to SUFFIX_SEI_NUT.

[0297] In some embodiments of methods 1800-1900(a), a specific payload type is the payload type of a decoded image hash supplemental enhancement message. In some embodiments of methods 1800-1900(a), a specific payload type is associated with a value equal to 132.

[0298] In some embodiments of methods 1300-1900, performing the conversion includes encoding video into a bitstream. In some embodiments of methods 1300-1900, performing the conversion includes generating a bitstream from video, and the method further includes storing the bitstream in a non-transitory computer-readable recording medium. In some embodiments of methods 1300-1900, performing the conversion includes decoding video from the bitstream. In some embodiments, a video decoding apparatus includes a processor configured to implement methods 1300-1900 or embodiments thereof. In some embodiments, a video encoding apparatus includes a processor configured to implement methods 1300-1900 or embodiments thereof. In some embodiments, a computer program product having computer instructions stored thereon, which, when executed by a processor, cause the processor to implement methods 1300-1900 or embodiments thereof. In some embodiments, a non-transitory computer-readable storage medium stores a bitstream generated according to methods 1300-1900 or embodiments thereof. In some embodiments, a non-transitory computer-readable storage medium stores instructions that enable a processor to implement methods(s) 1300-1900 or embodiments thereof. In some embodiments, a method for generating a bitstream includes: generating a bitstream of video according to methods(s) 1300-1900 or embodiments thereof, and storing the bitstream on a computer-readable program medium. In some embodiments, a method, apparatus, or bitstream generated according to the disclosed methods or systems described in this document.

[0299] Some embodiments of the disclosed technology involve making a decision or determination to enable a video processing tool or mode. In one example, when a video processing tool or mode is enabled, the encoder will use or implement the tool or mode in the processing of video blocks, but not necessarily modify the resulting bitstream based on the use of the tool or mode. That is, when a video processing tool or mode is enabled based on the decision or determination, the conversion from video blocks to a bitstream representation of the video will be performed using that video processing tool or mode. In another example, when a video processing tool or mode is enabled, the decoder will process the bitstream knowing that it has been modified based on the video processing tool or mode. That is, the conversion from the bitstream representation of the video to video blocks will be performed using the video processing tool or mode enabled based on the decision or determination.

[0300] Some embodiments of the technology disclosed herein include deciding or determining to disable video processing tools or modes. In one example, when video processing tools or modes are disabled, the encoder will not use the tools or modes in the conversion of video blocks to a bitstream representation of the video. In another example, when video processing tools or modes are disabled, the decoder will process the bitstream using knowledge that the bitstream has not yet been modified based on the video processing tools or modes that have been decided or determined to be disabled.

[0301] In this document, the term "video processing" can refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm can be applied during the conversion from the pixel representation of a video to the corresponding bitstream representation, and vice versa. As defined in the syntax, the bitstream representation of the current video block can, for example, correspond to bits that are co-occurring or scattered at different positions within the bitstream. For example, a macroblock can be encoded based on the error residual values ​​of the transformation and encoding / decoding, and also using bits in the header and other fields in the bitstream. Furthermore, during the conversion, the decoder can, based on this determination, parse the bitstream knowing that some fields may or may not be present, as described in the solutions above. Similarly, the encoder can determine whether to include or exclude certain syntax fields and generate the codec representation accordingly by including or excluding syntax fields from the codec representation.

[0302] The disclosures and other schemes, examples, embodiments, modules, and functional operations described in this document can be implemented in digital electronic circuits or in computer software, firmware, or hardware, containing the structures disclosed in this document and their equivalents, or combinations thereof. The disclosed and other embodiments can be implemented as one or more computer program products encoded on a computer-readable medium, i.e., one or more computer program instruction modules for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a complex influencing machine-readable propagating signals, or combinations thereof. The term "data processing apparatus" encompasses all means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or combinations thereof. Propagating signals are artificially generated signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device.

[0303] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on one computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a communications network.

[0304] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be performed by special-purpose logic circuitry (e.g., field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs)), and the apparatus can be implemented as special-purpose logic circuitry (e.g., FPGAs or ASICs).

[0305] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors in any type of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magneto-optical, magneto-optical, or optical disc) for storing data, or operatively coupled to receive data from or transfer data to a mass storage device (e.g., magneto-optical, magneto-optical, or optical disc), or both. However, a computer does not necessarily need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into special-purpose logic circuitry.

[0306] While this patent document contains numerous details, these details should not be construed as limiting any subject matter or the scope of the claims, but rather as descriptions of features specific to particular embodiments of a particular art. In this patent document, certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in various suitable sub-combinations. Furthermore, although features may be described above as operating in certain combinations and even initially claimed in the same manner, in certain circumstances one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0307] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order or sequence shown, or to perform all the operations shown, in order to achieve the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0308] Only a few implementations and examples are described, and other implementations, enhancements and variations can be made based on what is described and shown in this patent document.

Claims

1. A method for processing video data, comprising: Perform the conversion between the video and the video bitstream. The conversion is performed according to format rules, which stipulate that, in response to scalable nested supplemental enhancement information messages including supplemental enhancement information messages unrelated to a specific payload type, the supplemental enhancement information network abstraction layer unit including the scalable nested supplemental enhancement information message has a first network abstraction layer unit type equal to the prefix supplemental enhancement information network abstraction layer unit type. The formatting rules specify that one or more supplemental enhancement messages with padding payloads are not allowed to be included in scalable nested supplemental enhancement messages.

2. The method according to claim 1, wherein, The first network abstraction layer unit type is equal to PREFIX_SEI_NUT.

3. The method according to claim 1, wherein, The format rules specify that, in response to the scalable nested supplemental enhancement information message including a supplemental enhancement information message associated with the specific payload type, the supplemental enhancement information network abstraction layer unit including the scalable nested supplemental enhancement information message has a second network abstraction layer unit type equal to the suffix supplemental enhancement information network abstraction layer unit type.

4. The method according to claim 3, wherein, The second network abstraction layer unit type is equal to SUFFIX_SEI_NUT.

5. The method according to claim 1, wherein, The specific payload type is the payload type of the decoded image hash supplemental enhancement information message.

6. The method according to claim 1, wherein, The specific load type is associated with a value equal to 132.

7. The method according to claim 1, wherein, Performing the conversion includes encoding the video into the bitstream.

8. The method according to claim 1, wherein, Performing the conversion includes decoding the video from the bitstream.

9. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein, When the instruction is executed by the processor, the processor: Perform the conversion between the video and the video bitstream. The conversion is performed according to format rules, which stipulate that, in response to scalable nested supplemental enhancement information messages including supplemental enhancement information messages unrelated to a specific payload type, the supplemental enhancement information network abstraction layer unit including the scalable nested supplemental enhancement information message has a first network abstraction layer unit type equal to the prefix supplemental enhancement information network abstraction layer unit type. The formatting rules specify that one or more supplemental enhancement messages with padding payloads are not allowed to be included in scalable nested supplemental enhancement messages.

10. The apparatus according to claim 9, wherein, The first network abstraction layer unit type is equal to PREFIX_SEI_NUT.

11. The apparatus according to claim 9, wherein, The format rules specify that, in response to the scalable nested supplemental enhancement information message including supplemental enhancement information messages related to the specific payload type, the supplemental enhancement information network abstraction layer unit including the scalable nested supplemental enhancement information message has a second network abstraction layer unit type equal to the suffix supplemental enhancement information network abstraction layer unit type, and The second network abstraction layer unit type is equal to SUFFIX_SEI_NUT.

12. The apparatus according to claim 9, wherein, The specific payload type is the payload type of the decoded image hash supplemental enhancement message and is associated with a value equal to 132.

13. A non-transitory computer-readable storage medium for storing instructions, said instructions causing a processor to: Perform the conversion between the video and the video bitstream. in, The conversion is performed according to format rules, which specify that, in response to scalable nested supplemental enhancement information messages including supplemental enhancement information messages unrelated to a specific payload type, the supplemental enhancement information network abstraction layer unit including the scalable nested supplemental enhancement information message has a first network abstraction layer unit type equal to the prefix supplemental enhancement information network abstraction layer unit type. The formatting rules specify that one or more supplemental enhancement messages with padding payloads are not allowed to be included in scalable nested supplemental enhancement messages.

14. The non-transitory computer-readable storage medium according to claim 13, wherein, The first network abstraction layer unit type is equal to PREFIX_SEI_NUT.

15. The non-transitory computer-readable storage medium according to claim 13, wherein, The format rules specify that, in response to the scalable nested supplemental enhancement information message including a supplemental enhancement information message associated with the specific payload type, the supplemental enhancement information network abstraction layer unit including the scalable nested supplemental enhancement information message has a second network abstraction layer unit type equal to the suffix supplemental enhancement information network abstraction layer unit type, and The second network abstraction layer unit type is equal to SUFFIX_SEI_NUT.

16. The non-transitory computer-readable storage medium according to claim 13, wherein, The specific payload type is the payload type of the decoded image hash supplemental enhancement message and is associated with a value equal to 132.

17. A non-transitory computer-readable recording medium storing a bitstream of video, wherein a computer program is also stored thereon, When the computer program is executed by a processor, it generates the bit stream as described in claim 1.

18. The non-transitory computer-readable recording medium according to claim 17, wherein, The first network abstraction layer unit type is equal to PREFIX_SEI_NUT.

19. The non-transitory computer-readable recording medium according to claim 17, wherein, The format rules specify that, in response to the scalable nested supplemental enhancement information message including a supplemental enhancement information message associated with the specific payload type, the supplemental enhancement information network abstraction layer unit including the scalable nested supplemental enhancement information message has a second network abstraction layer unit type equal to the suffix supplemental enhancement information network abstraction layer unit type, and The second network abstraction layer unit type is equal to SUFFIX_SEI_NUT.

20. The non-transitory computer-readable recording medium according to claim 17, wherein, The specific payload type is the payload type of the decoded image hash supplemental enhancement message and is associated with a value equal to 132.

21. A method for storing a bitstream of video, comprising: The bitstream of the video is generated according to the format rules; as well as The bitstream is stored in a non-transitory computer-readable recording medium. The format rules specify that, in response to scalable nested supplemental enhancement information messages including supplemental enhancement information messages unrelated to a specific payload type, the supplemental enhancement information network abstraction layer unit including the scalable nested supplemental enhancement information message has a first network abstraction layer unit type equal to the prefix supplemental enhancement information network abstraction layer unit type. The formatting rules specify that one or more supplemental enhancement messages with padding payloads are not allowed to be included in scalable nested supplemental enhancement messages.

22. A video decoding apparatus comprising a processor configured to implement the method of any one of claims 1 to 8.

23. A video encoding apparatus comprising a processor configured to implement the method of any one of claims 1 to 8.

24. A bitstream generation method, comprising: The method according to any one of claims 1 to 8 generates a bitstream of video, and The bitstream is stored on a computer-readable program medium.

Images