System and method for signaling source picture timing information for temporal sublayers in video coding

Abstract

An apparatus may be configured to decode video data based on information included in a source picture timing information message. In one example, a source picture timing information message includes a syntax element specifying a highest temporal sub-layer, the source picture timing information message including source picture timing information about the highest temporal sub-layer, where a value of the first syntax element includes an end value in a range from an indicated temporal identifier to 7. In one example, the source picture timing information message includes a respective instance of a syntax element for each specified highest temporal sub-layer starting from the indicated temporal identifier, the syntax element specifying a scaling factor for determining a source picture interval for corresponding consecutive pictures in an output order.

Description

Technical Field

[0001] The present disclosure relates to video coding, and more particularly to techniques for signaling source picture timing information for encoded video. Background Art

[0002] Digital video capabilities may be incorporated into a variety of devices, including digital televisions, laptop or desktop computers, tablet computers, digital recording devices, digital media players, video gaming devices, cellular phones (including so-called smart phones), medical imaging devices, and the like. Digital video may be encoded according to a video coding standard. A video coding standard defines the format of a compatible bitstream that encapsulates encoded video data. A compatible bitstream is a data structure that can be received and decoded by a video decoding device to generate reconstructed video data. A video coding standard also defines a decoding process, and a decoder that follows the decoding process is said to be a compliant decoder. A video coding standard may incorporate video compression techniques. Examples of video coding standards include ISO / IEC MPEG-4 Visual and ITU-T H.264. (also known as ISO / IEC MPEG-4 AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC). HEVC is described in Rec. ITU-T H.265, High Efficiency Video Coding, November 2019, which is referred to herein as ITU-T H.265. VVC is described in Rec. ITU-T H.266, Versatile Video Coding, April 2022, which is incorporated herein by reference and is referred to herein as ITU-T H.266. Extensions and improvements to ITU-T H.266 are currently under consideration for the development of the next generation of video coding standards. For example, the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG), collectively referred to as the Joint Video Study Group (JVET), are working to standardize enhanced video coding techniques that go beyond the functionality of the VVC standard. The Enhanced Compression Model 7 (ECM 7), the algorithm description of the Enhanced Compression Model 7 (ECM 7), the ISO / IEC JTC1 / SC29 document: JVET-AB2025 (2022) incorporated herein by reference The 2011 Joint Test Model (JTM) study described the coding features under the JTM study, which is a potential enhancement of video coding technology beyond the capabilities of ITU-T H.266. It should be noted that the coding features of ECM 7 are implemented in the ECM reference software. As used herein, the term ECM may collectively refer to the algorithms included in ECM 7 and the specific implementation of the ECM reference software.

[0003] Video compression technology can reduce the data requirements for storing and transmitting video data. Video compression technology can reduce data requirements by exploiting the redundancy inherent in video sequences. Video compression technology can divide the video sequence into smaller and continuous parts (i.e., a group of pictures within a video sequence, pictures within a group of pictures, regions within pictures, sub-regions within regions, etc.). Intra-frame prediction coding techniques (e.g., spatial prediction techniques within pictures) and inter-frame prediction techniques (i.e., techniques (time) between pictures) can be used to generate the difference between the unit video data to be encoded and the reference unit of the video data. The difference can be called residual data. The residual data can be encoded as quantized transform coefficients. Syntax elements can involve residual data and reference coding units (e.g., intra-frame prediction mode indexes and motion information). The residual data and syntax elements can be entropy encoded. Entropy-encoded residual data and syntax elements can be included in the data structure that forms a compatible bitstream. Summary of the invention

[0004] In general, the present disclosure describes various techniques for encoding video data. Specifically, the present disclosure describes techniques for signaling source picture timing information in video encoding. It should be noted that although the techniques of the present disclosure are described with respect to ITU-T H.264, ITU-T H.265, ITU-T H.266, and ECM, the techniques of the present disclosure are generally applicable to video encoding. For example, in addition to those techniques included in ITU-T H.264, ITU-T H.265, ITU-H.266, and ECM, the encoding techniques described herein may be incorporated into video encoding systems (including video encoding systems based on future video encoding standards), including video block structures, intra-frame prediction techniques, inter-frame prediction techniques, transform techniques, filtering techniques, and / or other entropy coding techniques. Therefore, references to ITU-T H.264, ITU-T H.265, ITU-T H.266, and / or ECM are for descriptive purposes and should not be interpreted as limiting the scope of the techniques described herein. In addition, it should be noted that the incorporation of references herein is for descriptive purposes and should not be construed as limiting or creating ambiguity regarding the terms used herein. For example, where a definition of a term provided in an incorporated reference differs from that provided in another incorporated reference and / or as used herein, the term should be interpreted in a manner that broadly includes each corresponding definition and / or in a manner that includes each specific definition in alternatives.

[0005] In one example, a method for encoding video data includes: signaling a source picture timing information message; signaling a first syntax element specifying a highest temporal sublayer in the source picture timing information message, the source picture timing information message including source picture timing information about the highest temporal sublayer, wherein a value of the first syntax element is in a range from an indicated temporal identifier to 7, including the end value; and signaling a corresponding instance of a second syntax element for each specified highest temporal sublayer starting at the indicated temporal identifier in the source picture timing information message, wherein the second syntax element specifies a scaling factor for determining a source picture interval of corresponding consecutive pictures in output order.

[0006] In one example, a device includes one or more processors configured to perform the following operations: including: signaling a source picture timing information message; signaling in the source picture timing information message a first syntax element specifying a highest temporal sublayer, the source picture timing information message including source picture timing information about the highest temporal sublayer, wherein a value of the first syntax element is in a range from an indicated temporal identifier to 7, including the end value; and signaling in the source picture timing information message a corresponding instance of a second syntax element for each specified highest temporal sublayer starting at the indicated temporal identifier, wherein the second syntax element specifies a scaling factor for determining a source picture interval of corresponding consecutive pictures in output order.

[0007] In one example, a non-transitory computer-readable storage medium includes instructions stored thereon that, when executed, cause one or more processors of a device to: signal a source picture timing information message; signal a first syntax element specifying a highest temporal sublayer in the source picture timing information message, the source picture timing information message including source picture timing information about the highest temporal sublayer, wherein a value of the first syntax element is in a range from an indicated temporal identifier to 7, inclusive; and signal a corresponding instance of a second syntax element for each specified highest temporal sublayer starting at the indicated temporal identifier in the source picture timing information message, wherein the second syntax element specifies a scaling factor for determining a source picture interval of corresponding consecutive pictures in output order.

[0008] In one example, an apparatus includes: a component for performing the following operations: sending a signal to notify a source picture timing information message; a component for performing the following operations: sending a signal in the source picture timing information message to notify a first syntax element specifying a highest temporal sub-layer, the source picture timing information message including source picture timing information about the highest temporal sub-layer, wherein a value of the first syntax element is in a range from an indicated temporal identifier to 7, including the end value; and a component for performing the following operations: sending a signal in the source picture timing information message to notify a corresponding instance of a second syntax element for each specified highest temporal sub-layer starting at the indicated temporal identifier, wherein the second syntax element specifies a scaling factor for determining a source picture interval of corresponding consecutive pictures in output order.

[0009] In one example, a method for decoding video data includes: receiving a source picture timing information message; parsing a first syntax element specifying a highest temporal sublayer from the source picture timing information message, the source picture timing information message including source picture timing information about the highest temporal sublayer, wherein a value of the first syntax element is in a range from an indicated temporal identifier to 7, including the end value; and parsing a corresponding instance of a second syntax element for each specified highest temporal sublayer starting from the indicated temporal identifier from the source picture timing information message, wherein the second syntax element specifies a scaling factor for determining a source picture interval of corresponding consecutive pictures in output order.

[0010] In one example, a device includes one or more processors configured to: receive a source picture timing information message; parse from the source picture timing information message a first syntax element specifying a highest temporal sublayer, the source picture timing information message including source picture timing information about the highest temporal sublayer, wherein a value of the first syntax element is in a range from an indicated temporal identifier to 7, including the end value; and parse from the source picture timing information message a corresponding instance of a second syntax element for each specified highest temporal sublayer starting at the indicated temporal identifier, wherein the second syntax element specifies a scaling factor for determining a source picture interval of corresponding consecutive pictures in output order.

[0011] In one example, a non-transitory computer-readable storage medium includes instructions stored thereon that, when executed, cause one or more processors of a device to: receive a source picture timing information message; parse a first syntax element specifying a highest temporal sublayer from the source picture timing information message, the source picture timing information message including source picture timing information about the highest temporal sublayer, wherein a value of the first syntax element is in a range from an indicated temporal identifier to 7, inclusive; and parse a corresponding instance of a second syntax element for each specified highest temporal sublayer starting at the indicated temporal identifier from the source picture timing information message, wherein the second syntax element specifies a scaling factor for determining a source picture interval of corresponding consecutive pictures in output order.

[0012] In one example, an apparatus includes: a component for performing the following operations: receiving a source picture timing information message; a component for performing the following operations: parsing a first syntax element specifying a highest temporal sub-layer from the source picture timing information message, the source picture timing information message including source picture timing information about the highest temporal sub-layer, wherein a value of the first syntax element is in a range from an indicated temporal identifier to 7, including the end value; and a component for performing the following operations: parsing a corresponding instance of a second syntax element for each specified highest temporal sub-layer starting from the indicated temporal identifier from the source picture timing information message, wherein the second syntax element specifies a scaling factor for determining a source picture interval of corresponding consecutive pictures in output order.

[0013] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram illustrating an example of a system that may be configured to encode and decode video data in accordance with one or more techniques of this disclosure. Figure 2 is a conceptual diagram illustrating encoded video data and corresponding data structures in accordance with one or more techniques of this disclosure. Figure 3 is a conceptual diagram illustrating a data structure that encapsulates encoded video data and corresponding metadata in accordance with one or more techniques of this disclosure. Figure 4 is a conceptual diagram illustrating an example of components that may be included in an implementation of a system that may be configured to encode and decode video data in accordance with one or more techniques of this disclosure. Figure 5 is a block diagram illustrating an example of a video encoder that may be configured to encode video data in accordance with one or more techniques of this disclosure. Figure 6 is a block diagram illustrating an example of a video decoder that may be configured to decode video data in accordance with one or more techniques of this disclosure. DETAILED DESCRIPTION

[0015] The video content includes a video sequence consisting of a series of frames (or pictures). A series of frames may also be referred to as a group of pictures (GOP). Each video frame or picture may be divided into one or more regions. The region may be defined according to a set of rules for defining a basic unit (e.g., a video block) and a defined region. For example, the rule defining the region may be that the region must be an integer number of video blocks arranged in a rectangle. In addition, the video blocks in the region may be sorted according to a scanning mode (e.g., a raster scan). As used herein, the term "video block" may generally refer to a region of a picture, or may more specifically refer to a maximum array of sample values ​​that may be predictively encoded, its sub-partitions, and / or corresponding structures. In addition, the term "current video block" may refer to a region of a picture that is being encoded or decoded. A video block may be defined as an array of sample values. It should be noted that in some cases, a pixel value may be described as a sample value comprising a corresponding component of video data, which may also be referred to as a color component (e.g., brightness (Y) and chrominance (Cb and Cr) components or red, green, and blue components). It should be noted that in some cases, the terms "pixel value" and "sample value" may be used interchangeably. Additionally, in some cases, a pixel or sample may be referred to as a pel. A video sampling format (which may also be referred to as a chroma format) may define the number of chroma samples included in a video block relative to the number of luma samples included in the video block. For example, for a 4:2:0 sampling format, the sampling rate of the luma component is twice the sampling rate of the chroma components in both the horizontal and vertical directions. It should be noted that in some cases, the terms "luma" and "luminance" may be used interchangeably.

[0016] The video encoder may perform predictive coding on a video block and its sub-partitions. A video block and its sub-partitions may be referred to as a node. ITU-T H.264 specifies a macroblock including 16×16 luma samples. That is, in ITU-T H.264, a picture is segmented into macroblocks. ITU-T H.265 specifies a similar coding tree unit (CTU) structure (which may be referred to as a maximum coding unit (LCU)). In ITU-T H.265, a picture is segmented into CTUs. In ITU-T H.265, for a picture, the CTU size may be set to include 16×16, 32×32, or 64×64 luma samples. In ITU-T H.265, a CTU consists of a corresponding coding tree block (CTB) for each component of the video data, e.g., luma (Y) and chroma (Cb and Cr). It should be noted that a video having one luma component and two corresponding chroma components may be described as having two channels, i.e., a luma channel and a chroma channel. In addition, in ITU-T H.265, CTUs can be divided according to a quadtree (QT) partitioning structure, which allows the CTBs of a CTU to be divided into coding blocks (CBs). That is, in ITU-T H.265, CTUs can be divided into quadtree leaf nodes. According to ITU-T H.265, a luma CB together with two corresponding chroma CBs and associated syntax elements is referred to as a coding unit (CU). In ITU-TH.265, a signal can be sent to notify the minimum allowed size of a CB. In ITU-T H.265, the minimum allowed minimum size of a luma CB is 8×8 luma samples. In ITU-T H.265, the decision to encode a picture area using intra prediction or inter prediction is made at the CU level.

[0017] In ITU-T H.265, a CU is associated with a prediction unit structure having its root at the CU. In ITU-T H.265, the prediction unit structure allows the partitioning of luma CBs and chroma CBs to generate corresponding reference samples. That is, in ITU-T H.265, luma CBs and chroma CBs can be partitioned into corresponding luma prediction blocks and chroma prediction blocks (PBs), where PBs include blocks of sample values ​​to which the same prediction is applied. In ITU-T H.265, a CB can be divided into 1, 2, or 4 PBs. ITU-T H.265 supports PB sizes from 64×64 samples down to 4×4 samples. In ITU-T H.265, intra prediction data (e.g., intra prediction mode syntax elements) or inter prediction data (e.g., motion data syntax elements) corresponding to the PB are used to generate reference and / or predicted sample values ​​for the PB. ITU-T H.266 specifies a CTU with a maximum size of 128x128 luma samples. In ITU-H.266, CTU is divided according to a quadtree plus multi-type tree (QTMT or QT+MTT) structure. The QTMT structure in ITU-T H.266 allows the quadtree leaf nodes to be further divided by a binary tree (BT) structure. That is, in ITU-T H.266, the binary tree leaf nodes can be recursively divided vertically or horizontally. In addition, in ITU-T H.266, in addition to indicating binary segmentation, the multi-type tree can also indicate the so-called ternary (or ternary tree (TT)) segmentation. The ternary segmentation divides a block into three blocks vertically or horizontally. In the case of vertical TT segmentation, the block is segmented at one quarter of its width from the left edge and at one quarter of its width from the right edge, and in the case of horizontal TT segmentation, the block is segmented at one quarter of its height from the top edge and at one quarter of its height from the bottom edge.

[0018] As described above, each video frame or picture may be divided into one or more regions. For example, according to ITU-T H.265, each video frame or picture may be divided into one or more slices, and further divided into one or more tiles, wherein each slice includes a CTU sequence (e.g., arranged in raster scan order), and wherein the tile is a CTU sequence corresponding to a rectangular area of ​​the picture. It should be noted that in ITU-T H.265, a slice is a sequence of one or more slice segments starting from an independent slice segment and containing all subsequent subordinate slice segments (if any) before the next independent slice segment (if any). Slice segments (such as slices) are CTU sequences. Therefore, in some cases, the terms "slice" and "slice segment" may be used interchangeably to indicate a sequence of CTUs arranged in a raster scan order. In addition, it should be noted that in ITU-T H.265, a tile may be composed of CTUs contained in more than one slice, and a slice may be composed of CTUs contained in more than one tile. However, ITU-T H.265 specifies that one or both of the following conditions should be met: (1) all CTUs in a slice belong to the same tile; and (2) all CTUs in a tile belong to the same slice.

[0019] With respect to ITU-T H.266, a slice needs to consist of an integer number of complete tiles or an integer number of continuous complete CTU rows within a tile, rather than just an integer number of CTUs. It should be noted that in ITU-T H.266, the slice design does not include slice fragments (i.e., there are no independent / dependent slice fragments). Therefore, in ITU-T H.266, a picture may include a single tile, wherein the single tile is contained within a single slice, or a picture may include multiple tiles, wherein the multiple tiles (or their CTU rows) may be contained within one or more slices. In ITU-T H.266, a picture is divided into tiles by specifying the corresponding height of a tile row and the corresponding width of a tile column. Therefore, in ITU-T H.266, a tile is a rectangular CTU area within a specific tile row and a specific tile column position. In addition, it should be noted that ITU-T H.266 stipulates that a picture can be divided into sub-pictures, wherein a sub-picture is a rectangular CTU area within a picture. The upper left CTU of a sub-picture may be located at any CTU position within a picture, where the sub-picture is constrained to include one or more slices. Therefore, unlike tiles, sub-pictures do not have to be restricted to specific row and column positions. It should be noted that sub-pictures can be used to encapsulate regions of interest within a picture, and the sub-bitstream extraction process can be used to decode and display only specific regions of interest. That is, as described in further detail below, the bitstream of encoded video data includes a sequence of network abstraction layer (NAL) units, wherein the NAL unit encapsulates encoded video data (i.e., video data corresponding to a picture slice), or the NAL unit encapsulates metadata (e.g., parameter sets) for decoding the video data, and the sub-bitstream extraction process forms a new bitstream by removing one or more NAL units from the bitstream.

[0020] Figure 2 is a conceptual diagram showing examples of pictures within picture groups partitioned according to tiles, slices, and sub-pictures. It should be noted that the techniques described herein can be applicable to tiles, slices, sub-pictures, their sub-partitions, and / or their equivalent structures. That is, regardless of how the pictures are divided into regions, the techniques described herein are generally applicable. For example, in some cases, the techniques described herein can be applicable to cases where tiles can be partitioned into so-called bricks, where a brick is a rectangular CTU row region within a particular tile. In addition, for example, in some cases, the techniques described herein can be applicable to cases where one or more tiles can be included in a so-called tile group, where a tile group includes an integer number of adjacent tiles. Figure 2 In the example shown, Pic3 is shown to include 16 tiles (ie, tiles 0 to 15 ) and three slices (i.e., slice 0 to slice 2). Figure 2In the example shown, slice 0 includes four tiles (ie, tiles 0 to 3), and slice 1 includes eight tiles (ie, tiles 4 to 5). 11 ), and slice 2 includes four tiles (i.e., tiles 12 To Tile 15 ). In addition, if Figure 2 As shown in the example of , Pic3 is shown to include two sub-pictures (i.e., sub-picture 0 and sub-picture 1), where sub-picture 0 includes slices 0 and slice 1 and where sub-picture 1 includes slice 2. As described above, sub-pictures can be used to encapsulate regions of interest within a picture, and a sub-bitstream extraction process can be used to selectively decode (and display) the regions of interest. For example, referring to Figure 2 , sub-picture 0 may correspond to the action portion of a sporting event presentation (e.g., a view of a field), and sub-picture 1 may correspond to a scrolling banner displayed during the sporting event presentation. By organizing pictures into sub-pictures in this manner, a viewer may be able to disable the display of a scrolling banner. That is, through the sub-bitstream extraction process, slice 2 NAL units may be removed from the bitstream (and therefore not decoded and / or displayed), and slice 0 NAL units and slice 1 NAL units may be decoded and displayed. How slices of a picture are encapsulated into corresponding NAL unit data structures and sub-bitstream extraction is described in further detail below.

[0021] As described above, the video sampling format (which may also be referred to as the chroma format) may define the number of chroma samples included in the CU relative to the number of luma samples included in the CU. For example, for the 4:2:0 sampling format, the sampling rate of the luma component is twice the sampling rate of the chroma components in both the horizontal and vertical directions. Therefore, for a CU formatted according to the 4:2:0 format, the width and height of the sample array for the luma component are twice the width and height of each sample array for the chroma component. As described above, the CU is generally defined according to the number of horizontal and vertical luma samples. Therefore, a 16×16 CU formatted according to the 4:2:0 sample format includes 16×16 samples of the luma component and 8×8 samples for each chroma component. For a CU formatted according to the 4:2:2 format, the width of the sample array of the luma component is twice the width of the sample array of each chroma component, but the height of the sample array of the luma component is equal to the height of the sample array of each chroma component. In addition, for a CU formatted according to the 4:4:4 format, the sample array of the luma component has the same width and height as the sample array of each chroma component.

[0022] For intra-frame prediction coding, the intra-frame prediction mode may specify the location of the reference sample within the picture. In ITU-T H.265, the possible intra-frame prediction modes defined include a plane (i.e., surface fitting) prediction mode, a DC (i.e., flat overall average) prediction mode, and 33 angle prediction modes (predMode: 2-34). In ITU-T H.266, the possible intra-frame prediction modes defined include a plane prediction mode, a DC prediction mode, and 65 angle prediction modes. In addition, in ITU-T H.266, additional intra-frame prediction tools are enabled, such as, for example, intra-frame sub-division mode and matrix-based intra-frame prediction. It should be noted that the plane prediction mode and the DC prediction mode may be referred to as a non-directional prediction mode, and the angle prediction mode may be referred to as a directional prediction mode. It should be noted that regardless of the number of possible prediction modes defined, the techniques described herein may be generally applicable.

[0023] For inter-frame prediction coding, a reference picture is determined, and a motion vector (MV) identifies samples in the reference picture that are used to generate a prediction for the current video block. For example, the current video block may be predicted using reference sample values ​​located in one or more previously coded pictures, and a motion vector is used to indicate the position of the reference block relative to the current video block. The motion vector may describe, for example, the horizontal displacement component of the motion vector (i.e., the MV x ), the vertical displacement component of the motion vector (i.e., MV y) and the resolution of the motion vector (e.g., quarter pixel accuracy, half pixel accuracy, one pixel accuracy, two pixel accuracy, four pixel accuracy). Previously decoded pictures (which may include pictures output before or after the current picture) may be organized into one or more reference picture lists and identified using reference picture index values. In addition, in inter-frame prediction coding, uni-prediction refers to generating predictions using sample values ​​from a single reference picture, and bi-prediction refers to generating predictions using corresponding sample values ​​from two reference pictures. That is, in uni-prediction, a single reference picture and a corresponding motion vector are used to generate a prediction for the current video block, while in bi-prediction, a first reference picture and a corresponding first motion vector and a second reference picture and a corresponding second motion vector are used to generate a prediction for the current video block. In bi-prediction, corresponding sample values ​​are combined (e.g., added, rounded and cropped, or averaged according to weights) to generate a prediction. Pictures and their regions may be classified based on what types of prediction modes may be used to encode their video blocks. That is, for a region with a B type (e.g., a B slice), bi-prediction, uni-prediction, and intra-prediction modes can be utilized, for a region with a P type (e.g., a P slice), uni-prediction and intra-prediction modes can be utilized, and for a region with an I type (e.g., an I slice), only an intra-prediction mode can be utilized. As described above, a reference picture is identified by a reference index. For example, for a P slice, there may be a single reference picture list RefPicList0, and for a B slice, there may be a second independent reference picture list RefPicList1 in addition to RefPicList0. It should be noted that for a uni-prediction in a B slice, one of RefPicList0 or RefPicList1 may be used to generate a prediction. Furthermore, it should be noted that during the decoding process, when decoding of a picture begins, a reference picture list is generated from a previously decoded picture stored in a decoded picture buffer (DPB).

[0024] In addition, the coding standard may support various motion vector prediction modes. Motion vector prediction enables the value of the motion vector for the current video block to be derived based on another motion vector. For example, a set of candidate blocks with associated motion information can be derived from the spatial neighboring blocks and the temporal neighboring blocks of the current video block. In addition, the generated (or default) motion information can be used for motion vector prediction. Examples of motion vector prediction include advanced motion vector prediction (AMVP), temporal motion vector prediction (TMVP), so-called "merge" mode, and "skip" and "direct" motion reasoning. In addition, other examples of motion vector prediction include advanced temporal motion vector prediction (ATMVP) and space-time motion vector prediction (STMVP). In addition, in ITU-T H.266, the following inter-frame prediction modes are enabled: affine motion model, adaptive motion vector resolution, bidirectional optical flow, decoder-side motion vector refinement and partition mode.

[0025] As described above, for inter-frame prediction coding, reference samples in previously encoded pictures are used to encode video blocks in the current picture. Previously encoded pictures that can be used as references when encoding the current picture are called reference pictures. It should be noted that the decoding order does not necessarily correspond to the picture output order, that is, the temporal order of pictures in a video sequence. In ITU-T H.266, when a picture is decoded, it is stored in a decoded picture buffer (DPB) (which may be called a frame buffer, a reference buffer, a reference picture buffer, etc.). In ITU-T H.266, pictures stored in the DPB are removed from the DPB when they are output and are no longer needed for encoding subsequent pictures. In ITU-T H.266, the determination of whether the picture should be removed from the DPB is called once for each picture after decoding the slice header, that is, at the beginning of decoding the picture. For example, the reference Figure 2 , Pic2 is shown as referenced to Pic1. Similarly, Pic3 is shown as referenced to Pic0. Figure 2, assuming that the number of pictures corresponds to the decoding order, the DPB will be filled as follows: after decoding Pic0, the DPB will include {Pic0}; at the beginning of decoding Pic1, the DPB will include {Pic0}; after decoding Pic1, the DPB will include {Pic0, Pic1}; at the beginning of decoding Pic2, the DPB will include {Pic0, Pic1}. Pic2 will then be decoded with reference to Pic1, and after decoding Pic2, the DPB will include {Pic0, Pic1, Pic2}. At the beginning of decoding Pic3, pictures Pic0 and Pic1 will be marked for removal from the DPB because they are not required for decoding Pic3 (or any subsequent pictures, not shown), and assuming that Pic1 and Pic2 have been output, the DPB will be updated to include {Pic0}. Pic3 will then be decoded with reference to Pic0. The process of marking pictures to remove them from the DPB can be called reference picture set (RPS) management.

[0026] As described above, intra prediction data or inter prediction data is used to generate reference sample values ​​for a block of sample values. The difference between the sample values ​​included in the current PB or another type of picture region structure and the associated reference samples (e.g., those generated using prediction) may be referred to as residual data. The residual data may include a corresponding difference array corresponding to each component of the video data. The residual data may be in the pixel domain. A transform such as a discrete cosine transform (DCT), a discrete sine transform (DST), an integer transform, a wavelet transform, or a conceptually similar transform may be applied to the difference array to generate transform coefficients. It should be noted that in ITU-T H.265 and ITU-T H.266, a CU is associated with a transform tree structure having its root at the CU level. The transform tree is divided into one or more transform units (TUs). That is, in order to generate transform coefficients, the array of difference values ​​may be divided (e.g., four 8×8 transforms may be applied to a 16×16 residual value array). For each component of the video data, such a subdivision of the difference values ​​may be referred to as a transform block (TB). It should be noted that in some cases a core transform and a subsequent secondary transform may be applied (in a video encoder) to generate transform coefficients. For a video decoder, the order of transforms is reversed.

[0027] The quantization process may be performed directly on the transform coefficients or residual sample values ​​(e.g., in the case of palette coded quantization). Quantization approximates the transform coefficients by limiting the amplitude to a set of specified values. Quantization essentially scales the transform coefficients in order to change the amount of data required to represent a set of transform coefficients. Quantization may include dividing the transform coefficients (or the values ​​obtained by adding the offset values ​​to the transform coefficients) by a quantization scaling factor and any associated rounding function (e.g., rounding to the nearest integer). The quantized transform coefficients may be referred to as coefficient level values. Inverse quantization (or "dequantization") may include multiplying the coefficient level values ​​by the quantization scaling factor, as well as any reciprocal rounding or offset addition operations. It should be noted that, as used herein, the term quantization process may refer to dividing by a scaling factor in some cases to generate a level value, and may refer to multiplying by a scaling factor in some cases to recover the transform coefficients. That is, the quantization process may refer to quantization in some cases, and to inverse quantization in some cases. In addition, it should be noted that although the quantization process is described in some of the examples below with respect to arithmetic operations associated with decimal notation, such description is for illustrative purposes and should not be construed as limiting. For example, the techniques described herein can be implemented in devices that use binary operations, etc. For example, the multiplication and division operations described herein can be implemented using bit shift operations, etc.

[0028] The quantized transform coefficients and syntax elements (e.g., syntax elements indicating the coding structure of the video block) may be entropy encoded according to an entropy coding technique. The entropy coding process includes encoding the syntax element values ​​using a lossless data compression algorithm. Examples of entropy coding techniques include content adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), probability interval partitioning entropy coding (PIPE), etc. The entropy-coded quantized transform coefficients and the corresponding entropy-coded syntax elements may form a compatible bitstream that can be used to reproduce the video data at the video decoder. The entropy coding process, such as CABAC, may include binarizing the syntax elements. Binarization refers to the process of converting the value of the syntax element into a sequence of one or more bits. These bits may be referred to as "bins". Binarization may include one or a combination of the following coding techniques: fixed length coding, unary coding, truncated unary coding, truncated Rice coding, Golomb coding, k-order exponential Golomb coding, and Golomb-Rice coding. For example, binarization may include representing the integer value 5 of the syntax element as 00000101 using an 8-bit fixed-length binarization technique, or representing the integer value 5 as 11110 using a unary coding binarization technique. As used herein, each of the terms fixed-length coding, unary coding, truncated unary coding, truncated Rice coding, Golomb coding, k-order exponential Golomb coding, and Golomb-Rice coding may refer to general implementations of these techniques and / or more specific implementations of these coding techniques. For example, a Golomb-Rice coding implementation may be specifically defined according to a video coding standard. In the example of CABAC, for a particular bin, the context provides a maximum probability state (MPS) value of the bin (i.e., the MPS of the bin is one of 0 or 1), and a probability value that the bin is an MPS or a minimum probability state (LPS). For example, the context may indicate that the MPS of the bin is 0, and the probability that the bin is 1 is 0.3. It should be noted that the context may be determined based on the value of the previously encoded bin including the bin in the current syntax element and the previously encoded syntax element. For example, the values ​​of syntax elements associated with neighboring video blocks may be used to determine the context of the current bin.

[0029] As described above, the video content includes a video sequence consisting of a series of pictures, and each picture can be divided into one or more regions. In ITU-T H.266, the coding representation of a picture includes a VCL NAL unit of a specific layer within an AU and contains all CTUs of the picture. For example, referring again to Figure 2, the coded representation of Pic3 is encapsulated in three coded slice NAL units (i.e., slice 0 NAL unit, slice 1 NAL unit, and slice 2 NAL unit). It should be noted that the term video coding layer (VCL) NAL unit is used as a general term for coded slice NAL units, that is, VCL NAL is a general term for slice NAL units of all types. As described above, and described in further detail below, NAL units can encapsulate metadata for decoding video data. NAL units that encapsulate metadata for decoding video sequences are generally referred to as non-VCL NAL units. Therefore, in ITU-T H.266, NAL units can be VCL NAL units or non-VCL NAL units. It should be noted that the VCL NAL unit includes slice header data, which provides information for decoding a specific slice. Therefore, in ITU-T H.266, information for decoding video data (which may be referred to as metadata in some cases) is not limited to being included in non-VCL NAL units. ITU-T H.266 specifies that a picture unit (PU) is a group of NAL units that are associated with each other according to a specified classification rule, are continuous in decoding order, and contain exactly one coded picture, and an access unit (AU) is a group of PUs that belong to different layers and contain coded pictures associated with the same time output from the DPB. ITU-T H.266 further specifies that a layer is a group of VCL NAL units and associated non-VCL NAL units that all have a specific value of the layer identifier. In addition, in ITU-T H.266, a PU consists of zero or one PH NAL unit, one coded picture (which consists of one or more VCL NAL units), and zero or more other non-VCL NAL units. In addition, in ITU-T H.266, a coded video sequence (CVS) is an AU sequence consisting of a CVSS AU and zero or more subsequent AUs that are non-CVSS AUs in decoding order (including all subsequent AUs up to (excluding) any subsequent AU that is a CVSS AU), wherein a coded video sequence start (CVSS) AU is an AU in which a PU exists for each layer in the CVS, and a coded picture in each existing picture unit is a coded layer video sequence start (CLVSS) picture. In ITU-T H.266, a coded layer video sequence (CLVS) is a PU sequence within the same layer, consisting of a CLVSS PU and zero or more subsequent non-CLVSS PUs in decoding order (including all subsequent PUs up to (excluding) any subsequent PU that is a CLVSS PU). That is, in ITU-T H.266, a bitstream can be described as including a sequence of AUs forming one or more CVSs.

[0030] Multi-layer video coding enables video presentation to be decoded / displayed as a presentation corresponding to a base layer of video data, and decoded / displayed as one or more additional presentations corresponding to an enhancement layer of video data. For example, a base layer can enable presentation of a video presentation with a basic quality level (e.g., high-definition presentation and / or a 30Hz frame rate), and an enhancement layer can enable presentation of a video presentation with an enhanced quality level (e.g., ultra-high-definition rendering and / or a 60Hz frame rate). The enhancement layer can be encoded by reference to the base layer. That is, for example, the picture in the enhancement layer can be encoded (e.g., using inter-layer prediction techniques) by reference to one or more pictures in the base layer (including a scaled version thereof). It should be noted that the layers can also be encoded independently of each other. In this case, there may be no inter-layer prediction between the two layers. Each NAL unit may include an identifier indicating the video data layer to which the NAL unit is associated. As described above, a sub-bitstream extraction process can be used to decode and display only a specific region of interest of a picture. In addition, a sub-bitstream extraction process can be used to decode and display only a specific video layer. Sub-bitstream extraction may refer to a process by which a device receiving a compatible or compliant bitstream forms a new compatible or compliant bitstream by discarding and / or modifying data in the received bitstream. For example, sub-bitstream extraction may be used to form a new compatible or compliant bitstream corresponding to a particular video representation (e.g., a high-quality representation).

[0031] In ITU-T H.266, each of a video sequence, GOP, picture, slice, and CTU may be associated with metadata describing video coding properties, and some types of metadata are encapsulated in non-VCL NAL units. ITU-T H.266 defines parameter sets that can be used to describe video data and / or video coding properties. Specifically, ITU-T H.266 includes the following four parameter sets: video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), and adaptation parameter set (APS), where SPS applies to zero or more integer CVSs, PPS applies to zero or more integer coded pictures, APS applies to zero or more slices, and VPS may be optionally referenced by SPS. PPS applies to one or more single coded pictures that reference it. In ITU-T H.266, parameter sets may be encapsulated as non-VCL NAL units and / or may be signaled as messages. ITU-T H.266 also includes a picture header (PH), which is encapsulated in its own NAL unit as a non-VCL NAL unit when signaled, or is encapsulated in a slice header of a coded slice as part of a VCL NAL unit when signaled. In ITU-T H.266, the picture header applies to all slices of a coded picture. ITU-T H.266 further enables the signaling of decoding capability information (DCI) and supplementary enhancement information (SEI) messages. In ITU-T H.266, DCI and SEI messages assist processes related to decoding, display or other purposes, however, DCI and SEI messages may not be required to construct luminance or chrominance samples according to the decoding process. In ITU-T H.266, DCI and SEI messages can be signaled in the bitstream using non-VCL NAL units. In addition, DCI and SEI messages can be transmitted by some mechanism instead of being present in the bitstream (i.e., signaled out of band).

[0032] Figure 3 An example of a bitstream including a plurality of CVSs is shown, wherein the CVSs include AUs, and the AUs include picture units. Figure 3 The example shown in corresponds to encapsulating Figure 2 An example of a slice NAL unit is shown in the example of . Figure 3 In the example shown, the corresponding picture unit of Pic3 includes three VCL NAL coded slice NAL units, namely slice 0 NAL unit, slice 1 NAL unit and slice 2 NAL unit, and two non-VCL NAL units, namely PPS NAL unit and PH NAL unit. Figure 3In , the header is the NAL unit header (i.e., not to be confused with the slice header). In addition, it should be noted that in Figure 3 In the example, other non-VCL NAL units not shown may be included in the CVS, such as SPS NAL units, VPS NAL units, SEI message NAL units, etc. In addition, it should be noted that in other examples, the PPS NAL unit for decoding Pic3 may be included elsewhere in the bitstream, for example, in the picture unit corresponding to Pic0, or may be provided by an external agency. In ITU-T H.266, the PH syntax structure may be present in the slice header of the VCL NAL unit or in the PH NAL unit of the current PU.

[0033] For the formulas used in this article, the following arithmetic operators can be used: +Addition - Subtraction * Multiplication, including matrix multiplication x y Exponentiation. Raises x to the power of y. In other contexts, such symbols are used for superscripts and are not intended to be interpreted as exponentiation. / Integer division with truncation toward zero. For example, 7 / 4 and -7 / -4 are truncated to 1, and -7 / 4 and 7 / -4 are truncated to -1. ÷ is used to represent division in mathematical formulas when truncation or rounding is not intended. Used to represent division in mathematical formulas without the intent to truncate or round.

[0034] Additionally, the following math functions are available: Log2(x) is the logarithm of x with base 2; Ceil(x), the smallest integer greater than or equal to x.

[0035] With respect to the exemplary syntax used herein, the following definitions of logical operators may be applied: x&&y Boolean logical "AND" of x and y x||y Boolean logical OR of x and y ! Boolean logic "NO" x? y:z If x is TRUE or not equal to 0, it evaluates to y; otherwise, it evaluates to z.

[0036] Additionally, the following relational operators can be applied: > greater than > = greater than or equal to < <=less than or equal to == is equal to ! = not equal to

[0037] Furthermore, it should be noted that among the syntax descriptors used in this document, the following descriptors may be applied: -b(8): Byte (8 bits) with any bit string pattern. The parsing of this descriptor is specified by the return value of the function read_bit(8). -f(n): A fixed-pattern bit string written using n bits (from left to right) starting from the leftmost bit. The parsing process of this descriptor is specified by the return value of the function read_bit(n). -se(v): signed integer order 0 Exp-Golomb coded syntax element, starting from the leftmost bit. -tb(v): truncated binary code using at most maxVal bits, where maxVal is defined in the semantics of the syntax element. -tu(v): truncated unary code using at most maxVal bits, where maxVal is defined in the semantics of the syntax element. -u(n): Use n-bit unsigned integers. When n is "v" in the syntax table, the number of bits varies in a way that depends on the values ​​of other syntax elements. The parsing process of this descriptor is specified by the return value of the function read_bits(n), which is interpreted as the binary representation of the unsigned integer with the most significant bit written first. -ue(v): unsigned integer 0th order Exp-Golomb coded syntax element, starting from the leftmost bit. -st(v): A null-terminated string encoded as Universal Coded Character Set (UCS) Transmission Format-8 (UTF-8) characters, as specified in ISO / IEC 10646. The parsing process is specified as follows: st(v) starts at a byte-aligned position in the bitstream, and reads and returns a series of bytes from the bitstream (the series of bytes starting at the current position and continuing up to but not including the next byte-aligned byte equal to 0x00), and advances the bitstream pointer by (stringLength+1)*8 bit positions, where stringLength is equal to the number of bytes returned.

[0038] As described above, ITU-T H.266 defines NAL unit header semantics, which specifies the type of raw byte sequence payload (RBSP) data structure included in the NAL unit. Table 1 shows the syntax of the NAL unit header provided in ITU-T H.266. Table 1

[0039] ITU-T H.266 provides the following definitions for the corresponding syntax elements shown in Table 1. forbidden_zero_bit should be equal to 0. nuh_reserved_zero_bit shall be equal to 0. A value of 1 for nuh_reserved_zero_bit may be specified by ITU-T|ISO / IEC in the future. Although in this version of this specification, the value of nuh_reserved_zero_bit is required to be equal to 0, decoders conforming to this version of this specification shall allow a value of nuh_reserved_zero_bit equal to 1 to appear in the syntax and shall ignore (i.e., remove and discard from the bitstream) NAL units with nuh_reserved_zero_bit equal to 1. nuh_layer_id specifies the identifier of the layer to which the VCL NAL unit belongs or the identifier of the layer to which the non-VCL NAL unit applies. The value of nuh_layer_id shall be in the range of 0 to 55, inclusive. Other values ​​of nuh_layer_id are reserved for future use by ITU-T | ISO / IEC. Although in this version of this specification, the value of nuh_layer_id is required to be in the range of 0 to 55, inclusive, a decoder conforming to this version of this specification shall allow values ​​of nuh_layer_id greater than 55 to appear in the syntax and shall ignore (i.e., remove and discard from the bitstream) NAL units with nuh_layer_id greater than 55. The value of nuh_layer_id of all VCL NAL units of a coded picture shall be the same. The value of nuh_layer_id of a coded picture or PU is the value of nuh_layer_id of the VCL NAL unit of the coded picture or PU. When nal_unit_type is equal to PH_NUT or FD_NUT, nuh_layer_id shall be equal to the nuh_layer_id of the associated VCL NAL unit. When nal_unit_type is equal to EOS_NUT, nuh_layer_id shall be equal to one of the nuh_layer_id values ​​of the layer present in the CVS. NOTE—The value of nuh_layer_id for DCI, OPI, VPS, AUD, and EOB NAL units is not constrained. nuh_temporal_id_plus1 minus 1 specifies the temporal identifier of the NAL unit. The value of nuh_temporal_id_plus1 shall not be equal to 0. The variable TemporalId is derived as follows: TemporalId=nuh_temporal_id_plus1-1 When nal_unit_type is in the range of IDR_W_RADL to RSV_IRAP_11 (inclusive), TemporalId shall be equal to 0. When nal_unit_type is equal to STSA_NUT and vps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] is equal to 1, TemporalId shall be greater than 0. The value of TemporalId shall be the same for all VCL NAL units of an AU. The value of TemporalId of a coded picture, PU, ​​or AU is the value of TemporalId of the VCL NAL unit of the coded picture, PU, ​​or AU. The value of TemporalId of a sub-layer representation is the maximum value of the TemporalId of all VCL NAL units in the sub-layer representation. The value of TemporalId for non-VCL NAL units is constrained as follows: – If nal_unit_type is equal to DCI_NUT, OPI_NUT, VPS_NUT or SPS_NUT, TemporalId shall be equal to 0, and the TemporalId of the AU containing the NAL unit shall be equal to 0. – Otherwise, if nal_unit_type is equal to PH_NUT, TemporalId shall be equal to the TemporalId of the PU containing the NAL unit. – Otherwise, if nal_unit_type is equal to EOS_NUT or EOB_NUT, TemporalId shall be equal to 0. – Otherwise, if nal_unit_type is equal to AUD_NUT, FD_NUT, PREFIX_SEI_NUT, or SUFFIX_SEI_NUT, the TemporalId shall be equal to the TemporalId of the AU containing the NAL unit. – Otherwise, when nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, The TemporalId shall be greater than or equal to the TemporalId of the PU containing the NAL unit. NOTE—When the NAL unit is a non-VCL NAL unit, the value of TemporalId is equal to the minimum of the TemporalId values ​​of all AUs to which the non-VCL NAL unit applies. When nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, TemporalId may be greater than or equal to the TemporalId of the containing AU, since all PPSs and APSs may be included in the beginning of the bitstream (e.g., when they are transported out-of-band and the receiver places them at the beginning of the bitstream), with the first coded picture having TemporalId equal to 0. nal_unit_type specifies the NAL unit type, ie, the type of RBSP data structure contained in the NAL unit as specified in Table 2. NAL units with a nal_unit_type in the range of UNSPEC28..UNSPEC31 (inclusive), whose semantics are unspecified, shall not affect the decoding process specified in this specification. NOTE—NAL unit types in the range of UNSPEC_28..UNSPEC_31 may be used as determined by the application. The decoding process for these values ​​of nal_unit_type is not specified in this specification. Since different applications may use these NAL unit types for different purposes, special attention is expected when designing encoders that generate NAL units with these nal_unit_type values ​​and when designing decoders that interpret the content of NAL units with these nal_unit_type values. This specification does not define any management of these values. These nal_unit_type values ​​may only be used in contexts where "conflicts" in use (i.e., the meaning of the NAL unit content for the same nal_unit_type value is defined differently) are not important, or are not possible or are managed, for example, defined or managed in a controlling application or transport specification, or managed by the environment that controls the distribution of the bitstream. For purposes other than determining the amount of data in a DU of the bitstream, a decoder should ignore (remove from the bitstream and discard) the contents of all NAL units using reserved values ​​of nal_unit_type. NOTE—This requirement allows for compatible extensions of this specification to be defined in the future. Table 2 NOTE—A clean random access (CRA) picture may have associated RASL or RADL pictures present in the bitstream. NOTE—An Instantaneous Decoding Refresh (IDR) picture with nal_unit_type equal to IDR_N_LP does not have an associated leading picture present in the bitstream. An IDR picture with nal_unit_type equal to IDR_W_RADL does not have an associated RASL picture present in the bitstream, but may have an associated RADL picture in the bitstream. The value of nal_unit_type shall be the same for all VCL NAL units of a sub-picture. A sub-picture is said to have the same NAL unit type as the VCL NAL units of the sub-picture. For any particular picture's VCL NAL unit, the following applies: – If pps_mixed_nalu_types_in_pic_flag is equal to 0, then for all VCL NAL units of the picture, The value of nal_unit_type shall be the same, and a picture or PU is referred to as having the same NAL unit type as the coded slice NAL units of the picture or PU. – Otherwise (pps_mixed_nalu_types_in_pic_flag is equal to 1), all of the following constraints apply: – An image should have at least two sub-images. – A VCL NAL unit of a picture shall have two or more different nal_unit_type values. – There shall be no VCL NAL units for pictures with nal_unit_type equal to GDR_NUT. – When the nal_unit_type of the VCL NAL unit of the picture is equal to the value of IDR_W_RADL, IDR_N_LP or When nalUnitTypeA is set to CRA_NUT, the nal_unit_type of other VCL NAL units of the picture should be equal to nalUnitTypeA or TRAIL_NUT. The value of nal_unit_type shall be the same for all pictures of an IRAP or GDR AU. When sps_video_parameter_set_id is greater than 0, vps_max_tid_il_ref_pics_plus1[i][j] is equal to 0 (for j equal to GeneralLayerIdx[nuh_layer_id] and any value of i in the range of j+1 to vps_max_layers_minus1, inclusive), and pps_mixed_nalu_types_in_pic_flag is equal to 1, the value of nal_unit_type shall not be equal to IDR_W_RADL, IDR_N_LP, or CRA_NUT. Bitstream conformance requires that the following constraints apply: – When a picture is the leading picture of an IRAP picture, the picture shall be a RADL or RASL picture. – When a sub-picture is a leading sub-picture of an IRAP sub-picture, the sub-picture shall be a RADL or RASL sub-picture. – When a picture is not the leading picture of an IRAP picture, the picture shall not be a RADL or RASL picture. – When a sub-picture is not a leading sub-picture of an IRAP sub-picture, the sub-picture shall not be a RADL or RASL sub-picture. – RASL pictures shall not be present in the bitstream, these RASL pictures are associated with IDR pictures. – RASL sub-pictures shall not be present in the bitstream. These RASL sub-pictures are associated with IDR sub-pictures. – RADL pictures shall not be present in the bitstream that are associated with IDR pictures with nal_unit_type equal to IDR_N_LP. NOTE—Random access at the location of an IRAP PU can be performed by discarding all PUs preceding the IRAP AU (and correctly decoding non-RASL pictures in the IRAP AU and all subsequent AUs in decoding order), provided that each parameter set is available (in the bitstream or by external means not specified in this specification) when referenced. – RADL sub-pictures shall not be present in the bitstream that are associated with IDR sub-pictures with nal_unit_type equal to IDR_N_LP. – Any pictures with nuh_layer_id equal to a particular value layerId that precede the IRAP picture with nuh_llayer_id equal to layerId in decoding order shall precede the IRAP picture in output order, and shall precede any RADL pictures associated with the IRAP picture in output order. –nuh_layer_id is equal to the specified value layerId and the sub-picture index is equal to the specified value subpicIdx in decoding order Any sub-picture preceding the IRAP sub-picture with nuh_layer_id equal to layerId and sub-picture index equal to subpicIdx shall precede the IRAP sub-picture and all its associated RADL sub-pictures in output order. – Any pictures with nuh_layer_id equal to a particular value layerId that precede the recovery-point picture with nuh_layer_id equal to layerId in decoding order shall precede the recovery-point picture in output order. – Any sub-picture with nuh_layer_id equal to a particular value layerId and a sub-picture index equal to a particular value subpicIdx that precedes the sub-picture with nuh_layer_id equal to layerId and sub-picture index equal to subpicIdx in the recovery point picture in decoding order shall precede that sub-picture in the recovery point picture in output order. – Any RASL pictures associated with a CRA picture shall be placed before any RADL pictures associated with the CRA picture in output order. Before picture. – Any RASL sub-picture associated with a CRA sub-picture shall precede any RADL sub-picture associated with the CRA sub-picture in output order. - Any RASL picture with nuh_layer_id equal to a particular value layerId associated with a CRA picture shall follow, in output order, any IRAP or GDR picture with nuh_layer_id equal to layerId that precedes the CRA picture in decoding order. – nuh_layer_id associated with a CRA sub-picture is equal to a specific value layerId and the sub-picture index is equal to a specific value Any RASL sub-picture with nuh_layer_id equal to layerId and sub-picture index equal to subpicIdx shall follow in output order any IRAP or GDR sub-picture that precedes the CRA sub-picture in decoding order with nuh_layer_id equal to layerId and sub-picture index equal to subpicIdx. – If sps_field_seq_flag is equal to 0, the following applies: When the current picture with nuh_layer_id equal to a specific value layerId is a leading picture associated with an IRAP picture, then the current picture shall precede all non-leading pictures associated with the same IRAP picture in decoding order. Otherwise (sps_field_seq_flag is equal to 1), let picA and picB be the first leading picture and the last leading picture associated with the IRAP picture in decoding order, respectively, and there shall be at most one non-leading picture with nuh_layer_id equal to layerId before picA in decoding order, and there shall be at most one non-leading picture with nuh_layer_id equal to layerId before picA in decoding order, and there shall be at most one non-leading picture with nuh_layer_id equal to layerId before picA in decoding order. There should be no non-leading pictures with nuh_layer_id equal to layerId. – If sps_field_seq_flag is equal to 0, the following applies: when the current sub-picture with nuh_layer_id equal to a specific value layerId and sub-picture index equal to a specific value subpicIdx is a leading sub-picture associated with an IRAP sub-picture, then the current sub-picture shall precede all non-leading sub-pictures associated with the same IRAP sub-picture in decoding order. Otherwise (sps_field_seq_flag is equal to 1), let subpicA and subpicB be the first leading subpicture and the last leading subpicture associated with the IRAP subpicture in decoding order, respectively, and should exist before subpicA in decoding order At most one non-leading subpicture with nuh_layer_id equal to layerId and subpicture index equal to subpicIdx, and there shall not be between picA and picB in decoding order. The non-leading picture of subpicIdx.

[0040] As provided in Table 2, the NAL unit may include a supplemental enhancement information (SEI) syntax structure. Tables 3 and 4 show the supplemental enhancement information (SEI) syntax structure provided in ITU-T H.266. Table 3 Table 4

[0041] Regarding Tables 3 and 4, ITU-T H.266 provides the following semantics: Each SEI message consists of variables that specify the type payloadType and size payloadSize of the SEI message payload. The derived SEI message payload size payloadSize is specified in bytes and shall be equal to the number of RBSP bytes in the SEI message payload. NOTE—The NAL unit byte sequence containing the SEI message may include one or more emulation prevention bytes (indicated by the emulation_prevention_three_byte syntax element). Since the payload size of the SEI message is specified in the RBSP bytes, the number of emulation prevention bytes is not included in the size of the SEI payload, payloadSize. payload_type_byte is a byte of the payload type of the SEI message. payload_size_byte is the payload size of the SEI message in bytes.

[0042] It should be noted that ITU-T H.266 defines payload types for SEI messages. Generic Supplementary Enhancement Information Message for Coding Video Bitstream, Rec. ITU-T H.274 (May 2022, which is referred to as H.274) defines additional SEI message payload types. In addition, "Technologies under consideration for extensions of VSEI (draft 1)" ISO / IEC JTC1 / SC29 31st Meeting, July 11-19, 2023, Geneva, Switzerland, document JVET-AE2032-v1 (referred to as JVET-AE2032), defines additional SEI message payload types under consideration for future extensions to H.274. For example, JVET-AE2032 provides a source picture timing SEI message. Table 5 shows the source picture timing SEI message provided in JVET-AE2032. Table 6

[0043] Regarding Table 6, JVET-AE2032 provides the following semantics: The Source Picture Timing Information (SPTI) SEI message indicates the temporal distance between source pictures associated with corresponding decoded output pictures before encoding. For example, for content captured by a camera, the temporal distance between source pictures is the difference between the time the image sensor is exposed to produce the source picture associated with the currently decoded picture and the time the image sensor is exposed to produce the source picture associated with the previous decoded picture in output order. spti_cancel_flag equal to 1 indicates that the SPTI SEI message cancels the persistence of any previous SPTI SEI message in output order applied to the current layer. spti_cancel_flag equal to 0 indicates that source picture timing information follows. spti_persistence_flag specifies the persistence of the SPTI SEI message of the current layer. spti_persistence_flag equal to 0 specifies that the SPTI SEI message applies only to the currently decoded picture. spti_persistence_flag equal to 1 specifies that the SPTI SEI message applies to the currently decoded picture and persists for all subsequent pictures of the current layer in output order until one or more of the following conditions is true: – A new CLVS starts for the current layer. – End of bitstream. – Output the picture in the current layer in the AU associated with the SPTI SEI message, which follows the current picture in the output order. spti_source_picture_timing_type indicates the timing relationship between the source picture and the corresponding decoded output picture as specified in Table 7, where (spti_source_picture_timing_type and bitMask) not equal to 0 indicates that the timing relationship has the interpretation associated with the bitMask value in Table 7. When spti_source_picture_timing_type is greater than 0 and (spti_source_picture_timing_type and bitMask) is equal to 0, the interpretation associated with the bitMask value does not apply to the SPTI SEI message. When spti_source_picture_timing_type is equal to 0, the timing relationship can be specified by the application. In bitstreams conforming to this version of this document, the value of spti_source_picture_timing_type shall be in the range of 0 to 127, inclusive. Values ​​of spti_source_picture_timing_type in the range of 128 to 255, inclusive, are reserved for future use by ITU-T | ISO / IEC and shall not be present in bitstreams conforming to this version of this document. Decoders conforming to this version of this document shall ignore SPTI SEI messages with spti_source_picture_timing_type in the range of 128 to 255, inclusive. bitMask explain 0x01 Slow Motion 0x02 Frame rate conversion 0x04 High-speed imaging 0x08 Time-lapse imaging 0x10 Time Reversal 0x20 Still image / freeze frame 0x40 Episodic or event-driven Table 7 The variable temporalReversalFlag is equal to (spti_source_picture_timing_type & 0x10)? 1:0. spti_source_timing_equals_output_timing_flag equal to 1 indicates that the timing of the source picture is the same as the timing of the corresponding decoded output picture. spti_source_timing_equals_output_timing_flag equal to 0 indicates that the timing of the source picture may be different from the timing of the corresponding decoded output picture. When spti_source_timing_equals_output_timing_flag is equal to 1 and there is a picture timing SEI message for the current picture, the source picture timing may be determined according to the information conveyed in the picture timing SEI message. spti_time_scale specifies the number of time units that pass in one second. The value of spti_time_scale should not be equal to 0. For example, the spti_time_scale for a time coordinate system that measures time using a 27MHz clock is 27000000. spti_num_units_in_elemental_source_picture_interval specifies the number of time units of a clock running at frequency spti_time_scale Hz that corresponds to the indicated elemental source picture interval of consecutive pictures in output order in CLVS. The indicated elemental source picture interval, also represented by the variable ElementalSourcePictureInterval, is in seconds and is equal to the quotient of spti_num_units_in_elemental_source_picture_interval divided by spti_time_scale. For example, to represent an elemental source picture interval equal to 0.04 seconds, spti_time_scale may be equal to 27000000 and spti_num_units_in_elemental_source_picture_interval may be equal to 1080000. spti_max_sublayers_minus_1 plus 1 specifies the maximum number of temporal sublayers that may exist in the CLVS. When present, spti_sublayer_source_picture_interval_scale_factor[i] specifies the scale factor used to determine the source picture interval of corresponding consecutive pictures in output order in the CLVS with TemporalId less than or equal to i. A value of 0 may be used to indicate that the source picture corresponding to the currently decoded output picture is the same as the source picture corresponding to the previously decoded output picture. The indicated source picture interval (represented by the variable SourcePictureInterval[i], in seconds) associated with an output picture having a TemporalId less than or equal to i is derived as follows: SourcePictureInterval[i]=ElementalSourcePictureInterval*spti_sublayer_source_picture_interval_scale_factor[i] When picture n is an output picture with a TemporalId less than or equal to i and is not the first picture of the output bitstream, the value of the variable SourcePictureTime[n] is derived as follows: – If temporalReversalFlag is equal to 0, then SourcePictureTime[n] = SourcePictureTime[previousPicInOutputOrder]+SourcePictureInterval[i] – Otherwise, temporalReversalFlag is equal to 1, SourcePictureTime[n] = SourcePictureTime[previousPicInOutputOrder]-SourcePictureInterval[i] where previousPicInOutputOrder is the last picture (if any) output with a TemporalId less than or equal to i that precedes picture n in the output order. When the value of SourcePictureTime[0] is not provided by external means not specified in this document, the value of SourcePictureTime[0] is inferred to be 0. When present, spti_sublayer_synthesized_picture_flag[i] equal to 1 indicates that the decoded output picture belonging to the i-th temporal sublayer is synthesized and does not correspond to an unmodified original source picture. spti_sublayer_synthesized_picture_flag[i] equal to 0 does not provide such an indication. When not present, the value of spti_sublayer_synthesized_picture_flag[i] is inferred to be equal to 0.

[0044] The source picture timing SEI message provided in JVET-AE2032 may not be ideal.

[0045] Figure 1 1 is a block diagram illustrating an example of a system that may be configured to encode (e.g., encode and / or decode) video data in accordance with one or more techniques of this disclosure. System 100 represents an example of a system that may encapsulate video data in accordance with one or more techniques of this disclosure. Figure 1 As shown, system 100 includes source device 102, communication medium 110, and target device 120. Figure 1 In the example shown, source device 102 may include any device configured to encode video data and transmit the encoded video data to communication medium 110. Target device 120 may include any device configured to receive the encoded video data via communication medium 110 and decode the encoded video data. Source device 102 and / or target device 120 may include computing devices equipped for wired and / or wireless communication, and may include, for example, set-top boxes, digital video recorders, televisions, desktop, laptop or tablet computers, game consoles, medical imaging devices, and mobile devices (including, for example, smart phones, cellular phones, personal gaming devices).

[0046] The communication medium 110 may include any combination of wireless and wired communication media and / or storage devices. The communication medium 110 may include coaxial cables, fiber optic cables, twisted pair cables, wireless transmitters and receivers, routers, switches, repeaters, base stations, or any other device that can be used to facilitate communication between various devices and sites. The communication medium 110 may include one or more networks. For example, the communication medium 110 may include a network configured to allow access to the World Wide Web, such as the Internet. The network may operate according to a combination of one or more telecommunication protocols. The telecommunication protocol may include proprietary aspects and / or may include standardized telecommunication protocols. Examples of standardized telecommunication protocols include digital video broadcasting (DVB) standards, advanced television system committee (ATSC) standards, integrated service digital broadcasting (ISDB) standards, cable data service interface specifications (DOCSIS) standards, global mobile communication systems (GSM) standards, code division multiple access (CDMA) standards, 3rd generation partnership project (3GPP) standards, European Telecommunications Standards Institute (ETSI) standards, Internet Protocol (IP) standards, wireless application protocol (WAP) standards, and Institute of Electrical and Electronics Engineers (IEEE) standards.

[0047] The storage device may include any type of device or storage medium capable of storing data. The storage medium may include a tangible or non-transitory computer-readable medium. The computer-readable medium may include an optical disk, a flash memory, a magnetic memory, or any other suitable digital storage medium. In some examples, the memory device or a portion thereof may be described as a non-volatile memory, and in other examples, a portion of the memory device may be described as a volatile memory. Examples of volatile memory may include random access memory (RAM), dynamic random access memory (DRAM), and static random access memory (SRAM). Examples of non-volatile memory may include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or an electrically programmable memory (EPROM) or an electrically erasable and programmable (EEPROM) memory. The storage device may include a memory card (e.g., a secure digital (SD) memory card), an internal / external hard disk drive, and / or an internal / external solid-state drive. Data may be stored on the storage device according to a defined file format.

[0048] Figure 4 is a conceptual diagram illustrating examples of components that may be included in a specific implementation of system 100. Figure 4 In the exemplary implementation shown, system 100 includes one or more computing devices 402A-402N, a television service network 404, a television service provider site 406, a wide area network 408, a local area network 410, and one or more content provider sites 412A-412N. Figure 4The specific implementations shown in represent examples of a system that can be configured to allow digital media content (such as movies, live sporting events, etc.) and data and applications associated therewith and media presentations to be distributed to and accessed by multiple computing devices (such as computing devices 402A to 402N). Figure 4 In the example shown, computing devices 402A to 402N may include any device configured to receive data from one or more of television service network 404, wide area network 408, and / or local area network 410. For example, computing devices 402A to 402N may be equipped for wired and / or wireless communications and may be configured to receive services via one or more data channels, and may include televisions, including so-called smart televisions, set-top boxes, and digital video recorders. In addition, computing devices 402A to 402N may include desktop computers, laptops or tablet computers, game consoles, mobile devices (including, for example, "smart" phones, cellular phones, and personal gaming devices).

[0049] The television service network 404 is an example of a network configured to enable distribution of digital media content that may include television services. For example, the television service network 404 may include a public air television network, a public or subscription-based satellite television service provider network, and a public or subscription-based cable television provider network and / or a cloud or Internet service provider. It should be noted that, although in some examples, the television service network 404 may be primarily used to allow the provision of television services, the television service network 404 may also allow the provision of other types of data and services according to any combination of the telecommunications protocols described herein. In addition, it should be noted that in some examples, the television service network 404 may allow two-way communication between the television service provider site 406 and one or more of the computing devices 402A to 402N. The television service network 404 may include any combination of wireless and / or wired communication media. The television service network 404 may include coaxial cables, fiber optic cables, twisted pair cables, wireless transmitters and receivers, routers, switches, repeaters, base stations, or any other device that can be used to facilitate communication between various devices and sites. The television service network 404 may operate according to a combination of one or more telecommunications protocols. The telecommunication protocol may include proprietary aspects and / or may include a standardized telecommunication protocol. Examples of standardized telecommunication protocols include the DVB standard, the ATSC standard, the ISDB standard, the DTMB standard, the DMB standard, the Data Over Cable Service Interface Specification (DOCSIS) standard, the HbbTV standard, the W3C standard, and the UPnP standard.

[0050] Reference again Figure 4, the television service provider site 406 may be configured to distribute television services via the television service network 404. For example, the television service provider site 406 may include one or more broadcast stations, cable television providers or satellite television providers, or Internet-based television providers. For example, the television service provider site 406 may be configured to receive transmissions (including television programs) via satellite uplink / downlink. In addition, as Figure 4 As shown, television service provider site 406 may be in communication with wide area network 408 and may be configured to receive data from content provider sites 412A through 412N. It should be noted that in some examples, television service provider site 406 may include a television studio and the content may originate from the television studio.

[0051] Wide area network 408 may include a packet-based network and operate according to a combination of one or more telecommunication protocols. Telecommunication protocols may include proprietary aspects and / or may include standardized telecommunication protocols. Examples of standardized telecommunication protocols include global system mobile communications (GSM) standards, code division multiple access (CDMA) standards, 3rd generation partnership project (3GPP) standards, European Telecommunications Standards Institute (ETSI) standards, European standards (EN), IP standards, wireless application protocols (WAP) standards, and Institute of Electrical and Electronics Engineers (IEEE) standards, such as, one or more IEEE 802 standards (e.g., Wi-Fi). Wide area network 408 may include any combination of wireless and / or wired communication media. Wide area network 408 may include coaxial cables, fiber optic cables, twisted pair cables, Ethernet cables, wireless transmitters and receivers, routers, switches, repeaters, base stations, or any other equipment that can be used to facilitate communication between various devices and sites. In one example, wide area network 408 may include the Internet. Local area network 410 may include a packet-based network and operate according to a combination of one or more telecommunication protocols. The local area network 410 may be distinguished from the wide area network 408 based on access levels and / or physical infrastructure. For example, the local area network 410 may include a secure home network.

[0052] Reference again Figure 4, content provider sites 412A to 412N represent examples of sites that can provide multimedia content to television service provider sites 406 and / or computing devices 402A to 402N. For example, a content provider site may include a studio with one or more studio content servers, which are configured to provide multimedia files and / or streams to television service provider sites 406. In one example, content provider sites 412A to 412N may be configured to provide multimedia content using an IP suite. For example, a content provider site may be configured to provide multimedia content to a receiver device according to a real-time streaming protocol (RTSP), HTTP, etc. In addition, content provider sites 412A to 412N may be configured to provide data including hypertext-based content, etc. to one or more of receiver devices 402A to 402N and / or television service provider sites 406 via a wide area network 408. Content provider sites 412A to 412N may include one or more web servers. The data provided by data provider sites 412A to 412N may be defined according to a data format.

[0053] Reference again Figure 1 , source device 102 includes video source 104, video encoder 106, data encapsulator 107 and interface 108. Video source 104 may include any device configured to capture and / or store video data. For example, video source 104 may include a camera and a storage device operably coupled thereto. Video encoder 106 may include any device configured to receive video data and generate a compatible bitstream representing the video data. A compatible bitstream may refer to a bitstream from which a video decoder can receive and reproduce video data. Various aspects of a compatible bitstream may be defined according to a video coding standard. When generating a compatible bitstream, video encoder 106 may compress the video data. The compression may be lossy (perceptible or imperceptible to the observer) or lossless. Figure 5 is a block diagram illustrating an example of a video encoder 500 that can implement the techniques described herein for encoding video data. It should be noted that although the exemplary video encoder 500 is illustrated as having different functional blocks, such illustration is intended for descriptive purposes and does not limit the video encoder 500 and / or its subcomponents to a particular hardware or software architecture. The functionality of the video encoder 500 may be implemented using any combination of hardware, firmware, and / or software implementations.

[0054] The video encoder 500 can perform intra-frame prediction encoding and inter-frame prediction encoding of a picture region and thus can be referred to as a hybrid video encoder. Figure 5In the example shown, the video encoder 500 receives a source video block. In some examples, the source video block may include picture regions that have been divided according to a coding structure. For example, the source video data may include macroblocks, CTUs, CBs, sub-partitions thereof, and / or additional equivalent coding units. In some examples, the video encoder 500 may be configured to perform additional subdivisions of the source video block. It should be noted that the techniques described herein are generally applicable to video encoding regardless of how the source video data is divided before and / or during encoding. Figure 5 In the example shown, the video encoder 500 includes an adder 502, a transform coefficient generator 504, a coefficient quantization unit 506, an inverse quantization and transform coefficient processing unit 508, an adder 510, an intra-frame prediction processing unit 512, an inter-frame prediction processing unit 514, a filter unit 516, and an entropy coding unit 518. Figure 5 As shown, video encoder 500 receives source video blocks and outputs a bitstream.

[0055] exist Figure 5 In the example shown, the video encoder 500 can generate residual data by subtracting the predicted video block from the source video block. The selection of the predicted video block is described in detail below. The summer 502 represents a component configured to perform the subtraction operation. In one example, the subtraction of the video block occurs in the pixel domain. The transform coefficient generator 504 applies a transform such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform (for example, four 8×8 transforms can be applied to a 16×16 residual value array) to the residual block or its sub-partition to generate a set of residual transform coefficients. The transform coefficient generator 504 can be configured to perform any and all combinations of transforms included in the discrete triangular transform series, including its approximation. The transform coefficient generator 504 can output the transform coefficients to the coefficient quantization unit 506. The coefficient quantization unit 506 can be configured to perform quantization of the transform coefficients. The quantization process can reduce the bit depth associated with some or all coefficients. The degree of quantization can change the rate distortion (i.e., the relationship between bit rate and video quality) of the encoded video data. The degree of quantization can be modified by adjusting a quantization parameter (QP). The quantization parameter can be determined based on a slice level value and / or a CU level value (e.g., a CU delta QP value). The QP data may include any data used to determine the QP used to quantize a particular set of transform coefficients. Figure 5 As shown, the quantized transform coefficients (which may be referred to as level values) are output to the inverse quantization and transform coefficient processing unit 508. The inverse quantization and transform coefficient processing unit 508 may be configured to apply inverse quantization and inverse transformation to generate reconstructed residual data. Figure 5As shown, at the summer 510, the reconstructed residual data can be added to the predicted video block. In this way, the encoded video block can be reconstructed, and the resulting reconstructed video block can be used to evaluate the coding quality of a given prediction, transformation and / or quantization. The video encoder 500 can be configured to perform multiple encoding rounds (e.g., performing encoding while changing one or more of the prediction, transformation parameters and quantization parameters). The rate distortion or other system parameters of the bitstream can be optimized based on the evaluation of the reconstructed video block. In addition, the reconstructed video block can be stored and used as a reference for predicting subsequent blocks.

[0056] Reference again Figure 5 , the intra-frame prediction processing unit 512 may be configured to select an intra-frame prediction mode for a video block to be encoded. The intra-frame prediction processing unit 512 may be configured to evaluate a frame and determine an intra-frame prediction mode for encoding a current block. As described above, possible intra-frame prediction modes may include a planar prediction mode, a DC prediction mode, and an angular prediction mode. Furthermore, it should be noted that in some examples, the prediction mode of the chrominance component may be inferred based on the prediction mode of the luma prediction mode. The intra-frame prediction processing unit 512 may select an intra-frame prediction mode after performing one or more encoding rounds. Furthermore, in one example, the intra-frame prediction processing unit 512 may select a prediction mode based on a rate-distortion analysis. As Figure 5 As shown, the intra prediction processing unit 512 outputs intra prediction data (e.g., syntax elements) to the entropy coding unit 518 and the transform coefficient generator 504. As described above, the transform performed on the residual data can be mode-dependent (e.g., the secondary transform matrix can be determined based on the prediction mode).

[0057] Reference again Figure 5 , the inter-frame prediction processing unit 514 may be configured to perform inter-frame prediction encoding for the current video block. The inter-frame prediction processing unit 514 may be configured to receive a source video block and calculate a motion vector of a PU of the video block. The motion vector may indicate the displacement of a prediction unit of a video block within a current video frame relative to a prediction block within a reference frame. Inter-frame prediction encoding may use one or more reference pictures. In addition, motion prediction may be unidirectional prediction (using one motion vector) or bidirectional prediction (using two motion vectors). The inter-frame prediction processing unit 514 may be configured to select a prediction block by calculating a pixel difference determined by, for example, the sum of absolute differences (SAD), the sum of square differences (SSD), or other difference metrics. As described above, a motion vector may be determined and specified based on motion vector prediction. As described above, the inter-frame prediction processing unit 514 may be configured to perform motion vector prediction. The inter-frame prediction processing unit 514 may be configured to generate a prediction block using motion prediction data. For example, the inter-frame prediction processing unit 514 may locate a prediction video block ( Figure 5). It should be noted that the inter-frame prediction processing unit 514 can be further configured to apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values ​​for motion estimation. The inter-frame prediction processing unit 514 can output the motion prediction data of the calculated motion vector to the entropy coding unit 518.

[0058] like Figure 5 As shown, filter unit 516 receives the reconstructed video blocks and the encoding parameters and outputs the modified reconstructed video data. Filter unit 516 may be configured to perform deblocking and / or sample adaptive offset (SAO) filtering. SAO filtering is a nonlinear amplitude mapping that can be used to improve reconstruction by adding an offset to the reconstructed video data. It should be noted that, as Figure 5 As shown, the intra prediction processing unit 512 and the inter prediction processing unit 514 may receive the modified reconstructed video block via the filter unit 216. The entropy coding unit 518 receives the quantized transform coefficients and the prediction syntax data (i.e., the intra prediction data and the motion prediction data). It should be noted that in some examples, the coefficient quantization unit 506 may perform a scan of the matrix including the quantized transform coefficients before outputting the coefficients to the entropy coding unit 518. In other examples, the entropy coding unit 518 may perform the scan. The entropy coding unit 518 may be configured to perform entropy coding according to one or more of the techniques described herein. As such, the video encoder 500 represents an example of a device configured to generate encoded video data according to one or more techniques of the present disclosure.

[0059] Reference again Figure 1 , the data encapsulator 107 can receive the encoded video data and generate a compatible bitstream according to a defined data structure, for example, a sequence of NAL units. A device receiving the compatible bitstream can reproduce the video data therefrom. In addition, as described above, sub-bitstream extraction can refer to the process by which a device receiving a compatible bitstream forms a new compatible bitstream by discarding and / or modifying data in the received bitstream. It should be noted that the term conforming bitstream can be used instead of the term compatible bitstream. In one example, the data encapsulator 107 can be configured to generate syntax according to one or more techniques described herein. It should be noted that the data encapsulator 107 does not necessarily have to be located in the same physical device as the video encoder 106. For example, the functions described as being performed by the video encoder 106 and the data encapsulator 107 can be distributed in Figure 4 in the device shown.

[0060] As described above, the signaling provided in JVET-AE2032 may be less than ideal. For example, it may not be desirable to allow an SPTI SEI message with a TemporalId greater than 0 to contain information about the source picture timing of a picture with a TemporalId lower than the TemporalId of the SPTI SEI message. If this is allowed, the information about the source picture timing in the lower temporal layers may be different, depending on whether all temporal layers are received or whether a subset of the temporal layers are received. In one example, according to the techniques of this document, an SPTI SEI message with a TemporalId greater than 0 may not contain information about the source picture timing of a picture with a TemporalId lower than the TemporalId of the SPTI SEI message. Tables 8 and 9 show examples of the syntax of source picture timing SEI messages according to the techniques of this document. Table 8 Table 9

[0061] With respect to Table 8, in one example, the semantics may be based on the semantics provided above and the following semantics: spti_additional_sublayers plus 1 specifies the number of temporal sublayers about which this SPTI SEI message contains source picture timing information. The value of spti_additional_sublayers shall be in the range of 0 to 7 - TemporalId, inclusive.

[0062] With respect to Table 9, in one example, the semantics may be based on the semantics provided above and the following semantics: spti_max_sublayer specifies the highest temporal sublayer for which this SPTI SEI message contains source picture timing information. The value of spti_max_sublayer should be in the range of TemporalId to 7, inclusive.

[0063] In one example, according to the techniques herein, a signaling flag may be sent that specifies whether signaling information (e.g., element source picture duration, synthesis flag) is sent for each temporal sublayer from TemporalID to the highest temporal sublayer (i.e., spti_max_sublayer) or only for the highest temporal sublayer (i.e., spti_max_sublayer). Table 10 shows an example of the syntax of a source picture timing SEI message according to the techniques herein. Table 10

[0064] With respect to Table 10, in one example, the semantics may be based on the semantics provided above and the following semantics: spti_temporal_sublayer_info_present_flag equal to 1 indicates that the syntax elements spti_sublayer_source_picture_interval_scale_factor[i] and spti_sublayer_synthesized_flag[i] are present when i is in the range of TemporalId to spti_max_sublayer (inclusive). spti_temporal_sublayer_info_present_flag equal to 0 indicates that the syntax elements spti_sublayer_source_picture_interval_scale_factor[i] and spti_sublayer_synthesized_flag[i] are not present when i is in the range of TemporalId to spti_max_sublayer-1, and are present when i is equal to spti_max_sublayer.

[0065] With respect to Table 8, Table 9, and Table 10, the TemporalId in the message syntax may be the TemporalId of the SEI NAL unit containing the SPTI SEI message.

[0066] It should be noted that the text in JVET-AE2032 specifies that when spti_persistence_flag is equal to 0, the SPTI SEI message applies only to the currently decoded picture. Therefore, in this case, there is no need to signal any information related to other pictures other than the current picture. According to the techniques herein, the presence of the syntax element spti_max_sublayers_minus_1 may be conditional on spti_persistence_flag being equal to 1. Tables 11 and 12 show examples of syntax for source picture timing SEI messages according to the techniques herein. Table 11 Table 12

[0067] With respect to Table 11, in one example, the semantics may be based on the semantics provided above and the following semantics: spti_additional_sublayers plus 1 specifies the number of temporal sublayers for which this SPTI SEI message contains source picture timing information. When spti_additional_sublayers is not present, it is inferred to be equal to 0.

[0068] With respect to Table 12, in one example, the semantics may be based on the semantics provided above and the following semantics: spti_max_sublayer specifies the highest temporal sublayer for which this SPTI SEI message contains source picture timing information. The value of spti_max_sublayer shall be in the range of TemporalId to 7, inclusive. When spti_max_sublayer is not present, it is inferred to be equal to TemporalId.

[0069] It should be noted that the values ​​of the syntax elements spti_time_scale and spti_num_units_in_elemental_source_picture_interval may always be selected so that the scale factor of the highest temporal layer signaled is equal to 1. Thus, according to the techniques herein, in one example, there is no need to signal the value of spti_sublayer_source_picture_interval_scale_factor[i] when i is the highest value. Table 13 shows an example of the syntax of a source picture timing SEI message according to the techniques herein. Table 13

[0070] With respect to Table 13, in one example, the semantics may be based on the semantics provided above and the following semantics: When present, spti_sublayer_source_picture_interval_scale_factor[i] specifies the scale factor used to determine the source picture interval of corresponding consecutive pictures in output order with TemporalId less than or equal to i in the CLVS. A value of 0 may be used to indicate that the source picture corresponding to the currently decoded output picture is the same as the source picture corresponding to the previously decoded output picture. When spti_sublayer_source_picture_interval_scale_factor[i] is not present, it is inferred to be equal to 1.

[0071] It should be noted that one of the most important properties of a bitstream with multiple temporal layers is that one or more of the highest temporal layers can be removed without affecting the lower temporal layers. Therefore, it is important to be able to consistently determine the source picture timing information for a picture with a specific TemporalId, regardless of whether one or more temporal layers (higher than the TemporalId of that picture) have been removed. To do this, in one example, according to the techniques herein, the source picture interval may be determined based on the TemporalId of each picture rather than based on a global variable i. Therefore, in one example, the semantics of spti_sublayer_source_picture_interval_scale_factor[i] may be based on the following: When present, spti_sublayer_source_picture_interval_scale_factor[i] specifies the scale factor used to determine the source picture interval of corresponding consecutive pictures in output order in the CLVS with TemporalId equal to i. A value of 0 may be used to indicate that the source picture corresponding to the currently decoded output picture is the same as the source picture corresponding to the previously decoded output picture. The indicated source picture interval (represented by the variable SourcePictureInterval[i], in seconds) associated with an output picture having TemporalId equal to i is derived as follows: SourcePictureInterval[i]=ElementalSourcePictureInterval*spti_sublayer_source_picture_interval_scale_factor[i] When picture n is the output picture with TemporalId equal to i and is not the first picture of the output bitstream, the value of the variable SourcePictureTime[n] is derived as follows: – If temporalReversalFlag is equal to 0, then SourcePictureTime[n] = SourcePictureTime[previousPicInOutputOrder]+SourcePictureInterval[i] – Otherwise, temporalReversalFlag is equal to 1, SourcePictureTime[n] = SourcePictureTime[previousPicInOutputOrder]-SourcePictureInterval[i] where previousPicInOutputOrder is the last picture (if any) outputted that precedes picture n in the output order and has a TemporalId less than or equal to that of picture n. When the value of SourcePictureTime[0] is not provided by external means not specified in this document, the value of SourcePictureTime[0] is inferred to be 0.

[0072] Furthermore, in one example, according to the techniques herein, it may be clarified how to derive SourcePictureTime for the first picture (in output order). Thus, in one example, the semantics of spti_sublayer_source_picture_interval_scale_factor[i] may be based on the following: When present, spti_sublayer_source_picture_interval_scale_factor[i] specifies the scale factor used to determine the source picture interval of corresponding consecutive pictures in output order in the CLVS with TemporalId equal to i. A value of 0 may be used to indicate that the source picture corresponding to the currently decoded output picture is the same as the source picture corresponding to the previously decoded output picture. The indicated source picture interval (represented by the variable SourcePictureInterval[i], in seconds) associated with an output picture having TemporalId equal to i is derived as follows: SourcePictureInterval[i]=ElementalSourcePictureInterval*spti_sublayer_source_picture_interval_scale_factor[i] When picture n is the output picture with TemporalId equal to i and is not the first picture of the output bitstream, the value of the variable SourcePictureTime[n] is derived as follows: For each output image n (in output order), the following applies: If n is equal to 0, the value of SourcePictureTime[n] is derived as follows: – If external means are available to set the value of SourcePictureTime[n], the value of SourcePictureTime[n] is set by external means. – Otherwise, SourcePictureTime[n] is inferred to be equal to 0. Otherwise (n is greater than 0), when picture n is a picture with TemporalId equal to i, the value of the variable SourcePictureTime[n] is derived as follows: – If temporalReversalFlag is equal to 0, SourcePictureTime[n] = SourcePictureTime[previousPicInOutputOrder]+SourcePictureInterval[i] – Otherwise, temporalReversalFlag is equal to 1, SourcePictureTime[n] = SourcePictureTime[previousPicInOutputOrder]-SourcePictureInterval[i] where previousPicInOutputOrder is the last picture (if any) outputted before picture n in the output order that has a TemporalId less than or equal to i.

[0073] It should be noted that the SPTI SEI message is intended for use with VVC, HEVC, and AVC. In order to use the SPTI SEI message under these standards, a description of how to use the SEI message in the context of the corresponding standard should be provided. Specifically, the variables used in the SEI message should be assigned values ​​from the corresponding standard. In one example, according to the technology of this article, the variables used in the SEI message can be assigned values ​​from the corresponding standard as follows: Using source picture timing SEI messages in VVC For the purpose of interpreting the source picture timing SEI message, the following variables are specified: –TemporalId is set equal to TemporalId. Using source picture timing SEI messages in HEVC For the purpose of interpreting the source picture timing SEI message, the following variables are specified: –TemporalId is set equal to TemporalId. Using source picture timing SEI messages in AVC For the purpose of interpreting the source picture timing SEI message, the following variables are specified: – If the bitstream conforms to any of the profiles defined in Annexes G, H, I, or J, then TemporalId Set equal to temporal_id. – Otherwise (the bitstream conforms to the profile defined in Annex A), set TemporalId equal to (nal_ref_idc == 0?0:1).

[0074] It should be noted that the use of SPTI SEI messages can be divided into two cases: If the source picture timing does not have a fixed frame rate (eg, event-based timing), then the SPTI message may be sent for each picture and the persistence flag may be set to 0. Otherwise (source picture time has a fixed frame rate), it is possible to indicate source picture time for multiple pictures in the same SPTI SEI message.

[0075] For a fixed frame rate, if the source picture interval between any pair of output pictures is known, it is possible to determine the source picture time for all pictures: this source picture interval can be used to interpolate or extrapolate the source picture intervals of all other pictures. Based on this, Table 14 shows an example of the syntax of a source picture timing SEI message according to the techniques of this article: Table 14

[0076] With respect to Table 14, in one example, the semantics may be based on the semantics provided above and the following semantics: The indicated source picture interval (in seconds) is derived as follows: SourcePictureInterval=spti_num_units_in_elemental_source_picture_interval÷spti_time_scale For each output image n (in output order), the following applies: If n is equal to 0, the value of SourcePictureTime[n] is derived as follows: – If external means are available to set the value of SourcePictureTime[n], the value of SourcePictureTime[n] is set by external means. – Otherwise, SourcePictureTime[n] is inferred to be equal to 0. Otherwise (n is greater than 0), the value of the variable SourcePictureTime[n] is derived as follows: Let m be the first picture after the access unit containing the SPTI SEI message in output order, with a TemporalId lower than or equal to the TemporalId of the SPTI SEI message: – If temporalReversalFlag is equal to 0, SourcePictureTime[n] = SourcePictureTime[0] + SourcePictureInterval*n÷m – Otherwise, temporalReversalFlag is equal to 1, SourcePictureTime[n] = SourcePictureTime[0] - SourcePictureInterval*n÷m where previousPicInOutputOrder is the last picture (if any) outputted before picture n in the output order that has a TemporalId less than or equal to i.

[0077] In this way, the video encoder 500 represents an example of a device configured to perform the following operations: signaling a source picture timing information message; signaling in the source picture timing information message a first syntax element specifying a highest temporal sub-layer, the source picture timing information message including source picture timing information about the highest temporal sub-layer, wherein a value of the first syntax element is in a range from an indicated temporal identifier to 7, inclusive; and signaling in the source picture timing information message a corresponding instance of a second syntax element for each specified highest temporal sub-layer starting at the indicated temporal identifier, wherein the second syntax element specifies a scaling factor for determining a source picture interval of corresponding consecutive pictures in output order.

[0078] Reference again Figure 1 , interface 108 may include any device configured to receive data generated by data encapsulator 107 and transmit and / or store the data to a communication medium. Interface 108 may include a network interface card such as an Ethernet card, and may include an optical transceiver, a radio frequency transceiver, or any other type of device that can send and / or receive information. In addition, interface 108 may include a computer system interface that enables files to be stored on a storage device. For example, interface 108 may include support for peripheral component interconnect (PCI) and high-speed peripheral component interconnect (PCIe) bus protocols, proprietary bus protocols, universal serial bus (USB) protocols, I2 C's chipset, or any other logical and physical structure that can be used to interconnect peer devices.

[0079] Reference again Figure 1 , target device 120 includes interface 122, data decapsulator 123, video decoder 124, and display 126. Interface 122 may include any device configured to receive data from a communication medium. Interface 122 may include a network interface card such as an Ethernet card, and may include an optical transceiver, a radio frequency transceiver, or any other type of device that can receive and / or send information. In addition, interface 122 may include a computer system interface that allows a compatible video bitstream to be retrieved from a storage device. For example, interface 122 may include a computer system that supports PCI and PCIe bus protocols, proprietary bus protocols, USB protocols, I 2 The data decapsulator 123 may be configured to receive and parse any of the example grammatical structures described herein.

[0080] Video decoder 124 may include any device configured to receive a bitstream (e.g., sub-bitstream extraction) and / or acceptable variants thereof and reproduce video data therefrom. Display 126 may include any device configured to display video data. Display 126 may include one of a variety of display devices such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display. Display 126 may include a high definition display or an ultra high definition display. It should be noted that although in Figure 1 In the example shown, the video decoder 124 is described as outputting data to a display 126, but the video decoder 124 can be configured to output video data to various types of devices and / or subcomponents thereof. For example, the video decoder 124 can be configured to output video data to any communication medium, as described herein.

[0081] Figure 6 600 is a block diagram illustrating an example of a video decoder that may be configured to decode video data in accordance with one or more techniques of this disclosure (e.g., a decoding process for reference picture list construction described above). In one example, the video decoder 600 may be configured to decode transform data and reconstruct residual data from transform coefficients based on the decoded transform data. The video decoder 600 may be configured to perform intra-frame prediction decoding and inter-frame prediction decoding, and may therefore be referred to as a hybrid decoder. The video decoder 600 may be configured to parse any combination of the syntax elements described above in Tables 1 to 14. The video decoder 600 may decode video based on or in accordance with the above process and also based on the parsed values ​​in Tables 1 to 14.

[0082] exist Figure 6 In the example shown, the video decoder 600 includes an entropy decoding unit 602, an inverse quantization unit 604, an inverse transform coefficient processing unit 606, an intra-frame prediction processing unit 608, an inter-frame prediction processing unit 610, a summer 612, a post-filter unit 614, and a reference buffer 616. The video decoder 600 can be configured to decode video data in a manner consistent with a video encoding system. It should be noted that although the exemplary video decoder 600 shown has different functional blocks, such illustrations are intended for descriptive purposes and do not limit the video decoder 600 and / or its subcomponents to a specific hardware or software architecture. The functionality of the video decoder 600 can be implemented using any combination of hardware, firmware, and / or software implementations.

[0083] like Figure 6 As shown, the entropy decoding unit 602 receives an entropy-encoded bitstream. The entropy decoding unit 602 may be configured to decode syntax elements and quantized coefficients from the bitstream according to a process that is inverse to the entropy encoding process. The entropy decoding unit 602 may be configured to perform entropy decoding according to any of the entropy encoding techniques described above. The entropy decoding unit 602 may determine the values ​​of syntax elements in the encoded bitstream in a manner consistent with the video coding standard. Figure 6 As shown, the entropy decoding unit 602 can determine the quantization parameter, quantization coefficient value, transformation data and prediction data from the bit stream. Figure 6 In the example shown, the inverse quantization unit 604 and the inverse transform coefficient processing unit 606 receive the quantized coefficient values ​​from the entropy decoding unit 602 and output reconstructed residual data.

[0084] See again Figure 6, the reconstructed residual data may be provided to the summer 612. The summer 612 may add the reconstructed residual data to the predicted video block and generate the reconstructed video data. The predicted video block may be determined according to a predictive video technique, namely intra prediction and inter prediction. The intra prediction processing unit 608 may be configured to receive an intra prediction syntax element and retrieve the predicted video block from a reference buffer 616. The reference buffer 616 may include a memory device configured to store one or more frames of video data. The intra prediction syntax element may identify an intra prediction mode, such as the intra prediction mode described above. The inter prediction processing unit 610 may receive the inter prediction syntax element and generate a motion vector to identify a predicted block in one or more reference frames stored in the reference buffer 616. The inter prediction processing unit 610 may generate a motion compensated block, possibly performing interpolation based on an interpolation filter. An identifier of an interpolation filter for motion estimation with sub-pixel precision may be included in the syntax element. The inter prediction processing unit 610 may use an interpolation filter to calculate an interpolated value for a sub-integer pixel of a reference block. The post filter unit 614 may be configured to perform filtering on the reconstructed video data. For example, the post filter unit 614 may be configured to perform deblocking and / or sample adaptive offset (SAO) filtering, for example based on parameters specified in the bitstream. Furthermore, it should be noted that in some examples, the post filter unit 614 may be configured to perform dedicated arbitrary filtering (e.g., visual enhancement, such as mosquito noise removal). Figure 6 As shown, the video decoder 600 may output a reconstructed video block. In this way, the video decoder 600 represents an example of a device configured to perform the following operations: receiving a source picture timing information message; parsing a first syntax element specifying a top temporal sub-layer from the source picture timing information message, the source picture timing information message including source picture timing information about the top temporal sub-layer, wherein a value of the first syntax element is in a range from an indicated temporal identifier to 7, including the end value; and parsing a corresponding instance of a second syntax element for each specified top temporal sub-layer starting from the indicated temporal identifier from the source picture timing information message, wherein the second syntax element specifies a scale factor for determining a source picture interval of corresponding consecutive pictures in output order.

[0085] In one or more examples, the functions may be implemented by hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on or transmitted via a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media corresponding to tangible media such as data storage media, or propagation media including, for example, any medium that facilitates the transmission of a computer program from one place to another according to a communication protocol. Thus, computer-readable media may generally correspond to: (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0086] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, optical fiber cable, twisted pair, digital subscriber line (DSL) or wireless technology such as infrared, radio and microwave are used to transmit instructions from a website, server or other remote source, then the coaxial cable, optical fiber cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals or other transient media, but are directed to non-transient tangible storage media. As used herein, disks and optical disks include compact disks (CDs), laser disks, optical optical disks, digital versatile disks (DVDs), floppy disks and Blu-ray disks, wherein disks usually copy data magnetically, while optical disks use lasers to copy data optically. The above combination should also be included in the scope of computer-readable media.

[0087] Instructions may be executed by one or more processors such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functions described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Moreover, these techniques may be implemented entirely in one or more circuits or logic elements.

[0088] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units may be combined in a codec hardware unit, or provided by a collection of interoperating hardware units including one or more processors as described above, in combination with appropriate software and / or firmware.

[0089] In addition, each functional block or various features of the base station equipment and terminal equipment used in each of the above-mentioned embodiments can be realized or executed by a circuit (usually an integrated circuit or multiple integrated circuits). The circuit designed to perform the functions described in this specification may include a general-purpose processor, a digital signal processor (DSP), a dedicated or general-purpose integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices or discrete hardware components or a combination thereof. The general-purpose processor can be a microprocessor, or alternatively, the processor can be a conventional processor, a controller, a microcontroller or a state machine. The general-purpose processor or each of the above-mentioned circuits can be configured by a digital circuit, or can be configured by an analog circuit. In addition, when the technology of making an integrated circuit that replaces the current integrated circuit appears due to the progress of semiconductor technology, the integrated circuit produced by the technology can also be used.

[0090] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for decoding video data, the method comprising: Receive source picture timing information message; parsing a first syntax element specifying a top temporal sub-layer from the source picture timing information message, the source picture timing information message comprising source picture timing information about the top temporal sub-layer, wherein a value of the first syntax element is in the range of the indicated temporal identifier to 7, inclusive; and A respective instance of a second syntax element for each specified highest temporal sub-layer starting at the indicated temporal identifier is parsed from the source picture timing information message, wherein the second syntax element specifies a scaling factor for determining a source picture interval of corresponding consecutive pictures in output order.

2. A device comprising one or more processors, wherein the one or more processors are configured to: Receive source picture timing information message; parsing a first syntax element specifying a top temporal sub-layer from the source picture timing information message, the source picture timing information message comprising source picture timing information about the top temporal sub-layer, wherein a value of the first syntax element is in the range of the indicated temporal identifier to 7, inclusive; and A respective instance of a second syntax element for each specified highest temporal sub-layer starting at the indicated temporal identifier is parsed from the source picture timing information message, wherein the second syntax element specifies a scaling factor for determining a source picture interval of corresponding consecutive pictures in output order.

3. The device of claim 2, wherein the device comprises a video decoder.

Images