Green Metadata Signaling

Abstract

A system, method, apparatus, and computer-readable medium for processing video data are disclosed. For example, an apparatus for processing video data may include at least one memory and at least one processor coupled to the at least one memory, where the at least one processor is configured to: obtain a bitstream, extract a granularity type syntax element associated with the bitstream, extract a period type syntax element associated with the bitstream, where the granularity type syntax element specifies a type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable, and decode a portion of the bitstream based on the granularity type syntax element and the period type syntax element, where the period type syntax element indicates an upcoming time period or set of pictures to which the CM is applicable.

Description

[Technical field]

[0001]

[0001] The present application generally relates to video processing. For example, aspects of the present application relate to improving video coding techniques (e.g., encoding and / or decoding video) with respect to green metadata. [Background technology]

[0002]

[0002] Digital video capabilities may be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite radio telephones, so-called "smartphones," video teleconferencing devices, video streaming devices, and the like. Such devices allow video data to be processed and output for consumption. Digital video data comprises a large amount of data to meet the demands of consumers and video providers. For example, consumers of video data want the highest quality video, with high fidelity, resolution, frame rates, and the like. As a result, the large amount of video data required to meet these demands puts a strain on communication networks and devices that process and store the video data.

[0003]

[0003] Digital video devices may implement video coding techniques for compressing video data. Video coding is performed according to one or more video coding standards or formats. For example, video coding standards or formats include, among others, generic video coding (VVC), high efficiency video coding (HEVC), advanced video coding (AVC), MPEG-2 Part 2 coding (MPEG stands for Moving Picture Experts Group), as well as proprietary video coder-decoder (codec) / formats such as AOMedia Video1 (AV1) developed by the Alliance for Open Media. Video coding generally utilizes prediction methods (e.g., inter-prediction, intra-prediction, etc.) that exploit redundancy present in a video image or sequence. The goal of video coding techniques is to compress video data into a format that uses a lower bit rate while avoiding or minimizing degradation to video quality. As ever-evolving video services become available, coding techniques with better coding efficiency are needed. Summary of the Invention

[0004]

[0004] Systems and techniques for processing video data are described herein. According to at least one example, a method of processing a video is provided that includes obtaining a bitstream, extracting a granularity type syntax element associated with the bitstream, and the granularity type syntax element specifying a type of granularity to which a complexity metric (CM) associated with the bitstream is applicable.

[0005]

[0005] A system, method, apparatus, and computer-readable medium for processing video data are disclosed. In one illustrative example, an apparatus for processing video data is provided. The apparatus includes at least one memory and at least one processor coupled to the at least one memory (e.g., implemented in a circuit), where the at least one processor is configured to: obtain a bitstream; extract a granularity type syntax element associated with the bitstream; extract a period type syntax element associated with the bitstream, where the granularity type syntax element specifies a type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable; and decode a portion of the bitstream based on the granularity type syntax element and the period type syntax element, where the period type syntax element indicates a coming time period or set of pictures to which the CM is applicable.

[0006] In another example, a method for processing video data is provided, the method including obtaining a bitstream, retrieving a granularity type syntax element associated with the bitstream, retrieving a period type syntax element associated with the bitstream, the granularity type syntax element specifying a type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable, and decoding a portion of the bitstream based on the granularity type syntax element and the period type syntax element, the period type syntax element indicating an upcoming time period or set of pictures to which the CM is applicable.

[0007]

[0007] In another example, a non-transitory computer-readable medium is provided having instructions that, when executed by one or more processors, cause the one or more processors to obtain a bitstream, extract a granularity type syntax element associated with the bitstream, extract a period type syntax element associated with the bitstream, the granularity type syntax element specifying a type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable, and decode a portion of the bitstream based on the granularity type syntax element and the period type syntax element, the period type syntax element indicating an upcoming time period or set of pictures to which the CM is applicable.

[0008]

[0008] In another example, an apparatus for processing video data is provided, the apparatus including means for obtaining a bitstream, means for retrieving a granularity type syntax element associated with the bitstream, means for retrieving a period type syntax element associated with the bitstream, the granularity type syntax element specifying a type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable, and means for decoding a portion of the bitstream based on the granularity type syntax element and the period type syntax element, the period type syntax element indicating an upcoming time period or set of pictures to which the CM is applicable.

[0009]

[0009] In another example, an apparatus for processing video data is provided. The apparatus includes at least one memory and at least one processor coupled to the at least one memory (e.g., implemented in a circuit), the at least one processor is configured to: obtain video data, generate, for a bitstream, a granularity type syntax element that specifies a type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable, generate, for the bitstream, a period type syntax element associated with the bitstream, generate a bitstream associated with the video data, the period type syntax element indicating an upcoming time period or set of pictures to which the CM is applicable, and output the generated bitstream, the bitstream including the granularity type syntax element and the period type syntax element.

[0010]

[0010] In another example, a method for processing video data is provided, the method includes obtaining video data, generating, for a bitstream, a granularity type syntax element that specifies a type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable, generating, for the bitstream, a period type syntax element associated with the bitstream, generating a bitstream associated with the video data, the period type syntax element indicating an upcoming time period or set of pictures to which the CM is applicable, and outputting the generated bitstream, the bitstream including the granularity type syntax element and the period type syntax element.

[0011]

[0011] In another example, a non-transitory computer-readable medium is provided having instructions that, when executed by one or more processors, cause the one or more processors to obtain video data; generate, for the bitstream, a granularity type syntax element that specifies a type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable; generate, for the bitstream, a period type syntax element associated with the bitstream; generate a bitstream associated with the video data, the period type syntax element indicating an upcoming time period or set of pictures to which the CM is applicable; and output the generated bitstream, the bitstream including the granularity type syntax element and the period type syntax element.

[0012]

[0012] In another example, an apparatus for processing video data is provided, the apparatus including: means for obtaining the video data; means for generating, for a bitstream, a granularity type syntax element that specifies a type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable; means for generating, for the bitstream, a period type syntax element associated with the bitstream; means for generating a bitstream associated with the video data, the period type syntax element indicating an upcoming time period or set of pictures to which the CM is applicable; and means for outputting the generated bitstream, the bitstream including the granularity type syntax element and the period type syntax element.

[0013] According to at least one other example, an apparatus is provided for processing video data, the apparatus including at least one memory and one or more processors coupled to the memory (e.g., implemented in a circuit), the one or more processors configured to obtain a bitstream and to extract a granularity type syntax element associated with the bitstream, the granularity type syntax element specifying a type of granularity to which a complexity metric (CM) associated with the bitstream is applicable.

[0014]

[0014] According to at least one other example, a non-transitory computer-readable medium is provided that includes instructions that, when executed by one or more processors, cause the one or more processors to obtain a bitstream, extract a granularity type syntax element associated with the bitstream, and the granularity type syntax element specifies a type of granularity to which a complexity metric (CM) associated with the bitstream is applicable.

[0015]

[0015] According to at least one other example, an apparatus is provided for processing video data, comprising means for obtaining a bitstream, and means for extracting a granularity type syntax element associated with the bitstream, the granularity type syntax element specifying a type of granularity to which a complexity metric (CM) associated with the bitstream is applicable.

[0016] In some aspects, a value of the granularity type syntax element specifies that the CM is applicable to a picture of the bitstream.

[0017] In some aspects, a value of the granularity type syntax element specifies that the CM is applicable to a slice of the bitstream.

[0018] In some aspects, a value of the granularity type syntax element specifies that the CM is applicable to a tile of the bitstream.

[0019] In some aspects, the value of the granularity type syntax element specifies that the CM is applicable to sub-pictures of the bitstream.

[0020] In some aspects, a value of the granularity type syntax element specifies that the CM is applicable to a scalable layer of the bitstream.

[0021] In some aspects, the value of the granularity type syntax element specifies that the CM is applicable to a coding tree unit (CTU) row of the bitstream.

[0022]

[0022] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include extracting a period type syntax element associated with the bitstream, the period type syntax element specifying a type of upcoming period to which the CM is applicable.

[0023]

[0023] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include extracting a picture-level CM syntax structure associated with a bitstream, the picture-level CM syntax structure specifying a complexity metric for one or more pictures over a period of time.

[0024] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include extracting a granularity level CM syntax structure associated with a bitstream, the granularity level CM syntax structure specifying granularity level complexity metrics for one or more entities over a period of time. In some aspects, the one or more entities include at least one of a slice, a tile, a subpicture, and a layer.

[0025]

[0025] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include extracting a subpicture syntax element associated with the bitstream, the subpicture syntax element indicating that a subpicture identifier (ID) is signaled in the CM when a period spans multiple pictures.

[0026]

[0026] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include extracting a coding tree block (CTB) number syntax element associated with the bitstream, where the CTB number syntax element indicates that when the type of granularity is equal to slice or tile and the period spans multiple pictures, a total number of coding tree luma blocks over the period may be signaled in the CM.

[0027]

[0027] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include extracting an average coding tree block (CTB) number syntax element associated with the bitstream, the average CTB number syntax element indicating an average number of CTBs or 4x4 blocks per granularity per picture.

[0028]

[0028] In some aspects, intra-coded block statistics are signaled in association with at least a portion of a bitstream when there are intra-coded blocks available in at least a portion of the bitstream.

[0029]

[0029] In some aspects, inter-coded block statistics are signaled in association with at least a portion of a bitstream when there are inter-coded blocks available in at least a portion of the bitstream.

[0030] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include retrieving one or more quality restoration metrics associated with one or more granularity segments of the bitstream. In some aspects, the one or more granularity segments of the bitstream include at least one of a slice, a tile, and a subpicture.

[0031]

[0031] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include receiving a supplemental enhancement information (SEI) message and extracting a granularity type syntax element from the SEI message.

[0032]

[0032] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include determining an operating frequency of the device based on a CM associated with the bitstream.

[0033]

[0033] In some aspects, the apparatus includes a decoder.

[0034]

[0034] According to at least one other example, a method for processing video is provided, the method including obtaining video data, generating a bitstream associated with the video data, and generating, for the bitstream, a granularity type syntax element that specifies a type of granularity to which a complexity metric (CM) associated with the bitstream is applicable.

[0035] According to at least one other example, an apparatus is provided for processing video data, the apparatus including at least one memory and one or more processors coupled to the memory (e.g., implemented in a circuit), the one or more processors configured to obtain the video data, generate a bitstream associated with the video data, and generate, for the bitstream, a granularity type syntax element that specifies a type of granularity to which a complexity metric (CM) associated with the bitstream is applicable.

[0036]

[0036] According to at least one other example, a non-transitory computer-readable medium is provided that includes instructions that, when executed by one or more processors, cause the one or more processors to obtain video data, generate a bitstream associated with the video data, and generate, for the bitstream, a granularity type syntax element that specifies a type of granularity to which a complexity metric (CM) associated with the bitstream is applicable.

[0037]

[0037] According to at least one other example, an apparatus for processing video data is provided, the apparatus including means for obtaining video data, means for generating a bitstream associated with the video data, and means for generating, for the bitstream, a granularity type syntax element that specifies a type of granularity to which a complexity metric (CM) associated with the bitstream is applicable.

[0038]

[0038] In some aspects, the value of the granularity type syntax element specifies that the CM is applicable to a picture of the bitstream.

[0039]

[0039] In some aspects, the value of the granularity type syntax element specifies that the CM is applicable to a slice of the bitstream.

[0040]

[0040] In some aspects, the value of the granularity type syntax element specifies that the CM is applicable to a tile of the bitstream.

[0041]

[0041] In some aspects, the value of the granularity type syntax element specifies that the CM is applicable to sub-pictures of the bitstream.

[0042]

[0042] In some aspects, the value of the granularity type syntax element specifies that the CM is applicable to a scalable layer of the bitstream.

[0043]

[0043] In some aspects, the value of the granularity type syntax element specifies that the CM is applicable to a coding tree unit (CTU) row of the bitstream.

[0044]

[0044] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include generating a period type syntax element for the bitstream that specifies the type of upcoming period to which the CM is applicable.

[0045]

[0045] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include generating, for a bitstream, a picture-level CM syntax structure that specifies a complexity metric for one or more pictures over a period of time.

[0046] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include generating, for a bitstream, a granularity level CM syntax structure that specifies a granularity level complexity metric for one or more entities over a period of time. In some aspects, the one or more entities include at least one of a slice, a tile, a subpicture, and a layer.

[0047]

[0047] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include generating, for a bitstream, a sub-picture syntax element that indicates that a sub-picture identifier (ID) is signaled in the CM when a period spans multiple pictures.

[0048]

[0048] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include generating a coding tree block (CTB) number syntax element that indicates, for a bitstream, when the type of granularity is equal to slice or tile and the period spans multiple pictures, that the total number of coding tree luma blocks over the period may be signaled in the CM.

[0049]

[0049] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include generating an average CTB number syntax element indicating an average number of coding tree blocks (CTBs) or 4x4 blocks per granularity per picture for a bitstream.

[0050]

[0050] In some aspects, intra-coded block statistics are signaled in association with at least a portion of a bitstream when there are intra-coded blocks available in at least the portion of the bitstream.

[0051]

[0051] In some aspects, inter-coded block statistics are signaled in association with at least a portion of a bitstream when there are inter-coded blocks available in at least the portion of the bitstream.

[0052]

[0052] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include generating, for a bitstream, one or more quality restoration metrics associated with one or more granularity segments of the bitstream.

[0053]

[0053] In some aspects, one or more granularity segments of the bitstream include at least one of a slice, a tile, and a subpicture.

[0054]

[0054] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include generating a supplemental enhancement information (SEI) message and including a granularity type syntax element in the SEI message.

[0055]

[0055] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include storing the bitstream.

[0056]

[0056] In some aspects, the methods, apparatus, and non-transitory computer-readable medium described above may include transmitting a bitstream.

[0057]

[0057] In some aspects, the apparatus includes an encoder.

[0058] In some aspects, the device is, is a part of, and / or includes a mobile device (e.g., a mobile phone, or a so-called "smartphone" or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or component of a vehicle, a robotics device or system (television), or other device. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device includes one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0059]

[0059] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter, which should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0060]

[0060] The foregoing, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0061]

[0061] Illustrative examples of the present application are described in detail below with reference to the following figures: [Brief description of the drawings]

[0062] [Figure 1]

[0062] A block diagram illustrating an example of an encoding device and a decoding device in accordance with some examples of the present disclosure. [Diagram 2]

[0063] 1 illustrates an example use of granularity level complexity metrics for video pictures. [Diagram 3]

[0064] 1 is a flow diagram illustrating a technique for decoding encoded video according to an aspect of this disclosure. [Figure 4]

[0065] 1 is a flow diagram illustrating a technique for encoding video according to an aspect of this disclosure. [Diagram 5]

[0066] 1 is a block diagram illustrating an example video decoding device, in accordance with some examples of this disclosure. [Figure 6]

[0067] 1 is a block diagram illustrating an example video encoding device, in accordance with some examples of this disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0063]

[0068] Some aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments can be applied independently, and some of them can be applied in combination. In the following description, for the purpose of explanation, specific details are described to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments can be practiced without these specific details. The figures and descriptions are not limiting.

[0064]

[0069] The following description merely provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application, as set forth in the appended claims.

[0065]

[0070] Video coding devices implement video compression techniques for efficiently encoding and decoding video data. Video compression techniques may include applying different prediction modes, including spatial prediction (e.g., intra-frame prediction or intra prediction), temporal prediction (e.g., inter-frame prediction or inter prediction), inter-layer prediction (across different layers of video data), and / or other prediction techniques, to reduce or remove redundancy inherent in video sequences. A video encoder may partition each picture of an original video sequence into rectangular regions called video blocks or coding units (described in more detail below). These video blocks may be encoded using particular prediction modes.

[0066]

[0071] A video block may be divided into one or more groups of smaller blocks in one or more ways. A block may include a coding tree block, a prediction block, a transform block, or other suitable block. References to a "block" generally may refer to such a video block (e.g., a coding tree block, a coding block, a prediction block, a transform block, or other suitable block or sub-block, as will be understood by those skilled in the art) unless otherwise specified. Furthermore, each of these blocks may also be referred to interchangeably herein as a "unit" (e.g., a coding tree unit (CTU), a coding unit, a prediction unit (PU), a transform unit (TU), etc.). In some cases, a unit may refer to a coding logical unit that is encoded in the bitstream, and a block may refer to a portion of a video frame buffer that a process is targeted to.

[0067]

[0072] For inter-prediction modes, the video encoder may search for a block similar to a block encoded in a frame (or picture) in another temporal location, called a reference frame or picture. The video encoder may limit its search to a certain spatial displacement from the block to be encoded. The best match may be identified using a two-dimensional (2D) motion vector that includes a horizontal displacement component and a vertical displacement component. For intra-prediction modes, the video encoder may use spatial prediction techniques to form a predicted block based on data from previously encoded neighboring blocks in the same picture.

[0068]

[0073] The video encoder may determine a prediction error. For example, the prediction may be determined as a difference between pixel values ​​in a block being coded and pixel values ​​in a predicted block. The prediction error may also be referred to as a residual. The video encoder may also apply a transform (e.g., a discrete cosine transform (DCT) or other suitable transform) to the prediction error to generate transform coefficients. After the transform, the video encoder may quantize the transform coefficients. The quantized transform coefficients and the motion vectors may be represented using syntax elements and, together with the control information, form a coded representation of the video sequence. In some instances, the video encoder may entropy code the syntax elements, thereby further reducing the number of bits required for their representation.

[0069]

[0074] The video decoder may use the syntax elements and control information described above to construct prediction data (e.g., a prediction block) for decoding a current frame. For example, the video decoder may add the predicted block and the compressed prediction error. The video decoder may determine the compressed prediction error by weighting the transform basis functions using the quantized coefficients. The difference between the reconstructed frame and the original frame is called the reconstruction error.

[0070]

[0075] The Energy Efficient Media Consumption (Green Metadata or Green MPEG) standard, with International Standard Number ISO / IEC 23001-11 (herein incorporated by reference in its entirety for all purposes), specifies Green Metadata to promote the reduction of energy usage during media consumption. Green Metadata for Energy Efficient Decoding specifies two sets of information: Complexity Metric (CM) metadata and Decoding Operation Reduction Request (DOR-Req) metadata. For example, a decoder can use the CM metadata to vary the operating frequency of a processor and thus reduce decoder power consumption. In one illustrative example, in a point-to-point video conferencing application, a remote encoder (which can generate an encoded bitstream) can receive the DOR-Req metadata and use the DOR-Req metadata to modify the decoding complexity of the bitstream and thus reduce local decoder power consumption. By signaling the decoding complexity of a bitstream, a local decoder may be able to estimate the amount of power needed to decode the bitstream and potentially adapt the bitstream based on the amount of battery power remaining, for example, by requesting a less complex (or more complex) bitstream. In some cases, supplemental enhancement information (SEI) messages may be used to signal green metadata in a bitstream (e.g., AVC, HEVC, VVC, AV1, or other streams).

[0071]

[0076] Green metadata was specified for AVC and HEVC in the second edition of ISO / IEC 23001-11. In the working draft of the third edition of Green MPEG (MPEG MDS20584_WG03_N00330), new Green metadata was proposed to support VVC coder-decoders (codecs), and CM was specified in various granularities. The syntax structure may be improved to support more granularity types across various duration types. Furthermore, signaling CM for slice, tile, subpicture, or layer granularity using a single type may be problematic. In some cases, such as in the case of VVC, an encoder may divide a picture (e.g., a frame) of the video being encoded into one or more parts, such as slices, tiles, subpictures, layers, etc. For example, a picture may be divided into one or more tiles, and each tile may be divided into one or more blocks. A slice may include multiple tiles or multiple blocks within a tile. A subpicture may be one or more complete rectangular slices, with each rectangular slice covering a rectangular area of ​​the picture. A sub-picture may or may not be coded independently of other sub-pictures of the same picture.

[0072]

[0077] Currently, a decoder, such as in the case of AVC / HEVC, can use the number of slices and the number of tiles to identify whether the CM is calculated for slices or tiles. For example, when the number of slices is equal to the number of tiles, it is complicated to identify the CM granularity. Furthermore, AVC and HEVC do not support sub-picture granularity. It would be beneficial to define slice granularity and tile granularity in different types, along with defining sub-picture granularity and layer granularity.

[0073]

[0078] VVC allows a subpicture to be replaced with a different subpicture in a coded layer video sequence (CLVS). A coded video sequence (CVS) can be a set of layer units of a CLVS. In some cases, signaling is required to map a CM to a specific subpicture using a subpicture identifier (ID).

[0074]

[0079] VVC also allows for resolution changes in CLVS. In some cases, parsing each slice header to derive the number of total coding blocks over a period to interpret normalized coded statistics per slice or tile may be complicated. Slice headers may be included with slices, and slice headers may convey information about the associated slice. Information that applies to all slices of a picture may be conveyed in the picture header. A syntax element to indicate the total number of CTBs would be beneficial to simplify the derivation.

[0075]

[0080] Currently, CM provides intra-coded block statistics when all blocks are intra-coded. It is possible that P and B slices may have more intra-coded blocks than inter-coded blocks, or that P or B pictures may have more intra-coded blocks than inter-coded blocks. As a result, CM may not accurately represent the complexity. An intra-coded block refers to a block that is predicted based on another block in the same picture, and an inter-coded block refers to a block that is predicted based on another block from a different picture. An I slice refers to a slice that contains intra-coded blocks and no inter-coded blocks. P and B slices may contain both intra-coded and inter-coded blocks.

[0076]

[0081] Furthermore, rather than applying a quality metric to an entire picture, the quality metric may be applied to each individual sub-picture in the VVC.

[0077]

[0082] This disclosure describes systems, apparatus, methods, and computer-readable media (collectively "systems and techniques") for providing enhanced green metadata signaling, such as for improving complexity metric (CM) signaling. For example, in some cases, a granularity type indicator (e.g., a granularity type syntax element such as granularity_type) is provided to support various granularities, such as slice, tile, subpicture, scalable layer, and / or other granularities. In some examples, the semantics of a period type syntax element (e.g., period_type) is modified.

[0078]

[0083] In some cases, the systems and techniques provide improved complexity metric (CM) signaling. For example, the systems and techniques described herein provide the ability for a video codec (e.g., a video encoder, a video encoder, or a combined video encoder-decoder) to specify, for multiple pictures of a video, CM values ​​that are applicable to portions of a picture, such as slices, tiles, sub-pictures, and / or layers. For example, as previously described, a sub-picture may be defined for an encoded video. A sub-picture includes a portion of a picture, such as the upper right corner of the picture. A CM may be specified for the sub-picture, where the CM is different from at least one other CM specified for a slice (or other portion) of the picture. A CM value associated with a sub-picture may be defined once for multiple pictures, such as for 30 pictures from the first picture. The CM may be provided as part of metadata included with the encoded video. Allowing a single CM value to be specified for a sub-picture that spans multiple frames helps to reduce the size of metadata for coded video while allowing increased flexibility and granularity for defining CM for portions of a picture.

[0079]

[0084] In some aspects, CM signaling changes related to resolution changes are provided. In some cases, CM signaling changes related to intra-coded block statistics are provided. In some aspects, sub-picture quality metrics are provided.

[0080]

[0085] The systems and techniques described herein may be applied to any of the existing video codecs, such as Generic Video Coding (VVC), High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), VP9, ​​AV1 formats / codecs, and / or other video coding standards, codecs, formats, etc., that are or will be developed.

[0081]

[0086] FIG. 1 is a block diagram illustrating an example of a system 100 including an encoding device 104 and a decoding device 112. The encoding device 104 may be part of a source device, and the decoding device 112 may be part of a receiving device. The source device and / or the receiving device may include electronic devices such as a mobile or landline telephone handset (e.g., a smartphone, a cellular telephone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video gaming console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the source device and the receiving device may include one or more wireless transceivers for wireless communication. The coding techniques described herein are applicable to video coding in various multimedia applications, including streaming video transmission (e.g., over the Internet), television broadcast or transmission, encoding digital video for storage on a data storage medium, decoding digital video stored on a data storage medium, or other applications. The term coding as used herein may refer to encoding and / or decoding. In some examples, the system 100 may support one-way or two-way video transmission to support applications such as video conferencing, video streaming, video playback, video broadcasting, gaming, and / or video telephony.

[0082]

[0087] The encoding device 104 (or encoder) may be used to encode the video data using a video coding standard, format, codec, or protocol to generate an encoded video bitstream. Examples of video coding standards and formats / codecs include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual, ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC) including its Scalable Video Coding (SVC) extension and Multiview Video Coding (MVC) extension, High Efficiency Video Coding (HEVC) or ITU-T H.265, and Generic Video Coding (VVC) or ITU-T H.266. There are various extensions to HEVC that address multi-layer video coding, including range and screen content coding extensions, 3D video coding (3D-HEVC) and multiview extensions (MV-HEVC) and scalable extensions (SHVC). HEVC and its extensions were developed by the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG) Joint Collaboration Team on Video Coding (JCT-VC) and the Joint Collaboration Team on 3D Video Coding Extensions Development (JCT-3V). VP9, ​​AOMedia Video1 (AV1) developed by the Alliance for Open Media Alliance of Open Media (AOMedia), and Essential Video Coding (EVC) are other video coding standards to which the techniques described herein may be applied.

[0083]

[0088] The techniques described herein may be applied to any existing video codec (e.g., High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), or other suitable existing video codec) and / or may be efficient coding tools for any developing and / or future video coding standards, such as, for example, VVC, and / or other video coding standards under development or to be developed. For example, examples described herein may be implemented using video codecs such as VVC, HEVC, AVC, and / or extensions thereof. However, the techniques and systems described herein may also be applicable to other coding standards, codecs, or formats, such as MPEG, JPEG (or other coding standards for still images), VP9, ​​AV1, extensions thereof, or other suitable coding standards that are already available or not yet available or developed. For example, in some examples, the encoding device 104 and / or the decoding device 112 may operate according to a proprietary video codec / format, such as AV1, an extension of AVI, and / or a successor version of AV1 (e.g., AV2), or other proprietary formats or industry standards. Thus, although the techniques and systems described herein may be described with reference to a particular video coding standard, those skilled in the art will appreciate that the description should not be construed as applying only to that particular standard.

[0084]

[0089] 1, a video source 102 may provide video data to an encoding device 104. The video source 102 may be part of a source device or part of a device other than the source device. The video source 102 may include a video capture device (e.g., a video camera, a camera phone, a video phone, etc.), a video archive containing stored video, a video server or content provider providing video data, a video feed interface receiving video from a video server or content provider, a computer graphics system for generating computer graphics video data, a combination of such sources, or any other suitable video source.

[0085]

[0090] The video data from the video source 102 may include one or more input pictures or frames. A picture or frame is, in some cases, a still image that is part of a video. In some examples, the data from the video source 102 may be a still image that is not part of a video. In HEVC, VVC, and other video coding specifications, a video sequence may include a series of pictures. A picture may include three sample arrays, denoted SL, SCb, and SCr. SL is a two-dimensional array of luma samples, SCb is a two-dimensional array of Cb chrominance samples, and SCr is a two-dimensional array of Cr chrominance samples. The chrominance samples are sometimes referred to herein as "chroma" samples. A pixel may refer to all three components (luma and chroma samples) for a given location in the array of a picture. In other cases, a picture may be monochrome and include only an array of luma samples, in which case the terms pixel and sample may be used interchangeably. For example techniques described herein that refer to individual samples for purposes of explanation, the same techniques may be applied to pixels (e.g., all three sample components for a given location in an array of pictures). For example techniques described herein that refer to pixels (e.g., all three sample components for a given location in an array of pictures) for purposes of explanation, the same techniques may be applied to individual samples.

[0086]

[0091] The encoder engine 106 (or encoder) of the encoding device 104 encodes the video data to generate an encoded video bitstream. In some examples, the encoded video bitstream (or "video bitstream" or "bitstream") is a series of one or more coded video sequences. A coded video sequence (CVS) includes a series of AUs starting with an access unit (AU) having a random access point picture with some properties in a base layer up to and including the next AU having a random access point picture with some properties in the base layer. For example, some properties of the random access point picture starting the CVS may include a RASL flag (e.g., NoRaslOutputFlag) equal to 1. In other cases, the random access point picture (with a RASL flag equal to 0) does not start a CVS. An access unit (AU) includes one or more coded pictures and control information corresponding to the coded pictures that share the same output time. At the bitstream level, coded slices of a picture are encapsulated in data units called network abstraction layer (NAL) units. For example, an HEVC video bitstream may contain one or more CVSs that contain NAL units. Each of the NAL units has a NAL unit header. In one example, the header is one byte in H.264 / AVC (except for multi-layer extensions) and two bytes in HEVC. Syntax elements in the NAL unit header take designated bits and are therefore visible to all kinds of systems and transport layers, such as transport streams, real-time transport (RTP) protocols, file formats, among others.

[0087]

[0092] Two classes of NAL units exist in the HEVC standard, including video coding layer (VCL) NAL units and non-VCL NAL units. VCL NAL units contain coded picture data that form a coded video bitstream. For example, a sequence of bits that form a coded video bitstream resides in a VCL NAL unit. A VCL NAL unit may contain one slice or slice segment (described below) of coded picture data, and a non-VCL NAL unit contains control information related to one or more coded pictures. In some cases, a NAL unit may be referred to as a packet. A HEVC AU includes VCL NAL units that contain coded picture data and non-VCL NAL units that correspond to the coded picture data (if any). A non-VCL NAL unit may contain, in addition to other information, a parameter set with high-level information related to the coded video bitstream. For example, the parameter sets may include a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS). In some cases, each slice or other portion of the bitstream may reference a single active PPS, SPS, and / or VPS to enable the decoding device 112 to access information that can be used to decode the slice or other portion of the bitstream.

[0088]

[0093] A NAL unit may contain a sequence of bits (e.g., a coded video bitstream, CVS of a bitstream, etc.) that form a coded representation of video data, such as a coded representation of a picture in a video. The encoder engine 106 generates the coded representation of the picture by partitioning each picture into a number of slices. The slices are independent of other slices such that information in a slice is coded without dependency on data from other slices in the same picture. A slice includes one or more slice segments, including an independent slice segment, and one or more dependent slice segments, if present, that depend on a previous slice segment.

[0089]

[0094] In HEVC, slices are then partitioned into coding tree blocks (CTBs) of luma and chroma samples. A CTB of luma samples and one or more CTBs of chroma samples, together with syntax for the samples, are called a coding tree unit (CTU). A CTU is sometimes called a "tree block" or a "largest coding unit" (LCU). A CTU is the basic processing unit for HEVC encoding. A CTU may be split into multiple coding units (CUs) of various sizes. A CU contains luma and chroma sample arrays called coding blocks (CBs).

[0090]

[0095] The luma and chroma CBs may be further split into prediction blocks (PBs). A PB is a block of luma or chroma component samples that uses the same motion parameters for inter prediction or intra block copy (IBC) prediction (when available or enabled for use). The luma PB and one or more chroma PBs, together with associated syntax, form a prediction unit (PU). For inter prediction, a set of motion parameters (e.g., one or more motion vectors, reference indexes, etc.) is signaled in the bitstream for each PU and is used for inter prediction of the luma PB and one or more chroma PBs. The motion parameters are sometimes referred to as motion information. The CBs may also be partitioned into one or more transform blocks (TBs). A TB represents a square block of samples of a color component to which a residual transform (e.g., in some cases, the same two-dimensional transform) is applied to code the prediction residual signal. A transform unit (TU) represents a TB of luma and chroma samples and corresponding syntax elements. Transform coding is described in more detail below.

[0091]

[0096] The size of a CU corresponds to the size of a coding mode and may be square in shape. For example, the size of a CU may be 8×8 samples, 16×16 samples, 32×32 samples, 64×64 samples, or any other suitable size up to the size of a corresponding CTU. The phrase “N×N” is used herein to refer to pixel dimensions of a video block for vertical and horizontal dimensions (e.g., 8 pixels×8 pixels). The pixels in a block may be arranged in rows and columns. In some implementations, a block may not have the same number of pixels in the horizontal direction as in the vertical direction. Syntax data related to a CU may, for example, represent a partition of the CU into one or more PUs. The partition mode may differ between whether the CU is intra-prediction mode coded or inter-prediction mode coded. The PUs may be partitioned to be non-square in shape. Syntax data related to a CU may also, for example, represent a partition of a CU into one or more TUs according to a CTU. The TUs may be square or non-square in shape.

[0092]

[0097] According to the HEVC standard, the transform may be implemented using transform units (TUs). The TUs may be different for different CUs. The TUs may be sized based on the size of the PUs in a given CU. The TUs may be the same size as the PUs or smaller than the PUs. In some examples, the residual samples corresponding to a CU may be subdivided into smaller units using a quad tree structure known as a residual quad tree (RQT). The leaf nodes of the RQT may correspond to the TUs. The pixel difference values ​​associated with the TUs may be transformed to produce transform coefficients. The transform coefficients may then be quantized by the encoder engine 106.

[0093]

[0098] Once a picture of video data is partitioned into CUs, the encoder engine 106 predicts each PU using a prediction mode. The prediction unit or prediction block is then subtracted from the original video data to obtain a residual (described below). For each CU, a prediction mode may be signaled in the bitstream using syntax data. The prediction mode may include intra prediction (or intra-picture prediction) or inter prediction (or inter-picture prediction). Intra prediction exploits the correlation between spatially adjacent samples in a picture. For example, using intra prediction, each PU is predicted from neighboring image data in the same picture, for example, using DC prediction to find an average value for the PU, planar prediction to fit a flat surface to the PU, directional prediction to extrapolate from neighboring data, or any other suitable type of prediction. Inter prediction uses temporal correlation between pictures to derive a motion compensated prediction for a block of image samples. For example, using inter prediction, each PU is predicted using motion compensated prediction from image data in one or more reference pictures (before or after the current picture in output order). The decision of whether to code a picture area using inter-picture prediction or intra-picture prediction may be made, for example, at the CU level.

[0094]

[0099] The encoder engine 106 and the decoder engine 116 (described in more detail below) may be configured to operate according to VVC. According to VVC, a video coder (such as the encoder engine 106 and / or the decoder engine 116) partitions a picture into multiple coding tree units (CTUs) (where a CTB for luma samples and one or more CTBs for chroma samples, together with syntax for the samples, are referred to as a CTU). The video coder may partition the CTUs according to a tree structure, such as a quad-tree binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure eliminates the concept of multiple partition types, such as the separation between CUs, PUs, and TUs in HEVC. The QTBT structure includes two levels, including a first level partitioned according to a quad-tree partition and a second level partitioned according to a binary tree partition. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to coding units (CUs).

[0095]

[0100] In the MTT partitioning structure, blocks may be partitioned using quadtree partitioning, binary tree partitioning, and one or more types of tripletree partitioning. Tripletree partitioning is a partition in which a block is split into three sub-blocks. In some examples, tripletree partitioning splits a block into three sub-blocks without splitting the original block through the center. Partition types in MTT (e.g., quadtree, binary tree, and tripletree) can be symmetric or asymmetric.

[0096]

[0101] When operating according to the AV1 codec, the video encoder 200 and the video decoder 300 may be configured to code the video data in blocks. In AV1, the largest coding block that may be processed is called a superblock. In AV1, a superblock may be either 128×128 luma samples or 64×64 luma samples. However, in successor video coding formats (e.g., AV2), a superblock may be defined by a different (e.g., larger) luma sample size. In some examples, a superblock is the top level of a block quad tree. The video encoder 200 may further partition the superblock into smaller coding blocks. The video encoder 200 may partition the superblock and other coding blocks into smaller blocks using square or non-square partitions. The non-square blocks may include N / 2×N, N×N / 2, N / 4×N, and N×N / 4 blocks. The video encoder 200 and the video decoder 300 may perform a separate prediction and transform process for each of the coding blocks.

[0097]

[0102] AV1 also defines tiles of video data. A tile is a rectangular array of superblocks that may be coded independently of other tiles. That is, video encoder 200 and video decoder 300 may encode and decode coding blocks within a tile, respectively, without using video data from other tiles. However, video encoder 200 and video decoder 300 may perform filtering across tile boundaries. Tiles may be uniform or non-uniform in size. Tile-based coding may enable parallel processing and / or multithreading for encoder and decoder implementations.

[0098]

[0103] In some examples, the video coder may use a single QTBT or MTT structure to represent each of the luminance and chrominance components, and in other examples, the video coder may use two or more QTBT or MTT structures, such as one QTBT or MTT structure for the luminance component and another QTBT or MTT structure for both chrominance components (or two QTBT and / or MTT structures for each chrominance component).

[0099]

[0104] The video coder may be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, superblock partitioning, or other partitioning structures.

[0100]

[0105] In some examples, one or more slices of a picture are assigned a slice type. The slice types include an intra-coded slice (I slice), an inter-coded P slice, and an inter-coded B slice. An I slice (intra-coded frame, independently decodable) is a slice of a picture that is coded only by intra prediction, and therefore is independently decodable because an I slice only requires data in a frame to predict any prediction unit or prediction block of the slice. A P slice (unidirectionally predicted frame) is a slice of a picture that can be coded using intra prediction and using unidirectional inter prediction. Each prediction unit or prediction block in a P slice is coded using either intra prediction or inter prediction. When inter prediction is applied, a prediction unit or prediction block is predicted only by one reference picture, and therefore the reference samples are only from one reference region of one frame. A B slice (bidirectionally predicted frame) is a slice of a picture that can be coded using intra prediction and using inter prediction (e.g., either bi-predictive or uni-predictive). A prediction unit or prediction block of a B slice may be bidirectionally predicted from two reference pictures, where each picture contributes one reference region, and the sample sets of the two reference regions are weighted (e.g., with equal weights or with different weights) to produce a prediction signal of the bidirectionally predicted block. As described above, the slices of a picture are coded independently. In some cases, a picture may be coded as just one slice.

[0101]

[0106] As mentioned above, intra-picture prediction of a picture exploits correlation between spatially adjacent samples in a picture. There are multiple intra-prediction modes (also called "intra modes"). In some examples, intra-prediction of a luma block includes 35 modes, including a planar mode, a DC mode, and 33 angular modes (e.g., a diagonal intra-prediction mode and an angular mode adjacent to the diagonal intra-prediction mode). The 35 modes of intra-prediction are indexed as shown in Table 1 below. In other examples, more intra-modes may be defined that include prediction angles that may not yet be represented by the 33 angular modes. In other examples, the prediction angles associated with the angular modes may differ from those used in HEVC.

[0102] [Table 1]

[0103]

[0107] Inter-picture prediction uses temporal correlation between pictures to derive motion-compensated predictions for blocks of image samples. Using a translational motion model, the position of a block in a previously decoded picture (reference picture) is indicated by a motion vector (Δx, Δy), where Δx specifies the horizontal displacement of the reference block relative to the position of the current block, and Δy specifies its vertical displacement. In some cases, the motion vector (Δx, Δy) may be integer sample accurate (also called integer accuracy), in which case the motion vector points to an integer pel grid (or integer pixel sampling grid) of a reference frame. In some cases, the motion vector (Δx, Δy) may be of fractional sample accuracy (also called fractional pel accuracy or non-integer accuracy) to more accurately capture the movement of the underlying object without being restricted to the integer pel grid of the reference frame. The accuracy of the motion vector may be represented by the quantization level of the motion vector. For example, the quantization level may be integer accuracy (e.g., 1 pixel) or fractional pel accuracy (e.g., ¼ pixel, ½ pixel, or other sub-pixel value). When the corresponding motion vector has fractional sample accuracy, interpolation is applied to the reference picture to derive the prediction signal. For example, samples available at integer positions may be filtered (e.g., using one or more interpolation filters) to estimate values ​​at fractional positions. A previously decoded reference picture is indicated by a reference index (refIdx) into a reference picture list. The motion vector and the reference index may be referred to as motion parameters. Two types of inter-picture prediction may be implemented, including uni-prediction and bi-prediction.

[0104]

[0108] In the case of inter prediction using bi-prediction (also called bidirectional inter prediction), two sets of motion parameters (Δx0, y0, refIdx0 and Δx1, y1, refIdx1) are used to generate two motion compensated predictions (from the same reference picture or possibly from different reference pictures). For example, in the case of bi-prediction, each prediction block uses two motion compensated prediction signals to generate a B prediction unit. The two motion compensated predictions are then combined to obtain a final motion compensated prediction. For example, the two motion compensated predictions may be combined by averaging. In another example, weighted prediction may be used, in which case different weights may be applied to each motion compensated prediction. Reference pictures that may be used in bi-prediction are stored in two separate lists, denoted as list 0 and list 1. The motion parameters may be derived in the encoder using a motion estimation process.

[0105]

[0109] In the case of inter prediction using uni-prediction (also called unidirectional inter prediction), one set of motion parameters (Δx0, y0, refIdx0) is used to generate a motion compensated prediction from a reference picture. For example, in the case of uni-prediction, each prediction block uses at most one motion compensated prediction signal to generate a P prediction unit.

[0106]

[0110] A PU may include data related to a prediction process (e.g., motion parameters or other suitable data). For example, when a PU is encoded using intra prediction, the PU may include data representing an intra prediction mode for the PU. As another example, when a PU is encoded using inter prediction, the PU may include data defining a motion vector for the PU. The data defining a motion vector for a PU may represent, for example, a horizontal component (Δx) of the motion vector, a vertical component (Δy) of the motion vector, a resolution of the motion vector (e.g., integer precision, 1 / 4 pixel precision, or 1 / 8 pixel precision), a reference picture to which the motion vector points, a reference index, a reference picture list for the motion vector (e.g., list 0, list 1, or list C), or any combination thereof.

[0107]

[0111] AV1 includes two general techniques for encoding and decoding coding blocks of video data. The two general techniques are intra prediction (e.g., intra-frame prediction or spatial prediction) and inter prediction (e.g., inter-frame prediction or temporal prediction). In the context of AV1, when predicting a block of a current frame of video data using an intra-prediction mode, the video encoder 200 and the video decoder 300 do not use video data from other frames of the video data. In most intra-prediction modes, the video encoding device 104 encodes the block of the current frame based on the difference between a sample value in the current block and a predicted value generated from a reference sample in the same frame. The video encoding device 104 determines the predicted value generated from the reference sample based on the intra-prediction mode.

[0108]

[0112] After performing prediction using intra prediction and / or inter prediction, the encoding device 104 may perform transformation and quantization. For example, after prediction, the encoder engine 106 may calculate a residual value corresponding to the PU. The residual value may comprise pixel difference values ​​between a current block of pixels being coded (PU) and a predictive block (e.g., a predicted version of the current block) used to predict the current block. For example, after generating a predictive block (e.g., issuing an inter prediction or an intra prediction), the encoder engine 106 may generate a residual block by subtracting the predictive block produced by the prediction unit from the current block. The residual block includes a set of pixel difference values ​​that quantify differences between pixel values ​​of the current block and pixel values ​​of the predictive block. In some examples, the residual block may be represented in a two-dimensional block format (e.g., a two-dimensional matrix or array of pixel values). In such examples, the residual block is a two-dimensional representation of pixel values.

[0109]

[0113] Any residual data that may remain after prediction is performed is transformed using a block transform, which may be based on a discrete cosine transform, a discrete sine transform, an integer transform, a wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., size 32×32, 16×16, 8×8, 4×4, or other suitable size) may be applied to the residual data in each CU. In some embodiments, TUs may be used for the transform and quantization process implemented by the encoder engine 106. A given CU having one or more PUs may also include one or more TUs. As described in more detail below, the residual values ​​may be transformed into transform coefficients using a block transform, and then quantized and scanned using the TUs to produce serialized transform coefficients for entropy coding.

[0110]

[0114] In some embodiments, after intra-predictive coding or inter-predictive coding using the PU of the CU, the encoder engine 106 may calculate residual data for the TU of the CU. The PU may comprise pixel data in the spatial domain (or pixel domain). The TU may comprise coefficients in the transform domain after application of a block transform. As mentioned above, the residual data may correspond to pixel difference values ​​between pixels of the uncoded picture and predicted values ​​corresponding to the PU. The encoder engine 106 may form a TU including the residual data for the CU and then transform the TU to produce transform coefficients for the CU.

[0111]

[0115] The encoder engine 106 may perform quantization of the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient with an n-bit value may be truncated to an m-bit value during quantization, where n is greater than m.

[0112]

[0116] Once quantization is performed, the coded video bitstream includes the quantized transform coefficients, prediction information (e.g., prediction modes, motion vectors, block vectors, etc.), partition information, and any other suitable data, such as other syntax data. The different elements of the coded video bitstream may then be entropy coded by the encoder engine 106. In some examples, the encoder engine 106 may utilize a predefined scan order to scan the quantized transform coefficients to produce a serialized vector that may be entropy coded. In some examples, the encoder engine 106 may perform an adaptive scan. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), the encoder engine 106 may entropy code the vector. For example, the encoder engine 106 may use context-adaptive variable length coding, context-adaptive binary arithmetic coding, syntax-based context-adaptive binary arithmetic coding, probability interval partition entropy coding, or another suitable entropy coding technique.

[0113]

[0117] An output 110 of the encoding device 104 may send the NAL units constituting the encoded video bitstream data to a decoding device 112 of a receiving device via a communication link 120. An input 114 of the decoding device 112 may receive the NAL units. The communication link 120 may include channels provided by a wireless network, a wired network, or a combination of wired and wireless networks. The wireless network may include any wireless interface or combination of wireless interfaces, and may include any suitable wireless network (e.g., Internet or other wide area network, packet-based network, WiFi, radio frequency (RF), ultra-wideband (UWB), WiFi-Direct, cellular, 5G new wireless (NR), long-term evolution (LTE), WiMax, etc.). The wired network may include any wired interface (e.g., fiber, Ethernet, powerline Ethernet, Ethernet over coaxial cable, digital signal line (DSL), etc.). Wired and / or wireless networks may be implemented using a variety of equipment, such as base stations, routers, access points, bridges, gateways, switches, etc. The encoded video bitstream data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to a receiving device.

[0114]

[0118] In some examples, encoding device 104 may store the encoded video bitstream data in storage 108. Output unit 110 may retrieve the encoded video bitstream data from encoder engine 106 or from storage 108. Storage 108 may include any of a variety of distributed or locally accessed data storage media. For example, storage 108 may include a hard drive, a storage disk, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. Storage 108 may also include a decoded picture buffer (DPB) for storing reference pictures for use in inter prediction. In further examples, storage 108 may correspond to a file server or another intermediate storage device that may store the encoded video generated by the source device. In such cases, a receiving device, including decoding device 112, may access the stored video data from the storage device via streaming or download. The file server may be any type of server capable of storing the encoded video data and transmitting the encoded video data to a receiving device. Exemplary file servers include a web server (e.g., for a website), an FTP server, a network attached storage (NAS) device, or a local disk drive. The receiving device may access the encoded video data through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both, that is suitable for accessing the encoded video data stored in the file server. The transmission of the encoded video data from storage 108 may be a streaming transmission, a download transmission, or a combination thereof.

[0115]

[0119] An input 114 of the decoding device 112 may receive the encoded video bitstream data and provide the video bitstream data to the decoder engine 116 or to the storage 118 for later use by the decoder engine 116. For example, the storage 118 may include a DPB for storing reference pictures for use in inter prediction. A receiving device including the decoding device 112 may receive the encoded video data to be decoded via the storage 108. The encoded video data may be modulated and transmitted to the receiving device according to a communication standard, such as a wireless communication protocol. A communication medium for transmitting the encoded video data may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful for enabling communication from a source device to a receiving device.

[0116]

[0120] The decoder engine 116 may decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) to extract elements of one or more coded video sequences that make up the encoded video data. The decoder engine 116 may then rescale the encoded video bitstream data and perform an inverse transform on the encoded video bitstream data. The residual data is then passed to a prediction stage of the decoder engine 116. The decoder engine 116 then predicts blocks of pixels (e.g., PUs). In some examples, the prediction is added to the output of the inverse transform (the residual data).

[0117]

[0121] Video decoding device 112 may output the decoded video to video destination device 122, which may include a display or other output device for displaying the decoded video data to a content consumer. In some aspects, video destination device 122 may be part of a receiving device that includes decoding device 112. In some aspects, video destination device 122 may be part of a separate device other than the receiving device.

[0118]

[0122] In some embodiments, the video encoding device 104 and / or the video decoding device 112 may be integrated with an audio encoding device and an audio decoding device, respectively. The video encoding device 104 and / or the video decoding device 112 may also include other hardware or software necessary to implement the coding techniques described above, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. The video encoding device 104 and the video decoding device 112 may be integrated as part of a combined encoder / decoder (codec) in the respective devices.

[0119]

[0123] The exemplary system shown in FIG. 1 is one illustrative example that may be used herein. The techniques for processing video data using the techniques described herein may be implemented by any digital video encoding and / or decoding device. Generally, the techniques of this disclosure are implemented by a video encoding device or a video decoding device, but the techniques may also be implemented by a composite video encoder / decoder, commonly referred to as a "codec." Moreover, the techniques of this disclosure may also be implemented by a video preprocessor. The source device and the receiving device are merely examples of coding devices, such that the source device generates coded video data for transmission to the receiving device. In some examples, the source device and the receiving device may operate substantially symmetrically, such that each of the devices includes a video encoding component and a video decoding component. Thus, the exemplary system may support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0120]

[0124] Extensions to the HEVC standard include a multiview video coding extension called MV-HEVC and a scalable video coding extension called SHVC. MV-HEVC and SHVC extensions share the concept of layered coding, where different layers are included in the coded video bitstream. Each layer in a coded video sequence is addressed by a unique layer identifier (ID). A layer ID may be present in the header of a NAL unit to identify the layer to which the NAL unit is associated. In MV-HEVC, different layers usually represent different views of the same scene in the video bitstream. In SHVC, different scalable layers are provided that represent the video bitstream at different spatial resolutions (or picture resolutions) or with different reconstruction fidelity. The scalable layers may include a base layer (with layer ID=0) and one or more enhancement layers (with layer ID=1, 2, ... n). The base layer may conform to the profile of the first version of HEVC and represent the lowest available layer in the bitstream. Enhancement layers have increased spatial resolution, temporal resolution or frame rate, and / or reconstruction fidelity (or quality) compared to the base layer. Enhancement layers are hierarchically organized and may or may not depend on lower layers. In some examples, different layers may be coded using a single-standard codec (e.g., all layers are encoded using HEVC, SHVC, or other coding standard). In some examples, different layers may be coded using a multi-standard codec. For example, the base layer may be coded using AVC, while one or more enhancement layers may be coded using SHVC and / or MV-HEVC extensions to the HEVC standard.

[0121]

[0125] Generally, a layer includes a set of VCL NAL units and a corresponding set of non-VCL NAL units. The NAL units are assigned a specific layer ID value. The layers may be hierarchical in the sense that a layer may depend on a lower layer. A layer set refers to a set of layers represented in a bitstream that is self-contained, meaning that a layer in a layer set may depend on other layers in the layer set in the decoding process, but does not depend on any other layers for decoding. Thus, the layers in a layer set may form an independent bitstream that can represent video content. A set of layers in a layer set may be obtained from another bitstream by the operation of a sub-bitstream extraction process. A layer set may correspond to a set of layers that should be decoded when a decoder wishes to operate according to some parameters.

[0122]

[0126] As previously described, an HEVC bitstream includes a group of NAL units, including VCL NAL units and non-VCL NAL units. The VCL NAL units include coded picture data that form a coded video bitstream. For example, a sequence of bits that form a coded video bitstream resides in a VCL NAL unit. The non-VCL NAL units may include, among other information, parameter sets with high-level information related to the coded video bitstream. For example, the parameter sets may include a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS). Examples of parameter set goals include bitrate efficiency, error resiliency, and providing a system layer interface. Each slice references a single active PPS, SPS, and VPS to access information that the decoding device 112 may use to decode the slice. An ID, including a VPS identifier (ID), an SPS ID, and a PPS ID, may be coded for each parameter set. The SPS includes an SPS ID and a VPS ID. The PPS includes a PPS ID and an SPS ID. Each slice header includes a PPS ID. Using the IDs, the active parameter set can be identified for a given slice.

[0123]

[0127] A PPS contains information that applies to all slices in a given picture. Thus, all slices in a picture point to the same PPS. Slices in different pictures may also point to the same PPS. An SPS contains information that applies to all pictures in the same coded video sequence (CVS) or bitstream. As previously described, a coded video sequence is a series of AUs starting with a random access point picture (e.g., an instantaneous decode reference (IDR) picture or a broken link access (BLA) picture, or other suitable random access point picture) in a base layer (described above) with some property, up to and including the next access unit (AU) in the base layer that has a random access point picture with some property (or the end of the bitstream). The information in an SPS may not change from picture to picture within a coded video sequence. Pictures in a coded video sequence may use the same SPS. A VPS contains information that applies to all layers in a coded video sequence or bitstream. The VPS includes a syntax structure with syntax elements that apply to the entire coded video sequence. In some embodiments, the VPS, SPS, or PPS may be transmitted in-band with the encoded bitstream. In some embodiments, the VPS, SPS, or PPS may be transmitted out-of-band in a transmission separate from the NAL units containing the coded video data.

[0124]

[0128] This disclosure may generally refer to "signaling" certain information, such as syntax elements. The term "signaling" may generally refer to communication of values ​​for syntax elements and / or other data used to decode encoded video data. For example, video encoding device 104 may signal values ​​for syntax elements in a bitstream. Generally, signaling refers to generating values ​​in the bitstream. As mentioned above, video source 102 may transport the bitstream to video destination device 122 in substantially real-time or may transport the bitstream to video destination device 122 in non-real-time, such as may be done when storing syntax elements in storage 108 for later retrieval by video destination device 122.

[0125]

[0129] A video bitstream may also include supplemental enhancement information (SEI) messages. For example, an SEI NAL unit may be part of the video bitstream. In some cases, the SEI message may contain information that is not required by the decoding process. For example, the information in the SEI message may not be essential for a decoder to decode a video picture of the bitstream, but the decoder may use the information to improve the display or processing of the picture (e.g., the decoded output). The information in the SEI message may be embedded metadata. In one illustrative example, the information in the SEI message may be used by a decoder-side entity to improve the viewability of the content. In some cases, some application standards may specify the presence of such SEI messages in the bitstream (e.g., carrying frame packing SEI messages for frame compatible planar stereoscopic 3DTV video formats where a frame packing SEI message is carried for every frame of the video, handling of recovery point SEI messages, use of pan-scan rectangular SEI messages in DVB, among many other examples) so that quality improvements may be provided to all devices that comply with the application standard.

[0126]

[0130] As mentioned above, the Energy Efficient Media Consumption standard (ISO / IEC 23001-11) specifies green metadata to facilitate reduced energy usage during media consumption. Green metadata includes Complexity Metric (CM) metadata and Decoding Operation Reduction Request (DOR-Req) metadata. A decoder may use the CM metadata to help adjust the operating frequency of a processor performing the decoding to help reduce power consumption. As previously described, systems and techniques for improving green metadata, such as CM signaling, are described herein. For example, signaling CM for slice, tile, and / or sub-frame granularity using a single type across multiple pictures can be problematic. In some aspects, the systems and techniques described herein improve syntax structures for green metadata to support more granularity types (e.g., slice granularity, tile granularity, etc.), in some cases across various duration types. In some aspects, the systems and techniques described herein provide signaling to map one or more CMs to a particular sub-picture using a sub-picture identifier (ID). In some aspects, the systems and techniques described herein provide signaling (e.g., syntax elements) to indicate a total number of blocks (e.g., CTBs or other blocks). In some cases, such signaling (indicating the total number of blocks) can simplify the derivation of the number of total coding blocks over a period of time to interpret normalized coded statistics per slice or tile. The signaling may be included in CM metadata included with the coded video. In some aspects, the systems and techniques described herein provide signaling for intra-coded block statistics.In some cases, such intra-coded block statistics signaling can solve problems that arise when a CM provides intra-coded block statistics when all blocks are intra-coded (e.g., when P and B slices have more intra-coded blocks than inter-coded blocks, when P or B pictures have more intra-coded blocks than inter-coded blocks, etc.) In some aspects, the systems and techniques described herein provide mechanisms for applying quality metrics to individual portions of a picture (e.g., to individual sub-pictures), such as in VVC, respectively, rather than applying the quality metric to the entire picture.

[0127]

[0131] Next, various aspects of the above-mentioned complexity metric (CM) signaling are described. For example, in some aspects, a granularity type indicator (e.g., a granularity type syntax element such as granularity_type) is provided to support various granularities (e.g., granularity segments), such as slices, tiles, subpictures, scalable layers, and / or other granularities. For example, the encoding device 104 may signal a granularity type indicator in or with a bitstream. The granularity type indicator may be used in combination with a period type syntax element to support granularity CM signaling that applies for multiple pictures. In some examples, the semantics of the period type syntax element (e.g., period_type) is modified. In one illustrative example, CM signaling for VVC Green metadata is provided in Table 2 below (additions to ISO / IEC 23001-11 are indicated between <> (e.g., <added wording>)).

[0128] [Table 2]

[0129]

[0132] The period_type syntax element (eg, variable) specifies the type of upcoming period for which the complexity metric is applicable, and values ​​for the period_type syntax element may be defined in Table 3 below (as an illustrative example).

[0130] [Table 3]

[0131]

[0133] The granularity_type syntax element specifies the type of granularity to which the complexity metric is applicable, and values ​​for the granularity_type syntax element may be defined in Table 4 below (as an illustrative example).

[0132] [Table 4]

[0133]

[0134] The picture_level_CMs syntax structure specifies the complexity metrics for a particular picture over a period of time, and may be referred to herein as the picture level CM syntax structure.

[0134]

[0135] The granularity_level_CMs syntax structure specifies the granularity level complexity metrics for each entity, such as a slice, a tile, a subpicture, or a layer over a period of time. The granularity_level_CMs syntax structure may be referred to herein as a granularity level CM syntax structure.

[0135]

[0136] 2 is a diagram illustrating an example use of granularity level CM for a video picture 200 (also referred to as a frame or image) according to an aspect of the disclosure. The video picture 200 includes a bicyclist 202 riding and moving across a view of the video picture 200, where the bicyclist 202 appears in a set of pictures 204 of the video picture 200. Each picture of the video picture 200 may be divided (e.g., by an encoder such as the encoding device 104) into one or more portions, such as slices, tiles, sub-pictures, layers, etc. A picture 206 of the set of pictures 204 is shown divided into 16 slices 208, four tiles 210, and one sub-picture 212, where each tile 210 includes four slices 208 and the sub-picture 212 includes two tiles 210 on a lower portion of the picture.

[0136]

[0137] In some cases, being able to specify both the period type and the granularity type for a decoding device (e.g., decoding device 112) allows for greater flexibility and reduced signaling by allowing a granularity level CM to be defined once for slices, tiles, subpictures, or layers for multiple pictures. For example, instead of having to define a granularity level CM for the subpictures of each picture, an encoding device (e.g., encoding device 104) may apply a single granularity level CM to the subpictures of all pictures in a specified time interval. In a video picture 200, as a cyclist 202 moves in the video, areas in which the cyclist 202 appears may be more complex to encode / decode compared to other areas of the set of pictures 204 (which have little or no motion), and different CMs may be specified for those areas using the granularity level CM. For example, the granularity level CM may be specified once for a number of pictures (e.g., num_pictures=6), such as six pictures of the set of pictures 204, for the sub-picture 212 area (e.g., granularity_type=3). By allowing the granularity level CM to be set for a particular time interval (e.g., a set number of pictures, a time period, etc.), a single granularity level CM may be used in the metadata corresponding to the first picture of the set of pictures 204, and this granularity level CM may be applied to all pictures in the set of pictures 204 based on the specified time interval. After the set of pictures 204, the granularity level CM for the sub-picture 212 may be adjusted since there is no longer a bicyclist 202 in the area covered by the sub-picture 212, and then the area may be less complex to encode / decode.Similarly, multiple, possibly different, granularity levels CM may be specified for any number of slices, tiles, subpictures, or layers in a picture, where each granularity level CM may apply over a different time period (e.g., a single picture, all pictures during a specified time interval, a number of pictures, all pictures up to the picture containing the next slice, etc.).

[0137]

[0138] In some aspects, an encoding device (e.g., encoding device 104) may specify, for multiple pictures of a video, CM values ​​that are applicable to portions of the picture, such as slices, tiles, subpictures, and / or layers. For example, according to some aspects, an encoding device may generate and signal subpicture CM signaling. The subpicture CM signaling indicates, for one or more pictures, which subpictures the CM applies to. In one example, for subpicture granularity, a syntax element (e.g., referred to as a subpicture syntax element) indicates that a subpicture ID is signaled in the CM metadata when the duration spans multiple pictures (e.g., when the granularity level CM applies to multiple pictures). An example is shown in Table 5 below.

[0138] [Table 5]

[0139]

[0139] subpic_id[i] specifies the subpicture ID of the associated complexity metric (CM).

[0140]

[0140] subpic_CM is the i-th subpicture complexity metric structure.

[0141]

[0141] In some cases, subpic_id and / or subpic_CM(i) may be replaced by one or more syntax elements that point to a segment address. In some cases, a segment may be a slice, a tile, or a subpicture, and this segment address may identify, for example, a particular slice, tile, and / or subpicture of a picture. As an example, segment address [t] may indicate the address of the t-th segment. Thus, if the granularity type specifies subpicture granularity, segment address [t] may indicate the subpicture ID of the t-th subpicture.

[0142]

[0142] In some cases, aspects relate to resolution changes. For example, in VVC, resolution changes in coding layer video sequence (CLVS) are applicable to picture, slice, and tile granularity, but not sub-picture granularity. According to some aspects, a syntax element (e.g., referred to as coding tree block (CTB) number syntax element) indicating the total number of coding tree luma blocks over a period may be signaled in green metadata (e.g., in a CM syntax table, as one or more syntax elements, such as num_ctbs_minus1 in Table 6 below), such as when the granularity type is equal to slice (e.g., 0x01 from Table 4) or tile (e.g., 0x02 from Table 4) and the period type spans multiple pictures. An example is shown in Table 6 below.

[0143] [Table 6]

[0144]

[0143] num_ctbs_minus1 specifies the total number of coding tree blocks of associated complexity metric over the period.

[0145] In some aspects, an alternative syntax element (e.g., avg_number_ctbs_minus1) may indicate the average number of CTBs or 4x4 blocks (or other sized blocks) per granularity per picture instead of the total number of CTBs over a period of time to reduce overhead. Such a syntax element may be referred to as an average CTB number syntax element.

[0146]

[0145] In some cases, aspects relate to intra-coded block statistics. For example, the current Green Metadata CM syntax only signals intra-coded block statistics when all blocks are intra-coded blocks (e.g., portion_intra_predicted_blocks_area==255). Table 7 below shows the proposed CM signaling changes, where additions are indicated between <> (e.g., <text added>) and deletions are indicated with strikethrough text (e.g., «text deleted» (Note: «» is an alternative representation of strikethrough)). Intra-coded block statistics are signaled when there are intra-coded blocks available. Inter-coded block statistics are signaled when there are inter-coded blocks available.

[0147] [Table 7]

[0148]

[0146] Example definitions of various syntax elements from Table 7 are provided below for VVC.

[0149]

[0147] portion_intra_predicted_blocks_area indicates a portion of the area covered by intra-predicted blocks in a picture of a specified period using 4-sample granularity, and is defined as follows:

[0150]

number

[0151]

[0148] NumIntraPredictedBlocks is the number of intra predicted blocks in the specified period using 4-sample granularity. At the encoder side, it is calculated as follows:

[0152]

number

[0153]

[0149] Here, NumIntraPredictedBlocks_X is the number of blocks using intra prediction for the number of samples from X=4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 during the specified period.

[0154]

[0150] NumIntraPredictedBlocks is derived in the decoder from portion_intra_predicted_blocks_area and TotalNum4BlocksInPeriod.

[0155]

[0151] portion_planar_blocks_in_intra_area indicates a portion of the intra-planar predicted block area in the intra predicted area during the specified period and is defined as follows:

[0156]

number

[0157]

[0152] When not present, equals 0.

[0158]

[0153] NumPlanarPredictedBlocks is the number of intra-planar predicted blocks in a specified period using 4-sample granularity. At the encoder side, it is calculated as follows:

[0159]

number

[0160]

[0154] Here, NumIntraPlanarBlocks_X is the number of blocks using intra-plane prediction for the number of samples from X=4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 during the specified period.

[0161]

[0155] NumPlanarPredictedBlocks is derived in the decoder from portion_planar_blocks_in_intra_area and NumIntraPredictedBlocks.

[0162]

[0156] portion_dc_blocks_in_intra_area indicates a portion of the intra DC predicted block area in the intra predicted area during a specified period and is defined as follows:

[0163]

number

[0164]

[0157] When not present, equals 0.

[0165]

[0158] NumDcPredictedBlocks is the number of intra DC predicted blocks in a specified period using 4-sample granularity. At the encoder side, it is calculated as follows:

[0166]

number

[0167]

[0159] Here, NumIntraDcBlocks_X is the number of blocks using intra DC prediction for the number of samples from X=4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 during the specified period.

[0168]

[0160] NumDcPredictedBlocks is derived in the decoder from portion_dc_blocks_in_intra_area and NumIntraPredictedBlocks.

[0169]

[0161] The portion_angular_hv_blocks_in_intra_area (also called portion_hv_blocks_in_intra_area) indicates the portion of the intra horizontal and vertical predicted block area in the intra predicted area during a specified period and is defined as follows:

[0170]

number

[0171]

[0162] When not present, equals 0.

[0172]

[0163] NumHvPredictedBlocks is the number of intra horizontal and vertical predicted blocks in a specified period using 4-sample granularity. At the encoder side, it is calculated as follows:

[0173]

number

[0174]

[0164] Here, NumIntraHvBlocks_X is the number of blocks using intra horizontal and vertical prediction for the number of samples from X=4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 during the specified period.

[0175]

[0165] NumHvPredictedBlocks is derived in the decoder from portion_hv_blocks_in_intra_area and NumIntraPredictedBlocks.

[0176]

[0166] portion_mip_blocks_in_intra_area indicates a portion of the intra MIP predicted block area in the intra predicted area during a specified period and is defined as follows:

[0177]

number

[0178]

[0167] When not present, equals 0.

[0179]

[0168] NumMipPredictedBlocks is the number of intra MIP predicted blocks in the specified period using 4-sample granularity. On the encoder side, it is calculated as follows:

[0180]

number

[0181]

[0169] Here, NumIntraMipBlocks_X is the number of blocks using intra MIP prediction for the number of samples from X=4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 during the specified period.

[0182]

[0170] NumMipPredictedBlocks is derived in the decoder from portion_mip_blocks_in_intra_area and NumIntraPredictedBlocks.

[0183]

[0171] portion_bi_and_gpm_predicted_blocks_area indicates a portion of the area covered by inter bi-predictive or GPM predicted blocks in a picture of a specified period using 4-sample granularity, and is defined as follows:

[0184]

number

[0185]

[0172] NumBiAndGpmPredictedBlocks is the number of inter bi-predicted and GPM predicted blocks in a specified period using 4-sample granularity. On the encoder side, it is calculated as follows:

[0186]

number

[0187]

[0173] Here, NumBiPredictedXBlocks is the number of blocks using inter bi-prediction or GPM prediction for the number of samples from X=4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 during the specified period.

[0188]

[0174] NumBiPredictedXBlocks is derived in the decoder from portion_bi_and_gpm_predicted_blocks_area and TotalNum4BlocksInPeriod.

[0189]

[0175] portion_deblocking_instances indicates a portion of the deblocking filtering instances, as defined in the Terms and Definitions section of this specification, during a specified period of time, and is defined as follows:

[0190]

number

[0191]

[0176] NumDeblockingInstances is the number of deblocking filtering instances during the specified period. It is derived from portion_deblocking_instances and MaxNumDeblockingInstances at the decoder.

[0192] portion_sao_filtered_blocks indicates the portion of SAO filtered blocks in the specified period using 4-sample granularity. At the encoder side, it is calculated as follows:

[0193]

number

[0194]

[0178] NumSaoFilteredBlocks is the number of SAO filtered blocks in the specified period using 4-sample granularity. It is derived from portion_sao_filtered_blocks, TotalNum4BlocksInPeriod at the decoder.

[0195] portion_alf_filtered_blocks indicates the portion of ALF filtered blocks in the specified period using 4 sample granularity. At the encoder side, it is calculated as follows:

[0196]

number

[0197]

[0180] NumAlfFilteredBlocks is the number of ALF filtered blocks in the specified period using 4-sample granularity. It is derived from portion_alf_filtered_blocks, TotalNum4BlocksInPeriod at the decoder.

[0198] In some cases, aspects relate to sub-picture quality metrics. For example, a quality restoration metric may be applied to each granularity segment. In some cases, a segment may be a slice, a tile, or a sub-picture. Table 8 provides an example of a sub-picture-based metric for quality restoration proposed for Green MPEG, where additions are indicated between <> (e.g., <added text>).

[0199] [Table 8]

[0200]

[0182] xsd_subpic_number_minus1 specifies the number of subpictures available in the associated picture. When xsd_subpic_number_minus1 is equal to 0, the quality restoration metric applies to the entire picture.

[0201]

[0183] xsd_metric_type[i] indicates the type of the objective quality metric for the i-th objective quality metric.

[0202]

[0184] xsd_metric_value[i][j] contains the value of the i-th objective quality metric for the associated j-th subpicture.

[0203]

[0185] The current quality metric describes the quality of the last picture of each segment. The aspects described herein enable the SEI message to carry a quality metric that describes the quality of the associated picture. For example, an encoding device (e.g., encoding device 104 of FIG. 1 and FIG. 4) can add the quality metric to the SEI message.

[0204]

[0186] Figure 3 is a flow diagram illustrating a process for decoding an encoded video 300 according to an aspect of the disclosure. At operation 302, the process 300 may include obtaining a bitstream. At operation 304, the process 300 may include retrieving a granularity type syntax element associated with the bitstream, the granularity type syntax element specifying a type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable. In some cases, the value of the granularity type syntax element specifies that the CM is applicable to a picture or a portion of a picture of the bitstream, the portion of a picture being less than the entirety of the picture. In some cases, the value of the granularity type syntax element specifies that the CM is applicable to at least one of a slice, a tile, a subpicture, a scalable layer, or a coding tree unit (CTU) row of one or more pictures of the bitstream.

[0205]

[0187] At operation 306, the process 300 may include retrieving a period type syntax element associated with the bitstream, the period type syntax element indicating an upcoming time period or set of pictures to which the CM is applicable. In some cases, the period type syntax element indicates at least one of a specified time interval for the upcoming period, a number of pictures for the upcoming period, that the upcoming period includes all pictures up to the picture that includes the next slice, or that the upcoming period includes a single picture. In some cases, the process 300 may also include retrieving a granularity level CM syntax structure associated with the bitstream, the granularity level CM syntax structure specifying a granularity level complexity metric for one or more granularity segments of the bitstream over the upcoming period. In some cases, the process 300 may further include retrieving an additional period type syntax element associated with the bitstream, and decoding a portion of the bitstream based on the granularity type syntax element and the additional period type syntax element, where the additional period type syntax element is associated with the granularity type syntax element, and where the additional period type syntax element is different from the period type syntax element.

[0206]

[0188] In some cases, process 300 may also include retrieving at least one of: a subpicture syntax element associated with the bitstream, where the subpicture syntax element indicates that a subpicture identifier (ID) is signaled when the CM applies to multiple pictures; a coding tree block (CTB) number syntax element associated with the bitstream, where the CTB number syntax element indicates that the total number of coding tree luma blocks over the upcoming period may be signaled in the CM when the granularity type is equal to slice or tile and the upcoming period spans multiple pictures; or an average coding tree block (CTB) number syntax element associated with the bitstream, where the average CTB number syntax element indicates the average number of CTBs or 4x4 blocks per granularity per picture.

[0207]

[0189] In some cases, for the process 300, the intra-coded block statistics are signaled in association with at least a portion of the bitstream when there are intra-coded blocks available in at least a portion of the bitstream. In some cases, the inter-coded block statistics are signaled in association with at least a portion of the bitstream when there are inter-coded blocks available in at least a portion of the bitstream. In some cases, the process 300 may also include displaying at least a portion of the bitstream on a display. In some cases, the process 300 may further include determining an operating frequency of the device based on a CM associated with the bitstream.

[0208] At operation 308, the process 300 may include decoding the portion of the bitstream based on the granularity type syntax element and the period type syntax element. In some cases, the process 300 may be performed by one of a mobile device, a wearable device, an extended reality device, a camera, a personal computer, a vehicle, a robotic device, a television, or a computing device.

[0209]

[0191] Figure 4 is a flow diagram 400 illustrating a technique for encoding video according to an aspect of the disclosure. At operation 402, the process 400 may include obtaining video data. At operation 404, the process 400 may include generating, for the bitstream, a granularity type syntax element that specifies a type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable. In some cases, the value of the granularity type syntax element specifies that the CM is applicable to a picture or a portion of a picture of the bitstream, the portion of a picture being less than the entirety of the picture. In some cases, the value of the granularity type syntax element specifies that the CM is applicable to at least one of a slice, a tile, a subpicture, a scalable layer, or a coding tree unit (CTU) row of one or more pictures of the bitstream.

[0210]

[0192] At operation 406, the process 400 may include generating, for the bitstream, a period type syntax element associated with the bitstream, the period type syntax element indicating an upcoming time period or set of pictures to which the CM is applicable. In some cases, the period type syntax element indicates at least one of a specified time interval for the upcoming period, a number of pictures for the upcoming period, that the upcoming period includes all pictures up to the picture that includes the next slice, or that the upcoming period includes a single picture. In some cases, the period type syntax element indicates at least one of a specified time interval for the upcoming period, a number of pictures for the upcoming period, that the upcoming period includes all pictures up to the picture that includes the next slice, or that the upcoming period includes a single picture.

[0211]

[0193] In some cases, process 400 may also include generating, for the bitstream, a granularity-level CM syntax structure that specifies a granularity-level complexity metric for one or more entities over a coming period. In some cases, process 400 may further include generating, for the bitstream, an additional period type syntax element associated with the granularity type syntax element, where the additional period type syntax element is different from the period type syntax element, where the additional period type syntax element is for decoding a portion of the bitstream with the granularity type syntax element. In some cases, process 400 may also include generating, for the bitstream, at least one of: a sub-picture syntax element associated with the bitstream, the sub-picture syntax element indicating that a sub-picture identifier (ID) is signaled when the CM applies to multiple pictures; a coding tree block (CTB) number syntax element associated with the bitstream, the CTB number syntax element indicating that when the granularity type is equal to slice or tile and the upcoming period spans multiple pictures, the total number of coding tree luma blocks over the upcoming period may be signaled in the CM; or an average coding tree block (CTB) number syntax element associated with the bitstream, the average CTB number syntax element indicating the average number of CTBs or 4x4 blocks per granularity per picture.

[0212]

[0194] In some cases, for process 400, the intra-coded block statistics are signaled in association with at least a portion of a bitstream when there are intra-coded blocks available in at least a portion of the bitstream. In some cases, the inter-coded block statistics are signaled in association with at least a portion of a bitstream when there are inter-coded blocks available in at least a portion of the bitstream. In some cases, process 400 may be performed by one of a mobile device, a wearable device, an extended reality device, a camera, a personal computer, a vehicle, a robotic device, a television, or a computing device.

[0213] In some implementations, the processes (or methods) described herein may be implemented by a computing device or apparatus, such as the system 100 shown in FIG. 1. For example, the processes may be implemented by the encoding device 104 shown in FIG. 1 and FIG. 5, by another video source side device or video transmission device, by the decoding device 112 shown in FIG. 1 and FIG. 6, and / or by another client side device, such as a player device, a display, or any other client side device. In some cases, the computing device or apparatus may include one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other component(s) configured to perform steps of one or more processes described herein.

[0214]

[0196] In some examples, the computing device may include a mobile device, a desktop computer, a server computer and / or a server system, or other types of computing devices. The components of the computing device (e.g., one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components) may be implemented in a circuit. For example, the components may include and / or be implemented using electronic circuitry or other electronic hardware that may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits) and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein. In some examples, the computing device or apparatus may include a camera configured to capture video data (e.g., a video sequence) including video frames. In some examples, the camera or other capture device that captures the video data is separate from the computing device, in which case the computing device receives or obtains the captured video data. The computing device may include a network interface configured to communicate video data. The network interface may be configured to communicate Internet Protocol (IP)-based data or other types of data. In some examples, the computing device or apparatus may include a display for displaying output video content, such as samples of pictures of a video bitstream.

[0215]

[0197] A process may be described in terms of a logical flow diagram, whose operations represent a sequence of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the operations are described should not be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.

[0216]

[0198] Furthermore, the process may be performed under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored in a computer-readable or machine-readable storage medium, for example in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0217]

[0199] The coding techniques described herein may be implemented in an exemplary video encoding and decoding system (e.g., system 100). In some examples, the system includes a source device that provides encoded video data to be subsequently decoded by a destination device. In particular, the source device provides the video data to the destination device via a computer-readable medium. The source device and destination device may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, so-called "smart" pads, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, and the like. In some cases, the source device and destination device may be equipped for wireless communication.

[0218]

[0200] The destination device may receive the encoded video data to be decoded via a computer-readable medium. The computer-readable medium may comprise any type of medium or device capable of moving encoded video data from a source device to a destination device. In one example, the computer-readable medium may comprise a communication medium for enabling the source device to transmit the encoded video data directly to the destination device in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the destination device. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other equipment that may be useful for facilitating communication from the source device to the destination device.

[0219]

[0201] In some examples, the encoded data may be output from the output interface to a storage device. Similarly, the encoded data may be accessed from the storage device by the input interface. The storage device may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In a further example, the storage device may correspond to a file server or another intermediate storage device that may store the encoded video generated by the source device. The destination device may access the stored video data from the storage device via streaming or download. The file server may be any type of server capable of storing the encoded video data and transmitting the encoded video data to the destination device. Exemplary file servers include a web server (e.g., for a website), an FTP server, a network-attached storage (NAS) device, or a local disk drive. The destination device may access the encoded video data through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both, suitable for accessing the encoded video data stored on the file server. The transmission of the encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

[0220]

[0202] The techniques of this disclosure are not necessarily limited to wireless applications or settings. The techniques may be applied to video coding to support any of a variety of multimedia applications, such as over-the-air television broadcast, cable television transmission, satellite television transmission, Internet streaming video transmission such as Dynamic Adaptive Streaming over HTTP (DASH), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, the system may be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0221]

[0203] In one example, the source device includes a video source, a video encoder, and an output interface. The destination device may include an input interface, a video decoder, and a display device. The video encoder of the source device may be configured to apply the techniques disclosed herein. In other examples, the source device and the destination device may include other components or arrangements. For example, the source device may receive video data from an external video source, such as an external camera. Similarly, the destination device may interface with an external display device rather than including an integrated display device.

[0222]

[0204] The above exemplary system is only an example. The technique for processing video data in parallel may be implemented by any digital video encoding and / or decoding device. Generally, the techniques of this disclosure are implemented by a video encoding device, but the techniques may also be implemented by a video encoder / decoder, commonly referred to as a "codec". Moreover, the techniques of this disclosure may also be implemented by a video preprocessor. The source device and the destination device are only examples of coding devices, such that the source device generates coded video data for transmission to the destination device. In some examples, the source device and the destination device may operate substantially symmetrically, such that each of the devices includes a video encoding component and a video decoding component. Thus, the exemplary system may support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0223]

[0205] The video source may include a video capture device such as a video camera, a video archive containing previously captured video, and / or a video feed interface for receiving video from a video content provider. As a further alternative, the video source may generate computer graphics-based data as the source video, or a combination of live video, archive video, and computer-generated video. In some cases, when the video source is a video camera, the source device and the destination device may form a so-called camera phone or video phone. However, as mentioned above, the techniques described in this disclosure may be applicable to video coding in general, and may be applied to wireless and / or wired applications. In each case, the captured video, pre-captured video, or computer-generated video may be encoded by a video encoder. The encoded video information may then be output on a computer-readable medium by an output interface.

[0224]

[0206] As mentioned, computer-readable media may include a temporary medium, such as a wireless broadcast or wired network transmission, or a storage medium (i.e., a non-transitory storage medium), such as a hard disk, a flash drive, a compact disc, a digital video disc, a Blu-ray disc, or other computer-readable medium. In some examples, a network server (not shown) may receive encoded video data from a source device, for example, via a network transmission, and provide the encoded video data to a destination device. Similarly, a computing device of a media production facility, such as a disc stamping facility, may receive encoded video data from a source device and produce a disc including the encoded video data. Thus, a computer-readable medium may be understood to include one or more computer-readable media of various forms in various examples.

[0225]

[0207] The input interface of the destination device receives information from a computer-readable medium. The information of the computer-readable medium may include syntax information, including syntax elements that describe characteristics and / or processing of blocks and other coding units, e.g., groups of pictures (GOPs), defined by a video encoder and used by a video decoder. The display device displays the decoded video data to a user and may comprise any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device. Various embodiments of the present application have been described.

[0226] Specific details of the encoding device 104 and the decoding device 112 are illustrated in FIG. 5 and FIG. 6, respectively. FIG. 5 is a block diagram illustrating an example encoding device 104 that may implement one or more of the techniques described in this disclosure. The encoding device 104 may generate, for example, syntax elements and / or structures described herein (e.g., syntax elements and / or structures for green metadata, such as a complexity metric (CM), or other syntax elements and / or structures). The encoding device 104 may perform intra-predictive coding and inter-predictive coding of video blocks within a video slice, tile, sub-picture, etc. As previously described, intra-coding relies at least in part on spatial prediction to reduce or remove spatial redundancy within a given video frame or picture. Inter-coding relies at least in part on temporal prediction to reduce or remove temporal redundancy within adjacent or surrounding frames of a video sequence. Intra-mode (I-mode) may refer to any of several spatial-based compression modes. An inter-mode, such as uni-directional prediction (P mode) or bi-prediction (B mode), may refer to any of several temporal-based compression modes.

[0227]

[0209] Encoding device 104 includes partition unit 35, prediction processing unit 41, filter unit 63, picture memory 64, adder 50, transform processing unit 52, quantization unit 54, and entropy coding unit 56. Prediction processing unit 41 includes motion estimation unit 42, motion compensation unit 44, and intra-prediction processing unit 46. For video block reconstruction, encoding device 104 also includes inverse quantization unit 58, inverse transform processing unit 60, and adder 62. Filter unit 63 is intended to represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. In FIG. 5, filter unit 63 is illustrated as being an in-loop filter, but in other configurations, filter unit 63 may be implemented as a post-loop filter. Post-processing device 57 may perform additional processing on the encoded video data generated by encoding device 104. The techniques of this disclosure may, in some cases, be implemented by encoding device 104. However, in other cases, one or more of the techniques of this disclosure may be implemented by post-processing device 57.

[0228]

[0210] As shown in FIG. 5, encoding device 104 receives video data, and partition unit 35 partitions the data into video blocks. Partitioning may also include partitioning into slices, slice segments, tiles, or other larger units, as well as video block partitioning, for example, according to a quadtree structure of LCUs and CUs. Encoding device 104 generally illustrates components that encode video blocks in a video slice to be encoded. A slice may be divided into a number of video blocks (and possibly into a set of video blocks called tiles). Prediction processing unit 41 may select one of a number of possible coding modes, such as one of a number of intra-predictive coding modes or one of a number of inter-predictive coding modes, for a current video block based on an error result (e.g., a coding rate and a distortion level, etc.). Prediction processing unit 41 may provide the resulting intra-coded or inter-coded block to adder 50 to generate residual block data and to adder 62 to reconstruct a coding block for use as a reference picture.

[0229]

[0211] Intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-predictive coding of the current video block relative to one or more neighboring blocks in the same frame or slice as the current block to be coded to provide spatial compression. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 perform inter-predictive coding of the current video block relative to one or more predictive blocks in one or more reference pictures to provide temporal compression.

[0230]

[0212] Motion estimation unit 42 may be configured to determine an inter-prediction mode for a video slice according to a predetermined pattern for a video sequence. The predetermined pattern may designate a video slice in the sequence as a P slice, a B slice, or a GPB slice. Motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes. Motion estimation performed by motion estimation unit 42 is a process of generating motion vectors that estimate motion for a video block. A motion vector may indicate, for example, the displacement of a prediction unit (PU) of a video block in a current video frame or picture relative to a predictive block in a reference picture.

[0231]

[0213] A prediction block is a block that is found to closely match a PU of a video block to be coded in terms of pixel differences, which may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. In some examples, encoding device 104 may calculate values ​​for sub-integer pixel positions of a reference picture stored in picture memory 64. For example, encoding device 104 may interpolate values ​​for ¼ pixel positions, ⅛ pixel positions, or other fractional pixel positions of a reference picture. Thus, motion estimation unit 42 may perform motion search for full pixel positions and fractional pixel positions and output motion vectors with fractional pixel accuracy.

[0232]

[0214] Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-coded slice by comparing the position of the PU with the position of a predictive block of a reference picture. The reference picture may be selected from a first reference picture list (list 0) or a second reference picture list (list 1), each of which identifies one or more reference pictures stored in picture memory 64. Motion estimation unit 42 sends the calculated motion vector to entropy encoding unit 56 and motion compensation unit 44.

[0233]

[0215] Motion compensation performed by motion compensation unit 44 may involve fetching or generating a predictive block based on a motion vector determined by motion estimation, possibly performing interpolation to sub-pixel accuracy. Upon receiving a motion vector for a PU of a current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in a reference picture list. Encoding device 104 forms a residual video block by subtracting pixel values ​​of the predictive block from pixel values ​​of the current video block being coded to form pixel difference values. The pixel difference values ​​form residual data for the block and may include both luma and chroma difference components. Adder 50 represents one or more components that perform this subtraction operation. Motion compensation unit 44 may also generate syntax elements associated with the video block and the video slice for use by decoding device 112 in decoding the video block of the video slice.

[0234]

[0216] The intra-prediction processing unit 46 may intra-predict the current block as an alternative to the inter-prediction performed by the motion estimation unit 42 and the motion compensation unit 44 as described above. In particular, the intra-prediction processing unit 46 may determine an intra-prediction mode to be used to encode the current block. In some examples, the intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, e.g., during separate encoding passes, and the intra-prediction processing unit 46 may select an appropriate intra-prediction mode to use from the tested modes. For example, the intra-prediction processing unit 46 may calculate rate-distortion values ​​using a rate-distortion analysis for the various tested intra-prediction modes and select the intra-prediction mode with the best rate-distortion characteristics among the tested modes. The rate-distortion analysis generally determines the amount of distortion (or error) between the encoded block and the original uncoded block that was coded to produce the encoded block, as well as the bitrate (i.e., the number of bits) used to produce the encoded block. Intra-prediction processing unit 46 may calculate ratios from the distortions and rates for the various coding blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0235]

[0217] In either case, after selecting an intra-prediction mode for a block, intra-prediction processing unit 46 may provide information indicating the selected intra-prediction mode for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode. Encoding device 104 may include in the transmitted bitstream configuration data definitions of encoding contexts for various blocks, as well as an indication of the most probable intra-prediction mode, intra-prediction mode index table, and modified intra-prediction mode index table to be used for each of the contexts. The bitstream configuration data may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also referred to as codeword mapping tables).

[0236] After prediction processing unit 41 generates a predictive block for a current video block, either via inter prediction or intra prediction, encoding device 104 forms a residual video block by subtracting the predictive block from the current video block. The residual video data in the residual block may be included in one or more TUs and applied to transform processing unit 52. Transform processing unit 52 converts the residual video data into residual transform coefficients using a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform. Transform processing unit 52 may convert the residual video data from the pixel domain to a transform domain, such as the frequency domain.

[0237]

[0219] The transform processing unit 52 may send the resulting transform coefficients to the quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the quantization unit 54 may then perform a scan of a matrix including the quantized transform coefficients. Alternatively, the entropy coding unit 56 may perform the scan.

[0238]

[0220] After quantization, entropy coding unit 56 entropy codes the quantized transform coefficients. For example, entropy coding unit 56 may implement context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioned entropy (PIPE) coding, or another entropy coding technique. After entropy coding by entropy coding unit 56, the coded bitstream may be transmitted to decoding device 112 or archived for later transmission or retrieval by decoding device 112. Entropy coding unit 56 may also entropy code motion vectors and other syntax elements for the current video slice being coded.

[0239]

[0221] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain for later use as a reference block of a reference picture. Motion compensation unit 44 may calculate a reference block by adding the residual block to a predictive block of one of the reference pictures in the reference picture list. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values ​​for use in motion estimation. Adder 62 adds the reconstructed residual block to the motion compensated predictive block produced by motion compensation unit 44 to produce a reference block for storage in picture memory 64. The reference block may be used as a reference block by motion estimation unit 42 and motion compensation unit 44 to inter-predict a block in a subsequent video frame or picture.

[0240]

[0222] In this manner, encoding device 104 of Figure 5 represents an example of a video encoder configured to perform any of the techniques described herein. In some cases, some of the techniques of this disclosure may also be implemented by post-processing device 57.

[0241]

[0223] Figure 6 is a block diagram illustrating an example decoding device 112. The decoding device 112 includes an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, a filter unit 91, and a picture memory 92. The prediction processing unit 81 includes a motion compensation unit 82 and an intra-prediction processing unit 84. The decoding device 112 may, in some examples, perform a decoding path that is generally inverse to the encoding path described with respect to the encoding device 104 from Figure 5.

[0242] During the decoding process, the decode device 112 receives an encoded video bitstream representing video blocks of encoded video slices and associated syntax elements sent by the encoding device 104. In some embodiments, the decode device 112 may receive the encoded video bitstream from the encoding device 104. In some embodiments, the decode device 112 may receive the encoded video bitstream from a network entity 79, such as a server, a media-aware network element (MANE), a video editor / splitter, or other such device configured to implement one or more of the techniques described above. The network entity 79 may or may not include the encoding device 104. Some of the techniques described in this disclosure may be implemented by the network entity 79 prior to the network entity 79 sending the encoded video bitstream to the decode device 112. In some video decoding systems, the network entity 79 and the decode device 112 may be parts of separate devices, while in other cases, the functions described with respect to the network entity 79 may be performed by the same device that comprises the decode device 112.

[0243]

[0225] The entropy decoding unit 80 of the decoding device 112 entropy decodes the bitstream to generate quantized coefficients, motion vectors, and other syntax elements. The entropy decoding unit 80 forwards the motion vectors and other syntax elements to the prediction processing unit 81. The decoding device 112 may receive video slice level and / or video block level syntax elements. The entropy decoding unit 80 may process and parse both fixed length and variable length syntax elements, such as VPS, SPS, and PPS, or in multiple parameter sets.

[0244]

[0226] When a video slice is coded as an intra-coded (I) slice, intra-prediction processing unit 84 of prediction processing unit 81 may generate predictive data for video blocks of the current video slice based on the signaled intra-prediction mode and data from previously decoded blocks of the current frame or picture. When a video frame is coded as an inter-coded (i.e., B, P, or GPB) slice, motion compensation unit 82 of prediction processing unit 81 produces predictive blocks for video blocks of the current video slice based on motion vectors and other syntax elements received from entropy decoding unit 80. The predictive blocks may be produced from one of the reference pictures in the reference picture list. Decoding device 112 may construct the reference frame lists, i.e., list 0 and list 1, using a default construction technique based on the reference pictures stored in picture memory 92.

[0245]

[0227] Motion compensation unit 82 may determine prediction information for video blocks of a current video slice by parsing the motion vectors and other syntax elements, and use the prediction information to produce predictive blocks for the current video block being decoded. For example, motion compensation unit 82 may use one or more syntax elements in a parameter set to determine a prediction mode (e.g., intra or inter prediction) used to code the video blocks of the video slice, an inter-prediction slice type (e.g., a B slice, a P slice, or a GPB slice), construction information for one or more reference picture lists for the slice, a motion vector for each inter-coded video block of the slice, an inter-prediction status for each inter-coded video block of the slice, and other information for decoding video blocks in the current video slice.

[0246]

[0228] Motion compensation unit 82 may also perform interpolation based on an interpolation filter. Motion compensation unit 82 may use an interpolation filter used by encoding device 104 during encoding of the video block to calculate interpolated values ​​for sub-integer pixels of the reference block. In this case, motion compensation unit 82 may determine the interpolation filter used by encoding device 104 from a received syntax element and may use that interpolation filter to produce the predictive block.

[0247]

[0229] Inverse quantization unit 86 inverse quantizes, i.e., dequantizes, the quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit 80. The inverse quantization process may include the use of a quantization parameter calculated by encoding device 104 for each video block in a video slice to determine the degree of quantization, as well as the degree of inverse quantization that should be applied. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT or other suitable inverse transform), an inverse integer transform, or a conceptually similar inverse transform process to the transform coefficients to produce residual blocks in the pixel domain.

[0248] After motion compensation unit 82 generates a predictive block for a current video block based on the motion vector and other syntax elements, decoding device 112 forms a decoded video block by adding a residual block from inverse transform processing unit 88 with a corresponding predictive block generated by motion compensation unit 82. Adder 90 represents one or more components that perform this addition operation. If desired, a loop filter (either in the coding loop or after the coding loop) may also be used to smooth pixel transitions or otherwise improve video quality. Filter unit 91 is intended to represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. Although filter unit 91 is illustrated in FIG. 6 as being an in-loop filter, in other configurations filter unit 91 may be implemented as a post-loop filter. The decoded video blocks in a given frame or picture are then stored in picture memory 92, which stores reference pictures used for subsequent motion compensation. Picture memory 92 also stores decoded video for later presentation on a display device, such as video destination device 122 shown in FIG.

[0249]

[0231] Thus, the decoding device 112 of FIG. 6 represents an example of a video decoder configured to implement any of the techniques described herein.

[0250]

[0232] The term "computer-readable medium" as used herein includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instruction(s) and / or data. Computer-readable media may include non-transitory media on which data may be stored, which does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0251] In some embodiments, computer-readable storage devices, media, and memories may include cable or wireless signals containing bit streams, etc. However, when stated, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0252]

[0234] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that the embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the present technology may be presented as including individual functional blocks, including devices, device components, steps or routines in a method embodied in software, or functional blocks comprising a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0253]

[0235] Each embodiment may be described above as a process or method that is shown as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Furthermore, the order of operations may be rearranged. A process is terminated when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to a calling function or a main function.

[0254]

[0236] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0255]

[0237] A device implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Common examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rackmount devices, standalone devices, etc. The functions described herein may also be embodied in peripheral devices or add-in cards. Such functions may also be implemented on a circuit board among different chips or different processes executing in a single device, as further examples.

[0256]

[0238] The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing functionality described in this disclosure.

[0257]

[0239] In the above description, the aspects of the present application have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in various ways, and the appended claims are to be construed to include such variations, except as limited by the prior art. The various features and aspects of the above-described applications may be used individually or together. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be considered illustrative and not limiting. For purposes of explanation, the methods have been described in a particular order. It should be appreciated that in alternative embodiments, the methods may be performed in an order different from that described.

[0258]

[0240] Those skilled in the art will appreciate that the symbols or terminology used in this specification of less than ("<") and greater than (">") may be replaced with the symbols less than or equal to ("≦") and greater than or equal to ("≧"), respectively, without departing from the scope of this description.

[0259]

[0241] When a component is described as being "configured to" perform a certain operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or by any combination thereof.

[0260]

[0242] The phrase "coupled to" refers to any component that is physically connected to another component, either directly or indirectly, and / or any component that is in communication with another component, either directly or indirectly (e.g., connected to another component via a wired or wireless connection, and / or other suitable communications interface).

[0261]

[0243] Claim language reciting "at least one of" a set and / or "one or more" of a set or other language in this disclosure indicates that one member of a set or more than one member of a set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" can mean A, B, or A and B, and can further include items not listed in the set of A and B.

[0262]

[0244] The various exemplary logic blocks, modules, circuits, and algorithm steps described with respect to the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly show this compatibility of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally with respect to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functions in various ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of this application.

[0263]

[0245] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Features described as modules or components may be implemented together in an integrated logic device, or separately as separate but interoperable logic devices. When implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, or the like. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, that carries or communicates program code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer.

[0264]

[0246] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured for encoding and decoding, or may be incorporated into a composite video encoder / decoder (codec).

[0265]

[0247] Exemplary aspects of the present disclosure include the following.

[0266]

[0248] Aspect 1. An apparatus for processing video data, comprising at least one memory and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: obtain a bitstream; extract a granularity type syntax element associated with the bitstream; extract a period type syntax element associated with the bitstream, the granularity type syntax element specifying a type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable; and decode a portion of the bitstream based on the granularity type syntax element and the period type syntax element, the period type syntax element indicating an upcoming time period or set of pictures to which the CM is applicable.

[0267]

[0249] Aspect 2. The apparatus of claim 1, wherein a value of the granularity type syntax element specifies that the CM is applicable to a picture or a portion of a picture of the bitstream, the portion of a picture being less than the entire picture.

[0268]

[0250] Aspect 3. The apparatus of claim 1, wherein the value of the granularity type syntax element specifies that the CM is applicable to at least one of a slice, a tile, a subpicture, a scalable layer, or a coding tree unit (CTU) row of one or more pictures of the bitstream.

[0269]

[0251] Aspect 4. The apparatus of claim 1, wherein the period type syntax element indicates at least one of: a specified time interval for the upcoming period, a number of pictures for the upcoming period, that the upcoming period includes all pictures up to the picture containing the next slice, or that the upcoming period includes a single picture.

[0270]

[0252] Aspect 5. The apparatus of claim 1, wherein at least one processor is configured to retrieve a granularity level CM syntax structure associated with the bitstream, the granularity level CM syntax structure specifying a granularity level complexity metric for one or more granularity segments of the bitstream over a coming period.

[0271]

[0253] Aspect 6. The apparatus of claim 1, wherein at least one processor is configured to extract an additional period type syntax element associated with the bitstream, and decode a portion of the bitstream based on the granularity type syntax element and the additional period type syntax element, where the additional period type syntax element is associated with a granularity type syntax element and the additional period type syntax element is different from the period type syntax element.

[0272]

[0254] Aspect 7. The device of claim 1, wherein at least one processor is configured to extract at least one of: a subpicture syntax element associated with the bitstream, the subpicture syntax element indicating that a subpicture identifier (ID) is signaled when the CM applies to multiple pictures; a coding tree block (CTB) number syntax element associated with the bitstream, the CTB number syntax element indicating that when the granularity type is equal to slice or tile and the current period spans multiple pictures, the total number of coding tree luma blocks over the current period may be signaled in the CM; or an average coding tree block (CTB) number syntax element associated with the bitstream, the average CTB number syntax element indicating the average number of CTBs or 4x4 blocks per granularity per picture.

[0273]

[0255] Aspect 8. The apparatus of claim 1, wherein intra-coded block statistics are signaled in association with at least a portion of a bitstream when there are intra-coded blocks available in at least the portion of the bitstream.

[0274]

[0256] Aspect 9. The apparatus of claim 1, wherein inter-coded block statistics are signaled in association with at least a portion of a bitstream when there are inter-coded blocks available in at least the portion of the bitstream.

[0275]

[0257] Aspect 10. The apparatus of claim 1, wherein at least one processor is configured to determine an operating frequency of the apparatus based on a CM associated with the bitstream.

[0276]

[0258] Aspect 11. The apparatus of claim 1, further comprising a display configured to display at least a portion of the bitstream.

[0277]

[0259] Aspect 12. The apparatus of claim 1, wherein the apparatus is one of a mobile device, a wearable device, an extended reality device, a camera, a personal computer, a vehicle, a robotic device, a television, or a computing device.

[0278]

[0260] Aspect 13. A method for processing video data, comprising obtaining a bitstream, extracting a granularity type syntax element associated with the bitstream, extracting a period type syntax element associated with the bitstream, the granularity type syntax element specifying a type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable, and decoding a portion of the bitstream based on the granularity type syntax element and the period type syntax element, the period type syntax element indicating an upcoming time period or set of pictures to which the CM is applicable.

[0279]

[0261] Aspect 14. The method of claim 13, wherein the value of the granularity type syntax element specifies that the CM is applicable to a picture or a portion of a picture of the bitstream, the portion of a picture being less than the entire picture.

[0280]

[0262] Aspect 15. The method of claim 13, wherein the value of the granularity type syntax element specifies that the CM is applicable to at least one of a slice, a tile, a subpicture, a scalable layer, or a coding tree unit (CTU) row of one or more pictures of the bitstream.

[0281]

[0263] Aspect 16. The method of claim 13, wherein the period type syntax element indicates at least one of: a specified time interval for the upcoming period, a number of pictures for the upcoming period, that the upcoming period includes all pictures up to the picture containing the next slice, or that the upcoming period includes a single picture.

[0282]

[0264] Aspect 17. The method of claim 13, further comprising retrieving a granularity level CM syntax structure associated with the bitstream, the granularity level CM syntax structure specifying a granularity level complexity metric for one or more granularity segments of the bitstream over a coming period.

[0283]

[0265] Aspect 18. The method of claim 13, further comprising extracting an additional period type syntax element associated with the bitstream, and decoding a portion of the bitstream based on the granularity type syntax element and the additional period type syntax element, wherein the additional period type syntax element is associated with a granularity type syntax element, and wherein the additional period type syntax element is different from the period type syntax element.

[0284]

[0266] Aspect 19. The method of claim 13, further comprising extracting at least one of: a sub-picture syntax element associated with the bitstream, the sub-picture syntax element indicating that a sub-picture identifier (ID) is signaled when the CM applies to multiple pictures; a coding tree block (CTB) number syntax element associated with the bitstream, the CTB number syntax element indicating that the total number of coding tree luma blocks over the upcoming period may be signaled in the CM when the granularity type is equal to slice or tile and the upcoming period spans multiple pictures; or an average coding tree block (CTB) number syntax element associated with the bitstream, the average CTB number syntax element indicating the average number of CTBs or 4x4 blocks per granularity per picture.

[0285]

[0267] Aspect 20. The method of claim 13, wherein intra-coded block statistics are signaled in association with at least a portion of a bitstream when there are intra-coded blocks available in at least the portion of the bitstream.

[0286]

[0268] Aspect 21. The method of claim 13, wherein inter-coded block statistics are signaled in association with at least a portion of a bitstream when there are inter-coded blocks available in at least the portion of the bitstream.

[0287]

[0269] Aspect 22. The method of claim 13, further comprising displaying at least a portion of the bitstream on a display.

[0288]

[0270] Aspect 23. The method of claim 13, further comprising determining an operating frequency of the device based on a CM associated with the bitstream.

[0289]

[0271] Aspect 24. An apparatus for processing video data, comprising at least one memory and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: obtain video data; generate, for a bitstream, a granularity type syntax element that specifies a type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable; generate, for the bitstream, a period type syntax element associated with the bitstream; generate a bitstream associated with the video data, wherein the period type syntax element indicates an upcoming time period or set of pictures to which the CM is applicable; and output the generated bitstream, wherein the bitstream includes the granularity type syntax element and the period type syntax element.

[0290]

[0272] Aspect 25. The apparatus of claim 24, wherein a value of the granularity type syntax element specifies that the CM is applicable to a picture or a portion of a picture of the bitstream, the portion of a picture being smaller than an entire picture.

[0291]

[0273] Aspect 26. The apparatus of claim 24, wherein the value of the granularity type syntax element specifies that the CM is applicable to at least one of a slice, a tile, a subpicture, a scalable layer, or a coding tree unit (CTU) row of one or more pictures of the bitstream.

[0292]

[0274] Aspect 27. The apparatus of claim 24, wherein the period type syntax element indicates at least one of a specified time interval for the upcoming period, a number of pictures for the upcoming period, that the upcoming period includes all pictures up to the picture containing the next slice, or that the upcoming period includes a single picture.

[0293]

[0275] Aspect 28. The apparatus of claim 24, wherein the one or more processors are configured to generate, for the bitstream, a granularity level CM syntax structure that specifies a granularity level complexity metric for one or more entities over a coming period.

[0294]

[0276] Aspect 29. The apparatus of claim 24, wherein the period type syntax element indicates at least one of a specified time interval for the upcoming period, a number of pictures for the upcoming period, that the upcoming period includes all pictures up to the picture containing the next slice, or that the upcoming period includes a single picture.

[0295]

[0277] Aspect 30. The apparatus of claim 24, wherein at least one processor is configured to generate, for the bitstream, an associated additional period type syntax element, the additional period type syntax element being associated with the granularity type syntax element, where the additional period type syntax element is different from the period type syntax element, and where the additional period type syntax element is for decoding a portion of the bitstream with the granularity type syntax element.

[0296]

[0278] Aspect 31. The device of claim 24, wherein at least one processor is configured to generate, for the bitstream, at least one of: a subpicture syntax element associated with the bitstream, the subpicture syntax element indicating that a subpicture identifier (ID) is signaled when the CM applies to multiple pictures; a coding tree block (CTB) number syntax element associated with the bitstream, the CTB number syntax element indicating that when the granularity type is equal to slice or tile and the current period spans multiple pictures, the total number of coding tree luma blocks over the current period may be signaled in the CM; or an average coding tree block (CTB) number syntax element associated with the bitstream, the average CTB number syntax element indicating the average number of CTBs or 4x4 blocks per granularity per picture.

[0297]

[0279] Aspect 32. The apparatus of claim 24, wherein intra-coded block statistics are signaled in association with at least a portion of a bitstream when there are intra-coded blocks available in at least the portion of the bitstream.

[0298]

[0280] Aspect 33. The apparatus of claim 24, wherein inter-coded block statistics are signaled in association with at least a portion of a bitstream when there are inter-coded blocks available in at least the portion of the bitstream.

[0299]

[0281] Aspect 34. The apparatus of claim 24, further comprising a camera configured to capture video data.

[0300]

[0282] Aspect 35. The apparatus of claim 24, wherein the apparatus is one of a mobile device, a wearable device, an extended reality device, a camera, a personal computer, a vehicle, a robotic device, a television, or a computing device.

[0301]

[0283] Aspect 36. The apparatus of claim 1, wherein at least one processor is configured to retrieve one or more quality restoration metrics associated with one or more granularity segments of the bitstream.

[0302]

[0284] Aspect 37. The apparatus of claim 1, wherein at least one processor is configured to receive a supplemental enhancement information (SEI) message and extract a granularity type syntax element from the SEI message.

[0303]

[0285] Aspect 38. The apparatus of claim 1, wherein the apparatus includes a decoder.

[0304]

[0286] Aspect 39. The apparatus of claim 1, wherein the apparatus includes a camera configured to capture one or more pictures.

[0305]

[0287] Aspect 40. The method of claim 13, wherein at least one processor is configured to retrieve one or more quality restoration metrics associated with one or more granularity segments of the bitstream.

[0306]

[0288] Aspect 41. The method of claim 13, wherein at least one processor is configured to receive a supplemental enhancement information (SEI) message and extract a granularity type syntax element from the SEI message.

[0307]

[0289] Aspect 42. The method of claim 13, wherein the apparatus includes a decoder.

[0308]

[0290] Aspect 43. The method of claim 13, wherein the device includes a camera configured to capture one or more pictures.

[0309]

[0291] Aspect 44. The apparatus of claim 24, wherein at least one processor is configured to encode one or more quality restoration metrics associated with one or more granularity segments of the bitstream.

[0310]

[0292] Aspect 45. The apparatus of claim 24, wherein at least one processor is configured to encode a supplemental enhancement information (SEI) message with a granularity type syntax element.

[0311]

[0293] Aspect 46. The apparatus of claim 24, wherein the apparatus includes a decoder.

[0312]

[0294] Aspect 47. The apparatus of claim 24, wherein the apparatus includes a camera configured to capture one or more pictures of the video data.

[0313]

[0295] Aspect 48: A method of processing video data comprising one or more of the operations recited in any of aspects 24 to 47.

[0314]

[0296] Aspect 49: A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform one or more of the operations described in any of aspects 13 to 23 and aspect 48.

[0315]

[0297] Aspect 50: An apparatus for processing video data, comprising means for performing one or more of the operations described in any of aspects 13 to 23 and aspect 48.

[0316]

[0298] Aspect 1A: An apparatus for processing video data, comprising at least one memory and one or more processors coupled to the at least one memory, the one or more processors configured to obtain a bitstream, extract a granularity type syntax element associated with the bitstream, and the granularity type syntax element specifies a type of granularity to which a complexity metric (CM) associated with the bitstream is applicable.

[0317]

[0299] Aspect 2A: The apparatus of aspect 1A, wherein a value of the granularity type syntax element specifies that the CM is applicable to pictures of the bitstream.

[0318]

[0300] Aspect 3A: The apparatus of aspect 2A, wherein a value of the granularity type syntax element specifies that the CM is applicable to a slice of the bitstream.

[0319]

[0301] Aspect 4A: The apparatus of aspect 2A, wherein a value of the granularity type syntax element specifies that the CM is applicable to tiles of the bitstream.

[0320]

[0302] Aspect 5A: The apparatus of aspect 2A, wherein a value of the granularity type syntax element specifies that the CM is applicable to sub-pictures of the bitstream.

[0321]

[0303] Aspect 6A: The apparatus of aspect 2A, wherein a value of the granularity type syntax element specifies that the CM is applicable to a scalable layer of the bitstream.

[0322]

[0304] Aspect 7A: The apparatus of aspect 2A, wherein a value of the granularity type syntax element specifies that the CM is applicable to a coding tree unit (CTU) row of the bitstream.

[0323]

[0305] Aspect 8A: An apparatus described in any of aspects 1A to 7A, wherein one or more processors are configured to extract a period type syntax element associated with the bitstream, the period type syntax element specifying a type of upcoming period to which the CM is applicable.

[0324]

[0306] Aspect 9A: An apparatus described in any of aspects 1A to 8A, wherein one or more processors are configured to retrieve a picture level CM syntax structure associated with a bitstream, the picture level CM syntax structure specifying a complexity metric for one or more pictures over a period of time.

[0325]

[0307] Aspect 10A: An apparatus described in any of aspects 1A to 9A, wherein one or more processors are configured to retrieve a granularity level CM syntax structure associated with a bitstream, the granularity level CM syntax structure specifying a granularity level complexity metric for one or more entities over a period of time.

[0326]

[0308] Aspect 11A: The apparatus of aspect 10A, wherein the one or more entities include at least one of a slice, a tile, a subpicture, and a layer.

[0327]

[0309] Aspect 12A: An apparatus described in any of aspects 1A to 11A, wherein one or more processors are configured to extract a sub-picture syntax element associated with a bitstream, the sub-picture syntax element indicating that a sub-picture identifier (ID) is signaled in the CM when a period spans multiple pictures.

[0328]

[0310] Aspect 13A: An apparatus described in any of aspects 1A to 12A, wherein one or more processors are configured to extract a coding tree block (CTB) number syntax element associated with a bitstream, the CTB number syntax element indicating that when the type of granularity is equal to slice or tile and the period spans multiple pictures, the total number of coding tree luma blocks over the period may be signaled in the CM.

[0329]

[0311] Aspect 14A: An apparatus described in any of aspects 1A to 13A, wherein one or more processors are configured to retrieve an average coding tree block (CTB) number syntax element associated with the bitstream, the average CTB number syntax element indicating an average number of CTBs or 4x4 blocks per granularity per picture.

[0330]

[0312] Aspect 15A: An apparatus described in any of aspects 1A to 14A, wherein intra-coded block statistics are signaled in association with at least a portion of a bitstream when there are intra-coded blocks available in at least a portion of the bitstream.

[0331]

[0313] Aspect 16A: An apparatus described in any of aspects 1A to 15A, wherein inter-coded block statistics are signaled in association with at least a portion of a bitstream when there are inter-coded blocks available in at least a portion of the bitstream.

[0332]

[0314] Aspect 17A: An apparatus described in any of aspects 1A to 16A, wherein one or more processors are configured to retrieve one or more quality restoration metrics associated with one or more granularity segments of the bitstream.

[0333]

[0315] Aspect 18A: The apparatus of aspect 17A, wherein one or more granularity segments of the bitstream include at least one of a slice, a tile, and a subpicture.

[0334]

[0316] Aspect 19A: An apparatus described in any of aspects 1A to 18A, wherein one or more processors are configured to receive a supplemental enhancement information (SEI) message and extract a granularity type syntax element from the SEI message.

[0335]

[0317] Aspect 20A: An apparatus described in any of aspects 1A to 19A, wherein the one or more processors are configured to determine an operating frequency of the apparatus based on a CM associated with the bitstream.

[0336]

[0318] Aspect 21A: An apparatus described in any of aspects 1A to 20A, wherein the apparatus includes a decoder.

[0337]

[0319] Aspect 22A: An apparatus described in any of aspects 1A to 21A, further comprising a display configured to display one or more output pictures.

[0338]

[0320] Aspect 23A: An apparatus described in any of aspects 1A to 22A, further comprising a camera configured to capture one or more pictures.

[0339]

[0321] Aspect 24A: An apparatus described in any of aspects 1A to 23A, wherein the apparatus is a mobile device.

[0340]

[0322] Aspect 25A: A method of processing video data comprising one or more of the operations recited in any of aspects 1A to 24A.

[0341]

[0323] Aspect 26A: A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform one or more of the operations described in any of aspects 1A to 24A.

[0342]

[0324] Aspect 27A: An apparatus for processing video data, comprising means for performing one or more of the operations recited in any of aspects 1A to 24A.

[0343]

[0325] Aspect 28A: An apparatus for processing video data, comprising: a memory; and one or more processors coupled to the memory, the one or more processors configured to obtain video data, generate a bitstream associated with the video data, and generate, for the bitstream, a granularity type syntax element that specifies a type of granularity to which a complexity metric (CM) associated with the bitstream is applicable.

[0344]

[0326] Aspect 29A: The apparatus of aspect 28A, wherein a value of the granularity type syntax element specifies that the CM is applicable to pictures of the bitstream.

[0345]

[0327] Aspect 30A: The apparatus of aspect 29A, wherein a value of the granularity type syntax element specifies that the CM is applicable to a slice of the bitstream.

[0346]

[0328] Aspect 31A: The apparatus of aspect 29A, wherein a value of the granularity type syntax element specifies that the CM is applicable to tiles of the bitstream.

[0347]

[0329] Aspect 32A: The apparatus of aspect 29A, wherein a value of the granularity type syntax element specifies that the CM is applicable to subpictures of the bitstream.

[0348]

[0330] Aspect 33A: The apparatus of aspect 29A, wherein a value of the granularity type syntax element specifies that the CM is applicable to a scalable layer of the bitstream.

[0349]

[0331] Aspect 34A: The apparatus of aspect 29A, wherein a value of the granularity type syntax element specifies that the CM is applicable to a coding tree unit (CTU) row of the bitstream.

[0350]

[0332] Aspect 35A: An apparatus described in any of aspects 28A to 34A, wherein one or more processors are configured to generate, for the bitstream, a period type syntax element that specifies a type of upcoming period to which the CM is applicable.

[0351]

[0333] Aspect 36A: An apparatus described in any of aspects 28A to 35A, wherein one or more processors are configured to generate, for a bitstream, a picture-level CM syntax structure that specifies a complexity metric for one or more pictures over a period of time.

[0352]

[0334] Aspect 37A: An apparatus described in any of aspects 28A to 36A, wherein one or more processors are configured to generate, for a bitstream, a granularity level CM syntax structure that specifies a granularity level complexity metric for one or more entities over a period of time.

[0353]

[0335] Aspect 38A: The apparatus of aspect 37A, wherein the one or more entities include at least one of a slice, a tile, a subpicture, and a layer.

[0354]

[0336] Aspect 39A: An apparatus described in any of aspects 28A to 38A, wherein the one or more processors are configured to generate, for a bitstream, a sub-picture syntax element indicating that a sub-picture identifier (ID) is signaled in a CM when a duration spans multiple pictures.

[0355]

[0337] Aspect 40A: An apparatus described in any of aspects 28A to 39A, wherein one or more processors are configured to generate a coding tree block (CTB) number syntax element indicating that, for a bitstream, when the type of granularity is equal to slice or tile and the period spans multiple pictures, the total number of coding tree luma blocks over the period can be signaled in the CM.

[0356]

[0338] Aspect 41A: An apparatus described in any of aspects 28A to 40A, wherein one or more processors are configured to generate an average CTB number syntax element indicating an average number of coding tree blocks (CTBs) or 4x4 blocks per granularity per picture for the bitstream.

[0357]

[0339] Aspect 42A: An apparatus described in any of aspects 28A to 41A, wherein intra-coded block statistics are signaled in association with at least a portion of a bitstream when there are intra-coded blocks available in at least a portion of the bitstream.

[0358]

[0340] Aspect 43A: An apparatus described in any of aspects 28A to 42A, wherein inter-coded block statistics are signaled in association with at least a portion of a bitstream when there are inter-coded blocks available in at least a portion of the bitstream.

[0359]

[0341] Aspect 44A: An apparatus described in any of aspects 28A to 43A, wherein one or more processors are configured to generate, for a bitstream, one or more quality restoration metrics associated with one or more granularity segments of the bitstream.

[0360]

[0342] Aspect 45A: The apparatus of aspect 44A, wherein one or more granularity segments of the bitstream include at least one of a slice, a tile, and a subpicture.

[0361]

[0343] Aspect 46A: An apparatus described in any of aspects 28A to 45A, wherein one or more processors are configured to generate a supplemental enhancement information (SEI) message and include a granularity type syntax element in the SEI message.

[0362]

[0344] Aspect 47A: An apparatus described in any of aspects 28A to 46A, wherein the one or more processors are configured to store the bitstream.

[0363]

[0345] Aspect 48A: An apparatus described in any of aspects 28A to 47A, wherein the one or more processors are configured to transmit a bitstream.

[0364]

[0346] Aspect 49A: An apparatus described in any of aspects 28A to 48A, wherein the apparatus includes an encoder.

[0365]

[0347] Embodiment 50A: The apparatus of any of embodiments 28A to 49A, further comprising a display configured to display one or more output pictures.

[0366]

[0348] Aspect 51A: The apparatus of any of aspects 28A to 50A, further comprising a camera configured to capture one or more pictures.

[0367]

[0349] Aspect 52A: An apparatus described in any of aspects 28A to 51A, wherein the apparatus is a mobile device.

[0368]

[0350] Aspect 53A: A method of processing video data comprising one or more of the operations described in any of aspects 28A to 52A.

[0369]

[0351] Aspect 54A: A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform one or more of the operations described in any of aspects 28A to 52A.

[0370]

[0352] Aspect 55A: An apparatus for processing video data, comprising means for performing one or more of the operations recited in any of aspects 28A to 52A.

Claims

1. An apparatus for processing video data, comprising: at least one memory; at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: obtain a bitstream; extract a granularity type syntax element associated with the bitstream, the granularity type syntax element specifying a type of granularity of one or more pictures for which a complexity metric (CM) associated with the bitstream is applicable, wherein the CM is for determining an operating frequency of the apparatus, and a value of the granularity type syntax element specifies that the CM is applicable to a picture of the bitstream or a portion of the picture, the portion of the picture being smaller than the whole picture; extract a period type syntax element associated with the bitstream, the period type syntax element indicating a current time period or a set of pictures for which the CM is applicable; decode a portion of the bitstream based on the granularity type syntax element and the period type syntax element. An apparatus configured to perform the above.

2. The apparatus according to claim 1, wherein the at least one processor is configured to extract a granularity level CM syntax structure associated with the bitstream, the granularity level CM syntax structure specifying a granularity level complexity metric for one or more granularity segments of the bitstream over the current period.

3. The at least one processor of the apparatus according to claim 1 is further configured to: extract an additional period type syntax element associated with the bitstream, the additional period type syntax element being related to the granularity type syntax element and being different from the period type syntax element; decode the portion of the bitstream based on the granularity type syntax element and the additional period type syntax element. An apparatus configured to perform the above.

4. The at least one processor is configured to extract at least one of: a sub-picture syntax element related to the bitstream, the sub-picture syntax element indicating that a sub-picture identifier (ID) is signaled when the CM is applied to a plurality of pictures; a coding tree block (CTB) count syntax element related to the bitstream, the CTB count syntax element indicating that when the type of granularity is equal to a slice or a tile and the current period spans a plurality of pictures, the total number of coding tree luma blocks over the current period can be signaled in the CM; or an average CTB count syntax element related to the bitstream, the average CTB count syntax element indicating an average number of CTBs or 4×4 blocks per picture per granularity. The apparatus according to claim 1, wherein the at least one processor is configured to extract at least one of: a sub-picture syntax element related to the bitstream, the sub-picture syntax element indicating that a sub-picture identifier (ID) is signaled when the CM is applied to a plurality of pictures; a coding tree block (CTB) count syntax element related to the bitstream, the CTB count syntax element indicating that when the type of granularity is equal to a slice or a tile and the current period spans a plurality of pictures, the total number of coding tree luma blocks over the current period can be signaled in the CM; or an average CTB count syntax element related to the bitstream, the average CTB count syntax element indicating an average number of CTBs or 4×4 blocks per picture per granularity. The apparatus according to claim 1, wherein the at least one processor is configured to extract at least one of: a sub-picture syntax element related to the bitstream, the sub-picture syntax element indicating that a sub-picture identifier (ID) is signaled when the CM is applied to a plurality of pictures; a coding tree block (CTB) count syntax element related to the bitstream, the CTB count syntax element indicating that when the type of granularity is equal to a slice or a tile and the current period spans a plurality of pictures, the total number of coding tree luma blocks over the current period can be signaled in the CM; or an average CTB count syntax element related to the bitstream, the average CTB count syntax element indicating an average number of CTBs or 4×4 blocks per picture per granularity. The apparatus according to claim 1, wherein the at least one processor is configured to extract at least one of: a sub-picture syntax element related to the bitstream, the sub-picture syntax element indicating that a sub-picture identifier (ID) is signaled when the CM is applied to a plurality of pictures; a coding tree block (CTB) count syntax element related to the bitstream, the CTB count syntax element indicating that when the type of granularity is equal to a slice or a tile and the current period spans a plurality of pictures, the total number of coding tree luma blocks over the current period can be signaled in the CM; or an average CTB count syntax element related to the bitstream, the average CTB count syntax element indicating an average number of CTBs or 4×4 blocks per picture per granularity. Claim 5 The apparatus according to claim 1, wherein the at least one processor is configured to determine an operating frequency of the apparatus based on the CM related to the bitstream. Claim 6 The apparatus according to claim 1, further comprising a display configured to display at least the portion of the bitstream, wherein the apparatus is one of a mobile device, a wearable device, an extended reality device, a camera, a personal computer, a vehicle, a robotic device, a television, or a computing device. Claim 7 A method performed by an apparatus for processing video data, the method comprising: obtaining a bitstream; extracting a granularity type syntax element related to the bitstream, the granularity type syntax element specifying a type of granularity of one or more pictures to which a complexity metric (CM) related to the bitstream is applicable, wherein the CM is for determining an operating frequency of the apparatus, and a value of the granularity type syntax element specifies that the CM is applicable to a picture or a portion of the picture of the bitstream, the portion of the picture being smaller than the whole picture. extracting period type syntax elements associated with the bitstream, and the period type syntax elements indicating the current time period or set of pictures to which the CM is applicable; decoding a portion of the bitstream based on the granularity type syntax element and the period type syntax element A method comprising: **Claim 8** The value of the granularity type syntax element specifies that the CM is applicable to at least one of slices, tiles, sub-pictures, scalable layers, or coding tree units (CTUs) rows of the one or more pictures of the bitstream, and / or The method according to claim 7, wherein the period type syntax element indicates at least one of a specified time interval for the current period, the number of pictures for the current period, the current period including all pictures up to the picture including the next slice, or the current period including a single picture. **Claim 9** An apparatus for processing video data, comprising: at least one memory; at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: acquire video data; generate a granularity type syntax element for specifying a type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable, wherein the CM is for determining an operating frequency of a decoder, and the value of the granularity type syntax element specifies that the CM is applicable to a picture or a portion of the picture of the bitstream, and the portion of the picture is smaller than the whole picture; generate a period type syntax element for the bitstream, the period type syntax element indicating the current time period or set of pictures to which the CM is applicable; generate the bitstream associated with the video data, the bitstream including the granularity type syntax element and the period type syntax element; outputting the generated bitstream; An apparatus configured to perform the above. **Claim 10** The value of the granularity type syntax element specifies that the CM is applicable to at least one of the slices, tiles, sub-pictures, scalable layers, or coding tree units (CTUs) of the one or more pictures of the bitstream, and / or The period type syntax element indicates at least one of a specified time interval for the current period, the number of pictures for the current period, the current period including all pictures up to the picture including the next slice, or the current period including a single picture, according to the apparatus of Claim 1 or 9. **Claim 11** The at least one processor is configured to generate a granularity level CM syntax structure that specifies a granularity level complexity metric for one or more entities over the current period for the bitstream, and / or The period type syntax element indicates at least one of a specified time interval for the current period, the number of pictures for the current period, the current period including all pictures up to the picture including the next slice, or the current period including a single picture, according to the apparatus of Claim 9. **Claim 12** The at least one processor is configured to generate an additional period type syntax element related to the bitstream, where the additional period type syntax element is related to the granularity type syntax element, and where the additional period type syntax element is different from the period type syntax element, and where the additional period type syntax element is for decoding a portion of the bitstream with the granularity type syntax element, according to the apparatus of Claim 9, or The at least one processor is for the bitstream A sub-picture syntax element related to the bitstream, the sub-picture syntax element indicating that a sub-picture identifier (ID) is signaled when the CM is applied to a plurality of pictures. A coding tree block (CTB) count syntax element related to the bitstream, the CTB count syntax element indicating that when the type of granularity is equal to a slice or a tile and the current period spans a plurality of pictures, the total number of coding tree luma blocks over the current period can be signaled in the CM, or An average CTB count syntax element related to the bitstream, the average CTB count syntax element indicating the average number of CTBs or 4×4 blocks per picture per granularity. The apparatus according to claim 9, configured to generate at least one of the above.

13. The intra-coded block statistic value is signaled in relation to at least the portion of the bitstream when there are intra-coded blocks available in at least a portion of the bitstream, or The inter-coded block statistic value is signaled in relation to at least the portion of the bitstream when there are inter-coded blocks available in at least a portion of the bitstream, the apparatus according to claim 1 or 9.

14. The apparatus according to claim 9, further comprising a camera configured to capture the video data, the apparatus being one of a mobile device, a wearable device, an extended reality device, a camera, a personal computer, a vehicle, a robotic device, a television, or a computing device.

15. A method for processing video data, comprising: Obtaining video data; For a bitstream, generating a granularity type syntax element that specifies the type of granularity of one or more pictures to which a complexity metric (CM) associated with the bitstream is applicable, where the CM is for determining the operating frequency of a decoder, and the value of the granularity type syntax element specifies that the CM is applicable to a picture of the bitstream or a portion of the picture, and the portion of the picture is smaller than the whole picture, For the bitstream, generating a period type syntax element, where the period type syntax element indicates the current time period or set of pictures to which the CM is applicable, Generating the bitstream associated with the video data, where the bitstream includes the granularity type syntax element and the period type syntax element, Outputting the generated bitstream A method comprising.

Images